LOTS OF FALSE 'GLOBAL CACHE BLOCKS LOST' ALERTS REPORTED BY ENTERPRISE MANAGER CLOUD CONTROL 13C
This incident is reported by the Oracle Enterprise Manager 13c every 5 minutes when monitoring a Cluster Database
Target type=Database Instance
Target name=GPPRD_GPPRD1
Categories=Error
Message=Total global cache block lost is 15.
Severity=Warning
Event reported time=Aug 21, 2020 2:30:09 AM MSK
Operating System=Linux
Platform=x86_64
Associated Incident Id=138883
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Metric Alert
Event name=rac_global_cache:lost
Metric Group=Global Cache Statistics
Metric=Global Cache Blocks Lost
Metric value=15
Key Value=
Rule Name=ROOT_NOTIFICATION_RULE,ALL TARGET EVENTS
Rule Owner=SYSMAN
Update Details:
Total global cache block lost is 15.
There is a high number of sent Global Cache Blocks Lost alerts for the problematic target in the alert notifications summary view
## OMS Repository
SELECT TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY') "RECEIVED AT",
COUNT(COLLECTION_TIMESTAMP) "ALERTS"
FROM
MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE
METRIC_NAME='rac_global_cache' AND
METRIC_COLUMN='lost' AND
COLUMN_LABEL = 'Global Cache Blocks Lost' AND
TARGET_NAME = '&INSTANCE_NAME'
GROUP BY
TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY')
ORDER BY 1;
Enter value for instance_name: GPPRD_GPPRD1
old 7: TARGET_NAME = '&INSTANCE_NAME'
new 7: TARGET_NAME = 'GPPRD_GPPRD1'
RECEIVED AT ALERTS
------------------ ----------
14-AUG-20 142
15-AUG-20 202
16-AUG-20 202
17-AUG-20 202
18-AUG-20 202
19-AUG-20 202
20-AUG-20 202
21-AUG-20 64
8 rows selected.
In this demo, the Enterprise Manager sent 202 alerts mostly every day for the problematic instance.
## OMS Repository
SET PAGES 999
SET LINES 300
COL MESSAGE FOR A60
SELECT TO_CHAR(COLLECTION_TIMESTAMP, 'HH24:MI DD-MM-YYYY') ALERTED_AT,
MESSAGE
FROM
MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE
TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY')=TO_DATE('&DDMMYYYY', 'DD-MM-YYYY') AND
TARGET_NAME = '&INSTANCE_NAME'
ORDER BY COLLECTION_TIMESTAMP;
Enter value for ddmmyyyy: 20-AUG-20
old 7: TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY')=TO_DATE('&DDMMYYYY', 'DD-MM-YYYY') AND
new 7: TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY')=TO_DATE('20-AUG-20', 'DD-MM-YYYY') AND
Enter value for instance_name: GPPRD_GPPRD1
old 8: TARGET_NAME = '&INSTANCE_NAME'
new 8: TARGET_NAME = 'GPPRD_GPPRD1'
ALERTED_AT MESSAGE
---------------- ------------------------------------------------------------
00:00 20-08-2020 Total global cache block lost is 15.
00:00 20-08-2020 Total global cache block lost is 15.
00:05 20-08-2020 Total global cache block lost is 15.
...
23:50 20-08-2020 Total global cache block lost is 15.
23:50 20-08-2020 Total global cache block lost is 15.
23:55 20-08-2020 Total global cache block lost is 15.
23:55 20-08-2020 Total global cache block lost is 15.
202 rows selected.
Even though there is 0 lost block during the day, the Enterprise Manager continues generating alerts.
Why does Enterprise Manager keep sending alerts even though there are no lost blocks for the last hours (days, weeks)?
Well,
The Global Cache Blocks Lost alert is generated by the Global Cache Blocks Lost Metric in the Enterprise Manager.
To get the number of lost blocks in a target, the Metric queries the gc blocks lost statistic from the V$SYSSTAT view of the target instance.
## Problematic Target
SET PAGES 999
SET LINES 300
COL NAME FOR A20
COL VALUE FOR 999999
SELECT NAME, VALUE FROM V$SYSSTAT WHERE NAME='gc blocks lost';
NAME VALUE
-------------------- -------
gc blocks lost 15
The V$SYSSTAT view stores the value of lost blocks (gc blocks lost) since the instance startup. It means the value of gc blocks lost statistic just increases. Statistics resets only with instance's restart.
When lost blocks (gc blocks lost) exceeds the threshold value of the Metric, the Enterprise Manager starts to send alerts and it keeps sending alerts until instance's restart.
If you are sure there are no issues with losing blocks (gc blocks lost statistic) in your instance, disable the Global Cache Blocks Lost Metric.
To do so, set empty values for the Metric thresholds
To find lost blocks ratio the following query can be used in the target instance
## Problematic Target
SET PAGES 999
SET LINES 300
COL RATIO FOR 99999999
SELECT A.INST_ID "INSTANCE",
A.VALUE "GC BLOCKS LOST",
B.VALUE "GC CUR BLOCKS SERVED",
C.VALUE "GC CR BLOCKS SERVED",
A.VALUE/(B.VALUE+C.VALUE) RATIO
FROM
GV$SYSSTAT A,
GV$SYSSTAT B,
GV$SYSSTAT C
WHERE
A.NAME='gc blocks lost' AND
B.NAME='gc current blocks served' AND
C.NAME='gc cr blocks served' and
B.INST_ID=a.inst_id AND
C.INST_ID = a.inst_id;
INSTANCE GC BLOCKS LOST GC CUR BLOCKS SERVED GC CR BLOCKS SERVED RATIO
---------- -------------- -------------------- ------------------- ---------
1 15 32576274 42979 0
Troubleshooting gc block lost and Poor Network Performance in a RAC Environment (Doc ID 563566.1)
WAITEVENT: "gc current/cr block lost" Reference Note (Doc ID 2296681.1)
Tuning Inter-Instance Performance in RAC and OPS (Doc ID 181489.1)
EM 13c: How to disable "Global Cache Blocks Lost Metric" Using EMCLI (Doc ID 2543134.1)
False increase of 'Global Cache Blocks Lost' or 'gc blocks lost' after upgrade to 12c (Doc ID 2096299.1)
EM13c: Repeating Alerts for the Global Cache Blocks Lost Metric Generated after Threshold Exceeded Once with Incident Report: Message=Metrics Global Cache Blocks Lost is at XXX (Doc ID 2350104.1)
