The Oracle Enterprise Manager 13c frequently generates Global Cache Blocks Lost alerts for a RAC target.
SYMPTOMS

This incident is reported by the Oracle Enterprise Manager 13c every 5 minutes when monitoring a Cluster Database

Host=db-gp01.local
Target type=Database Instance
Target name=GPPRD_GPPRD1
Categories=Error
Message=Total global cache block lost is 15.
Severity=Warning
Event reported time=Aug 21, 2020 2:30:09 AM MSK
Operating System=Linux
Platform=x86_64
Associated Incident Id=138883
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Metric Alert
Event name=rac_global_cache:lost
Metric Group=Global Cache Statistics
Metric=Global Cache Blocks Lost
Metric value=15
Key Value=
Rule Name=ROOT_NOTIFICATION_RULE,ALL TARGET EVENTS
Rule Owner=SYSMAN
Update Details:
Total global cache block lost is 15.

There is a high number of sent Global Cache Blocks Lost alerts for the problematic target in the alert notifications summary view

## OMS Repository

SELECT TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY') "RECEIVED AT",
       COUNT(COLLECTION_TIMESTAMP) "ALERTS"
FROM
       MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE
       METRIC_NAME='rac_global_cache' AND
       METRIC_COLUMN='lost' AND
       COLUMN_LABEL = 'Global Cache Blocks Lost' AND
       TARGET_NAME = '&INSTANCE_NAME' 
GROUP BY 
       TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY')
ORDER BY 1;
Enter value for instance_name: GPPRD_GPPRD1
old   7:        TARGET_NAME = '&INSTANCE_NAME'
new   7:        TARGET_NAME = 'GPPRD_GPPRD1'

RECEIVED AT            ALERTS
------------------ ----------
14-AUG-20                 142
15-AUG-20                 202
16-AUG-20                 202
17-AUG-20                 202
18-AUG-20                 202
19-AUG-20                 202
20-AUG-20                 202
21-AUG-20                  64

8 rows selected.

In this demo, the Enterprise Manager sent 202 alerts mostly every day for the problematic instance.

## OMS Repository

SET PAGES 999
SET LINES 300
COL MESSAGE FOR A60

SELECT TO_CHAR(COLLECTION_TIMESTAMP, 'HH24:MI DD-MM-YYYY') ALERTED_AT,
       MESSAGE 
FROM
       MGMT_VIEW.MGMT$ALERT_NOTIF_LOG 
WHERE
      TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY')=TO_DATE('&DDMMYYYY', 'DD-MM-YYYY') AND 
      TARGET_NAME = '&INSTANCE_NAME'
ORDER BY COLLECTION_TIMESTAMP;
Enter value for ddmmyyyy: 20-AUG-20
old   7:       TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY')=TO_DATE('&DDMMYYYY', 'DD-MM-YYYY') AND
new   7:       TO_DATE(COLLECTION_TIMESTAMP, 'DD-MM-YYYY')=TO_DATE('20-AUG-20', 'DD-MM-YYYY') AND
Enter value for instance_name: GPPRD_GPPRD1
old   8:       TARGET_NAME = '&INSTANCE_NAME'
new   8:       TARGET_NAME = 'GPPRD_GPPRD1'

ALERTED_AT       MESSAGE
---------------- ------------------------------------------------------------
00:00 20-08-2020 Total global cache block lost is 15.
00:00 20-08-2020 Total global cache block lost is 15.
00:05 20-08-2020 Total global cache block lost is 15.

...

23:50 20-08-2020 Total global cache block lost is 15.
23:50 20-08-2020 Total global cache block lost is 15.
23:55 20-08-2020 Total global cache block lost is 15.
23:55 20-08-2020 Total global cache block lost is 15.

202 rows selected.

Even though there is 0 lost block during the day, the Enterprise Manager continues generating alerts.

CAUSE

Why does Enterprise Manager keep sending alerts even though there are no lost blocks for the last hours (days, weeks)?

Well,
The Global Cache Blocks Lost alert is generated by the Global Cache Blocks Lost Metric in the Enterprise Manager.

GLOBAL_CACHE_BLOCKS_LOST_001.png

To get the number of lost blocks in a target, the Metric queries the gc blocks lost statistic from the V$SYSSTAT view of the target instance.

## Problematic Target

SET PAGES 999
SET LINES 300
COL NAME FOR A20
COL VALUE FOR 999999
SELECT NAME, VALUE FROM V$SYSSTAT WHERE NAME='gc blocks lost';
NAME                   VALUE
-------------------- -------
gc blocks lost            15

The V$SYSSTAT view stores the value of lost blocks (gc blocks lost) since the instance startup. It means the value of gc blocks lost statistic just increases. Statistics resets only with instance's restart.

When lost blocks (gc blocks lost) exceeds the threshold value of the Metric, the Enterprise Manager starts to send alerts and it keeps sending alerts until instance's restart.

WORKAROUND

If you are sure there are no issues with losing blocks (gc blocks lost statistic) in your instance, disable the Global Cache Blocks Lost Metric.
To do so, set empty values for the Metric thresholds

GLOBAL_CACHE_BLOCKS_LOST_002.png

NOTE

To find lost blocks ratio the following query can be used in the target instance

## Problematic Target

SET PAGES 999
SET LINES 300
COL RATIO FOR 99999999

SELECT A.INST_ID "INSTANCE",
       A.VALUE "GC BLOCKS LOST",
       B.VALUE "GC CUR BLOCKS SERVED",
       C.VALUE "GC CR BLOCKS SERVED",
       A.VALUE/(B.VALUE+C.VALUE) RATIO
FROM
       GV$SYSSTAT A, 
       GV$SYSSTAT B,
       GV$SYSSTAT C
WHERE
       A.NAME='gc blocks lost' AND
       B.NAME='gc current blocks served' AND
       C.NAME='gc cr blocks served' and
       B.INST_ID=a.inst_id AND
       C.INST_ID = a.inst_id;
  INSTANCE GC BLOCKS LOST GC CUR BLOCKS SERVED GC CR BLOCKS SERVED     RATIO
---------- -------------- -------------------- ------------------- ---------
         1             15             32576274               42979         0
REFERENCES

Troubleshooting gc block lost and Poor Network Performance in a RAC Environment (Doc ID 563566.1)
WAITEVENT: "gc current/cr block lost" Reference Note (Doc ID 2296681.1)
Tuning Inter-Instance Performance in RAC and OPS (Doc ID 181489.1)
EM 13c: How to disable "Global Cache Blocks Lost Metric" Using EMCLI (Doc ID 2543134.1)
False increase of 'Global Cache Blocks Lost' or 'gc blocks lost' after upgrade to 12c (Doc ID 2096299.1)
EM13c: Repeating Alerts for the Global Cache Blocks Lost Metric Generated after Threshold Exceeded Once with Incident Report: Message=Metrics Global Cache Blocks Lost is at XXX (Doc ID 2350104.1)

Written At
23 AUG 202021:00
Red Hat Release
7.x x64
Enterprise Manager
13.1.0.0.0 - 13.3.0.0.0
Database Release
12.1.0.2 - 19.13.0.0.0