HOW TO SUPPRESS BEA-101020 ALERTING
REQUEST
I receive lots of emails with these errors reported as Critical from Oracle Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4). I want to stop them to be generated.
RELATED
SYMPTOMS
Host=dbpilot.net Target type=Oracle WebLogic Server Target name=/b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node1 Categories=Diagnostics, Fault Message=Incident (BEA-101020 [HTTP][java.lang.NumberFormatException]) detected in /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/alert/log.xml at time/line number: Mon Dec 25 12:45:54 2017/114796 Severity=Critical Event reported time=Dec 25, 2017 12:50:14 PM MSK Target Lifecycle Status=Production Comment=B2B Production Operating System=Linux Platform=x86_64 Associated Incident Id=256229 Associated Incident Status=New Associated Incident Owner= Associated Incident Acknowledged By Owner=No Associated Incident Priority=None Associated Incident Escalation Level=0 Event Type=Metric Alert Event name=alertLogAdrIncident:adr_problemKey Metric Group=Incident Metric=Problem Key Metric value=BEA-101020 [HTTP][java.lang.NumberFormatException] Key Value=Mon Dec 25 12:45:54 2017/114796 Key Column 1=Timeline Rule Name=Notifications All,rule 156 Rule Owner=SYSMAN Update Details: Incident (BEA-101020 [HTTP][java.lang.NumberFormatException]) detected in /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/alert/log.xml at time/line number: Mon Dec 25 12:45:54 2017/114796
The error details at log.xml on weblogic server
[root@dbpilot ~]# tail /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/alert/log.xml
</msg>
<msg time='2017-12-25T12:45:54.040+03:00' org_id='oracle' comp_id='ofm'
msg_id='1530680151' type='INCIDENT_ERROR' level='1'
host_id='dbpilot.net' host_addr='10.10.10.10' prob_key='BEA-101020 [HTTP][java.lang.NumberFormatException]'
upstream_comp='' downstream_comp='' ecid=''
errid='28167' detail_path='/d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/incident/incdir_28167'>
<txt>Errors in directory: /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/incident/incdir_28167 (incident=28167):
null
</txt>
</msg>
A content of readme file in incdir_28167 directory
[root@dbpilot ~]# cd /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/incident/incdir_28167
[root@dbpilot incdir_28167]# cat readme.txt
Incident Id: 28167
Incident Source: SYSTEM
Create Time: Mon Dec 25 12:45:54 MSK 2017
Problem Key: BEA-101020 [HTTP][java.lang.NumberFormatException]
Application Name: b2b_content_new_en
Error Message Id: BEA-101020
Description
-----------
Incident detected using watch rule "UncheckedException":
Watch time: 25.12.2017 12:45:54 MSK
Watch ServerName: B2B_CMSServer_Node1
Watch RuleType: Log
Watch Rule: (SEVERITY = 'Error') AND ((MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))
Watch DomainName: b2b_domain
Watch Data:
DATE : 25.12.2017 12:45:54 MSK
SERVER : B2B_CMSServer_Node1
MESSAGE : [ServletContext@291958422[app:b2b_content_new_en module:pages_new_en path:/pages_new_en spec-version:2.5]] Servlet failed with Exception
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at jsp_servlet._catalog.__catalog_product._jspService(__catalog_product.java:531)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:35)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:280)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at com.bea.content.manager.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:178)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:160)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:330)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
SUBSYSTEM : HTTP
USERID :
SEVERITY : Error
THREAD : [ACTIVE] ExecuteThread: '106' for queue: 'weblogic.kernel.Default (self-tuning)'
MSGID : BEA-101020
MACHINE : dbpilot.net
TXID :
CONTEXTID :
TIMESTAMP : 1514195154039
Diagnostic Dumps
----------------
Name: jvm.threads
Dump Files: jvm_threads433_i28167.dmp
Name: dms.metrics
Dump Files: dms_metrics434_i28167.dmp
Name: odl.logs
Dump Arguments: timestamp=1514195154040
Dump Files: odl_logs435_i28167.dmp
Name: wls.image
Dump Files: diagnostic_image_B2B_CMSServer_Node1_2017_12_25_12_45_54.zip
GATHER DETAILS
By querying the OMS database get details about the latest alert
ALTER SESSION SET NLS_DATE_FORMAT='DD-MM-YYYY HH24:MI:SS';
SET LINES 300
SET PAGES 999
COL MESSAGE FOR A150
SELECT * FROM (
SELECT TO_DATE(COLLECTION_TIMESTAMP,'DD-MM-YYYY HH24:MI:SS') "COLLECTION_TIMESTAMP", MESSAGE
FROM MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE MESSAGE LIKE '%BEA-101020 %' AND
UPPER(TARGET_NAME) LIKE UPPER('%/b2b_domain/ %') ORDER BY 1 DESC
) WHERE ROWNUM < 3;
COLLECTION_TIMESTAM MESSAGE
------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
25-12-2017 12:50:14 Incident (BEA-101020 [HTTP][java.lang.NumberFormatException]) detected in /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/alert/log.xml at time/line number: Mon Dec 25 12:45:54 2017/114796
25-12-2017 12:41:10 Incident (BEA-101020 [HTTP][java.lang.NumberFormatException]) detected in /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node2/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node2/alert/log.xml at time/line number: Mon Dec 25 12:41:01 2017/117235
Number of errors by month
COL SYSTIMESTAMP FOR A35
SELECT SYSTIMESTAMP FROM DUAL;
SYSTIMESTAMP
-----------------------------------
25.12.17 13:05:24,786012 +03:00
ALTER SESSION SET NLS_DATE_FORMAT='MM-YYYY';
SELECT TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') "MONTH",
COUNT(*) "GENERATED ERRORS"
FROM MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE MESSAGE LIKE '%BEA-101020 %' AND
UPPER(TARGET_NAME) LIKE UPPER('%/b2b_domain/ %')
GROUP BY TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') ORDER BY 1;
MONTH GENERATED ERRORS
------- ----------------
11-2017 338
12-2017 1432
Number of errors by target name
SET LINES 300
COL TARGET_NAME FOR A60
BREAK ON "MONTH"
SELECT TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') "MONTH",
TARGET_NAME,
COUNT(*) "GENERATED ERRORS"
FROM MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE MESSAGE LIKE '%BEA-101020%' AND
UPPER(TARGET_NAME) LIKE UPPER('%/b2b_domain/%')
GROUP BY TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY'),TARGET_NAME
ORDER BY 1,2;
MONTH TARGET_NAME GENERATED ERRORS
------- ------------------------------------------------------------ ----------------
11-2017 /b2b_domain_b2b_domain/b2b_domain/B2B_AppsServer_Node1 68
/b2b_domain_b2b_domain/b2b_domain/B2B_AppsServer_Node2 55
/b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node1 106
/b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node2 109
12-2017 /b2b_domain_b2b_domain/b2b_domain/AdminServer 1
/b2b_domain_b2b_domain/b2b_domain/B2B_AppsServer_Node1 241
/b2b_domain_b2b_domain/b2b_domain/B2B_AppsServer_Node2 237
/b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node1 463
/b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node2 490
SOLUTION
On weblogic host(s) modify Module-FMWDFW-2818.xml file. Set UncheckedException to false and restart weblogic.
[oracle@dbpilot ~]# locate Module-FMWDFW-2818.xml
/d00/app/oracle/Middleware/user_projects/domains/b2b_domain/config/diagnostics/Module-FMWDFW-2818.xml
[oracle@dbpilot ~]# cd /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/config/diagnostics
[oracle@dbpilot diagnostics]# egrep UncheckedException Module-FMWDFW-2818.xml -A 1
<name>UncheckedException</name>
<enabled>true</enabled>
[oracle@dbpilot diagnostics]# cp Module-FMWDFW-2818.xml Module-FMWDFW-2818.xml.$(date +%F)
[oracle@dbpilot diagnostics]# ls -lh Module-FMWDFW-2818.xml* --time-style='+'
-rw-r----- 1 oracle oinstall 1.8K Module-FMWDFW-2818.xml
-rw-r----- 1 oracle oinstall 1.8K Module-FMWDFW-2818.xml.2017-12-25
-rw-r----- 1 oracle oinstall 1.8K Module-FMWDFW-2818.xml.orig.22AUG2017
[oracle@dbpilot diagnostics]# egrep UncheckedException Module-FMWDFW-2818.xml -A 1
<name>UncheckedException</name>
<enabled>false</enabled>
The UncheckedException is responsible for generating BEA-101020 alerts
[root@dbpilot diagnostics]# egrep UncheckedException Module-FMWDFW-2818.xml -A 3
<name>UncheckedException</name>
<enabled>false</enabled>
<rule-type>Log</rule-type>
<rule-expression>(SEVERITY = 'Error') AND ((MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))</rule-expression>
After service restart no new BEA-101020 error will be spooled into log.xml anymore.
Version : 16:15 25.12.2017