HOW TO SUPPRESS BEA-101020 ALERTING

REQUEST

I receive lots of emails with these errors reported as Critical from Oracle Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4). I want to stop them to be generated.

HOW TO SUPPRESS BEA-000337 ALERTING

SYMPTOMS

Host=dbpilot.net 
Target type=Oracle WebLogic Server
Target name=/b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node1 
Categories=Diagnostics, Fault 
Message=Incident (BEA-101020 [HTTP][java.lang.NumberFormatException]) detected in /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/alert/log.xml at time/line number: Mon Dec 25 12:45:54 2017/114796 
Severity=Critical 
Event reported time=Dec 25, 2017 12:50:14 PM MSK 
Target Lifecycle Status=Production 
Comment=B2B Production
Operating System=Linux
Platform=x86_64
Associated Incident Id=256229 
Associated Incident Status=New 
Associated Incident Owner= 
Associated Incident Acknowledged By Owner=No 
Associated Incident Priority=None 
Associated Incident Escalation Level=0 
Event Type=Metric Alert 
Event name=alertLogAdrIncident:adr_problemKey 
Metric Group=Incident
Metric=Problem Key
Metric value=BEA-101020 [HTTP][java.lang.NumberFormatException]
Key Value=Mon Dec 25 12:45:54 2017/114796
Key Column 1=Timeline
Rule Name=Notifications All,rule 156 
Rule Owner=SYSMAN 
Update Details:
Incident (BEA-101020 [HTTP][java.lang.NumberFormatException]) detected in /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/alert/log.xml at time/line number: Mon Dec 25 12:45:54 2017/114796

The error details at log.xml on weblogic server

[root@dbpilot ~]# tail /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/alert/log.xml
</msg>
<msg time='2017-12-25T12:45:54.040+03:00' org_id='oracle' comp_id='ofm'
 msg_id='1530680151' type='INCIDENT_ERROR' level='1'
 host_id='dbpilot.net' host_addr='10.10.10.10' prob_key='BEA-101020 [HTTP][java.lang.NumberFormatException]'
 upstream_comp='' downstream_comp='' ecid=''
 errid='28167' detail_path='/d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/incident/incdir_28167'>
 <txt>Errors in directory: /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/incident/incdir_28167  (incident=28167):
null
 </txt>
</msg>

A content of readme file in incdir_28167 directory

[root@dbpilot ~]# cd /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/incident/incdir_28167
[root@dbpilot incdir_28167]# cat readme.txt
Incident Id: 28167
Incident Source: SYSTEM
Create Time: Mon Dec 25 12:45:54 MSK 2017
Problem Key: BEA-101020 [HTTP][java.lang.NumberFormatException]
Application Name: b2b_content_new_en
Error Message Id: BEA-101020

Description
-----------
Incident detected using watch rule "UncheckedException":
Watch time:             25.12.2017 12:45:54 MSK
Watch ServerName:       B2B_CMSServer_Node1
Watch RuleType:         Log
Watch Rule:             (SEVERITY = 'Error') AND ((MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))
Watch DomainName:       b2b_domain
Watch Data:
   DATE : 25.12.2017 12:45:54 MSK
   SERVER : B2B_CMSServer_Node1
   MESSAGE : [ServletContext@291958422[app:b2b_content_new_en module:pages_new_en path:/pages_new_en spec-version:2.5]] Servlet failed with Exception
java.lang.NumberFormatException: null
        at java.lang.Integer.parseInt(Integer.java:417)
        at java.lang.Integer.parseInt(Integer.java:499)
        at jsp_servlet._catalog.__catalog_product._jspService(__catalog_product.java:531)
        at weblogic.servlet.jsp.JspBase.service(JspBase.java:35)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:27)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:280)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at com.bea.content.manager.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:178)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:160)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:330)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

   SUBSYSTEM : HTTP
   USERID : 
   SEVERITY : Error
   THREAD : [ACTIVE] ExecuteThread: '106' for queue: 'weblogic.kernel.Default (self-tuning)'
   MSGID : BEA-101020
   MACHINE : dbpilot.net
   TXID :
   CONTEXTID :
   TIMESTAMP : 1514195154039

Diagnostic Dumps
----------------

Name: jvm.threads
Dump Files: jvm_threads433_i28167.dmp

Name: dms.metrics
Dump Files: dms_metrics434_i28167.dmp

Name: odl.logs
Dump Arguments: timestamp=1514195154040
Dump Files: odl_logs435_i28167.dmp

Name: wls.image
Dump Files: diagnostic_image_B2B_CMSServer_Node1_2017_12_25_12_45_54.zip

GATHER DETAILS

By querying the OMS database get details about the latest alert

ALTER SESSION SET NLS_DATE_FORMAT='DD-MM-YYYY HH24:MI:SS';
SET LINES 300
SET PAGES 999
COL MESSAGE FOR A150

SELECT * FROM (
   SELECT TO_DATE(COLLECTION_TIMESTAMP,'DD-MM-YYYY HH24:MI:SS') "COLLECTION_TIMESTAMP", MESSAGE 
   FROM   MGMT_VIEW.MGMT$ALERT_NOTIF_LOG 
   WHERE  MESSAGE LIKE '%BEA-101020%' AND 
   UPPER(TARGET_NAME) LIKE UPPER('%/b2b_domain/%') ORDER BY 1 DESC
) WHERE ROWNUM < 3; 

COLLECTION_TIMESTAM MESSAGE
------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
25-12-2017 12:50:14 Incident (BEA-101020 [HTTP][java.lang.NumberFormatException]) detected in /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node1/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node1/alert/log.xml at time/line number: Mon Dec 25 12:45:54 2017/114796

25-12-2017 12:41:10 Incident (BEA-101020 [HTTP][java.lang.NumberFormatException]) detected in /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/servers/B2B_CMSServer_Node2/adr/diag/ofm/b2b_domain/B2B_CMSServer_Node2/alert/log.xml at time/line number: Mon Dec 25 12:41:01 2017/117235

Number of errors by month

COL SYSTIMESTAMP FOR A35 
SELECT SYSTIMESTAMP FROM DUAL;
 
SYSTIMESTAMP
-----------------------------------
25.12.17 13:05:24,786012 +03:00

ALTER SESSION SET NLS_DATE_FORMAT='MM-YYYY';

SELECT TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') "MONTH",
       COUNT(*) "GENERATED ERRORS"
FROM   MGMT_VIEW.MGMT$ALERT_NOTIF_LOG 
WHERE  MESSAGE LIKE '%BEA-101020%' AND 
       UPPER(TARGET_NAME) LIKE UPPER('%/b2b_domain/%') 
GROUP BY TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') ORDER BY 1;

MONTH   GENERATED ERRORS
------- ----------------
11-2017              338
12-2017             1432

Number of errors by target name

SET LINES 300
COL TARGET_NAME FOR A60 
BREAK ON "MONTH"
SELECT TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') "MONTH",
       TARGET_NAME,
       COUNT(*) "GENERATED ERRORS"
FROM   MGMT_VIEW.MGMT$ALERT_NOTIF_LOG 
WHERE  MESSAGE LIKE '%BEA-101020%' AND 
       UPPER(TARGET_NAME) LIKE UPPER('%/b2b_domain/%') 
GROUP BY TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY'),TARGET_NAME
ORDER BY 1,2;

MONTH   TARGET_NAME                                                  GENERATED ERRORS
------- ------------------------------------------------------------ ----------------
11-2017 /b2b_domain_b2b_domain/b2b_domain/B2B_AppsServer_Node1                     68
        /b2b_domain_b2b_domain/b2b_domain/B2B_AppsServer_Node2                     55
        /b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node1                     106
        /b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node2                     109
12-2017 /b2b_domain_b2b_domain/b2b_domain/AdminServer                               1
        /b2b_domain_b2b_domain/b2b_domain/B2B_AppsServer_Node1                    241
        /b2b_domain_b2b_domain/b2b_domain/B2B_AppsServer_Node2                    237
        /b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node1                     463
        /b2b_domain_b2b_domain/b2b_domain/B2B_CMSServer_Node2                     490

SOLUTION

On weblogic host(s) modify Module-FMWDFW-2818.xml file. Set UncheckedException to false and restart weblogic.

[oracle@dbpilot ~]# locate Module-FMWDFW-2818.xml
/d00/app/oracle/Middleware/user_projects/domains/b2b_domain/config/diagnostics/Module-FMWDFW-2818.xml

[oracle@dbpilot ~]# cd /d00/app/oracle/Middleware/user_projects/domains/b2b_domain/config/diagnostics
[oracle@dbpilot diagnostics]# egrep UncheckedException Module-FMWDFW-2818.xml -A 1
      <name>UncheckedException</name>
      <enabled>true</enabled>
	  
[oracle@dbpilot diagnostics]# cp Module-FMWDFW-2818.xml Module-FMWDFW-2818.xml.$(date +%F)
[oracle@dbpilot diagnostics]# ls -lh Module-FMWDFW-2818.xml* --time-style='+'
-rw-r----- 1 oracle oinstall 1.8K  Module-FMWDFW-2818.xml
-rw-r----- 1 oracle oinstall 1.8K  Module-FMWDFW-2818.xml.2017-12-25
-rw-r----- 1 oracle oinstall 1.8K  Module-FMWDFW-2818.xml.orig.22AUG2017

[oracle@dbpilot diagnostics]# egrep UncheckedException Module-FMWDFW-2818.xml -A 1
      <name>UncheckedException</name>
      <enabled>false</enabled>

The UncheckedException is responsible for generating BEA-101020 alerts

[root@dbpilot diagnostics]# egrep UncheckedException Module-FMWDFW-2818.xml -A 3
<name>UncheckedException</name>
<enabled>false</enabled>
<rule-type>Log</rule-type>
<rule-expression>(SEVERITY = 'Error') AND ((MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))</rule-expression>

After service restart no new BEA-101020 error will be spooled into log.xml anymore.

Version : 16:15 25.12.2017

HOW TO SUPPRESS BEA-101020 ALERTING

Tags In

Leave a Reply Cancel reply

Categories

HOW TO SUPPRESS BEA-101020 ALERTING

Tags In

Related Posts

THE AUD$ TABLE IS VERY LARGE

STANDARD QUERY g0bggfqrddc4w CAUSES THE PL/SQL lock timer EVENT TO CONSUME MOST PART OF DBTIME

A 13c AGENT DEPLOYMENT PROCESS CRASHES ON REDHAT/OEL 7.X TARGET HOSTS

Leave a Reply Cancel reply

Categories

Tags