HOW TO SUPPRESS BEA-000337 ALERTING
Enterprise Manager
0
QUESTIONS
I receive lots of emails with these errors reported as Critical either from Oracle Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4) and from Oracle Enterprise Manager Cloud Control 13c Release 1 (13.1.0.0.0). I want to stop them to be generated.
RELATED
SYMPTOMS
Host=dbpilot.net Target type=Oracle WebLogic Server Target name=/EMGC_GCDomain/GCDomain/EMGC_OMS1 Categories=Diagnostics, Fault Message=Incident (BEA-000337 [/em/websvcs/emws/ConsoleJobStepExecutorService]) detected in /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml at time/line number: Tue Dec 20 10:10:23 2016/1622 Severity=Critical Event reported time=Dec 20, 2016 10:13:14 AM MSK Customer Support Identifier=10000001 Operating System=Linux Platform=x86_64 Associated Incident Id=13093 Associated Incident Status=New Associated Incident Owner= Associated Incident Acknowledged By Owner=No Associated Incident Priority=None Associated Incident Escalation Level=0 Event Type=Metric Alert Event name=alertLogAdrIncident:adr_problemKey Metric Group=Incident Metric=Problem Key Metric value=BEA-000337 [/em/websvcs/emws/ConsoleJobStepExecutorService] Key Value=Tue Dec 20 10:10:23 2016/1622 Key Column 1=Timeline Rule Name=Notifications All,rule 156 Rule Owner=SYSMAN Update Details: Incident (BEA-000337 [/em/websvcs/emws/ConsoleJobStepExecutorService]) detected in /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml at time/line number: Tue Dec 20 10:10:23 2016/1622
The error details at log.xml on weblogic server
[oracle@dbpilot ~] tail /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml
</msg>
<msg time='2016-12-20T10:10:23.377+03:00' org_id='oracle' comp_id='ofm'
msg_id='1235543872' type='INCIDENT_ERROR' level='1'
host_id='dbpilot.net' host_addr='10.10.10.10' prob_key='BEA-000337 [/em/websvcs/emws/ConsoleJobStepExecutorService]'
upstream_comp='' downstream_comp='' ecid='3feb7623-3f11-4fdc-8654-90eda224f5c1-00000005'
errid='153' detail_path='/orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/incident/incdir_153'>
<txt>Errors in directory: /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/incident/incdir_153 (incident=153):
stuck thread detected: [STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'
</txt>
</msg>
GATHER DETAILS
By querying the OMS database get details about the latest alert
SET LINES 300
SET PAGES 999
COL MESSAGE FOR A150
SELECT TO_DATE(COLLECTION_TIMESTAMP,'DD-MM-YYYY HH24:MI:SS') "DATE" ,MESSAGE "NUMBER" FROM MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE MESSAGE LIKE '%BEA-000337%'
GROUP BY TO_DATE(COLLECTION_TIMESTAMP,'DD-MM-YYYY HH24:MI:SS'),MESSAGE ORDER BY 1,2;
20-DEC-16
Incident (BEA-000337 [/em/websvcs/emws/ConsoleJobStepExecutorService]) detected in /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml at time/line number: Tue Dec 20 09:10:23 2016/1613
20-DEC-16
Incident (BEA-000337 [/em/websvcs/emws/ConsoleJobStepExecutorService]) detected in /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml at time/line number: Tue Dec 20 10:10:23 2016/1622
SOLUTION
Set StuckThread to false in Module-FMWDFW-2818.xml file and restart service (OMS, weblogic).
[oracle@dbpilot ~] locate Module-FMWDFW-2818.xml
/orastage/oem13100/gc_inst/user_projects/domains/GCDomain/config/diagnostics/Module-FMWDFW-2818.xml
[oracle@dbpilot ~] cd /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/config/diagnostics/
[oracle@dbpilot ~] cp Module-FMWDFW-2818.xml Module-FMWDFW-2818.xml_orig_21Dec2016
.. Before
[oracle@dbpilot ~] egrep -i stuck Module-FMWDFW-2818.xml -A 4
<name>StuckThread</name>
<enabled>true</enabled>
<rule-type>Log</rule-type>
<rule-expression>(SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))</rule-expression>
<alarm-type>AutomaticReset</alarm-type>
.. After
[oracle@dbpilot ~] egrep -i stuck Module-FMWDFW-2818.xml -A 4
<name>StuckThread</name>
<enabled>false</enabled>
<rule-type>Log</rule-type>
<rule-expression>(SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))</rule-expression>
<alarm-type>AutomaticReset</alarm-type>
After service restart no new BEA-000337 error will be spooled into log.xml anymore.
Version : 17:38 25.12.2017
Tags In
Related Posts
Leave a Reply Cancel reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Certificates
Categories
- Linux (8)
- Iptables (2)
- Memory Usage (1)
- Postfix (1)
- Yum (4)
- Networker (2)
- Oracle (63)
- Account (2)
- AWR (2)
- Cipher Suites (1)
- Database Errors (5)
- Database Performance (9)
- Datapump (3)
- Enterprise Manager (24)
- Enterprise Manager Metrics (4)
- Environments (4)
- High CPU Usage (2)
- High IO Load (4)
- Initialization Parameters (5)
- job_queue_processes (1)
- Processes (3)
- NLS (1)
- Objects (1)
- Objects DDL (2)
- OJVM/PSU/DRU Patches (1)
- PL/SQL (5)
- UTL_FILE (4)
- Prerequisites (1)
- Profile (1)
- Queries (2)
- Redologs (1)
- Resource Limits (1)
- Segment Shrink (1)
- Session (6)
- Spfile (3)
- SQL PLUS (1)
- Spooling (1)
- Standard Queries (7)
- Statistics (2)
- Temporary Tablespace (2)
Tags
/etc/sudoers
Account
Agent
Agent 12c Deinstall
Agent 13c Deployment
Automated Maintenance Tasks
AWR
BLOCKING SESSION
Database Auditing
Database Directory
Database Performance
Datapump
DBMS_SYSTEM
Enterprise Manager
Enterprise Manager Metrics
Failed Logins (Historical) Metric
Failed Logins Metric
Gather Statistics Job
GET_ENV
High IO Load
Idle Sessions
Initialization Parameters
LOCKED(TIMED)
OEM
OEM 12.1.0.4
OEM 13.1.0.0.0
OMS
Processes
Repository
Retrieve Object DDL
ROW LOCK
Scheduler
Schema Statistics
Scripts
Sessions
spfile
SQL_ID
Standard Query
TRANSACTION
User Grants
UTL_FILE
WAITING SESSION
Window Group
Window Name
Yum