FIXING ADFC-00032 [ADFc] IN OEM 13.1.0.0.0
REQUEST
I receive lots of emails with these errors reported as Critical from Oracle Enterprise Manager Cloud Control 13c Release 1 (13.1.0.0.0). I want to stop them to be generated.
SYMPTOMS
Host=dbpilot.net Target type=Oracle Management Service Target name=dbpilot.net:4889_Management_Service Categories=Diagnostics, Error Message=Incident (ADFC-00032 [ADFc]) detected in /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/EMGC_DOMAIN/EMOMS/alert/log.xml at time/line number: Tue Dec 26 13:07:22 2017/11012. Severity=Critical Event reported time=Dec 26, 2017 1:07:43 PM MSK Customer Support Identifier=10000001 Operating System=Linux Platform=x86_64 Associated Incident Id=45967 Associated Incident Status=New Associated Incident Owner= Associated Incident Acknowledged By Owner=No Associated Incident Priority=None Associated Incident Escalation Level=0 Event Type=Metric Alert Event name=adrAlertLogIncidentError:genericIncidentErrStack Metric Group=Incident Metric=Diagnostic Incident Metric value=Errors in directory: /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/EMGC_DOMAIN/EMOMS/incident/incdir_1191 (incident=1191):~ADF Controller encountered a serious error. Key Value=Tue Dec 26 13:07:22 2017/11012 Key Column 1=Time/Line Number Rule Name=Notifications All,rule 156 Rule Owner=SYSMAN Update Details: Incident (ADFC-00032 [ADFc]) detected in /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/EMGC_DOMAIN/EMOMS/alert/log.xml at time/line number: Tue Dec 26 13:07:22 2017/11012.
The error details at log.xml on OMS server
[oracle@dbpilot ~] cd /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/EMGC_DOMAIN/EMOMS/alert
[oracle@dbpilot ~] tail log.xml
</msg>
<msg time='2017-12-26T13:07:22.187+03:00' org_id='oracle' comp_id='ofm'
msg_id='1809474518' type='INCIDENT_ERROR' level='1'
host_id='dbpilot.net' host_addr='10.10.10.10' prob_key='ADFC-00032 [ADFc]'
upstream_comp='' downstream_comp='' ecid='005OJhmavbmFg485zj1FiW0009Nm0006cv'
errid='1191' detail_path='/orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/EMGC_DOMAIN/EMOMS/incident/incdir_1191'>
<txt>Errors in directory: /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/EMGC_DOMAIN/EMOMS/incident/incdir_1191 (incident=1191):
ADF Controller encountered a serious error.
</txt>
</msg>
A content of readme file in incdir_1191 directory
[oracle@dbpilot ~] cat readme.txt
Incident Id: 1191
Incident Source: SYSTEM
Create Time: Tue Dec 26 13:07:22 MSK 2017
Problem Key: ADFC-00032 [ADFc]
ECID: 005OJhmavbmFg485zj1FiW0009Nm0006cv
Thread Id: 1775376
Application Name: emgc
User Name: SYSMAN
Error Message Id: ADFC-00032
Context Values
--------------
ApplicationName : emgc
SubjectUsername : SYSMAN
DSID : 0000M2IIQMPFg485zj1FiW1QAF^c0001Mr
ServletUsername : SYSMAN
DFW_SERVER_NAME : EMGC_OMS1
DFW_DOMAIN_NAME : GCDomain
ServletURI : /em/faces/core-jobs-results-executionDetails
DFW_USER_NAME : SYSMAN
DFW_APP_NAME : emgc
HTTP Request
------------
StartTime: 2017-12-26 13:07:17.327
User: SYSMANECID: 005OJhmavbmFg485zj1FiW0009Nm0006cv,0:1
Context Values:
ApplicationName: emgc
SubjectUsername: SYSMAN
ServletUsername: SYSMAN
DSID: 0000M2IIQMPFg485zj1FiW1QAF^c0001Mr
ServletURI: /em/faces/core-jobs-results-executionDetails
URI: /em/core/jobs/results/executionDetails.jspx
RequestType: GET
QueryString: Adf-Rich-Message=true&unique=1514282804577&oracle.adf.view.rich.STREAM=emT:r1:exec_table:pc1:exectt,emT:r1:0:r1:0:opTbl&Adf-Page-Id=3&javax.faces.ViewState=!-197klq3ezk&Adf-Window-Id=w0
All Headers Names:
Host, User-Agent, Accept, Accept-Language, Accept-Encoding, Referer, Cookie, ECID-Context, Connection, X-Forwarded-For, X-WebLogic-KeepAliveSecs, X-WebLogic-Force-JVMID
Selected Header Values:
ECID-Context : 1.005OJhmavbmFg485zj1FiW0009Nm0006cv;kXjE
Accept-Language : ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
Host : dbpilot.net:7802
Referer : https://dbpilot.net:7802/em/faces/core-jobs-results-executionDetails?execId=612C3DBC81E583C8E0531E01040A863E&_afrRedirect=3733355492352977
Accept-Encoding : gzip, deflate
User-Agent : Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Connection : Keep-Alive
Accept : text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Stack Trace
-----------
java.lang.Throwable: dummy exception to generate stacktrace leading up to incident
at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl.createIncident(DiagnosticsDataExtractorImpl.java:287)
at oracle.adfinternal.controller.util.LogUtils.reportIncident(LogUtils.java:844)
at oracle.adfinternal.controller.util.LogUtils.reportIncident(LogUtils.java:793)
at oracle.adf.controller.internal.binding.TaskFlowRegionModelViewPort.doProcessBeginRegion(TaskFlowRegionModelViewPort.java:237)
at oracle.adf.controller.internal.binding.TaskFlowRegionModelViewPort.processBeginRegion(TaskFlowRegionModelViewPort.java:163)
at oracle.adf.controller.internal.binding.TaskFlowRegionModelLocal.processBeginRegion(TaskFlowRegionModelLocal.java:12)
at oracle.adf.view.rich.component.fragment.UIXRegion$RegionContextChange.doChangeImpl(UIXRegion.java:1436)
at oracle.adf.view.rich.context.DoableContextChange.doChange(DoableContextChange.java:91)
at oracle.adf.view.rich.component.fragment.UIXRegion._beginInterruptibleRegion(UIXRegion.java:777)
at oracle.adf.view.rich.component.fragment.UIXRegion.processRegion(UIXRegion.java:520)
at oracle.adfinternal.view.faces.taglib.region.AbstractRegionTag.doStartTag(AbstractRegionTag.java:70)
at oracle.adfinternal.view.faces.taglib.region.RegionTag.doStartTag(RegionTag.java:92)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:50)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:45)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:45)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:89)
at oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:89)
at oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:424)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:845)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.sysman.eml.app.JSPFilter.doFilter(JSPFilter.java:86)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:137)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:460)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:120)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:217)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:81)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:588)
at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:481)
at weblogic.servlet.jsp.PageContextImpl.include(PageContextImpl.java:162)
at weblogic.servlet.jsp.PageContextImpl.include(PageContextImpl.java:183)
at oracle.adfinternal.view.faces.taglib.region.IncludeTag.__include(IncludeTag.java:473)
at oracle.adfinternal.view.faces.taglib.region.AbstractRegionTag$1.call(AbstractRegionTag.java:101)
at oracle.adfinternal.view.faces.taglib.region.AbstractRegionTag$1.call(AbstractRegionTag.java:71)
at oracle.adf.view.rich.component.fragment.UIXRegion.processRegion(UIXRegion.java:526)
at oracle.adfinternal.view.faces.taglib.region.AbstractRegionTag.doStartTag(AbstractRegionTag.java:70)
at oracle.adfinternal.view.faces.taglib.region.RegionTag.doStartTag(RegionTag.java:92)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:50)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:45)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:45)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspIterationTagNode.executeHandler(OracleJspIterationTagNode.java:45)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspClassicTagNode.evalBody(OracleJspClassicTagNode.java:87)
at oracle.jsp.runtime.tree.OracleJspBodyTagNode.executeHandler(OracleJspBodyTagNode.java:58)
at oracle.jsp.runtime.tree.OracleJspCustomTagNode.execute(OracleJspCustomTagNode.java:263)
at oracle.jsp.runtime.tree.OracleJspNode.execute(OracleJspNode.java:89)
at oracle.jsp.runtimev2.ShortCutServlet._jspService(ShortCutServlet.java:89)
at oracle.jsp.runtime.OracleJspBase.service(OracleJspBase.java:29)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:424)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:845)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.sysman.eml.app.JSPFilter.doFilter(JSPFilter.java:86)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:202)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:137)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:460)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:120)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:217)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:81)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:588)
at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:273)
at com.sun.faces.context.ExternalContextImpl.dispatch(Unknown Source)
at javax.faces.context.ExternalContextWrapper.dispatch(Unknown Source)
at javax.faces.context.ExternalContextWrapper.dispatch(Unknown Source)
at javax.faces.context.ExternalContextWrapper.dispatch(Unknown Source)
at javax.faces.context.ExternalContextWrapper.dispatch(Unknown Source)
at org.apache.myfaces.trinidadinternal.context.FacesContextFactoryImpl$OverrideDispatch.dispatch(FacesContextFactoryImpl.java:167)
at com.sun.faces.application.view.JspViewHandlingStrategy.executePageToBuildView(Unknown Source)
at com.sun.faces.application.view.JspViewHandlingStrategy.buildView(Unknown Source)
at org.apache.myfaces.trinidad.view.ViewDeclarationLanguageWrapper.buildView(ViewDeclarationLanguageWrapper.java:94)
at org.apache.myfaces.trinidad.view.ViewDeclarationLanguageWrapper.buildView(ViewDeclarationLanguageWrapper.java:94)
at org.apache.myfaces.trinidadinternal.application.ViewDeclarationLanguageFactoryImpl$ChangeApplyingVDLWrapper.buildView(ViewDeclarationLanguageFactoryImpl.java:322)
at oracle.adfinternal.view.faces.lifecycle.ResponseRenderManager._processViewDefinitionLanguage(ResponseRenderManager.java:105)
at oracle.adfinternal.view.faces.lifecycle.ResponseRenderManager.runRenderView(ResponseRenderManager.java:41)
at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._renderResponse(LifecycleImpl.java:1095)
at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:389)
at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:255)
at javax.faces.webapp.FacesServlet.service(Unknown Source)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:192)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.adf.view.page.editor.webapp.WebCenterComposerFilter.doFilter(WebCenterComposerFilter.java:117)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.adf.share.http.ServletADFFilter.doFilter(ServletADFFilter.java:77)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:105)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:502)
at oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:60)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:502)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:327)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:229)
at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:800)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.sysman.eml.app.MBeanServerConnFilter.doFilter(MBeanServerConnFilter.java:43)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:202)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.sysman.emInternalSDK.core.model.utils.EntityManagerFilter.doFilter(EntityManagerFilter.java:46)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:561)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.adfinternal.view.faces.caching.filter.AdfFacesCachingFilterImpl.doFilter(AdfFacesCachingFilterImpl.java:142)
at oracle.adf.view.rich.webapp.AdfFacesCachingFilter.doFilter(AdfFacesCachingFilter.java:66)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.sysman.core.app.exception.EMRuntimeExceptionFilter.doFilter(EMRuntimeExceptionFilter.java:150)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:137)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:460)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:120)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:217)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:81)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:220)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3436)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3402)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2285)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2201)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1575)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
Diagnostic Dumps
----------------
Name: jvm.threads
Dump Arguments: context=true, timing=true
Dump Files: jvm_threads276_i1191.txt
Name: adf.AdfcDiagnosticDump
Application Name: emgc
Dump Files: adf_AdfcDiagnosticDump281_i1191.txt
Name: adf.ADFConfigDiagnosticDump
Application Name: emgc
Dump Files: adf_ADFConfigDiagnosticDump282_i1191.txt
Name: adf.ADFConfigPropertiesDump
Application Name: emgc
Dump Files: adf_ADFConfigPropertiesDump284_i1191.txt
Name: dms.metrics
Dump Arguments: dump=LocalDMS
Dump Files: dms_metrics277_i1191.txt
Name: dfw.samplingArchive
Dump Files: dfw_samplingArchive279_i1191.JVMThreadDump.txt,dfw_samplingArchive279_i1191.readme.txt
Name: odl.logs
Dump Arguments: match_all=false, timestamp=1514282842187, ecid=005OJhmavbmFg485zj1FiW0009Nm0006cv
Dump Files: odl_logs280_i1191.txt
Name: adf.DiagnosticsJarsVersionDump
Dump Files: adf_DiagnosticsJarsVersionDump283_i1191.txt
GATHER DETAILS
By querying the OMS database get details about the latest alert before any changes
ALTER SESSION SET NLS_DATE_FORMAT='DD-MM-YYYY HH24:MI:SS';
SET line 300
SET pages 999
COL MESSAGE FOR A150
SELECT * FROM (
SELECT TO_DATE(COLLECTION_TIMESTAMP,'DD-MM-YYYY HH24:MI:SS') "COLLECTION_TIMESTAMP", MESSAGE
FROM MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE MESSAGE LIKE '%ADFC-00032 %' ORDER BY 1 DESC
) WHERE ROWNUM < 3;
COLLECTION_TIMESTAM MESSAGE
------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
26-12-2017 13:07:43 Incident (ADFC-00032 [ADFc]) detected in /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/EMGC_DOMAIN/EMOMS/alert/log.xml at time/line number: Tue Dec 26 13:07:22 2017/11012.
26-12-2017 13:07:43 Incident (ADFC-00032 [ADFc]) detected in /orastage/oem13100/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/EMGC_DOMAIN/EMOMS/alert/log.xml at time/line number: Tue Dec 26 13:07:20 2017/11003.
COL SYSTIMESTAMP FOR A35
SELECT SYSTIMESTAMP FROM DUAL;
SYSTIMESTAMP
-----------------------------------
26-DEC-17 02.17.43.885503 PM +03:00
Total errors
SET LINES 300
COL TARGET_NAME FOR A50
COL TARGET_TYPE FOR A10
SELECT TARGET_NAME, TARGET_TYPE, COUNT(*) "GENERATED ERRORS"
FROM MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE MESSAGE LIKE '%ADFC-00032%'
GROUP BY TARGET_NAME, TARGET_TYPE ORDER BY 1;
TARGET_NAME TARGET_TYP GENERATED ERRORS
------------------------------------ ---------- ----------------
dbpilot.net:4889_Management_Service oracle_oms 1011
Number of errors by month
ALTER SESSION SET NLS_DATE_FORMAT='MM-YYYY';
SELECT TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') "MONTH",
COUNT(*) "GENERATED ERRORS"
FROM MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE MESSAGE LIKE '%ADFC-00032%'
GROUP BY TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') ORDER BY 1;
MONTH GENERATED ERRORS
------- ----------------
03-2017 72
04-2017 65
05-2017 107
06-2017 20
07-2017 25
08-2017 45
09-2017 48
10-2017 100
11-2017 55
12-2017 45
Number of errors by target name
ALTER SESSION SET NLS_DATE_FORMAT='MM-YYYY';
SET LINES 300
COL TARGET_NAME FOR A60
BREAK ON TARGET_NAME
SELECT TARGET_NAME,TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY') "MONTH",
COUNT(*) "GENERATED ERRORS"
FROM MGMT_VIEW.MGMT$ALERT_NOTIF_LOG
WHERE MESSAGE LIKE '%ADFC-00032%'
GROUP BY TARGET_NAME,TO_DATE(COLLECTION_TIMESTAMP,'MM-YYYY')
ORDER BY 1,2;
TARGET_NAME MONTH GENERATED ERRORS
-------------------------------------- ------- ----------------
dbpilot.net:4889_Management_Service 03-2017 72
04-2017 65
05-2017 107
06-2017 20
07-2017 25
08-2017 45
09-2017 48
10-2017 100
11-2017 55
12-2017 45
SOLUTION
A) Perform a full backup of all components (OMS, Database, Agent)
B) Apply patches
For EM 13.2 Cloud Control:
Apply the following PSU for the EM 13.2 Cloud Control OMS (or later):
EMBPPSU 13.2.0.0.170418,
Alternatively, apply Patch 24742172 to the EM 13.2 OMS Oracle Home according to the Patch README file.
For EM 13.1 Cloud Control:
Apply the following PSU for the EM 13.1 Cloud Control OMS (or later):
EMBPPSU 13.1.0.0.170718, Patch 25904755, or later
Alternatively, apply Patch 24742172 to the EM 13.1 OMS Oracle Home according to the Patch README file.
REFERENCES
EM 13c: ADFC-00032 [ADFc] Error Messages Reported Frequently in Enterprise Manager 13c Cloud Control (Doc ID 2141011.1)
EM 13c: Enterprise Manager 13.2 Cloud Control Repeatedly Reports Log Monitor Incident with ADFC-00032 Errors (Doc ID 2240806.1)
NOTES for EM 13.1 Cloud Control
(a) Patch 23069794 (Doc ID 2141011.1) does not fix the ADFC-00032 [ADFc] issue
(b) Before installing patch 24742172 (Doc ID 2240806.1) refer to 2335997.1
EM 13: EM Middleware (EMGC_GCDomain) Targets and System Infrastructure Server Target show down/pending with Incidents after EM PSUs/OMS/Agent Bundle Patches are applied using opatch 13.9.2.1.0 (Doc ID 2335997.1)
(c) Don’t apply any patches with faulty 6880880 (OPatch Version: 13.9.2.1.0) until Opatch patch 27155076 is applied.
The patch 24742172 must be installed in the following order :
1. OMS, Update OMSPatcher utility by installing patch 19999993 (OMSPatcher Version: 13.6.0.0.4).
2. OMS and Agent, Update OPatch utility by installing patch 6880880 (OPatch Version: 13.9.2.1.0)
3. OMS and Agent, Apply the OPatch Patch 27155076
4. OMS, Apply the patch 24742172
These steps were actually at the time of the post writing (17:20 28.12.2017)
In this test case I’ve applied 24742172 on EM 13.1 Cloud Control and it’s fixed the issue.
(d) Added later, after 1 month [16:21 31.01.2018]
The 24742172 patch hasn’t fixed the issue completely, but a number of generated errors have noticeably decreased.
TARGET_NAME MONTH GENERATED ERRORS
------------------------------------------------------------ ------- ----------------
dbpilot.net:4889_Management_Service 04-2017 65
05-2017 107
06-2017 20
07-2017 25
08-2017 45
09-2017 48
10-2017 100
11-2017 55
12-2017 50
01-2018 8
Version : 17:20 28.12.2017
Platform : Red Hat 6.8
OEM : 13.1.0.0.0
Platform : Red Hat 6.8
OEM : 13.1.0.0.0