Print Friendly, PDF & Email
By the following steps in this post, you can fix the Oracle Management Agent status when it’s stopped working and doesn’t send any data to the OMS anymore.

PROBLEM
The emctl status agent command output shows that the agent is Running and Ready but its collection status is in the COLLECTIONS_HALTED

SYMPTOMS
The agent status shows

oracle@dbpilot|09:23|~/scripts: emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.5.0
OMS Version            : 13.3.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /agent12c/agent_inst
Agent Log Directory    : /agent12c/agent_inst/sysman/log
Agent Binaries         : /agent12c/core/12.1.0.5.0
Agent Process ID       : 21684
Parent Process ID      : 21574
Agent URL              : https://dbpilot.net:3872/emd/main/
Local Agent URL in NAT : https://dbpilot.net:3872/emd/main/
Repository URL         : https://oem13c-r3.net:4903/empbs/upload
Started at             : 2019-07-23 10:22:59
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.215.10.el5uek (amd64)
Last Reload            : (none)
Last successful upload                       : 2019-07-23 10:23:24
Last attempted upload                        : 2019-07-23 10:23:24
Total Megabytes of XML files uploaded so far : 0.03
Number of XML files pending upload           : 4,911
Size of XML files pending upload(MB)         : 2.48
Available disk space on upload filesystem    : 49.52%
Collection Status                            : [COLLECTIONS_HALTED
(UPLOAD SYSTEM Threshold - unable to purge files in upload system)]
Backoff Expiration                           : 2019-07-23 10:23:56
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2019-07-23 10:23:20
Last successful heartbeat to OMS             : 2019-07-23 10:23:20
Next scheduled heartbeat to OMS              : 2019-07-23 10:24:21

---------------------------------------------------------------
Agent is Running and Ready

Because of the COLLECTIONS_HALTED status, all related Enterprise Manager targets of that host will be in DOWN state.

SOLUTION
The Doc ID 2462039.1 recommends executing the steps 1-5 against the Oracle Management Server(OMS) and the steps 6-7 against a problem agent

1 Stop the OMS.

$OMS_HOME/bin/emctl stop oms -all -force

2 After the OMS is stopped verify no processes are left running

{
ps -ef | grep EMGC_ADMINSERVER | grep -v grep
ps -ef | grep EMGC_OMS1 | grep -v grep
ps -ef | grep java | grep -v grep
ps -ef | grep opmn | grep -v grep
}

If there are any left OMS java processes, kill them by using the kill command

kill -9 PID

where PID is a process identifier for a left process

3 Restart the OMS repository database and the database listener

-- Shutdown the database

sqlplus / as sysdba
SQL> shutdown immediate;
SQL> exit 

-- Restart the listener

lsnrctl  stop
lsnrctl  start

-- Start up the database 

sqlplus / as sysdba
SQL> startup;
SQL> exit

4 Login to the OMS repository database and bounce the job subsystem

-- Connect as the SYS user 

SQL> CONNECT / AS SYSDBA
Connected.
SQL> SHOW USER
USER is "SYS"

-- Save the current value of the job_queue_processes parameter 

SQL> sho parameter  job_queue_processes

-- Set the job_queue_processes parameter to 0

SQL> alter system set job_queue_processes=0;

-- Connect as the SYSMAN user and disable the job

SQL> connect sysman/your_sysman_password
Connected.
SQL> sho user
USER is "SYSMAN"

SQL> exec emd_maintenance.remove_em_dbms_jobs;
SQL> commit;

-- Connect as the SYS user and restore the job_queue_processes parameter

SQL> CONNECT / AS SYSDBA
Connected.
SQL> SHOW USER
USER is "SYS"

SQL> alter system set job_queue_processes=1000 scope=BOTH;

-- Connect as the SYSMAN user and enable the job

SQL> connect sysman/your_sysman_password
Connected.
SQL> sho user
USER is "SYSMAN"

SQL> exec emd_maintenance.submit_em_dbms_jobs;
SQL> commit;

5 Start up the OMS

$OMS_HOME/bin/emctl start oms

6 When the OMS is started run the following commands on the servers with the agent(s) having the collection issue

{
${AGENT_HOME}/bin/emctl stop agent
${AGENT_HOME}/bin/emctl clearstate agent
${AGENT_HOME}/bin/emctl start agent
${AGENT_HOME}/bin/emctl upload agent
}
NOTE

The agent may have many files to upload, and you might need to issue the upload command several times before all files have been uploaded.

 
7Verify that the issue has gone

oracle@dbpilot|10:14|~:<agent12c><strong>${AGENT_HOME}/bin/emctl status agent</strong>
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.5.0
OMS Version            : 13.3.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /agent12c/agent_inst
Agent Log Directory    : /agent12c/agent_inst/sysman/log
Agent Binaries         : /agent12c/core/12.1.0.5.0
Agent Process ID       : 31671
Parent Process ID      : 31488
Agent URL              : https://dbpilot.net:3872/emd/main/
Local Agent URL in NAT : https://dbpilot.net:3872/emd/main/
Repository URL         : https://oem13c-r3.net:4903/empbs/upload
Started at             : 2019-07-23 11:12:26
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.215.10.el5uek (amd64)
Last Reload            : (none)
Last successful upload                       : 2019-07-23 11:15:04
Last attempted upload                        : 2019-07-23 11:15:04
Total Megabytes of XML files uploaded so far : 0.74
Number of XML files pending upload           : 1
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 49.51%
<span class="blue">Collection Status                            : Collections enabled</span>
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2019-07-23 11:14:22
Last successful heartbeat to OMS             : 2019-07-23 11:14:22
Next scheduled heartbeat to OMS              : 2019-07-23 11:15:22

---------------------------------------------------------------
Agent is Running and Ready

As the result the Collection Status of the agent is changed to the Collections enabled.

REFERENCES
EM 13.2 Agent Unreachable – Halted – UPLOAD SYSTEM Threshold – Unable To Purge Files In Upload System (Doc ID 2462039.1)

 
 

Version  : 10:20 23.07.2019
Revision : 17:12 25.01.2021
Platform : Red Hat 7.6
OEM      : 13.3.0.0.0