Print Friendly, PDF & Email

 
ISSUE

emctl status agent‘s output shows that an agent is “Running and Ready” but its collection status is [COLLECTIONS_HALTED(UPLOAD SYSTEM Threshold – unable to purge files in upload system)]
 

 
SYMPTOMS

oracle@dbpilot|09:23|~/scripts:emctl status  agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.5.0
OMS Version            : 13.3.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /agent12c/agent_inst
Agent Log Directory    : /agent12c/agent_inst/sysman/log
Agent Binaries         : /agent12c/core/12.1.0.5.0
Agent Process ID       : 21684
Parent Process ID      : 21574
Agent URL              : https://dbpilot.net:3872/emd/main/
Local Agent URL in NAT : https://dbpilot.net:3872/emd/main/
Repository URL         : https://oem13c-r3.net:4903/empbs/upload
Started at             : 2019-07-23 10:22:59
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.215.10.el5uek (amd64)
Last Reload            : (none)
Last successful upload                       : 2019-07-23 10:23:24
Last attempted upload                        : 2019-07-23 10:23:24
Total Megabytes of XML files uploaded so far : 0.03
Number of XML files pending upload           : 4,911
Size of XML files pending upload(MB)         : 2.48
Available disk space on upload filesystem    : 49.52%
Collection Status                            : [COLLECTIONS_HALTED(
  UPLOAD SYSTEM Threshold - unable to purge files in upload system)]
Backoff Expiration                           : 2019-07-23 10:23:56
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2019-07-23 10:23:20
Last successful heartbeat to OMS             : 2019-07-23 10:23:20
Next scheduled heartbeat to OMS              : 2019-07-23 10:24:21

---------------------------------------------------------------
Agent is Running and Ready

Because of this status, OEM targets that are related to this host will be in DOWN state.

 
SOLUTION

Based on the Doc ID 2462039.1 the step to fix the issue are the following :

1. Stop the Oracle Management Server ( OMS). After the OMS is stopped, please verify no processes are left over running

$OMS_HOME/bin/emctl stop oms -all -force

2. Verify no processes related to the OMS are hanging:

{
ps -ef | grep EMGC_ADMINSERVER | grep -v grep
ps -ef | grep EMGC_OMS1 | grep -v grep
ps -ef | grep java | grep -v grep
ps -ef | grep opmn | grep -v grep
}

If there are any left OMS java processes, kill them by using kill command

kill -9 PID

where PID is a process identifier for a left process

3. Restart the OMS’s repository database and the database listener

sqlplus / as sysdba
SQL> shutdown immediate;
exit 

lsnrctl  stop
lsnrctl  start

sqlplus / as sysdba
SQL> startup;

4. Login to the OMS’s repository database as SYS and bounce the job subsystem

-- Get and remember the current value of the job_queue_processes parameter 
SQL> sho parameter  job_queue_processes

-- Set the job_queue_processes parameter to 0
SQL> alter system set job_queue_processes=0;

-- Connect as SYSMAN user and disable the job
SQL> connect sysman/your_sysman_password
Connected.
SQL> sho user
USER is "SYSMAN"

SQL> exec emd_maintenance.remove_em_dbms_jobs;
SQL> commit;

-- Connect as SYS user again and reset the job_queue_processes parameter to its original value that you've got in previous step
SQL> sho user
USER is "SYS"

SQL> alter system set job_queue_processes=1000 scope=BOTH;

-- Connect as SYSMAN user again and enable the job
SQL> sho user
USER is "SYSMAN"

SQL> exec emd_maintenance.submit_em_dbms_jobs;
SQL> commit;

5. Start the OMS

$OMS_HOME/bin/emctl start oms

6. When the OMS is started run the following commands on the servers for the affected agents

{
${AGENT_HOME}/bin/emctl stop agent
${AGENT_HOME}/bin/emctl clearstate agent
${AGENT_HOME}/bin/emctl start agent
${AGENT_HOME}/bin/emctl upload agent
}

Note: The agent may have many files to upload, and this may take several times to upload all the files.

7.Verify that the issue has gone

oracle@dbpilot|10:14|~:${AGENT_HOME}/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.5.0
OMS Version            : 13.3.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /agent12c/agent_inst
Agent Log Directory    : /agent12c/agent_inst/sysman/log
Agent Binaries         : /agent12c/core/12.1.0.5.0
Agent Process ID       : 31671
Parent Process ID      : 31488
Agent URL              : https://dbpilot.net:3872/emd/main/
Local Agent URL in NAT : https://dbpilot.net:3872/emd/main/
Repository URL         : https://oem13c-r3.net:4903/empbs/upload
Started at             : 2019-07-23 11:12:26
Started by user        : oracle
Operating System       : Linux version 2.6.39-400.215.10.el5uek (amd64)
Last Reload            : (none)
Last successful upload                       : 2019-07-23 11:15:04
Last attempted upload                        : 2019-07-23 11:15:04
Total Megabytes of XML files uploaded so far : 0.74
Number of XML files pending upload           : 1
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 49.51%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2019-07-23 11:14:22
Last successful heartbeat to OMS             : 2019-07-23 11:14:22
Next scheduled heartbeat to OMS              : 2019-07-23 11:15:22

---------------------------------------------------------------
Agent is Running and Ready

 
REFERENCES

EM 13.2 Agent Unreachable – Halted – UPLOAD SYSTEM Threshold – Unable To Purge Files In Upload System (Doc ID 2462039.1)

 
 

Version  : 10:20 AM 23.07.2019
Platform : Red Hat 7.6
OEM      : 13.3.0.0.0