Management Agent is blocked on a target host because it was reinstalled or recovered.
SYMPTOMS

When a Management Agent is requested to upload collected metric data to an OMS, the following error occurs

[oracle@dbpilot ~] emctl upload agent

Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: The agent is blocked by the OMS. Agent is out-of-sync with repository.
This most likely means that the agent was reinstalled or recovered.
Please contact an EM administrator to unblock
the agent by performing an agent resync from the console. (AGENT_BLOCKED)

A status of Management Agent is BLOCKED as well

[oracle@dbpilot ~] emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.1.0.0.0
OMS Version            : (unknown)
Protocol Version       : 12.1.0.1.0
Agent Home             : /agent13c/agent_inst
Agent Log Directory    : /agent13c/agent_inst/sysman/log
Agent Binaries         : /agent13c/agent_13.1.0.0.0
Core JAR Location      : /agent13c/agent_13.1.0.0.0/jlib
Agent Process ID       : 15031
Parent Process ID      : 14974
Agent URL              : https://dbpilot.net:3872/emd/main/
Local Agent URL in NAT : https://dbpilot.net:3872/emd/main/
Repository URL         : https://s001as-oem13c.net:4903/empbs/upload
Started at             : 2018-05-21 12:25:47
Started by user        : oracle
Operating System       : Linux version 3.10.0-514.el7.x86_64 (amd64)
Number of Targets      : 4
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 515
Size of XML files pending upload(MB)         : 0.56
Available disk space on upload filesystem    : 80.78%
Collection Status                            : Collections enabled
Heartbeat Status                             : Agent is blocked 
Blocked Reason                               : Agent is out-of-sync with repository.
                This most likely means that the agent was reinstalled or recovered.
                Please contact an EM administrator to unblock the agent by performing 
                an agent resync from the console.
Last attempted heartbeat to OMS              : 2018-05-21 12:43:52
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2018-05-21 12:46:53
---------------------------------------------------------------
Agent is Running and Ready
CAUSE

A Management Agent was reinstalled or restored from a backup. Or there is a clone of a Virtual Machine that blocked a Management Agent of an original VM in an OMS.

NOTE

An existing copy of a Virtual Machine in the same VLAN is the most likely reason for the issue.

SOLUTION

Perform the Agent Resynchronization in the Enterprise Manager Cloud Control Console

1 If you have copy of original VM in the same VLAN then stop an agent on that cloned machine and remove agent from it. See HOW TO DEINSTALL THE ENTERPRISE MANAGER 13C CLOUD CONTROL AGENT to know how to do it.

2 Start agent on original machine

## Original machine 

emclt start agent

3 In Enterprise Manager go to Targets > All Targets

AGENT_IS_OUT_OF_SYNC_WITH_REPOSITORY_001.png

4 Search for the problematic Management Agent and click on it to access the Agent's home page

AGENT_IS_OUT_OF_SYNC_WITH_REPOSITORY_002.png

5 On the Agent's home page the following error message is shown

Agent has been blocked due to Bounce Counter mismatch. If Agent has been restored from a backup perform an Agent Resync.

6 Click Agent > Resynchronization

Click "Continue" to start the Agent Resynchronization

AGENT_IS_OUT_OF_SYNC_WITH_REPOSITORY_004.png

A new job is submitted to perform the Agent Resynchronization. Click "OK" to confirm

AGENT_IS_OUT_OF_SYNC_WITH_REPOSITORY_005.png

7 Find the job details by choosing Enterprise > Job > Activity

AGENT_IS_OUT_OF_SYNC_WITH_REPOSITORY_006.png

In Available Criteria field search for the job name. The job status must be as Succeeded

AGENT_IS_OUT_OF_SYNC_WITH_REPOSITORY_007.png

NOTE

The Agent Resynchronization can be started with emcli as well

###############################
## OMS : Login as SYSMAN user
###############################

oracle@s001as-oem13c.net <OMS> emcli login -username=sysman
Enter password :

Login successful

###############################
## OMS : Start the Resynchronization 
##     : for the problematic Agent
###############################

oracle@s001as-oem13c.net <OMS> emcli resyncAgent -agent="dbpilot.net:3872"
Resync job RESYNC_20200228082544416 successfully submitted

###############################
## OMS : Verify a status of the submitted job
###############################

oracle@s001as-oem13c.net <OMS> emcli get_jobs -name="RESYNC_20200228082544416" -owner="SYSMAN"

Name                      Type            Job ID                            Execution ID                      Scheduled            Completed            TZ Offset  Status     Status ID  Owner   Target Type  Target Name
RESYNC_20200228082544416  ResyncStateJob  9F9CFD428CC747CFE0530201040A30FD  9F9CFD428CCA47CFE0530201040A30FD  2020-02-28 08:25:44  2020-02-28 08:25:49  GMT+03:00  Succeeded  5          SYSMAN  oracle_emd   dbpilot.net:3872

8 Once the job succeeds, verify the status of the Managemet Agent. The agent status must be shown as UP (a green up arrow next to the agent name)

AGENT_IS_OUT_OF_SYNC_WITH_REPOSITORY_008.png

9 On the monitored host verify the status of the Agent

[oracle@dbpilot ~] emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.1.0.0.0
OMS Version            : 13.1.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /agent13c/agent_inst
Agent Log Directory    : /agent13c/agent_inst/sysman/log
Agent Binaries         : /agent13c/agent_13.1.0.0.0
Core JAR Location      : /agent13c/agent_13.1.0.0.0/jlib
Agent Process ID       : 15031
Parent Process ID      : 14974
Agent URL              : https://dbpilot.net:3872/emd/main/
Local Agent URL in NAT : https://dbpilot.net:3872/emd/main/
Repository URL         : https://s001as-oem13c.net:4903/empbs/upload
Started at             : 2018-05-21 12:25:47
Started by user        : oracle
Operating System       : Linux version 3.10.0-514.el7.x86_64 (amd64)
Number of Targets      : 4
Last Reload            : (none)
Last successful upload                       : 2018-05-21 13:47:35
Last attempted upload                        : 2018-05-21 13:47:35
Total Megabytes of XML files uploaded so far : 0.25
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 80.79%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2018-05-21 13:47:57
Last successful heartbeat to OMS             : 2018-05-21 13:47:57
Next scheduled heartbeat to OMS              : 2018-05-21 13:48:58

---------------------------------------------------------------
Agent is Running and Ready
NOTE

In some cases, the Agent Resynchronization won't help. If the above plan fails to unblock the agent, the only option you have is to reinstall the agent.

Written At
19 FEB 201912:30
Red Hat Release
7.x x64
Enterprise Manager
13.1.0.0.0 - 13.3.0.0.0