Print Friendly, PDF & Email

 
PROBLEM

Networker does not start after VM cloning if a cloned host with a different hostname.

 
SYMPTOMPS

Networker works fine on original VM and does not start on cloned one.

Original : original.net
Cloned   : cloned.net
root@original.net: service networker start
root@original.net: ps -fe |grep nsr
root     27102     1  0 Jan26 ?        00:00:24 /usr/sbin/nsrexecd
root     28597 28565  0 12:31 pts/0    00:00:00 grep nsr

root@cloned.net: service networker start
root@cloned.net: ps -fe |grep nsr
root     23949 17304  0 15:22 pts/4    00:00:00 grep nsr

Installed Networker packages are the same on both machines. Packages re-install does not help

root@cloned.net: uname -ri
2.6.39-400.297.3.el5uek x86_64

root@cloned.net: rpm -qa --queryformat '%{NAME}-%{VERSION}-%{RELEASE} %{ARCH}\n'  |grep lgto
lgtoclnt-8.2.3.5-1 x86_64
lgtoman-8.2.3.5-1 x86_64
lgtonmda-8.2.3.0-1 x86_64

Running nsrexecd with debug option helps to identify a possible root cause of the problem

root@cloned.net: export LD_LIBRARY_PATH=/usr/lib/nsr/lib64/cst:/usr/lib/nsr/lib64
root@cloned.net: nsrexecd -D9
01/26/18 17:05:26.774560 clu_is_cluster_mode:ENTRY
01/26/18 17:05:26.774883 app_name=NetWorker, app_dir=/usr/sbin/
01/26/18 17:05:26.774960 b_clusmode=0
01/26/18 17:05:26.775002 clu_is_cluster_mode:EXIT
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.775740 0 20 Origin is /usr/sbin/
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.776641 0 152 Desired file descriptor limit (5632) cannot be set, setting to max file descriptor limit (4096) instead.  Please increase the max file descriptor limit.
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.776870 0 47 File descriptor limit successfully set to 4096.
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.777135 0 38 Bytes 0x73 0x30 0x30 are not UTF-8 BOM
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.779898 0 35 clu_is_cluster_host_lc(): ENTRY ...
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.780317 0 25 clu_is_cluster_host:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.780518 0 40 clu_is_cluster_host:EXIT:returning FALSE
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.782380 0 25 clu_is_cluster_host:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.782672 0 40 clu_is_cluster_host:EXIT:returning FALSE
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.783108 0 24 clus_get_hostname:ENTRY:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.783354 0 28 hostname=null, path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.783607 0 23 clu_pathl_gethost:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.783795 0 19 input path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.783951 0 52 clu_pathl_gethost:EXIT:Clu_cluster_type is undefined
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.784064 0 36 Server_name=cloned.net
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.784159 0 23 clus_get_hostname:EXIT:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788105 0 24 clus_get_hostname:ENTRY:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788312 0 28 hostname=null, path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788412 0 23 clu_pathl_gethost:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788483 0 19 input path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788551 0 52 clu_pathl_gethost:EXIT:Clu_cluster_type is undefined
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788616 0 36 Server_name=cloned.net
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788680 0 23 clus_get_hostname:EXIT:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788888 0 24 clus_get_hostname:ENTRY:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.788986 0 28 hostname=null, path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789056 0 23 clu_pathl_gethost:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789121 0 19 input path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789212 0 52 clu_pathl_gethost:EXIT:Clu_cluster_type is undefined
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789306 0 36 Server_name=cloned.net
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789374 0 23 clus_get_hostname:EXIT:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789533 0 37 could not find GUI metadata directory
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789746 0 24 clus_get_hostname:ENTRY:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789819 0 28 hostname=null, path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789881 0 23 clu_pathl_gethost:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.789943 0 19 input path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.790020 0 52 clu_pathl_gethost:EXIT:Clu_cluster_type is undefined
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.790095 0 36 Server_name=cloned.net
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.790156 0 23 clus_get_hostname:EXIT:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.796494 0 56 Did not find NetWorker portmapper port, setting to 7938.
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.811147 0 45 Found system portmapper port, setting to 111.
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.811727 0 24 clus_get_hostname:ENTRY:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.811857 0 28 hostname=null, path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.811930 0 23 clu_pathl_gethost:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.812014 0 19 input path=/nsr/res
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.812079 0 52 clu_pathl_gethost:EXIT:Clu_cluster_type is undefined
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.812145 0 36 Server_name=cloned.net
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.812236 0 23 clus_get_hostname:EXIT:
138906 1516975526 5 3 20 49800944 10559 0 cloned.net nsrexecd RAP critical 162 Attributes '%s' and/or '%s' of the %s resource do not resolve to the machine's hostname '%s'. To correct the error, it may be necessary to delete the %s database. 5 11 9 2335:name 11 17 51954:my hostname 11 11 89897:NSRLA 12 24 cloned.net 11 11 89897:NSRLA
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.819130 0 25 clu_is_cluster_mode:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.819452 0 38 app_name=NetWorker, app_dir=/usr/sbin/
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.819771 0 12 b_clusmode=0
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.819924 0 24 clu_is_cluster_mode:EXIT
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.820040 0 19 clu_dflt_name:ENTRY
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.820781 0 46 Adding range 7937-9936 to "service" port type.
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.820998 0 91 Setting DNS cache TTLs to 1800 secs for positive lookups and 1800 secs for negative lookups
0 1516975526 1 5 0 4221196608 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.828328 0 28 Creating UDP/IPv4 portmapper
0 1516975526 1 5 0 4221196608 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.828481 0 54 Creating IPv4 UDP RPC service on address [0.0.0.0]:543
0 1516975526 1 5 0 4221196608 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.853137 0 28 Creating TCP/IPv4 portmapper
0 1516975526 1 5 0 4221196608 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.853240 0 54 Creating IPv4 TCP RPC service on address [0.0.0.0]:543
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.859830 0 43 Attempting to bind IPv4 socket descriptor 9
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.859896 0 25 Socket bound to port 9673
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.859931 0 56 Creating IPv4 TCP RPC service on address [0.0.0.0]:51493
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.872728 0 54 Creating TCP/IPv4 RPC client handle prog 100000 vers 2
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.872781 0 44 Attempting to bind IPv4 socket descriptor 11
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.872812 0 34 Socket bound to OS determined port
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.872860 0 53 Setting default RPC socket send buffer size to 262144
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.872887 0 56 Setting default RPC socket receive buffer size to 262144
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.872911 0 45 Setting RPC socket send buffer size to 262144
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.872944 0 45 Setting RPC socket recv buffer size to 262144
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.873062 0 48 Bound TCP/IPv4 socket descriptor 11 to port 7938
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.873257 0 69 Attempting to register 390436 (vers 1) service with portmapper (7938)
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.873422 0 61 Successfully registered service 390436 with portmapper (7938)
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.873906 0 22 Reduced cluster query.
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874083 0 24 clus_get_hostname:ENTRY:
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874213 0 28 hostname=null, path=/nsr/res
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874262 0 23 clu_pathl_gethost:ENTRY
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874342 0 19 input path=/nsr/res
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874388 0 52 clu_pathl_gethost:EXIT:Clu_cluster_type is undefined
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874421 0 36 Server_name=cloned.net
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874453 0 23 clus_get_hostname:EXIT:
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874483 0 77 nsr_getserverhost(): returning clus_get_hostname() = cloned.net
0 1516975526 1 5 0 4196018496 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.874529 0 93 Started audit log thread, waiting 30 seconds for nsrexecd to be fully ready before launching.
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.879814 0 54 Creating TCP/IPv4 RPC client handle prog 100000 vers 2
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.879864 0 44 Attempting to bind IPv4 socket descriptor 11
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.879891 0 34 Socket bound to OS determined port
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.879916 0 45 Setting RPC socket send buffer size to 262144
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.879939 0 45 Setting RPC socket recv buffer size to 262144
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880012 0 48 Bound TCP/IPv4 socket descriptor 11 to port 7938
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880136 0 43 0 child authentication threads are running.
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880170 0 53 execd_authserver_main(): waiting for 0 child threads.
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880212 0 43 0 child authentication threads are running.
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880237 0 43 0 child authentication threads are running.
0 1516975526 1 5 0 4212803904 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880259 0 60 execd_authserver_main(): 0 child threads have been left out.
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880785 0 54 Creating TCP/IPv4 RPC client handle prog 100000 vers 2
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880900 0 43 Attempting to bind IPv4 socket descriptor 9
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.880988 0 34 Socket bound to OS determined port
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.881043 0 45 Setting RPC socket send buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.881096 0 45 Setting RPC socket recv buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.881310 0 47 Bound TCP/IPv4 socket descriptor 9 to port 7938
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.887955 0 43 Attempting to bind IPv4 socket descriptor 9
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.888023 0 30 Creating IPv4 nsrexecd service
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.888102 0 25 Socket bound to port 9673
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.888203 0 54 Creating IPv4 TCP RPC service on address [0.0.0.0]:287
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.888314 0 56 Creating IPv4 TCP RPC service on address [0.0.0.0]:51493
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.894915 0 54 Creating TCP/IPv4 RPC client handle prog 100000 vers 2
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895010 0 54 Creating TCP/IPv4 RPC client handle prog 100000 vers 2
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895060 0 44 Attempting to bind IPv4 socket descriptor 13
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895329 0 34 Socket bound to OS determined port
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895433 0 45 Setting RPC socket send buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895220 0 44 Attempting to bind IPv4 socket descriptor 14
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895503 0 45 Setting RPC socket recv buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895647 0 34 Socket bound to OS determined port
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895761 0 45 Setting RPC socket send buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895830 0 45 Setting RPC socket recv buffer size to 262144
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895833 0 48 Bound TCP/IPv4 socket descriptor 13 to port 7938
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895958 0 69 Attempting to register 390435 (vers 1) service with portmapper (7938)
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.895991 0 48 Bound TCP/IPv4 socket descriptor 14 to port 7938
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.896164 0 69 Attempting to register 390113 (vers 1) service with portmapper (7938)
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.896346 0 61 Successfully registered service 390435 with portmapper (7938)
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.896502 0 54 Creating TCP/IPv4 RPC client handle prog 100000 vers 2
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.896616 0 44 Attempting to bind IPv4 socket descriptor 13
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.896672 0 61 Successfully registered service 390113 with portmapper (7938)
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.896716 0 34 Socket bound to OS determined port
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.896900 0 45 Setting RPC socket send buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.896934 0 17 checking file ...
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.897100 0 16 checking file ..
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.897014 0 45 Setting RPC socket recv buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 2 %s 1 0 28 @(#) Product:      NetWorker
0 1516975526 1 5 0 4204411200 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.897541 0 48 Bound TCP/IPv4 socket descriptor 13 to port 7938
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 2 %s 1 0 37 @(#) Release:      8.2.3.5.Build.1142
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 2 %s 1 0 23 @(#) Build number: 1142
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 2 %s 1 0 47 @(#) Build date:   Thu Aug  4 00:58:05 PDT 2016
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 2 %s 1 0 27 @(#) Build arch.:  linux86w
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 2 %s 1 0 29 @(#) Build info:   DBG=0,OPT=
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.898441 0 35 clu_is_cluster_host_lc(): ENTRY ...
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.898671 0 43 hourly check of daemon.raw rollover invoked
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.898981 0 41 periodic reaping after 1516975526 seconds
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.899091 0 29 md_check_orphaned_snmd: entry
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.899411 0 17 checking file ...
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.899699 0 16 checking file ..
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.899876 0 27 mondaemon_kill_check: entry
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.900109 0 21 md_reaper, remaining:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.900261 0 21 md_reaper, remaining:
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.900476 0 43 0 child authentication threads are running.
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.900746 0 54 Creating TCP/IPv4 RPC client handle prog 100000 vers 2
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.900983 0 43 Attempting to bind IPv4 socket descriptor 9
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.901212 0 34 Socket bound to OS determined port
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.901368 0 45 Setting RPC socket send buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.901504 0 45 Setting RPC socket recv buffer size to 262144
0 1516975526 1 5 0 49800944 10559 0 cloned.net nsrexecd NSR notice 5 %s %s 2 0 24 01/26/18 17:05:26.901900 0 47 Bound TCP/IPv4 socket descriptor 9 to port 7938

The interesting thing from the output is that the nsrexecd utility failed to resolve the hostname.

Attributes '%s' and/or '%s' of the %s resource do not resolve to the machine's hostname '%s'. To correct the error, it may be necessary to delete the %s database.

The same debug information can be found in the daemon.raw log

root@cloned.net: tail -8 /nsr/logs/daemon.raw
0 1516974447 1 5 0 328705776 9097 0 cloned.net nsrexecd NSR notice 2 %s 1 0 29 @(#) Build info:   DBG=0,OPT=
138906 1516974848 5 3 20 2488600304 9501 0 cloned.net nsrexecd RAP critical 162 Attributes '%s' and/or '%s' of the %s resource do not resolve to the machine's hostname '%s'. To correct the error, it may be necessary to delete the %s database. 5 11 9 2335:name 11 17 51954:my hostname 11 11 89897:NSRLA 12 24 cloned.net 11 11 89897:NSRLA
0 1516974848 1 5 0 2488600304 9501 0 cloned.net nsrexecd NSR notice 2 %s 1 0 28 @(#) Product:      NetWorker
0 1516974848 1 5 0 2488600304 9501 0 cloned.net nsrexecd NSR notice 2 %s 1 0 37 @(#) Release:      8.2.3.5.Build.1142
0 1516974848 1 5 0 2488600304 9501 0 cloned.net nsrexecd NSR notice 2 %s 1 0 23 @(#) Build number: 1142
0 1516974848 1 5 0 2488600304 9501 0 cloned.net nsrexecd NSR notice 2 %s 1 0 47 @(#) Build date:   Thu Aug  4 00:58:05 PDT 2016
0 1516974848 1 5 0 2488600304 9501 0 cloned.net nsrexecd NSR notice 2 %s 1 0 27 @(#) Build arch.:  linux86w
0 1516974848 1 5 0 2488600304 9501 0 cloned.net nsrexecd NSR notice 2 %s 1 0 29 @(#) Build info:   DBG=0,OPT=

Despite this the hostname is resolved by OS successfully

root@cloned.net: cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.localdomain   localhost
::1             localhost.localdomain   localhost

root@cloned.net: nslookup cloned.net
Server:         10.1.1.50
Address:        10.1.1.50#53

Name:   cloned.net
Address: 10.1.1.20

root@cloned.net: nslookup 10.1.1.20
Server:         10.1.1.50
Address:        10.1.1.50#53

20.1.1.10.in-addr.arpa  name = cloned.net.

 
SOLUTION

Recreate the /nsr directory

root@cloned.net: mv /nsr/ /nsr.orig.26JAN2018
root@cloned.net: mkdir /nsr

Now Networker should start

root@cloned.net: ps -fe |grep nsr
root     30565 30338  0 12:52 pts/2    00:00:00 grep nsr

root@cloned.net: service networker start

root@cloned.net: ps -fe |grep nsr
root     30578     1  2 12:52 ?        00:00:00 /usr/sbin/nsrexecd
root     30611 30338  0 12:52 pts/2    00:00:00 grep nsr

 
 

Version   : 13:51 29.01.2018
Platform  : Oracle Linux Server release 5.11
Networker : 8.2.3.5