Failed to connect Device error shown in CNC - NSO shows different IP : In CNC the following error message shown for one the device - Major - NSO Sync-from/to failed. NSO Reported Error: Failed to connect to device PE-A: connection refused: NEDCOM CONNECT: No route to host (Host unreachable) in new stateOn checking further found the different IP configuration information in NSO than the one configured in device. Checked the device configuration all looks fine. Setup detail - CW - 10.22.179.235 - admin/Cwork123!NSO - 10.194.60.130 - xrd1/Cwork123!
Hi Murali, apologies for delay. I will take a look today~Hi Murali,After a quick look at your NSO configuration, it seems like PE-A is defined as192.168.5.11. Im not sure why CW GUI is showing 192.168.5.21.admin@ncs> show devices listNAME   ADDRESS  DESCRIPTION NED ID    ADMIN STATE----------------------------------------------------------------------------P-BOTTOM-LEFT 192.168.5.26 -   cisco-iosxr-cli-7.43 unlocked P-BOTTOM-RIGHT 192.168.5.27 -   cisco-iosxr-cli-7.43 unlocked P-TOP-LEFT  192.168.5.24 -   cisco-iosxr-cli-7.43 unlocked P-TOP-RIGHT  192.168.5.25 -   cisco-iosxr-cli-7.43 unlocked PE-A   192.168.5.11 -   cisco-iosxr-cli-7.43 unlocked PE-B   192.168.5.22 -   cisco-iosxr-cli-7.43 unlocked PE-C   192.168.5.23 -   cisco-iosxr-cli-7.43 unlocked SR-PCE   192.168.5.28 -   cisco-iosxr-cli-7.43 unlocked When executing check-sync on PE-A, it showsadmin@ncs> request devices device PE-A check-syncresult errorinfo Failed to authenticate towards device PE-A: Unknown SSH host keyI manually updated ssh host keys usingadmin@ncs> request devices fetch-ssh-host-keysfetch-result {...After this I executed a check-sync on PE-A again and it showsadmin@ncs> request devices device PE-A check-syncresult errorinfo Failed to connect to device PE-A: connection refused: NEDCOM CONNECT: Authenticate: Exhausted available authentication methods. Server allowed: [ publickey password ] in new stateI am not sure how these devices were setup, but prior to running CNC or NSO operations, I would make sure that all devices are correctly onboarded.From our testing, device operations and connectivity on NSO 6.1 is working fine. These errors you are seeing are likely due to your setup or onboarding bootstrap config. Please make sure you are able to ssh into each device using the authgroup credentials from XRDevices.Thanks,Isaac KimHi IsaacI haven't configured that IP address 192.168.5.11 to my device. It was working fine earlier configured IP for PE-A device is 192.168.5.21. I have attached screen shot of devices onboarded and configured IP address.I checked completely and not sure how and why PE-A devices shows different IP address rather than one configured in device.On same NSO or CNC I was able to provision services and policies earlier.Hi Murali,I cannot think of any scenario where a device will be onboarded with a different IP from what was configured. It could potentially be that certain changes were not committed.If you would like, we can get on a call to try to resolve this.Thanks,Isaac KimHi Isaac,Sure we can discuss over the call to resolve this.Regards,Murali BGot on call, but decided we need CNC instance to be up to continue with RCA. No relevant info found from NSO logs.Murali to reach out when he has another CNC instance and able to reproduce issue.If you are not able to reproduce, please let me know here as well.Thanks,Isaac KimGot on another call, found that there is old DLM config from cloned NSO instance which could be source of issue. Could not reproduce on CNC UI. We will keep watch again, but next time we clone NSO and add to a new CNC instance we should try to cleanup old DLM config.Check for old DLM config:1.enter NSO CLI2.enter config mode ("configure")3.show dlm config ("show cisco-dlm")4.delete any stale entries ("delete cisco-dlm crosswork-vm <CW IP>")5.commit changes ("commit")6.continue with onboarding NSO to CNCIf Murali faces this again, we will reconvene and investigate further.If we dont face this for some time (assuming we follow above steps), I think we can assume this issue is resolved.Thanks,Isaac KimHi Murali, I believe this ticket has been resolved.To prevent the ticket from being open too long, I will mark this as resolved.If you face this issue again, please open a new JSD and reference this JSD for context.Thanks,Isaac Kim