NSO upgrade fails: Existing setup NCS 5.5.2.9 with TSDN 3.0 RC upgrade to TSDN 4.0 M6 Upgrade fails. Followed the wiki, and in step 5, start of NCS 5.7.3 , NCS fails with following error message.Raised ticket , but marked as Duplicate. this setup now has python3, python pointing to python3.8. Still this CRIT message is seen after NSO restart.Setup : 172.20.63.189 - cnctest/Ovatest123@ This is a different setup than .ncs.log:<CRIT> 23-May-2022::02:11:26.505 cnc-scale-nso2 ncs: - CDB: Upgrade failed: Upgrade transaction failed to validate: illegal reference /cisco-flat-L2vpn-fp:flat-L2vpn{L2NM-3-srpeb-pec-internal}/flat-L2vpn-p2p/local-site/pe
Hi Ramakrishnan,From the error, it seems that device that used l2vpn service are not there. I searched for 'request device lsa dispatch-map sync' in audit.log after you brought in lsa package. I did not see you apply that.To me, you should do the upgrade from fresh again since the data is corrupted now. You did used a wrong python version originally.Also I saw this setup have swap size is too small compare to the memory $ free -h       total    used    free   shared buff/cache  availableMem:      129G    3.1G     96G    128M     29G    125GSwap:     6.0G     0B    6.0GThe swap size should be the same size of RAM memoryPlease fix this issue before you run upgrade again.Note: NSO upgrade should be in maintenance mode ( no northbount calls and logs should not in trace mode). Otherwise it could effect the performanceThanks,LinhI tried deleting the devices only through ncs_cli. "no device Node_*" , and "commit"Hello Rama, because there is no crash log from NSO upon crash, we suspect that the process was immediately killed for some reason. Maybe due to resource overload or some other system related cause. To verify NSO's capability in this scenario,our QA team has cloned your CDB and executed a bulk device delete (all devices) and NSO did not crash despite taking 1 hour 15 minutes for the delete to complete. For consistency, they have run through this scenario two times and encountered no crash.QA has recommended to try to switch to different vm and esxi. At this point we believe NSO crash is related to system, not NSO itself.Please let me know if this recommendation works. If not, we can discuss further!Thanks,Isaac KimWhen i delete device (bulk) as such, NSO return the prompt after 1 hour, plus few minutes. Only when commit is issued, i see the crash. Did they try with commit as well?Hi Rama, I apologize for the late update.I have been able to reproduce your crash upon commit. We have tried to look for the source of crash, but there is no evidence in any system logs or NSO crash dump logs.In this case, I will be creating a platform ticket so that the platform team can take a look. Once the ticket is raised, I will update this comment section.Thanks,Isaac KimHi Rama, I have filed the ticket with platform team with reproduction setup. I will update you if we hear back from them.Thanks,Isaac KimHi Rama, apologies for the late reply.The NSO platform team have found two relevant issues in regards to the bulk delete NSO crash issue. Both have been fixed starting NSO 5.5.3 and NSO 5.7.4.Because this issue stems from resource overload (combination of host system and NSO causing crash), the workaround for TSDN 3.0 is to delete devices in batches (commit each batch).Thanks,Isaac Kim