Upgrade from TSDN 3.0 to TSDN 4.0 fails: We have an NSO 5.5.2.9 with TSDN 3.0 RC CFP. It is scale setup with 200K flat L2VPN, flat L3VPN. We are upgrading this to NCS 5.7.3 with M6. We followed step 2b as per the wiki. In step no 2b, i lost connectivity to NSO.Setup : 172.20.63.190 - cnctest/Ovatest123@I can see following error message in devel.log<INFO> 17-May-2022::23:58:12.861 cnc-scale-nso-190 ncs: devel-cdb upgrade: revision: cisco-flat-L2vpn-fp@2021-09-28<INFO> 18-May-2022::00:06:17.284 cnc-scale-nso-190 ncs: devel-cdb position before/at failure: undefinedHelp us in upgrading the setup to CNC 4.0
Hello Ramakrishna, the Redeploy is expected to take 40-50 mins with no connectivity to NSO during that time since upgrade is taking place. Please let us if any other errors observed during the reload (step 2B). When I logged into your setup I see all the packages up and running successfully.I moved on to step 5 in the wiki - Installed 5.7.3. This is a scale setup with 200K L2VPN/L3VPN services. NCS did not start within timeout. I could not start it again, since 4569 port is already in use. How to move forward, Is the NSO starting up or not. i see following error message in ncs.log<ERR> 19-May-2022::04:29:34.423 cnc-scale-nso-190 ncs: - The dependency node '/ncs:devices/device/config/Cisco-IOS-XR-hw-module-cfg:hw-module/nodes/node/config/attention-led' for node '/ncs:devices/device/config/Cisco-IOS-XR-hw-module-cfg:hw-module/nodes/node/config' in module 'tailf-ncs' does not exist<ERR> 19-May-2022::04:29:34.424 cnc-scale-nso-190 ncs: - The dependency node '/ncs:devices/device/config/Cisco-IOS-XR-hw-module-cfg:hw-module/nodes/node/config/auto-reset-disable' for node '/ncs:devices/device/config/Cisco-IOS-XR-hw-module-cfg:hw-module/nodes/node/config' in module 'tailf-ncs' does not exist<ERR> 19-May-2022::04:29:34.464 cnc-scale-nso-190 ncs: - The dependency node '/ncs:devices/device/config/Cisco-IOS-XR-hw-module-cfg:hw-module/nodes/node/config/attention-led' for node '/ncs:devices/device/config/Cisco-IOS-XR-hw-module-cfg:hw-module/nodes/node/config' in module 'tailf-ncs' does not exist<ERR> 19-May-2022::04:29:34.464 cnc-scale-nso-190 ncs: - The dependency node '/ncs:devices/device/config/Cisco-IOS-XR-hw-module-cfg:hw-module/nodes/node/config/auto-reset-disable' for node '/ncs:devices/device/config/Cisco-IOS-XR-hw-module-cfg:hw-module/nodes/node/config' in module 'tailf-ncs' does not existHi Ramakrishnan, $ cat /etc/systemd/system/ncs.service.d/startup-timeout.confTimeoutStartSec=4800$ sudo systemctl daemon-reload$ sudo systemctl show ncs | grep ^Timeout. > it was 5 min .TimeoutStopUSec=5minThe ncs is up on your set up after upgrade.But packages are not coming up as you have not followed NCS installation prerequisites.I have updated python3 on your lab to make packages up, These are basic CNC prerequisites.Your lab is successfully upgraded.Please read the documentation carefully in the future for right python version.ThanksPriyankaHi Priyanka,Thanks for fixing. So currently 172.20.63.190 is in proper state to use for testing ? I tried logging into its UI () but it's not accepting previous password (cnctest/Ovatest123@). For CLI login, it's working. It was same NSO which you have recommended 20 min startup time during CNC2.0 scale testing. Then during upgrade to RC 3.0 last release, we have made this to 3hr to make NCS come up. So 1hr 20 min is recommended standard setting for scale we should follow ? Now to upgrade this to M6, Ramakrishna followed the steps as mentioned in "M6 - CFP Upgrade from 3.0 to 4.0" wiki (some steps have changed in this wiki 1 or 2 days back after he attempted). During the process of upgrade, some of these settings are coming back to default. We may need to mention these notes/pre-requisites (including linking right python version) in Upgrade steps as well. I think it may be good to have links/pointers added here as well for troubleshooting common failures during upgrade as a reference.Thank You !!-Sreedhar.Hi Sreedhar,The following setting in ncs.conf will fix the UI login issue. <pam>   <enabled>true</enabled>   <service>system-auth</service>  </pam>I have restarted ncs after above change The reason for increased time in your NSO as during CNC 2.0 time your cdb was not this huge .This time it was 23 GB as these numbers are not fixed, I increased it incrementally to make it work .for now, this is 1 hr 20 min and is working fine.Also one more thing - if it's your scale lab - I wonder why all debugging is turned on, that's also one reason it was taking too long to start. Sure, will add these points to the wiki. Also, make sure wiki is read by the team as even those pointers like pre requisites which are documented are also not been read and JSD has been opened for similar issues multiple times.Appreciate it if everyone does their part diligently.ThanksPriyankaWorking UI Screenshot Working UI screenshot added and also troubleshooting section added :Hi Priyanka,Great !! Highly Appreciate your support (Right from CNC 2.0, You have helped us change many settings on this machine as per the requirements of growing scale. Without that could not have reached this scale) and help in bringing up this NSO with large scale of 300K VPN services. Yes agreed definitely we (COE folks) are lacking in knowledge on NSO (especially upgrades) due to limited use. Sure, we will check the updated wiki's. Purposefully with our knowledge, no debugs are enabled. May be as part of troubleshooting, it may be added after case was raised. I hope these debugs are stopped now to avoid any future restart issues.Thanks Again !!-Sreedhar.One quick question. How do I disable 'Debug' messages in this setup.