TSDN 4.0.0 - M3 - CFP Upgrade from 3.0 to 4.0 Failed at "5. NSO Startup with package reload force + ignore initial validation": TSDN 4.0.0 - M3 - CFP Upgrade from 3.0 to 4.0 Failed at "5. NSO Startup with package reload force + ignore initial validation" with error message in attached screenshotAdditionally 1) steps 3 (3. Remove old packages from TSDN CFP 3.0) and 4 (4. Copy new packages from TSDN CFP 4.0 with new ncs.conf) are not clear enough to follow e.g it need to help withfor #3, how to identify legacy NEDs on 5.5.2.12-TSDN 3.0.0 RCsetup?For #3, (To migrate to : 6.77) and (To migrate to : 7.38) - what is meaning of this ?</ul>for #4, does this setup require all packages and all NEDs to be moved and need to identify subdirectory structure too for clarity.For #4, there are more NEDS in this package, with overlapping information from step #3, are these all required along with one from #3?</ul>2) For 5. NSO Startup with package reload force + ignore initial validationscript snippet does not correspond to one found on this upgrade scenario on system. See attached screenshot for what is found on system. 
setup details10.194.63.180 ssh nso/Vtsisco@123!UI admin/Vtsisco@123!Hello Vipin,Below are replies to your questions1. For NED migration, both old NED and new NED need to be loaded into NSO. Because the old NED was compiled with old NSO version (NSO 5.5.2 from TSDN 3.0), to migrate from old NED to new NED, we need to load the old NED with new NSO version (NSO 5.7 from TSDN 4.0). This can be done by recompiling old NED with new NSO version. For your typical TSDN 3.0 -> 4.0 upgrade, there is no need to analyze which NED require migration as we provide instructions for packaged NEDs. We also provide old NEDs recompiled with new NSO version. From this point, we consider the old NED to be a "legacy" NED.Lets walk through the upgrade steps:3. Remove old packages from TSDN CFP 3.0Here we should remove old packages from TSDN CFP 3.0 from following path (path depends on type of install)NSO System Install: /var/opt/ncs/packages/NSO Local Install runtime directory: packages//// For NED packages that will be upgraded (major version upgrade), the "legacy" NED must be recompiled with new NSO version and loaded to migrate device data/// To load legacy NED with new NSO version, remove legacy NED package and replace with recompiled NED belowLegacy XE CLI NED 6.74 compiled with .7 (To migrate to : 6.77) : XE-CLI-6.74-compiled-with-.7Legacy XR CLI NED 7.33 compiled with .7 (To migrate to : 7.38) : XR-CLI-7.33-compiled-with-.7To "migrate" XE CLI NED 6.74 to 6.77, we need "legacy" XE CLI 6.74 NED recompiled with new NSO version 5.7. We provide this with the download link "XE-CLI-6.74-compiled-with-.7". Download the recompiled NED from the link and place it in path defined above. To "migrate" XR CLI NED 7.33 to 7.38, we need "legacy" NED 7.33 recompiled with new NSO version 5.7. We provide this with the download link "XR-CLI-7.33-compiled-with-.7". Download the recompiled NED from the link and place it in path defined above. 4. Copy new packages from TSDN CFP 4.0 with new ncs.conf/// Copy over 4.0 packages but do not remove any legacy NEDs compiled with .7With the recompiled "legacy" NEDs defined in <NSO PATH>/packages, copy TSDN 4.0 packages.At this point, you should have two NEDs per NED type. XR CLI 7.33 -> 7.38 NEDs:<NSO PATH>/packages/ncs-5.7-cisco-iosxr-7.38.X.tar.gz<NSO PATH>/packages/ncs-5.7-cisco-iosxr-7.33.X.tar.gzXE CLI 6.74 -> 6.77 NEDs:<NSO PATH>/packages/ncs-5.7-cisco-ios-6.77.X.tar.gz<NSO PATH>/packages/ncs-5.7-cisco-ios-6.74.X.tar.gzIn the next steps, you will reload NSO so that both "legacy" NED and "new major version" NED will be loaded into NSO. During migration upgrade step, you will migrate from the loaded "legacy" NED to the loaded "new major version" NED.I hope this clarifies the procedure for NED migration. 2. I can see that the script is a bit different. I will update the wiki to clarify the objective of the step which is to add "--ignore-initial-validation flag".Please let me know if you have any other questions.Thanks,Isaac KimHi Isaac, Blocker issue TSDN 4.0.0 - M3 - CFP Upgrade from 3.0 to 4.0 Failed at "5. NSO Startup with package reload force + ignore initial validation" with error message in attached screenshot is still not addressed. I believe you are still troubleshooting it?Thanks,VipinHello Vipin,I have discussed with Quy about some confusion about upgrade wiki steps. We identified few areas which I reworded for clarity. We identified that one issue on your setup is that your system is still pointing to old NSO version: It should be pointing to NSO 5.7 before you do a reload.Please refer to Step B-2 of the upgrade wiki and let me know if you are still running into the error.Thanks,Isaac Kimhi Isaac,We (Vipin and I) have discussed about the current nso version on .180 - the upgrade system - thanks for your help with the wiki steps - this will help all users during the upgrade process. We will follow the revised steps you have provided and will update you if any issue occurs. Thanks,-Quyhi Isaac - adding the email from Vipin lastnight since the ticket was closed:Hi Isaac, SahithiJIRA is closed pre-maturely so resorting to emails.Fixed ncs-5.7.2.1 install following steps Step-B, 2 -1 thru 3c.Error from 5, 3 change toroot@nso-10-194-63-180:/opt/ncs/ncs-5.7.2.1/etc/ncs# sudo NCS_RELOAD_PACKAGES=force /etc/init.d/ncs startStarting ncs: Bad configuration: /etc/ncs/ncs.conf:0: Cannot access directory ./ncs-cdb, definedfor/ncs-config/cdb/db-dirDaemon died status=21STEP-B, 2-4 is unclear about how to merge two ncs.conf (/opt/ncs/nso_5.7.2.1/etc/ncs/ncs.conf and/etc/ncs/ncs.conf). Please note there is typo here nso to fix (should be ncs).Quy ; I, interpreteda. merge changes from /etc/ncs/ncs.conf INTO/opt/ncs/nsoc_5.7.2.1/etc/ncs/ncs.confb. copy merged/opt/ncs/nsoc_5.7.2.1/etc/ncs/ncs.conf to/etc/ncs/ncs.conf i.e. overwrite /etc/ncs/ncs.conf. To identify changes to merge we two have to refer to changes required in ncs.conf per install for NSO TSDN 3.0 FP if you want to troubleshoot, setup details are:IP 10.194.63.180ssh nso/Vtsisco@123!Ui admin/ Vtsisco@123!Thanks,Vipin Sharma]Hi Quy,Not sure why the ticket was closed. We haven't closed it from our side. I see it still is open. Hopefully, we should be able to communicate in comments.Updating my reply on this comment. Hope it resolved the issue.```Hi Vipin,The below is occurring because your current ncs.conf is not pointing to the correct cdb folder.Try changing the below variable.<db-dir>{NCS_RUN_DIR}</db-dir> -> <db-dir>{NCS_RUN_DIR}/cdb</db-dir>Regarding the steps, I believe Isaac would have better input here.@Isaac Kim (isaakim) Any thoughts about steps for merging the ncs.conf files? ```ThanksSahithiHi Vipin and Quy!I have updated the wiki step B-2 with more explicit and clear steps.Please review and let me know if this change is satisfactory.Thank you for the feedback!Thanks,Isaac KimFailure At STEP-B, 8. Sync-from Migrated Devices for new NED capabilitiesadmin@ncs> request devices device TB4-PE-A sync-from dry-run  result falseinfo Failed to connect to device TB4-PE-A: connection refused: NEDCOM CONNECT: Authenticate: Exhausted available authentication methods. Server allowed: [ keyboard-interactive password ] in new stateadmin@ncs>Hi Vipin,Would you be able to confirm what authentication method was used for this device prior to upgrade? Was it ssh-rsa, ssh-dss, etc?Thanks,Isaac KimHi Isaac,I do not know this, is there a way to find out ?I believe it should be default for device from cnc fp 3.0.0 RC Thanks,VipinHi Vipin,No problem, we will try to investigate now. I hope its okay if we validate few things on your machine. We may also try setting some NSO config to debug but will not push anything southbound. Thanks,Isaac KimHi Vipin,Do you by any chance still have a copy of your pre-upgrade ncs.conf? If so, would you be able to share it here?Thanks,Isaac KimHi Issac,thanks, Please take setup for your investigation. Thanks,VipinHere is pre-upgrade ncs.conf /home/nso/tsdn3.0.0-backup/etc_nsc_ncs.conf/home/nso/tsdn3.0.0-backup/ ncs-5.5.2.12@2022-03-18T17:07:00.backup.gz has entire backup before upgrade, may be you can find it here tooHi Quy and Vipin,I have RCAed this with the help of Linh and we discovered that the source of issue mentioned above was due to the fact that encrypted-strings in old ncs.conf was not transferred over to new ncs.conf which meant that ssh-rsa host keys could not be decrypted as the new ncs.conf default encryption keys were being used.old ncs.conf:<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"><pre class="code-xml">      ${NCS_DIR}/bin/ncs_crypto_keys   ${NCS_CONFIG_DIR}/ncs.crypto_keys   </pre></div></div>new ncs.conf:<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"><pre class="code-xml">      0123456789abcdef   0123456789abcdef   0123456789abcdef   0123456789abcdef       0123456789abcdef0123456789abcdef   0123456789abcdef0123456789abcdef       0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef   </pre></div></div>The solution to this problem is to transfer your old encryption keys to new ncs.conf:<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"><pre class="code-xml">      ${NCS_DIR}/bin/ncs_crypto_keys   ${NCS_CONFIG_DIR}/ncs.crypto_keys   </pre></div></div>We have done this for you during the debug process and now devices can be reached from your NSO setup.Thanks,Isaac KimHi Quy and Vipin,I hope the upgrade procedure went well!If everything is good, I will close this ticket in next few days.Please let me know if you have any other questions in this regard.Thanks,Isaac KimHi Isaac, The upgrade went well, thanks for updating the wiki per our requested - appreciated your help - you can close the ticket for us please.Thanks and regards,-QuyNotes: the most time consuming was the merging the old ncs.conf with the new version - there are so many configurations need to carry over. Hope we can simplify this step somehow. Designated closed. For additional queries on ncs.conf migration, please open a new ticket.