migration of devices takes lot of time in upgrade: We are upgrading NSO 5.5.2.9 (TSDN 3.0) to NSO 5.7.3 M6 (TSDN 4.0) In step 6, migration of devices to new ned-id is taking around 5 mins in the setup. This is a scale setup with 25K devices. In this speed, it will take about 86 days to migrate the setup.Is there a way to increase the speed of migrating the ned id. Is this step mandatory for the upgrade?Setup: 172.20.63.190 - cnctest/Ovatest123@
Hi Ramakrishnan, When you update TSDN between release, please diligently think what could affect the performance of the upgrade. Right now I saw this setup still under stress of a lot of restconf call from other server<INFO> 24-May-2022::11:04:22.639 cnc-scale-nso-190 ncs: audit user: cnctest/14572 assigned to groups: ncsadmin,cnctest,wheel<INFO> 24-May-2022::11:04:22.639 cnc-scale-nso-190 ncs: audit user: cnctest/14572 created new session via rest from 172.20.63.218:60353 with http<INFO> 24-May-2022::11:04:22.639 cnc-scale-nso-190 ncs: audit user: cnctest/14572 RESTCONF: request with http: GET /restconf/data HTTP/1.1<INFO> 24-May-2022::11:04:22.646 cnc-scale-nso-190 ncs: audit user: cnctest/14572 terminated session (reason: normal)<INFO> 24-May-2022::11:04:22.654 cnc-scale-nso-190 ncs: audit user: cnctest/14575 assigned to groups: ncsadmin,cnctest,wheel<INFO> 24-May-2022::11:04:22.654 cnc-scale-nso-190 ncs: audit user: cnctest/14575 created new session via rest from 172.20.63.218:35560 with https<INFO> 24-May-2022::11:04:22.654 cnc-scale-nso-190 ncs: audit user: cnctest/14575 RESTCONF: request with https: GET /restconf/data HTTP/1.1<INFO> 24-May-2022::11:04:22.658 cnc-scale-nso-190 ncs: audit user: cnctest/14575 terminated session (reason: normal)Please disable northbound calls during migration process.This is your current log settings  <developer-log>   <enabled>true</enabled>   <file>    <name>${NCS_LOG_DIR}/devel.log</name>    <enabled>true</enabled>   </file>  </developer-log>  <developer-log-level>trace</developer-log-level>Please don't use trace level for log. It would degrade the performance for sure. Please fix the issue I mentioned above.Thanks,-LinhHi Linh,Thanks for taking a look at the setup. Since Ramakrishna has signed-off for the day - to check the current services, just added this NSO to one of CNC. Due to full-sync, we are seeing that stress now. Otherwise while upgrading trails, this stress was not there. Is it ok if I remove this line "<developer-log-level>trace</developer-log-level>" save it ? Or we do need to restart ncs to get this into effect ? Please let me know.Thanks,Sreedhar.Hi Sreedhar,You could change it back info log level an then restart ncs.Regards,LinhI followed the steps in the Upgrade wiki. These debug logs were enabled in the config file /opt/ncs/ncs-5.7.3/etc/ncs/ncs.conf. xpath trace and developer logs. Also, there were many other changes like wrong log folders in this config file. I have to fix these issues, since ncs did not start. Can you fix these issues in the specified ncs.conf file. As defined in the stiep 2 of, it is responsible of the user to migrate configuration from old ncs.conf to the new ncs.conf template at /opt/ncs/ncs-5.7.3/etc/ncs/ncs.conf. Then the new ncs.conf needs to be coped to the location /etc/ncs/. The configuration is depending on each individual setup. There is no such one ncs.conf that works for all setups.My humble request is to disable trace level log for both xpath trace and developer logs in the /opt/ncs/ncs-5.7.3/etc/ncs/ncs.conf. This will avoid trace level logs for all setups when people follow the steps. It seems your use case is different from other people where they need these configuration to be enabled. You should take a note for yourself so you should not hit to these mistake again.  Again there is no one ncs.conf that fit in for all use cases. Users need diligently to modify ncs.conf to fit in their use cases.How long it does take to migrate ned-id is depending on how many entries that need to update. I saw your system has quite a number of entries to updatecnctest@ncs% request devices device Node_100_68 migrate new-ned-id cisco-iosxr-cli-7.39 no-networkingmodified-path {  path /cisco-ios-xr:segment-routing/traffic-eng/affinity-map/color/bit-position  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:policy-map/class/class-default/police-rate-unit/police/rate/percent  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:policy-map/class/class-ref/police-rate-unit/police/rate/percent  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:snmp-server/timeouts/duplicate  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:dhcp/ipv4/database  info node type has changed from non-presence container to presence container}modified-path {  path /cisco-ios-xr:key/chain/key/key-string/type  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:ptp/clock/profile/clock-type  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:ntp/access-group/vrf  info sub-tree has been deleted}modified-path {  path /cisco-ios-xr:ethernet/oam/profile/action/wiring-conflict  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:vrf/vrf-list/address-family/ipv6/flowspec/import/from/default-vrf/route-policy  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:vrf/vrf-list/address-family/ipv6/multicast/import/from/default-vrf/route-policy  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:vrf/vrf-list/address-family/ipv6/unicast/import/from/default-vrf/route-policy  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:vrf/vrf-list/address-family/ipv4/flowspec/import/from/default-vrf/route-policy  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:vrf/vrf-list/address-family/ipv4/multicast/import/from/default-vrf/route-policy  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:vrf/vrf-list/address-family/ipv4/unicast/import/from/default-vrf/route-policy  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:group/config/segment-routing/traffic-eng/affinity-map/color/bit-position  info leaf/leaf-list type has changed}modified-path {  path /cisco-ios-xr:admin/dsc/serial/rack  info leaf/leaf-list type has changed}And devel.log shows<INFO> 25-May-2022::13:04:04.897 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli device=Node_100_68 migrating<INFO> 25-May-2022::13:04:04.899 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli device=Node_100_68 migrating device configuration<INFO> 25-May-2022::13:04:04.905 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli device=Node_100_68 migrating device configuration: ok (0.005 s)<INFO> 25-May-2022::13:04:04.905 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli device=Node_100_68 finding affected services<INFO> 25-May-2022::13:04:04.906 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli device=Node_100_68 finding affected services: ok (0.000 s)<INFO> 25-May-2022::13:04:04.906 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli applying transaction<INFO> 25-May-2022::13:04:04.906 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli entering validate phase<INFO> 25-May-2022::13:07:47.971 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli grabbing transaction lock<INFO> 25-May-2022::13:07:47.971 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli grabbing transaction lock: ok (0.000 s)<INFO> 25-May-2022::13:07:47.972 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli run transforms and transaction hooks<INFO> 25-May-2022::13:07:47.977 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli run transforms and transaction hooks: ok (0.005 s)<INFO> 25-May-2022::13:07:47.977 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli mark inactive<INFO> 25-May-2022::13:07:47.978 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli mark inactive: ok (0.000 s)<INFO> 25-May-2022::13:07:47.978 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli pre validate<INFO> 25-May-2022::13:07:47.979 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli pre validate: ok (0.000 s)<INFO> 25-May-2022::13:07:47.979 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli run validation over the changeset<INFO> 25-May-2022::13:07:47.985 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli run validation over the changeset: ok (0.006 s)<INFO> 25-May-2022::13:07:47.985 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli run dependency-triggered validation<INFO> 25-May-2022::13:07:48.017 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli run dependency-triggered validation: ok (0.032 s)<INFO> 25-May-2022::13:07:48.017 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli check configuration policies<INFO> 25-May-2022::13:07:48.017 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli check configuration policies: ok (0.000 s)<INFO> 25-May-2022::13:07:48.206 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli leaving validate phase (223.300 s)<INFO> 25-May-2022::13:07:48.207 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli entering write-start phase<INFO> 25-May-2022::13:07:48.207 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli subsystem=cdb write-start<INFO> 25-May-2022::13:07:48.207 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli subsystem=ncs-internal-service-mux write-start<INFO> 25-May-2022::13:07:48.240 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli check data kickers<INFO> 25-May-2022::13:07:48.240 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli check data kickers: ok (0.000 s)<INFO> 25-May-2022::13:07:48.247 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli leaving write-start phase (0.040 s)<INFO> 25-May-2022::13:07:48.247 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli entering prepare phase<INFO> 25-May-2022::13:07:48.247 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli subsystem=cdb prepare<INFO> 25-May-2022::13:07:48.247 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli subsystem=ncs-internal-device-mgr prepare<INFO> 25-May-2022::13:07:48.248 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli leaving prepare phase (0.001 s)<INFO> 25-May-2022::13:07:48.248 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli entering commit phase<INFO> 25-May-2022::13:07:48.248 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli subsystem=cdb commit<INFO> 25-May-2022::13:07:48.250 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli subsystem=ncs-internal-device-mgr commit<INFO> 25-May-2022::13:07:48.251 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli releasing transaction lock<INFO> 25-May-2022::13:07:48.251 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli leaving commit phase (0.002 s)<INFO> 25-May-2022::13:07:48.251 cnc-scale-nso-190 ncs: devel-cdb Device migration validate for mount-id: {549854595,1044521224}<INFO> 25-May-2022::13:07:48.251 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli applying transaction: ok (223.345 s)<INFO> 25-May-2022::13:07:48.251 cnc-scale-nso-190 ncs: devel-cdb Device migration validation succeeded<INFO> 25-May-2022::13:07:48.252 cnc-scale-nso-190 ncs: ncs progress usid=109 tid=98887276 datastore=running context=cli device=Node_100_68 migrating: ok (223.354 s)So it is expected. It would take under 4 minutes to migrate one device