delete device crashes NSO in scale setup: I have a CNC 3.0 TSDN setup with NSO 5.5.2.9. It has 25K devices. I want to delete these devices, since migration is taking time. I tried deleting 25K devices. It took 1 hour 5 mins for the response. When i committed the change, NSO is restarted. I tried this twice. Can you help me to delete these devices starting with Node_*setup : 172.20.63.189 - cnctest/Ovatest123@
Hello, just to help isolate the crash through logs, approximately what day and time did you encounter the crash? Doesnt have to be exact, but can help me filter through other error logs. Also just want to confirm you are committing through CNC WebUI when you encounter the crash.Thanks,Isaac KimHi Isaac, Ramakrishna has signed-off for the day. He has tried this today IST day time. Not sure how the commit was done. FYI.Thanks,Sreedhar.Thank you, I will try to go off of this.I will post update if I find anything.Thanks,Isaac KimHi as requested to help with device deletion, can I try to delete Node_101_1 device to see behavior? There seems to be connection timeout with Node_X devices.Thanks,Isaac KimYes. Please go ahead and delete all nodes with Node_X. These are sheer simulator nodes and don't have network connectivity directly to NSO. They are reachable to DLM only.Seems like deleting device Node_101_1 was successful, but seems like dependency validation took much longer than expected (253.609s).Hi,When deleting devices in batches of Node_10X and Node_11X, there is some delay but it goes through fine. When deleting devices in bulk (all), I have observed the crash. Whats strange is that there is no indication of a NSO crash as there is no NSO crash dump logs.In any case, NSO crash is not acceptable. But not sure if this crash is due to resource overload or due to some NSO internal mechanism. This is definitely not a CFP issue, so I will have to raise an NSO platform RT for further investigation.In the meantime, to move forward with your device cleanup, I would suggest to delete the devices in batches. I would try 1K devices first and if that crashes NSO then 500. Unfortunately, with no crash logs, there is nothing to indicate what is source of crash.I will post an update here when the RT is created.Thanks,Isaac KimI tried deleting the devices only through ncs_cli. "no device Node_*" , and "commit"Hello Rama, because there is no crash log from NSO upon crash, we suspect that the process was immediately killed for some reason. Maybe due to resource overload or some other system related cause. To verify NSO's capability in this scenario,our QA team has cloned your CDB and executed a bulk device delete (all devices) and NSO did not crash despite taking 1 hour 15 minutes for the delete to complete. For consistency, they have run through this scenario two times and encountered no crash.QA has recommended to try to switch to different vm and esxi. At this point we believe NSO crash is related to system, not NSO itself.Please let me know if this recommendation works. If not, we can discuss further!Thanks,Isaac KimWhen i delete device (bulk) as such, NSO return the prompt after 1 hour, plus few minutes. Only when commit is issued, i see the crash. Did they try with commit as well?Hi Rama, I apologize for the late update.I have been able to reproduce your crash upon commit. We have tried to look for the source of crash, but there is no evidence in any system logs or NSO crash dump logs.In this case, I will be creating a platform ticket so that the platform team can take a look. Once the ticket is raised, I will update this comment section.Thanks,Isaac KimHi Rama, I have filed the ticket with platform team with reproduction setup. I will update you if we hear back from them.Thanks,Isaac KimHi Rama, apologies for the late reply.The NSO platform team have found two relevant issues in regards to the bulk delete NSO crash issue. Both have been fixed starting NSO 5.5.3 and NSO 5.7.4.Because this issue stems from resource overload (combination of host system and NSO causing crash), the workaround for TSDN 3.0 is to delete devices in batches (commit each batch).Thanks,Isaac Kim