NSO crashed and won't start, after services deletion commands: As part of SH automation execution the NSO crashed and won't start, after executing the following commands:"delete flat-L3vpn","delete flat-L2vpn","delete l3vpn-ntw","delete l3vpn-route-policy","delete l2vpn-ntw","delete l2vpn-route-policy","delete sr-te","delete te"VM details:10.56.112.77auto1/Public1234!system installationNSO 5.7.2.1 + CFP M3/var/log/ncs/ncs.logJob logs:Link:console logs:...0:35:19 admin@ncs% 00:35:19 2022-03-22T00:35:19: %TESTCASES-INFO: Command to be executed: delete te00:35:19 delete te00:35:19 00:35:19 00:35:19 00:35:19 admin@ncs% 00:35:19 commit00:35:20 Commit complete.00:35:20 00:35:20 00:35:20 00:35:20 admin@ncs% 00:35:20 exit00:35:20 00:35:20 admin@ncs> 00:35:20 exit00:35:20 00:35:32 2022-03-22T00:35:31: %TESTCASES-INFO: Waiting 6.0 minutes for services to be deleted...FAILED TO CONNECT THE NSO CLI:...00:41:33 Last login: Mon Mar 21 22:35:07 2022 from 10.56.22.10100:41:33 ]00:41:33 00:41:33 2022-03-22 00:41:33,350: %UNICON-INFO: +++ initializing handle +++00:41:33 1337;RemoteHost=auto1@gd-oashery-nso]1337;CurrentDir=/home/auto1]1337;ShellIntegrationVersion=16;shell=bash]133;C;00:41:33 ]1337;RemoteHost=auto1@gd-oashery-nso]1337;CurrentDir=/home/auto1133;D;0]133;Aauto1@gd-oashery-nso:~$ ]133;Bncs_cli -u admin00:41:33 [?2004l]133;C;Failed to connect to server00:41:33 ]1337;RemoteHost=auto1@gd-oashery-nso]1337;CurrentDir=/home/auto1133;D;1]133;Aauto1@gd-oashery-nso:~$ ]133;B00:42:33 00:42:33 2022-03-22 00:42:33,500: %UNICON-ERROR: Timeout occurred : Timeout value : 6000:42:33 Command sent: ncs_cli -u admin00:42:33 Pattern: 00:42:33 Got: '1337;RemoteHost=auto1@gd-oashery-nsox07x1b]1337;CurrentDir=/home/auto1x07x1b]1337;ShellIntegrationVersion=16;shell=bashx07x1b]133;C;x07x1b]1337;RemoteHost=auto1@gd-oashery-nsox07x1b]1337;CurrentDir=/home/auto1x07x1b133;D;0x07x1b]133;Ax07auto1@gd-oashery-nso:<sub>$ x1b]133;Bx07ncs_cli -u adminx1b133;C;x07Failed to connect to serverx1b]1337;RemoteHost=auto1@gd-oashery-nsox07x1b]1337;CurrentDir=/home/auto1x07x1b133;D;1x07x1b]133;Ax07auto1@gd-oashery-nso:</sub>$ x1b]133;Bx07'00:42:33 2022-03-22T00:42:33: %TESTCASES-ERROR: [31mERROR occurred in send_line: ('Failed to match buffer on spawn', TimeoutError('Timeout occurred'))[0m[39m...
 Hi Ofir,With the help from Isaac, we could not conclude what is root cause for the crash from the attached logs. We will try to reproduce this issue in the local setup. Please share all payloads of services before massive service deletion. Let see if we can reproduce the issue.Please confirm is it CNC 4.0 M4 build? Is there any settings that need to know to make it happens like CQ etc ...Thanks,LinhHello Ofir,I am assisting with RCA for this failure you are seeing.Would you be able to give me access to your Jenkins console logs : Thanks,Isaac KimHi Isaac, I don't have permissions for the Jenkins, but you can watch these logs at: or . btw, I re-installed the NSO (5.7.2.1 + M3) from scratch and when when the services have created via CW nso proxy REST API, in CW they displayed as in progress, now I'm not sure whether it's related to the same issue but it's worth to check that from NSO POV the services created properly, and the new NSO installation are correct, can you please help us verifying that ? it's the same NSO details:10.56.112.77auto1/Public1234!CW:10.56.112.51( ssh: cw-admin/Public1234! , UI: admin/Public1234!)Hi Ofir, All services that you push to NSO are not in 'reached' state because you don't set a proper username in nacm. This user 'auto1' (see the screen shot) is used to communicate to with NSO but it is not set proper access in nacm auto1@ncs% show nacm groups group ncsadmin user-name [ admin cw-admin private ]; 'auto1' should be member of nacm ncsadmin group-LinhHi Linh,we dont have such steps in installation confluence page.attached snapshot.Hi Arun,That is the same command in general. Only difference is it is going deep down inner level ( like a filter) so you could see it clearly.Thanks,LinhHi Linh, after fixing the nacm permissions, I re-deploy all the services, the status was green for all services, but then I started the automation script, after the execution of the delete commands, the nso server was crashed again, can you please take a look on it now ? the nso is down.Hi Ofir, Please leave the system as it for now. The team and I will take a look. We will let you know when we are done!Thanks,LinhSure, Linh. you can do the investigation, until Sunday we must have a solution so we won't be blocked.Thanks,Ofir. Can I have Jenkins console url again?Thanks,LinhHello Ofir,Would you be able to provide alternative links for new console logs? I cannot access the above link.Thanks,Isaac Kimis it ok ? Yes thank you!Will take a look now.Thanks,Isaac KimHello Ofir,Looking at your console logs, I can see something that stood out:<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"><pre class="code-java">480: 2022-03-25T17:20:17: Cleanup CFP Services481: 2022-03-25T17:20:17: Command to be executed: delete flat-L3vpndelete flat-L3vpn[ok][2022-03-25 14:20:14][edit]admin@ncs% 485: 2022-03-25T17:20:17: Command to be executed: delete flat-L2vpndelete flat-L2vpn[ok][2022-03-25 14:20:15][edit]admin@ncs% 489: 2022-03-25T17:20:18: Command to be executed: delete l3vpn-ntwdelete l3vpn-ntw[ok][2022-03-25 14:20:15][edit]</pre></div></div>We can see here that both flat-L3vpn and l3vpn-ntw servicepoints are being deleted in the same transaction. This is not supported behavior.With TSDN stacked services, only the top level NB servicepoint that is being used for provisioning can be interacted with.For example:l3vpn-ntw => If you are using l3vpn-ntw for L3 provisioning, then you can only use l3vpn-ntw (no flat-L3vpn)flat-L3vpn => If you are using flat-L3vpn for L3 provisioning, then you can only use flat-L3vpn (no l3vpn-ntw)l2vpn-ntw => If you are using l2vpn-ntw for L2 provisioning, then you can only use l2vpn-ntw (no flat-L2vpn)flat-L2vpn => If you are using flat-L2vpn for L2 provisioning, then you can only use flat-L2vpn (no l2vpn-ntw)From the above console logs, you can see that both l3vpn-ntw and flat-L3vpn servicepoints (and both L2 servicepoints) are being targeted. This will lead to undefined behavior and will definitely cause race conditions.Could you please try to reinstall your backup and try to remove the conflicting operations?Thanks,Isaac KimHi Isaac, This automation flow is being executed for a long time, I can send you prev logs, after upgrading to 5.7.2.1 + M3 it started to crash, this deletion script is been used by other teams as well, nothing has changed from our side but the NSO version which was upgraded, what do you mean by installing backup ? should I re-install everything all over again ? sorry but it's not acceptable solution, as the process is long and it takes the entire day for that, if it's not supported operation, I think you need to block it, anyway, crashing the entire NSO server and re-install is required, not acceptable.Hi Issac,As Ofir said, it is not acceptable for the System to crash in any race condition.we will modify the automation steps . but lets have CDETS on it to fix the issue.Ofir., please riase CDETS referring to this Jira ticket.Thanks,Arun.M.GHi Arun,Because the delete operation has conflicting servicepoints, the behavior afterwards is undefined. I can see from NSO crash dump that the issue stems from undefined plan path for ietf te from NSO side effect of trying to reactive redeploy zombie. Now we have to step through the entire delete flow to see where the race condition lies.To reiterate, it may not be acceptable for race conditions, but this is undefined behavior because there is conflict in servicepoint interaction. Again, with undefined behavior, the outcome is unpredictable which can cause crash like this.To be frank, it is unacceptable to continue using conflicting operations as part of automation flow. This needs to be resolved to have valid automation runs.I am continuing to look into this issue despite the undefined nature of the outcome.I will update on this ticket once we confirm exact root of issue.Thanks,Isaac KimPlease do not raise a CDETS for this ticket yet as we do not know if the issue stems from TSDN service logic, from platform failure, or from the undefined behavior due to servicepoint conflict.Until we identify root cause of issue, we should not jump to any conclusions.Because the automation NSO setup is reused across runs, the issue can stem from previous state. Because the NSO setup is "cleared" using conflicting delete operations, any run after the first run on virgin NSO setup is invalid.I wil be debugging on the earliest NSO logs provided as this should be the cleanest state. I can see that IETF TE oper-data was able to be referenced by platform correctly before the mass delete operation (which has conflicting servicepoint delete). After this transaction is processed, I can see NSO cannot access IETF TE oper-data plan path anymore. Any logs or automation "runs" after this point must be assumed invalid due to the conflicting operations.Hello Ofir and Arun,Thank you for your patience on this issue.With your input, we have been able to reproduce this issue consistently on our local setups and isolate the source of the issue.Reproduction Steps:1. Load merge all L2 payloads (as defined in Jenkins console logs) and all IETF TE payloads2. commit3. Wait for plan to be reached for all services4a. Failed Delete Commands :delete flat-L2vpndelete l2vpn-ntwdelete l2vpn-route-policydelete tecommitRESULT : NSO crash OR stuck IETF TE zombie4b. Successful Delete Commands :delete l2vpn-ntwdelete l2vpn-route-policydelete tecommitRESULT : Services Deleted, Zombies clearedIf you would like to have more details on our reproduction setup, please let me know and I can provide exact steps.Based on these results, I believe that fixing the conflicting delete operations can resolve your issue. If you fix the conflicting delete operations but still face NSO crash, please let us know so we can investigate further.For your existing TSDN + NSO automation setup, I have tried to recover the CDB, but because there are pending zombies with illegal references, I was not able to load the CDB files into a running NSO (which is required for CDB manipulation with ncs_load). Unfortunately, it seems like you will have to reinstall TSDN.For preventing this conflicting delete operations from occurring on service side, please feel free to open an enhancement request and we will see how we can implement some checks to prevent this from happening again.Thanks,Isaac KimHi Isaac,Thanks for the comments and suggestion. we agree on the steps. but NSO crash is not accepted in any Race condition. we should need to have a fix to avoid the issue hitting at customer side with this race condition.Thanks,Arun.M.GHi, raised CDETS for tracking, please keep this Jira ticket as a reference to cdets bug for further investigation.Thanks,Ofir.Hello Arun and Ofir,After discussing with team, we have come to conclusion that despite the undefined nature of the delete operations, it is not acceptable for NSO to crash in any circumstance.Thank you for raising the CDETs. We will look into the crash further now that we have a reproducible procedure.Yes, we will leave this ticket open for now and I will update this ticket if we have any progress on the CDETs.Thanks,Isaac KimThank you Isaac and team for the understanding.Late update, reproduction setup was provided to platform team few week ago. Waiting to get assigned to engineer.Hello Ofir and Arun, we have gotten a reply from platform team and we are working to bring a fix for TSDN 5.0.This issue seems quite rare. Despite this, the workaround for TSDN 4.0 is the following:<ol>Only delete top layer serviceWhen deleting service, use the full path to service instance</ol>I have added a warning in our TSDN 4.0 documentation for this.For TSDN 5.0, we will absorb the platform fix for this.Thanks,Isaac KimWorkaround provided for TSDN 4.0 and fix will be absorbed for TSDN 5.0