sareddyd-esxi-01-vm2: Device is locked in a commit operation by session 1272984: NSO: 5.7.3build: nso-5.7.3-tsdn-4.0.0-M6device: /auto/iox-ucs-077-san2/prod/7.7.1.15I.SIT_IMAGE/xrv9k/wslogs : 10.56.116.151://tmp/data_061022_065731_dirlogin: nso/Public1234!Steps:I was deleting EVPN services. Two services stuck with the "Device is locked in a commit operation by session" Number as mentioned above.Logs are present in the same NSO CFS : 10.56.116.151. Login provided.RFS1: 10.56.116.152RFS2: 10.56.116.153I request you to help in releasing the lock and continue to delete the services. I am not able to operate any action on the devices. Please help.Same log attached with this ticket
 /etc/init.d/ncs stop and start solved the issue. But the two evpn services: l2nm-evpn-404,l2nm-evpn-405 are in zombies state. In device they got deleted from NSO before this issue raised. Please check ...Hi Sampath,I would suspect that the stuck zombies are due to the session lock and NSO stop start.I can try to help with unblocking your setup.You mentioned that the config was deleted from the network but still exists in NSO? In this case I would advise to try to execute L2NM cleanup action with no-networking true. Afterwards sync device and your setup should be ready to go.Let me know if this works.Thanks,Isaac KimHello Sampath, I believe we have been able to provide a suitable recovery option for your issue. If there is no reply in few days, I will close this ticket.Thanks,Isaac KimHey, Sorry Isaac, thought of talking to you on this. I have done deleting services in the morning as it has not resolved for a long time I have done stop and start in the evening, hence I am able to delete other service, I have just clean up zombie service. So We need to concentrate only on the issue where I am not able to delete the services because of locking. Let me know if we have investigated on that partHi Sampath, I ran across this ticket when cleaning up my support queue.Apologies, I seemed to have missed the reply for a long time. I will try to analyze the logs and see if there is any unexpected behavior so we can have some closure on this issue.Looking back on the log dump you posted to this ticket,we can see that DLM and/or CNC GUI is having trouble clearing lock on transaction handle 1 as seen below:<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"><pre class="code-java">2-Jun-2022::04:55:23.983 <0.1337.0> <0.6529.272> ** Generic server <0.6529.272> terminating ** Last message in was {jsonrpc,,[{,1}]}** When Server state == {webui_sess,,       <0.6529.272>,https,760286,0,0,[],</pre></div></div>Afterwards, we can see that this th 1 is invalid as it does not exist as seen below:<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"><pre class="code-java">{, {error_report,<0.1337.0>, {<0.18671.272>,crash_report,  [[{initial_call,{cs_trans,init,[]}},  {pid,<0.18671.272>},  {registered_name,[]},  {error_info,   {exit,    {bad_return_value,     {error,      {jsonrpc_error,       {-32602,,       },       [],[],       {struct,[{param,}]},       }}},</pre></div></div>This indeed is something we have seen in rare cases in TSDN 4.0 where CNC UI's session transaction lock on NSO is not cleared as described in the ticket. Its strange that the transaction handler id is not found. It seems like due to NSO start/stop during an active commit may have caused CNC UI's state data to be out of sync with NSO. The logical assumption here is that start/stop cleared the transaction, but CNC UI is still attempting to clear what it thinks is an active transaction. In the logs, we can see the above two crash logs repeatedly. This is likely why CNC UI is in unresponsive state.Although changing NSO state during active commits/transactions is not recommended, the workaround would be to log out of CNC GUI (clear CNC state data), restart NSO (force close any locked sessions), and then continue.I hope this helps provide some clarity on this issue.If you face this issue again anytime in the future, please feel free to open a new ticket and reference this JSD for context.Thanks,Isaac KimAnalysis and workaround provided.