Provisioning Service from CAT UI throws error " server didnot respond to the request within 25 sec": Hi,While trying to provision l2vpn p2p srte service from CNC UI , facing the commit error "service did not respond to the request within 25 seconds".
payload atatched Hello Arun,I would need more information to RCA this issue.Would you be able to provide ssh credentials to your NSO node so I can check a few things? I will not modify anything.Thanks,Isaac KimHi Isaac,NSO: 10.56.116.240 ( nso/Public1234!)Thanks,Arun.M.GHi Arun,I can see in your logs the following:<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"><pre class="code-java">% Failed to commit one or more configuration items during a pseudo-atomic operation. All changes made have been reverted.!! SEMANTIC ERRORS: This configuration was rejected by!! the system due to semantic errors. The individual!! errors with each failed configuration command can be!! found below.l2vpnxconnect group L2NM-P2p-SRTE-742 p2p L2NM-P2p-SRTE-742 neighbor ipv4 126.1.1.30 pw-id 742 mpls label local 742 remote 753!!% Invalid argument: Remote label already assigned to neighbor ! !!!end</pre></div></div>It seems like NED error : "Remote label already assigned to this neighbor".It seems like later, there was a service update made to this service "L2NM-P2p-SRTE-742" and the plan became reached.Could you confirm the following?<ol>Did service provisioning show failed status when checking CNC UI after the timeout?Did service update make service go into provisioned state?Are you using CQ or no-CQ commits for these operations? I assume no-CQ was used which caused all changes to be reverted. This makes more sense in regards to the current plan status which is reached with a service update.</ol>Thanks,Isaac KimHi Isaac,yes. there was a commit failed due to the below mentioned label conflict. i have corrected after that.but sometime, even for proper service also, we are facing the commit error for time out.below are the answer .,1] Did service provisioning show failed status when checking CNC UI after the timeout? >> Service provisioning doesnt happen at all. it will stay on the same provisioning page.2] Did service update make service go into provisioned state?  >> yes. i tried these to modify the SR policy in l2vpn p2p sr service. it worked.3] Are you using CQ or no-CQ commits for these operations? I assume no-CQ was used which caused all changes to be reverted. This makes more sense in regards to the current plan status which is reached with a service update.  >> i am not modifying anything in the Advanced option in CNC UI during the commit.it is always by default.Note: this issue is intermittent. sometimes it commits properly.Hi Arun,Thank you for the quick reply. Seems like there is a system slowdown which might be the cause of the symptoms described. Let me take another look at your system and see if we can isolate the cause of slowdown.I may execute few operations on NSO to check NSO system responsiveness. I will not make any changes.Thanks,Isaac KimHi Arun,I ran a few checks and things seem to be okay:<ol>L2 Service Callback max time delta is 0.05 seconds (across all services)NSO L2 service get-modification reverse responds quickly on NSO CLINSO L2 service get-modification reverse responds quickly on NSO Web UI"top" shows low overall system resource utilizationCorrect ulimit fd limit</ol>So it didnt seem like NSO itself was having any slowdowns.Next I tried to test southbound operations such as device connect, device config pull, etc. Here we saw some greater levels of delay:<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"><pre class="code-java">admin@ncs% request devices device NCS55A2-SDN-112 compare-config | details very-verbose2022-04-06T20:27:03.585 device NCS55A2-SDN-112: compare-config...2022-04-06T20:27:03.586 device NCS55A2-SDN-112: taking device lock... ok (0.000 s)2022-04-06T20:27:03.599 device NCS55A2-SDN-112: connect...2022-04-06T20:27:04.410 device NCS55A2-SDN-112: cisco-iosxr-cli-7.38: connecting... ok (0.466 s)2022-04-06T20:27:05.370 device NCS55A2-SDN-112: connect: ok (1.771 s)2022-04-06T20:27:05.370 device NCS55A2-SDN-112: show...2022-04-06T20:27:05.459 device NCS55A2-SDN-112: cisco-iosxr-cli-7.38: reading config...2022-04-06T20:27:10.281 device NCS55A2-SDN-112: cisco-iosxr-cli-7.38: transforming input... ok (0.009 s)2022-04-06T20:27:10.293 device NCS55A2-SDN-112: cisco-iosxr-cli-7.38: reading config: ok (4.834 s)2022-04-06T20:27:10.296 device NCS55A2-SDN-112: cisco-iosxr-cli-7.38: retrieving transaction-id... ok (6.791 s)2022-04-06T20:27:17.090 device NCS55A2-SDN-112: cisco-iosxr-cli-7.38: extended parsing... ok (0.017 s)2022-04-06T20:27:17.108 device NCS55A2-SDN-112: cisco-iosxr-cli-7.38: populating cdb... ok (0.599 s)2022-04-06T20:27:17.708 device NCS55A2-SDN-112: show: ok (12.338 s)2022-04-06T20:27:17.726 device NCS55A2-SDN-112: releasing device lock2022-04-06T20:27:17.726 device NCS55A2-SDN-112: compare-config: ok (14.141 s)[ok][2022-04-06 20:27:17][edit]admin@ncs% request devices device ASR9901-SDN-30 compare-config | details very-verbose2022-04-06T20:29:42.512 device ASR9901-SDN-30: compare-config...2022-04-06T20:29:42.513 device ASR9901-SDN-30: taking device lock... ok (0.000 s)2022-04-06T20:29:42.524 device ASR9901-SDN-30: connect...2022-04-06T20:29:43.326 device ASR9901-SDN-30: cisco-iosxr-cli-7.38: connecting... ok (1.954 s)2022-04-06T20:29:46.278 device ASR9901-SDN-30: connect: ok (3.753 s)2022-04-06T20:29:46.278 device ASR9901-SDN-30: show...2022-04-06T20:29:46.353 device ASR9901-SDN-30: cisco-iosxr-cli-7.38: reading config...2022-04-06T20:29:51.034 device ASR9901-SDN-30: cisco-iosxr-cli-7.38: transforming input... ok (0.009 s)2022-04-06T20:29:51.047 device ASR9901-SDN-30: cisco-iosxr-cli-7.38: reading config: ok (4.693 s)2022-04-06T20:29:51.049 device ASR9901-SDN-30: cisco-iosxr-cli-7.38: retrieving transaction-id... ok (4.059 s)2022-04-06T20:29:55.112 device ASR9901-SDN-30: cisco-iosxr-cli-7.38: extended parsing... ok (0.013 s)2022-04-06T20:29:55.126 device ASR9901-SDN-30: cisco-iosxr-cli-7.38: populating cdb... ok (0.429 s)2022-04-06T20:29:55.556 device ASR9901-SDN-30: show: ok (9.277 s)2022-04-06T20:29:55.576 device ASR9901-SDN-30: releasing device lock2022-04-06T20:29:55.576 device ASR9901-SDN-30: compare-config: ok (13.063 s)[ok][2022-04-06 20:29:55][edit]</pre></div></div>We typically see half the response time for compare-config our real devices, but this is dependent on device and setup. But now I can see how this can potentially cause a problem, especially if NSO pull device config as part of the service provisioning transaction.This combined with the fact you are using no-CQ (default commit setting) and the fact that CNC UI has 25 second timeout, maybe CNC UI is waiting too long (over 25 seconds) for both device to fully provision.If you would like to see a full breakdown of timings for your commit, you can do the following:1. Log into NSO CLI 2. Enter config mode3. load merge L2 service payload (or set config) 4. run => "commit | details very-verbose"5. paste output hereThen I think it will be easier to isolate exactly which device operation is causing the most delay. It will also reveal if we are exceeding CNC UI timeout value.Thanks,Isaac KimHI Isaac,Thanks for the details and clarification.I will try to do from Ncs_cli using load merge L2 service payload and let u know. our labs are down . if will be back on monday . once it is back, i will check and let you know.Thanks,Arun.M.GDevice transaction phases revealed device response exceeded UI timeout value.