provisioning L3VPN slow in scale setup: Our NSO scale setup has 25K devices, 200K VPN services.Compared to earlier NSO 5.4.x, current NSO 6.0 is slow in provisioning L3VPN services.In this release, we are trying L3VPN with 5 end points.L3VPN 5 EP - 1 service - 240 to 250 sec (without actually sending configs to routers -no-networking - true)L3VPN (point-to-point) - between 2 devices - 1 service - 104 to 110 sec. Is there any bug on L3VPN? Can we speed this up in setup?Question 2: The setup has only 3 months license. This setup needs to be tested over 5 months. Can we extend the license? Can you share the steps.Setup: 172.20.63.133Creds: cnctest/Ovatest
Hi ,Cannot login in your NSO .Could you please share correct working credentials ?Also can you please share usecase and comparison for NSO 5.4 vs NSO 6.0 test you executed .Please update steps of testing used came in both ?Types of devices and ned types same ?I see in one comparison no networking true - In this case you see different numbers in NSO 5.4 vs NSO 6.0 .Please share details and set up .I will look into .ThanksPriyankaWe use REST API through shell script and create L3VPN service through payloads.Device types are XR device ( simulated ) Major difference is for L3VPN, we were only using 2 end points in NSO 5.4.x. For NSO 6.0, we create L3VPN with 5 end points and also L3VPN with 2 end point. In both NSO cases, we were only executing no-networking is true.Other difference is updated payload between new NSO 6.0 and old NSO 5.4.x.In terms of memory, CPU, Swap, everything is same. It is the same machine.Setup details 172.20.63.133UI, CLI - cnctest/Ovatest123@ Hi Rama,First question comparing 2 EP (NSO 5.4.x ) vs 5 EP (NSO 6.0 ) is not apple to apple comparison.Please share a comparison for 2 EP with the same payload for both use cases.As there is no point in checking where orange is compared to apple.Usecase makes lots of difference in results.I hope you got my point.ThanksPriyankaAgree for 5 EP and 2 EP difference. Even comparing 2 end point L3VPN between NSO 6.0 and NSO 5.4.x also yields the same result, Time is higher in NSO 6.0. Please clarify on the license part also.Hi Rama ,Can you be specific what is the exact difference you see between NSO 5.4.x and NSO 6.0 2 EP in terms of commit times and also share exact outputs along with plan for both usecase.I am really not sure what licensing you are asking us ?We as CFP team not responsible for any license.ThanksPriyankaProvided setup details for both NSO and the scripts that are used along with payload. Kindly check why NSO 6.0 is taking long time for L3VPN.  Hi Rama,I compared both your labs .Here is my observation :1. All RFS nodes on NSO 5.7 x -> has 26 packages 2.All RFS nodes on NSO 6.0 > has 31 packages .Here the packages which are causing degradation of performance are NETCONF NED packages.Please remove all netconf ned packages from both nodes .Then only we can say these set ups are comparable .right now NSO 6.0 has more number of netconf neds.Here is the platform ticket with reference :Here is my recommendation :for faster performance :1.Remove all NETCONF neds from RFS ndoes .2.Make sure to keep same number of packages on both set up if you are comparing them .We should keep identical labs if we are comparing them .Also please close CDET which you have raised :As without confirmation no point in keeping duplicate tracking of issues.ThanksPriyankaDevices in both setup use only cli ned. How can the unused netconf packages can cause performance issues?NSO 5.7.5.1devices use 'device-type cli ned-id cisco-iosxr-cli-7.39'NSO 6.0 device use 'device-type cli ned-id cisco-iosxr-cli-7.43'We used only the default packages available in TSDN FP for both NSO version. Removed netconf related packages in NSO. But there are no changes in time taken for provisioning.Hi Ramakrishnan,Priyanka mentioned the RT(Ticket on NSO platform) in the previous reply on degradation of performance due to netconf NED presence. Regarding "how", We believe NSO Platform folks are more well versed with the technical implementation here. You should be able to file RT with help of Crystal and find out the how.You mentioned that you removed netconf packages and tested. Please provide the following details to proceed further. Can you please let us know the exact time degradation?What is time to provision in 5.7.x vs time to provision in 6.0 ? Please give us the times. Along with "show packages package oper-status up" on the system.Also are you using 5.4 or 5.7 ? In the description I see 5.4 while you mention in the reply to be 5.7.We do not see any kind of degradation from our testing. So, we would need as much info specific as possible. If it would be easier to webex, let us know that as well.Hi Rama,I tried to create service using your payload in your labs and it ended up in ned error in NSO 5.7.5.1 lab and rpc error in NSO 6.0 lab.Can we do webex and get closure as no point of back and forth .ThanksPriyankaHi Team,I just tried now and able to provision L3VPN services in both NSO's. Will attach the snaps to show exact times for both including "show packages package oper-status up" o/p. In NSO 6.0, we see a huge delay of 10 to 20 times than earlier releases. We have one more standalone NSO 6.1, there also same issue seen. This delay is seen ONLY for L3VPN and not for L2VPN. Every release from the time* NSO is introduced into COE/CNC, am doing the scale testing using these scripts for scaling up 200K ; 300K VPN services. And never hit this huge delay for L3VPN.Thanks for sharing Platform tickets but these are not transparent to every one like CDETS. As we don't have access to platform tickets, neither description nor the content are visible to us.Please refer the attached notepad "repro-CSCwe21126" for exact times and show package o/p along with NSO logins.Will be available online for 1 more hour before i sign-off for the day. Please ping me, if required to repro again. "ramasrin" will be available tomorrow morning IST to provide more info. if needed for you to help us triage this at earliest.Thanks for your support !!-Sreedhar. Hi Sreedhar ,Please join :ThanksPriyankaHi Sreedhar,As per our webex discussion :Please try 50 service creation using multi-thread on NSO 6.x - Please share a comparison of 50 service (single thread ) vs multi thread (using 5 thread .each thread 10 services without device conflicts between threads.)Let's first focus on standalone for this.'ThanksPriyankaHi Priyanka, Our script has the payload for 10 services each with same device. We can modify it so that all devices used are different. When you mean by multi threading, should we run 5 instance of that script simultaneously from 5 terminals? By single thread, do you mean single terminal executing 10 service one by one? Or when i refer the following link, it talks about changes in package-meta-data.xml. Should we do anything of that sort ? I will try to sync up with you. with regards,Srinivasan RHi Rama,As per our webex discussion - Try the way i suggested distributing devices across services on different threads.Please try that and let me know how it goes.ThanksPriyankal3vpn.xml is the payload that caused lot of delay. scale-l3nm_fast is the one that NSO team gave, that is faster.  We observed that vpn-instance-profile created for each service is not referenced in the vpn-node in scale-l3nm_fast.xml. If we refer the vpn-instance-profile under vpn-node, it takes around 25 seconds for each service. If we dont refer it, it takes just 1 second to create the service.Hi Team, With the modified payload (removed the reference of vpn-instance-profile under vpn-node), provisioning of L3VPN is very fast. Able to provision 10 L3VPN services in less than 20 seconds currently. Unblocked with the issue with this workaround. Thanks,Sreedhar.changing payload helped team to unblocked.