Scale -Plan completion Stuck -Plan completion Not happening for L2vpn services.( 1250 services created and 1250 services creation is running): Hi Team,We are trying to stabilize the XRV9K devices for the scale VPN count, We have 60 device topology, Initially, we created services for only 10 devices, and all the services are provisioned and it moved to success on the Crosswork, Later we powered ON 10 more devices and synced the devices to NSO and created 1250 L2vpn P2P services, but the Plan completion Not happening for those 1250 services. Attached are the service details. Please take a look at the issue.Please reach me for the setup details, I will unicast the details since it's a scale setup.FYI: We observed the issue after powering ON and provisioning the services, we didn't see any issue for the first 10 devices, It was already in the powered ON state and we provisioned the services.
 Hi., This is blocking us to proceed further.can someone take a look on this issue?Thanks,Arun.M.GHi Ragesh,We see that a lot of netconfs are loaded in the setup which would be impacting the performance. Please delete all the netconfs if they are not being used along with the multi-vendor packages. This would help to increase the plan completion turnaround time.Please help us with the following information as well:- What is the test strategy?- How many and what kind of devices are being used? Are the devices shared across services and are they hosted on same or different server?- What is the plan wait time?- How many lines of config is pushed by each service?- What is the payload being used?Thanks,NikhilHi NIkhil,this is NSO LSA setup. and customer might have all NEDs configured.,when there is Unused NEds, why does the performance impacts? anyhow, the NEDs are unused. lets have a webex meeting to do that together as i dont want to end up in some unexpected things to happen.Below are the answers for your queries.<ul class="alternate" type="square">What is the test strategy?   >> Actually we have a 60 devices topology having 48 PE devices and 12 Core devices includes 2 XTC and 2 RR . all devices having default resources 4vCPu, 16GB Ram and 64gb HD. All are Xrv9k devices that are created in Single UCS. It has the devices versions mixed with 7.7.1, 7.3.2, 7.5.2.while having all devices switched on, we were not able to scale more than 500 services ( 160 p2p, 160 evpn, 180 L3vpn vrf)so, to scale more , we switched off all devices except 10 PE devices and 12 core devices. we were able to reach 2k services per device pair. ( 2k was our target to reach per device pair)then. switched on another set of 10 PE devices, and started creating services and we ended up in this issue where Plan completion got stuck for 6 hours and after that , it started happening very very dead slow.Attached image has devices pairs and the count of services.<ul class="alternate" type="square">How many and what kind of devices are being used? Are the devices shared across services and are they hosted on same or different server?<ul class="alternate" type="square">What is the plan wait time?    >> as per the script, we have 30 seconds interval between the services creation. it will take 2.5 hours to create 1250 services. ( 250 services between 5 pair of services).<ul class="alternate" type="square">How many lines of config is pushed by each service?it is as normal L2vpn p2p, l2vpn evpn service payload with 2 endpoints.<ul class="alternate" type="square">What is the payload being used?Attached payloads. Advised the user to remove the unused NEDS and related multi-vendor packages from the system to improve the performance.This is actually Bug in the product and it is valid scenario.not accepting that this has been moved to Invalid.removing the unused NEDs is just a workaround. not a real solution.when the neds are ununsed ,why does it impacts the performance. this is bug .CSCwd26703 is being tracked for this. There is a ticket already in progress with the platform for this issue. We will update CSCwd26703 once we get a solution from the platform team. Closed this ticket since we already have a bug open for this to track the progress.