Would like to know more about Zombie-service pending delete state from NSO: Hi Team,Please help to clarify for the following NSO behaviour.1. When a clean up action is performed directly for a service(any service/any payload), it still removes the plan status from the device/CNC server/NSO 2. And after performing the Clean-up action for the service, the following is the observation from the NSO.admin@ncs> show zombieszombies service /cisco-flat-L3vpn-fp-internal:flat-L3vpn pending-deletezombies service /cisco-flat-L3vpn-fp-internal:flat-L3vpn pending-deleteadmin@ncs>Please confirm if the above said behaviors are expected from NSO.Thanks,Dinesh
HI Dinesh,Here is the troubleshooting guide for zombie handling.We can perform cleanup action on the deleted service to remove the leftover plan and zombies. Cleanup action before service deletion is not valid.--RaghaviWiki for zombie handling:Cleanup action before service deletion is not valid. - This is perfect.Do you have any solution steps for the below NSO state?admin@ncs> show zombieszombies service /cisco-flat-L3vpn-fp-internal:flat-L3vpnpending-deletezombies service /cisco-flat-L3vpn-fp-internal:flat-L3vpnpending-deleteadmin@ncs>Can you add the steps to reproduce how you landed into this issue..?Also please provide the details about ned-id's and device versions when payload is given this help to reproduce the issue.1. Create a L2NM service2. Click on Clean-up option from the (L2VPN Service) provisioing NSO tablePlease do find the payload in the enclosure.NED-ID - cisco-iosxr-cli-7.33     7.33.12NCS Version: 5.5.2.7TSDN Package - nso-5.5.2.3-tsdn-3.0.0-M5XRV Version: 7.3.2.37I-0 and 7.4.1-0 HI Dinesh, Can we have a quiickcall..? Hello Dinesh,Please attach the NSO logs and plan state and any errors in plan and any missing items from the above wiki.Also please try out the same with no-networking to true and share the results.Also when you click on cleanup action whats the response you received from NSO. is your cleanup successful..?I also tried to reproduce this on my machine its not reproducible. So please provide the details of the above requested logs. --RaghaviNSO version : 5.5.2.7CFP build version : nso-5.5.2.3-tsdn-3.0.0-M5Device type and version used for testing : XRV9K | XRV Versions: 7.3.2.37I-0 and 7.4.1-0STEPS TO REPRODUCE:admin@ncs% run show l3vpn-ntw vpn-services vpn-service-plan L3VPN_NM-IGP-IPV4-60 plan | tab                                                                         POST                        BACK     STATUS                                        ACTIONTYPE                  NAME  TRACK GOAL CODE  STATE                   STATUS  WHEN         ref STATUS----------------------------------------------------------------------------------------------------------------------------------------------------------self                  self  false -   -    init                    reached 2021-08-31T06:31:19 -  -                                  ietf-l3vpn-ntw-nano-services:config-apply reached 2021-08-31T06:31:19 -  -                                  ready                   reached 2021-08-31T06:31:35 -  -ietf-l3vpn-ntw-nano-services:vpn-node PE-A_1 false -   -    init                    reached 2021-08-31T06:31:19 -  -                                  ietf-l3vpn-ntw-nano-services:config-apply reached 2021-08-31T06:31:19 -  -                                  ready                   reached 2021-08-31T06:31:35 -  -ietf-l3vpn-ntw-nano-services:vpn-node PE-C_1 false -   -    init                    reached 2021-08-31T06:31:19 -  -                                  ietf-l3vpn-ntw-nano-services:config-apply reached 2021-08-31T06:31:19 -  -                                  ready                   reached 2021-08-31T06:31:35 -  -admin@ncs% request l3nm-actions cleanup servicePossible completions: <string>admin@ncs% request l3nm-actions cleanup service L3VPN_NM-IGP-IPV4-60 no-networkingPossible completions: false trueadmin@ncs% request l3nm-actions cleanup service L3VPN_NM-IGP-IPV4-60 no-networking true########################<ol>Warning    #########################You are about to forcefully cleanup a T-SDN service.This will affect the deploying service and leave network device(s) ; NSO out-of-sync (for no-networking=true).Are you sure you want to proceed? yessuccess truedetail Removing service /l3vpn-ntw/vpn-services/vpn-service{L3VPN_NM-IGP-IPV4-60} Removed service /l3vpn-ntw/vpn-services/vpn-service{L3VPN_NM-IGP-IPV4-60}Cleaning up L3vpn service: L3NM-L3VPN_NM-IGP-IPV4-60-internal Removing service /cisco-flat-L3vpn-fp:flat-L3vpn{L3NM-L3VPN_NM-IGP-IPV4-60-internal} Removed service /cisco-flat-L3vpn-fp:flat-L3vpn{L3NM-L3VPN_NM-IGP-IPV4-60-internal}Cleaning up L3VPN Internal Services: L3NM-L3VPN_NM-IGP-IPV4-60-internal Removed all internal plan components Removing zombie service: /ncs:zombies/ncs:service{"/cisco-flat-L3vpn-fp-internal:flat-L3vpn[name='L3NM-L3VPN_NM-IGP-IPV4-60-internal'][endpoint-name='PE-A_1']"} Removed zombie service Removing zombie service: /ncs:zombies/ncs:service{"/cisco-flat-L3vpn-fp-internal:flat-L3vpn[name='L3NM-L3VPN_NM-IGP-IPV4-60-internal'][endpoint-name='PE-C_1']"} Removed zombie service Removed commit-queue-recovery-data Cleanup Successful for L3VPN Internal Services</ol> Removed all plan components Removing zombie service: /ncs:zombies/ncs:service{"/cisco-flat-L3vpn-fp:flat-L3vpn[name='L3NM-L3VPN_NM-IGP-IPV4-60-internal']"} Removed zombie service Removing zombie service: /ncs:zombies/ncs:service{"/cisco-flat-L3vpn-fp:flat-L3vpn[name='L3NM-L3VPN_NM-IGP-IPV4-60-internal']"} Removed zombie service Cleanup Successful Removed all plan components Removing zombie service: /ncs:zombies/ncs:service{"/l3vpn-ntw/vpn-services/vpn-service[vpn-id='L3VPN_NM-IGP-IPV4-60']"} Removed zombie service Removing zombie service: /ncs:zombies/ncs:service{"/l3vpn-ntw/vpn-services/vpn-service[vpn-id='L3VPN_NM-IGP-IPV4-60']"} Removed zombie service Cleanup Successful for L3NMadmin@ncs% *** ALARM service-activation-failure: Operation failed because: Not able to retrieve diffset for zombie nano service : '/cisco-flat-L3vpn-fp-internal:flat-L3vpn'admin@ncs% run show zombieszombies service /cisco-flat-L3vpn-fp-internal:flat-L3vpn pending-deletezombies service /cisco-flat-L3vpn-fp-internal:flat-L3vpn pending-deleteadmin@ncs% exitadmin@ncs> request zombies service /cisco-flat-L3vpn-fp-internal:flat-L3vpn re-deployError: Operation failed because: Not able to retrieve diffset for zombie nano service : '/cisco-flat-L3vpn-fp-internal:flat-L3vpn'admin@ncs> configureEntering configuration mode privateCurrent configuration users:admin https (webui from 172.24.97.150) on since 2021-08-30 23:32:47 private modeadmin https (webui from 172.24.97.150) on since 2021-08-30 22:50:27 private modeadmin https (rest from 10.194.57.24) on since 2021-08-26 01:16:22 private modeadmin@ncs% run show l3vpn-ntw vpn-services vpn-service-plan L3VPN_NM-IGP-IPV4-60 plan | tab------------------------------------------------------------^syntax error: unknown argumentadmin@ncs%System message at 2021-08-30 23:36:25...Commit performed by admin via https using rest.admin@ncs%NSO IP: 10.194.57.50 admin/Ovatest123@hi Dinesh i don't see any logs attached here.Please refer the wiki--Raghavi.Hi Raghavi,Please be specific to which log you might require from the NSO and it's respective directory.Also update the Wiki link as well.Thanks,DineshHi Raghavi,FYI - ncs.log has been enclosed for now.You can access the NSO with the following details for any more logs if required. NSO IP: 10.194.57.50 admin/Ovatest123@Thanks,Dinesh Hi Dinesh,Thanks you, we updated the wikiPlease tar and share the complete NSO logs, the above provided logs are not enough as this is not reproducible on our setup.--Raghavi. Dinesh,The reason we ask you to upload logs to the ticket is so all information is co-located and we can have it for reference in future if needed. Let us know if you have trouble uploading complete technical support data.Thanks,Nimisha.Nimisha,Yes, there is a trouble in uploading the tar'ed folder here as it says "Unknown server error (502)."--DineshConsidering the ticket progress gap, have filed CSCvz58592This is being handled by a CDETS now so closing this ticket