On ncs 5.4.6, clean up of service is leading to zombie:   we create the service in single transaction ( stack mode ). I have observed that NSO responds with ok and then config is pushed to device which is contrary to previous functionality on 5.4.2 where only after config is pushed to device, nso responds with ok. Also, after cleaning up the service, we have observed that service stays as zombie. Below is the ip and creds of NSO, attached payload and command to create ; cleanup service  IP: 10.64.96.124 ( nso / Cwork123! )  command to create service : netconf-console --host=10.64.96.124 -u nso -p Cwork123! --port=2022 --rpc=payload.xml  command to clean up service: tm-tc-actions cleanup service Crosswork_1001 no-networking false
 Hi Raja,I logged on your setup and saw you did not configure nacm user correctlynso@ncs% show nacmread-default   permit;write-default  permit;exec-default   permit;groups {  group ncsadmin {    user-name [ private ];  }  group ncsoper {    user-name [ public ];  }Please correct it according to the wiki Also zombie that I saw is from tm-tc packagenso@ncs> show zombieszombies service /cisco-tm-tc-fp-internal:tm-tc pending-deleteWe are NOT the owner of the TM-TC package. Please contact the owner of this package for precise steps/command to cleanup the zombieThanks,LinhHi Linh, I added the following nacm config set nacm groups group ncsadmin user-name nsoi still see the same issue. Can you please test same usecase on a clean setup with proper nacm permissions and confirm.I see that it was tested on corrupted setup with existing zombies.We faced similar issue of plan not-reached and zombies not back tracking correctly when nacm permissions are not set correctly.ThanksSahithiHi Raja,I was able to cleanup zombie by going down to device levelnso@ncs% request tm-tc-actions cleanup service Crosswork_1001 device 10.64.96.123 no-networking truesuccess truedetailCleaning up TMTC service: Crosswork_1001 Removed all plan components Removing zombie service: /ncs:zombies/ncs:service{"/cisco-tm-tc-fp-internal:tm-tc[name='Crosswork_1001'][node-name='10.64.96.123']"} Removed zombie service Removed service oper Removed side-effects Removed side-effects Removed kickers Removed kickers Cleanup Successful for Crosswork_1001nso@ncs% run show zombies% No entries found.Please let me know if I can close the ticketRegards,LinhHi LinhUp on running the command for first time, we can observe that there is a zombie with pending-delete. After running the command again, the zombie is getting removed. please find screenshot in attachments. I have created another setup, please use it to debug10.104.127.17  nso / Cwork123!Hi Raja,As the rule of thumb, cleanup action should be used as the last resort. These are steps you should follow1. Delete the service2. If zombie stay behind then do a cleanup action.It seems to me that you perform a cleanup action without deleting the service first because I saw this warning "...Not able to retrieve diffset for zombie ..."Thanks,LinhHi Moksha,Glad the clean up worked with change . If you want me to file an RT Below details are required to file a RT:Problem statement.Expected behavior.NSO logs.Clear steps to reproduce the issue.Sample package/reproduction. Priority.Found in Version.When you provide the sample package. Please give the steps on how to reproduce the issue with it when installed locally.ThanksSahithiProblem statement : Up on cleaning up the service, there is a zombie created despite config delete is successful. When clean-up is retried, the zombie is getting cleared. Expected behaviour: There should not be any zombie if device config delete is successful NSO logs : Clear steps to reproduce the issue: Install the following functional pack on ncs version 5.4.6. - Set the following commands on ncs_cli 1.set cisco-tm-tc-fp:cfp-configurations dynamic-device-mapping cisco-iosxr-cli-7.33:cisco-iosxr-cli-7.33 python-impl-class-name tm_tc_multi_vendors.IosXR2.set cisco-tm-tc-fp:cfp-configurations stacked-service-enabledRun the following command to create the service ( the device key is 10.64.96.113 in given payload, change it as required ) - netconf-console -u nso -p Cwork123! --port=2022 --rpc=payload.xmlRun the following command to clean up the service ( the device key is 10.64.96.113 in given command, change it as required ) - tm-tc-actions cleanup service Crosswork device 10.64.96.113 no-networking falseShow zombies should display the zombie created due to cleanupRun again the same cleanup command Show zombies output should be empty, refer to attached screenshotSample package/reproduction : Priority :Found in Version : 5.4.6 Hi Moksha,The package you have given is with tm tc functionality. NSO team would not understand the tm-tc functionality. They will not be able to debug the clean up action. This ticket would be rejected if I file with FP specific logic. is something which is generic just for the NSO i.e remove all the TM-TC code, isolating just the issue with NSO here.Please create a sample package with example service and action which is replicates this issue.You can find example sample packages under -> nso-5.4.6/examples.ncs/getting-started/developing-with-ncsClean up is something which is used as last resort.ThanksSahithiHi Could you please answer the question by Sahithi?ThanksNipaPlease reply to Sahithi's question above.Thanks,NipaEmail from Moksha on 04/28+ TeamHi Sahithi / Nipa,Is there a way we can go ahead with tm-tc package for now, we never tried this and it might take some more time to understand and build the sample package.Regarding cleanup, we do it only when the devices gets removed from our application, which is not frequent.Regards,MokshaHi Moksha,Do you have a queue on RT system to file a defect? if so, please go ahead and file it. If not, let me know I will try and get you an access.Thanks,NipaHi Nipa, I don't have details on how to file a RT. Please provide the link, i can try and let you know if i have access. Regarding the cleanup, i already responded on email we do it only when the devices gets removed from our application, which is not frequent. During cleanup, we removed the configuration and remove the service in single transaction. Here is the link for RTClose this since the issue is not on TSDN CFGRegards,Linh