Not able to delete devices from RFS node after cleaning up services : I cleaned up all the services from CFS , TM-TC service from RFS and trying to delete devices from RFS and hitting the below:admin@ncs% delete devices device *admin@ncs% commitAborted: illegal reference 'core-fp-delete-shutdown-service ASR-903.cisco.com GigabitEthernet 0/0/7 device'admin@ncs% exitThere are uncommitted changes.Discard changes and continue? yesadmin@ncs> exitcopied show tech @ /ws/shbalu-sjc/NSO<del>rw-r</del><del>r</del>- 1 shbalu eng 276M Jun 1 03:56 data_060122_035155.tgz <del>rw-r</del><del>r</del>- 1 shbalu eng 1.1G Jun 1 03:56 data_060122_035147.tgz 
Hi Shanmugavel,Please share your setup info. I would check on your setup settings.Thanks,LinhHi Linh,Here is the setup info, FYI: I was doing cleanup and that is when i hit the issue while deleting devices from RFSCFS: 172.20.63.112 (admin/Vtsisco@123!)RFS: 172.20.63.120 (admin/Vtsisco@123!)I looked at ned-cisco-ios-cli-6.77-ASR-903.cisco.com.trace in RFS. The service that used the interface GigabitEthernet 0/0/7 is created >> 1-Jun-2022::00:41:30.004 user: admin/132617 thandle 3437741 hostname nso-rfs1 device ASR-903.cisco.com PREPARE 2:vrf definition L3VPN-9901-1-4-903 rd 180:180 address-family ipv4 route-target export 180:180 route-target import 180:180 exit-address-family !!ip prefix-list SET_COLORv4_RPL-903_5432 seq 5 permit 90.6.0.0/24interface GigabitEthernet0/0/7 service instance 1002 ethernet encapsulation dot1q 1002 bridge-domain 1002 exitexitinterface BDI1002 vrf forwarding L3VPN-9901-1-4-903 ip address 10.14.182.12 255.255.255.0 no shutdownexitroute-map SET_COLORv4_RPL-903 permit 5 match ip address prefix-list SET_COLORv4_RPL-903_5432 set extcommunity color 5432!mpls label mode vrf L3VPN-9901-1-4-903 protocol all-afs per-vrfrouter bgp 65001 address-family ipv4 unicast vrf L3VPN-9901-1-4-903 redistribute connected neighbor 10.14.182.18 remote-as 65007 neighbor 10.14.182.18 activate exit-address-family ! address-family vpnv4 unicast neighbor 10.14.182.18 activate neighbor 10.14.182.18 route-map SET_COLORv4_RPL-903 out exit-address-family !!<< 1-Jun-2022::00:41:30.007 user: admin/132617 thandle 3437741 hostname nso-rfs1 device ASR-903.cisco.com SET_TIMEOUT<< 1-Jun-2022::00:41:30.008 user: admin/132617 thandle 3437741 hostname nso-rfs1 device ASR-903.cisco.com SET_TIMEOUT<< 1-Jun-2022::00:41:30.062 user: admin/132617 thandle 3437741 hostname nso-rfs1 device ASR-903.cisco.com SET_TIMEOUTThen this service is never delete after this. Then this is expected behavior since it is still have reference to ASR-903.Correct steps is deleting the services that reference the device before deleting the deviceIt was removed from the CFS node:admin@ncs% show l3vpn-ntw No entries found.admin@ncs% admin@ncs% Since i already gave you the setup info, let me know if you still see the service on the CFS node, if the service is deleted and config was not removed from Device then this is an issue ned to be debugged.I see the device delete issue with 2 device (ASR-903.cisco.com, ASR9901-1) which were part of the same service and able to delete one device ASR9901-4 which was also part of the same serviceadmin@ncs> show devices listNAME        ADDRESS  DESCRIPTION NED ID        ADMIN STATE ---------------------------------------------------------------------------ASR-903.cisco.com 1.1.1.12 -      cisco-ios-cli-6.77  unlocked   ASR9901-1     1.1.1.1  -      cisco-iosxr-cli-7.39 unlocked   ASR9901-4     1.1.1.4  -      cisco-iosxr-cli-7.39 unlocked   admin@ncs> confEntering configuration mode privateadmin@ncs% delete devices device ASR9901-4admin@ncs% commitCommit complete.admin@ncs% admin@ncs% delete devices device ASR9901-1admin@ncs% commitAborted: illegal reference 'core-fp-delete-shutdown-service ASR9901-1 TenGigE 0/0/0/10 device'admin@ncs% admin@ncs% exitThere are uncommitted changes.Discard changes and continue? yesadmin@ncs> confEntering configuration mode privateadmin@ncs% delete devices device ASR-903.cisco.com admin@ncs% commitAborted: illegal reference 'core-fp-delete-shutdown-service ASR-903.cisco.com GigabitEthernet 0/0/7 device'admin@ncs% exitThere are uncommitted changes.Discard changes and continue? yesadmin@ncs> show devices listNAME        ADDRESS  DESCRIPTION NED ID        ADMIN STATE ---------------------------------------------------------------------------ASR-903.cisco.com 1.1.1.12 -      cisco-ios-cli-6.77  unlocked   ASR9901-1     1.1.1.1  -      cisco-iosxr-cli-7.39 unlocked   admin@ncs> Hi Shanmugavel, just wanted to update this ticket with latest information. We have unblocked your setup and are looking into reproducing and RCA the leftover core-fp-delete-shutdown-service instance. I will mark this ticket as in-progress.Hi Shanmugavel,I have tried an exhaustive variety of scenarios to try to reproduce this issue (leftover core-fp-delete-shutdown-service), and the only scenario that I can reproduce the artifact is to do a cleanup on Flat-L3 service without first executing delete. In this case, I can see one of the core-fp-delete-shutdown-service being cleaned up, but the second is leftover from race conditions.I would like to note, cleanup without first executing delete is an unsupported flow from T-SDN CFP viewpoint, so we will not pursue this scenario further. In all other scenarios like leftover zombie re-deploy, failed plan due to unreachable + cleanup, out of sync device, error recovery, multiple service update and delete, etc, there are no leftover artifacts.At this point, as I am out of ideas and from our discussion before, we heard that you made sure to delete Flat L3 before cleanup action. There does not seem to be any other way for me to RCA this issue without a proper reproduction flow as the logs have a variety of errors, and we cannot distinguish the reason for artifact from the logs itself as there is no outstanding error message.Unless if you can give me some general steps on what you suspect could have caused this artifact, I would not be able continue with RCA.The next step would be to kindly ask team to keep an eye out for this artifact issue. If encountered again in the future, it will be very important to note the steps taken prior to the artifact being generated. This way we can try to have a reproducible flow which will help with RCA.Thanks,Isaac Kim