Zombies showed up post policy delete via UI: During regression run on build combo (CW#38, CAT#27), we encountered zombies post deletion of a policy via UI.Please find this regression log file for reference:Steps :1. TSDN policy testcases executed via CNC API ( task#4 through task #28, includes SR,RSVP,L2VPN,L3VPN,ODN)2. Execute the same TSDN policy testcases via CNC UI ( task@29 through task#53)3. Failure encountered in task #45 :  steps: 3.1 Create l2vpn_untagged_srpolicy_R1_R10_dynamic_igp policy via UI :  Values provided are :For SR -> ([('policy', [OrderedDict([('Name', 'srte_c_9066_ep_6.1.1.10'), ('head-end', OrderedDict()), ('tail-end', '6.1.1.10'), ('color', '9066'), ('path', [OrderedDict()])]), OrderedDict([('Name', 'srte_c_9066_ep_6.1.1.1'), ('head-end', OrderedDict()), ('tail-end', '6.1.1.1'), ('color', '9066'), ('path', [OrderedDict()])])])])For L2vpn -> ([('ietf-l2vpn-ntw:vpn-service', [OrderedDict([('Vpn-id', 'L2vpn_tldp_R1_R10_untagged_sr'), ('vpn-svc-type', 'vpn-common:t-ldp'), ('vpn-nodes', OrderedDict([('vpn-node', [OrderedDict([('vpn-node-id', 'R1'), ('ne-id', 'R1'), ('te-service-mapping', OrderedDict([('te-mapping', OrderedDict([('sr-policy', OrderedDict())]))])), ('signaling-options', [OrderedDict([('type', 'vpn-common:t-ldp'), ('t-ldp-pwe', OrderedDict([('ac-pw-list', [OrderedDict()])]))])]), ('vpn-network-accesses', OrderedDict([('vpn-network-access', [OrderedDict([('id', '3334'), ('Interface-mtu', '9000'), ('connection', OrderedDict([('encapsulation-type', 'vpn-common:untagged-int'), ('untagged-interface', OrderedDict([('l2-access-type', 'vpn-common:untagged-int'), ('untagged', OrderedDict())]))]))])])]))]), OrderedDict([('vpn-node-id', 'R10'), ('ne-id', 'R10'), ('te-service-mapping', OrderedDict([('te-mapping', OrderedDict([('sr-policy', OrderedDict())]))])), ('signaling-options', [OrderedDict([('type', 'vpn-common:t-ldp'), ('t-ldp-pwe', OrderedDict([('ac-pw-list', [OrderedDict()])]))])]), ('vpn-network-accesses', OrderedDict([('vpn-network-access', [OrderedDict([('id', '3334'), ('Interface-mtu', '9000'), ('connection', OrderedDict([('encapsulation-type', 'vpn-common:untagged-int'), ('untagged-interface', OrderedDict([('l2-access-type', 'vpn-common:untagged-int'), ('untagged', OrderedDict())]))]))])])]))])])]))])])])3.2 Create policy successful, verified the policy status is up via UI.3.3. Update color value in the policy.3.4 Update policy successful and verified updated data and status from UI.3.5 Delete the policies from UI ( both sr and l2vpn)3.6 Post delete, verified in UI both the SR and L2VPN services are not found.3.7 Check for zombies and zombies got found for both :show zombieszombies service /flat-L2vpn-internal-local-site-service delete-path  /flat-L2vpn-internal-local-site-service pending-delete                                                      POST      BACK                                              ACTIONTYPE NAME TRACK GOAL STATE                           STATUS    WHEN ref STATUS--------------------------------------------------------------------------------------------------------------------self self true  -   init                            not-reached -   -  -             cisco-flat-L2vpn-fp-local-site-nano-services:config-apply not-reached -   -  -             ready                           not-reached -   -  -zombies service /flat-L2vpn-internal-remote-site-service delete-path /flat-L2vpn-internal-remote-site-service                                                            POST      BACK                                                    ACTIONTYPE NAME TRACK GOAL STATE                            STATUS  WHEN         ref STATUS--------------------------------------------------------------------------------------------------------------------------------self self true  -   init                            reached 2022-07-26T21:03:24 -  -             cisco-flat-L2vpn-fp-remote-site-nano-services:config-apply reached 2022-07-26T21:03:24 -  -             ready                            reached 2022-07-26T21:07:11 -  -zombies service /sr-te/cisco-sr-te-cfp-sr-policies-internal:policies/policy delete-path /sr-te/cisco-sr-te-cfp-sr-policies-internal:policies/policy                                                          POST      BACK                                                  ACTIONTYPE NAME TRACK GOAL STATE                          STATUS  WHEN         ref STATUS----------------------------------------------------------------------------------------------------------------------------self self true  -   init                          reached 2022-07-26T20:59:40 -  -             cisco-sr-te-cfp-sr-policies-nano-services:config-apply reached 2022-07-26T20:59:40 -  -             ready                          reached 2022-07-26T21:05:03 -  -Please refer to atleast the log of task #45, all steps are clearly mentioned including payload and everything. Let me know if any other info is required.Please find the NSO showtech collected : /auto/ncahi/Users/mesarma/cwork_proj/nso_st_27july/data_072722_062505.tgzConcern is : Even after successful deletion of the policy via CNC UI, why should zombies even get created.And if there is this possibility to keep getting this in future as well, then please suggest some possible Actions that we can automate in the regression run itself to make sure this kind of issues do not occur.
Hi I saw this error in the Jenkins logrequest devices device R10 compare-configinfo R10: Device is locked in a commit operation by session 17368admin@ncs>503: Zombies found on devices hence aborting the script.504: Failed reason: check_zombies failedIt looks like your automation code perform an deleting action during the time that device is locked from the previous command. That is why deleting is failed. This is not NSO issue, it is an user operation issue. This issue would happen intermittently since device response time is depending on network connectivity. If the network has long delay and would not response in time as expected in automation logic then another command is kicked in when the previous command is not finish yet.Again, it is not NSO issue and you need to adjust your automation code to deal with the situation like this.Regards,LinhHi Linh,So could you please suggest any solution to handle such intermittent unpredictable issues in our regression automation ?Please note: We are already putting good amount of sleep before and after performing any operation to avoid any conflict.Thanks,MeenakshiHi Meenakshi,Few 2cent suggestions:1. From your jenkins log, I saw most of time you give 30s sleep time. This value is not sufficient for create/deleting a service. In my automation code, I gave it 40 seconds.2. It seems that you automation codes does not handle zombies (I could be wrong by looking at Jenkins log) because most of test cases are failed after zombies exist. You should clean up the zombies and bring back devices in sync. So the next test cases could go on ...Regards,Linhit seem automation code issue. The operation step in too soon while others still is in progress