[TM-TC] Post cleanup , unable to create new plan again.: In tm-tc fp, cleanup is successfully deleting the config that is intended to be removed. But after retrying creation and cleanup multiple times, the cleanup is throwing below alarm on CLI admin@ncs# *** ALARM service-activation-failure: Operation failed because: Not able to retrieve diffset for zombie nano service : '/cisco-tm-tc-fp-internal:tm-tc'After throwing this alarm on CLI, the cleanup returns success but new plan creation fails. This happens after trying for few times and creating new plan is blocked after we hit this. 
Hi Raja, thank you for the information.Seems like there are some leftover artifacts from cleanup action which may be interfering with your service creation.Leftover artifacts from cleanup action are typically the result of race conditions. If you see the TM TC cleanup action, we do a delete operation and afterwards force-backtrack plan components. These two operations can result in a race condition which can cause certain artifacts to regenerate during cleanup. Otherwise there could be leftover artifacts that can result in unexpected behavior.To avoid this, for TSDN, we mandate users to first attempt to delete service prior to using cleanup action as a last resort option.Could you confirm if service was deleted prior to cleanup?Thanks,Isaac KimHi Raja,After some discussion, we have decided that the best solution would be to use NSO's native delete operation.As Sahithi corrected me last call, NSO delete is not a synchronous operation. We must keep in mind, cleanup action also has a async delete operation at the beginning of the action. This is what is causing the race conditions and artifacts you mentioned in this ticket. If we do not delete the TMTC service prior to force backtrack, service configuration will remain as leftover artifacts. In addition, in my testing, I have seen TMTC cleanup action leave artifacts when executed on created service with out of sync device. So a delete + converged plan prior to cleanup is a . Otherwise behavior is undefined. : plan must be in backtrack mode and fully converged (which means no active operations on plan are running).To meet this pre-condition, we can do a NSO delete + commit. Once we commit, service config is deleted and zombies will spawn which will backtrack the plan asynchronously.If there are no errors during backtrack, the plan should eventually be cleaned up. In this case, cleanup action is not required.In case there are errors (such as connection refused, out of sync), the plan will fail and remain in backtrack mode. This meets the pre-condition stated above and cleanup action should be executed next.In order to ensure that plan has fully converged, we will need to introduce some kind of external poller mechanism to check plan state. Here is an example flow:<ol>Delete Service ACommitExternally Poll NSO for Service A plan state<ol>Service A plan has no failure during backtrack<ol>Plan is deleted<ol>DONE</ol></ol>Service A plan has failure during backtrack<ol>Plan is stuck in failed state and in backtrack mode<ol>Execute cleanup action<ol>DONE</ol></ol></ol></ol></ol>Based on this flow, we can see that the main condition for the poller is to determine if the plan has failed or not. If the plan does not fail during backtrack, the plan should not exist. If the plan does fail during backtrack, the plan will fail and converge. If the plan is currently backtracking, plan should exist and there should be no plan failures.So to determine if we need to execute cleanup action, all we have to do is check if plan has failed or not.To check plan status, you can check two items:<tm tc plan> -> plan -> error-info -> message<tm tc plan> -> plan -> failedIn most cases, there will be an error-info message on plan for failed delete. But in edge cases, there are times when message is empty.I hope this is a viable solution to this problem. Please let us know what you think.Thanks,Isaac KimThanks for looking into the issue. I see, conclusion is that there is no way we can achieve synchronous deletion of plan which is a must as per design of collection service which triggers TM-TC FP. I have concern in polling externally to check if plan deletion is successful or not. This can work in case of standalone installations but there is no plan created on CFS in case of LSA deployments. Given this, we want to do all the changes with in tm-tc fp instead of handling it on client side. Is there any polling API available in ncs api that we can use ? That would be helpful to achieve whatever you are suggesting but within fp code. Hi Moksha,As discussed on call, I have tried to find a way to stream notifications from RFS node to CFS node stream. But I do not think that is possible. We can check the received RFS service-state-change notifications on the CFS node through NSO device tree, but this will require an external poller.To utilize notification stream, you will need to interface with notification stream on the NSO node where the service is located. In the case of LSA, the RFS service-state-change notification stream should be used. A custom kicker can be utilized to watch the received notifications for each RFS node and execute a custom action which can copy over the notifications to CFS notification stream. But this will be more complicated.Thanks,Isaac KimA solution was proposed to submitter and abosorbed.