One of the 2nd clusters in NSO LSA HA with L3 network shows RFS nodes as down: Issue: One of the 2nd clusters in NSO LSA HA with L3 network shows RFS nodes as downSteps:1. I have setup LSA on two pairs of VMs - 1 cluster with 1 CFS, 2 RFS and another cluster with 1 CFS, 2 RFS.2. 1st cluster came up fine and status is UP for RFS nodes in "show cluster".3. 2nd cluster displays the status as down for RFS nodes.4. I checked CFS is able to connect to RFS nodes via port 2022. sync-from and ssh fetch-host keys returned true as well.5. admin@ncs> show cluster              LAST  RECEIVEDNAME     NAME STATUS EVENT NOTIFICATIONS--------------------------------------------------nso-cw2-rfs1nso-cw2-rfs2                      LOCAL REMOTEREMOTE NODE  ADDRESS    PORT CHANNELS USER  USER  STATUS TRACE-----------------------------------------------------------------------------nso-cw2-rfs2 192.168.6.63 2022 -     admin admin  down  disablednso-cw2-rfs1 192.168.6.62 2022 -     admin admin  down  disabledJanaki and Linh helped to check the setup and everything seems to correct with configurations. Hence, need help to resolve this issue so that NSO LSA with L3 can with HA mode.
Assigning to Van to please check the cluster.Hi Gomathi,After I enable device notificationset cluster device-notifications enabledThe cluster is upadmin@ncs% run show cluster                                    RECEIVEDNAME     NAME         STATUS LAST EVENT         NOTIFICATIONS--------------------------------------------------------------------------------------nso-cw2-rfs1 device-notifications up   0000-01-01T00:00:00-00:00 0nso-cw2-rfs2 device-notifications up   0000-01-01T00:00:00-00:00 0                      LOCAL REMOTEREMOTE NODE  ADDRESS    PORT CHANNELS USER  USER  STATUS TRACE-----------------------------------------------------------------------------nso-cw2-rfs1 192.168.6.62 2022 -     admin admin  up   disablednso-cw2-rfs2 192.168.6.63 2022 -     admin admin  up   disabledRegards,LinhHi Linh,After enabling the device notification on cw2-cfs1, show cluster shows rfs nodes as up which is different setting from tsdn wiki.I enabled back HA on both cw1-cfs1 and cw2-cfs2 below is the output:cw1-cfs1:admin@ncs> show cluster                                    RECEIVEDNAME     NAME         STATUS LAST EVENT         NOTIFICATIONS--------------------------------------------------------------------------------------nso-cw1-rfs1 device-notifications up   0000-01-01T00:00:00-00:00 0       ncs-events      up   0000-01-01T00:00:00-00:00 0nso-cw1-rfs2 device-notifications up   0000-01-01T00:00:00-00:00 0       ncs-events      up   0000-01-01T00:00:00-00:00 0                      LOCAL REMOTEREMOTE NODE  ADDRESS    PORT CHANNELS USER  USER  STATUS TRACE-----------------------------------------------------------------------------nso-cw1-rfs2 192.168.5.32 2022 -     admin admin  up   disablednso-cw1-rfs1 192.168.5.31 2022 -     admin admin  up   disabledcw2-cfs1:dmin@ncs> show cluster                                    RECEIVEDNAME     NAME         STATUS LAST EVENT         NOTIFICATIONS--------------------------------------------------------------------------------------nso-cw1-rfs1 device-notifications -    0000-01-01T00:00:00-00:00 -       ncs-events      -    0000-01-01T00:00:00-00:00 -nso-cw1-rfs2 device-notifications -    0000-01-01T00:00:00-00:00 -       ncs-events      -    0000-01-01T00:00:00-00:00 -Is the above output on cw2-cfs1 expected?Before enabling HA, cw2-cfs1 showed as below:admin@ncs% run show cluster                                    RECEIVEDNAME     NAME         STATUS LAST EVENT         NOTIFICATIONS--------------------------------------------------------------------------------------nso-cw2-rfs1 device-notifications up   0000-01-01T00:00:00-00:00 0nso-cw2-rfs2 device-notifications up   0000-01-01T00:00:00-00:00 0                      LOCAL REMOTEREMOTE NODE  ADDRESS    PORT CHANNELS USER  USER  STATUS TRACE-----------------------------------------------------------------------------nso-cw2-rfs1 192.168.6.62 2022 -     admin admin  up   disablednso-cw2-rfs2 192.168.6.63 2022 -     admin admin  up   disabledadmin@ncs% exitadmin@ncs> request high-availability enableresult enabledI enabled HA on all NSO LSA nodes, except, cw1-rfs1 and cw2-rfs1 all other nodes are returning expected hcc and ha configs in L3 mode. cw2-rfs1 gives below misconfiguration error:admin@ncs> show high-availability statushigh-availability status mode nonehigh-availability status assigned-role secondaryhigh-availability status be-secondary-result "error (69) - configuration mismatch"high-availability status primary-id masterhigh-availability status read-only-mode falseI double checked all configs they seem to be configured correctly. CSR router is able to ping corresponding VIP IP on each related pair of cfs and rfs nodes.Can you check and confirm on above secondary node HA config error on cw2-rfs1.Thanks,Gomathi.Hi Gomathi,Can you check bgp router connection and configuration?nso-cw2-rfs1 (192.168.5.31)admin@ncs% show high-availability token $9$i61J6fDpauBiYapug5zczA4QRcuXHc1A/4+s7Lde8oI=;ha-node master { address 192.168.5.31; nominal-role primary;}ha-node standby { address 192.168.6.62; nominal-role secondary; failover-primary true;}admin@nso-cw1-rfs1:~$ Peer AS Up/Down State |#Received Accepted 65137 3d 11:02:52 Establ | 3 3-----------------------nso-cw2-rfs1 (192.168.6.62)admin@ncs% show high-availability token $9$hMmeGxncwdpTtiNNsNpU/w42hZ3BjEztZawyaVoTBug=;ha-node master { address 192.168.5.31; nominal-role primary;}ha-node standby { address 192.168.6.62; nominal-role secondary; failover-primary true;}admin@nso-cw2-rfs1:~$ Peer AS Up/Down State |#Received Accepted 65137 00:21:34 Establ | 4 4Hi Van,Below is the bgp configs on cw2-rfs1 - csr:router bgp 65137 template peer-policy ha-peer route-reflector-client send-community both exit-peer-policy ! template peer-session rfs1-peer remote-as 65138 exit-peer-session ! template peer-session rfs2-peer remote-as 65139 exit-peer-session ! bgp log-neighbor-changes timers bgp 1 10 10 neighbor 192.168.5.31 remote-as 65138 neighbor 192.168.6.62 remote-as 65139 ! address-family ipv4 redistribute connected neighbor 192.168.5.31 activate neighbor 192.168.5.31 send-community both neighbor 192.168.6.62 activate neighbor 192.168.6.62 send-community extended exit-address-familycsr-cw1-cw2-rfs1#ping 192.168.5.31Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 192.168.5.31, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 mscsr-cw1-cw2-rfs1#ping 12.12.12.12Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 12.12.12.12, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 mscsr-cw1-cw2-rfs1#ping 192.168.6.62Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 192.168.6.62, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 mscsr-cw1-cw2-rfs1#ping 13.13.13.13 >>>> VIP of cw1-rfs2/cw2-rfs2 not pingable from csr.Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 13.13.13.13, timeout is 2 seconds:.....Success rate is 0 percent (0/5)csr-cw1-cw2-rfs1#csr-cw1-cw2-rfs1#ping 11.11.11.11Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 11.11.11.11, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 msThanks,Gomathi.Hi Gomathi,Found duplicate stream notification entries in ncs.conf,"dispatch-map-update" in secondary node and "service-aa-changes" in primary node. HA is up after removing the extra config and restarting...admin@ncs> show high-availability high-availability enabledhigh-availability status mode primaryhigh-availability status current-id masterhigh-availability status assigned-role primaryhigh-availability status read-only-mode falseID ADDRESS -----------------------standby 192.168.6.62 Hi Van,Thanks for looking into the issue and fixing it!There was no error during ncs packages reload operation after ncs.conf file update on cw2-rfs2 node. Hence, not sure how the duplicate entries got added in ncs.conf file.I remember during ansible script execution the duplicate entries error was thrown during ncs packages reload. I removed them and restarted ncs with manual installation and it worked fine.On this VM, show hcc was fine and only HA had the issue. As the error was generic and ncs restart also worked didn't have clue that ncs.conf file had duplicate entries.It would help user if error was specific during HA config issue.Thanks,Gomathi.Hi Gomathi,The config syntax was fine as NSO can handle the duplications at start up but not forgiving when doing config comparison for HA. Best practice for setting up HA is replicating exact application and config on both nodes (NSO version, package versions, ncs.conf, HA/hcc config).It's a little vauge but "error (69) - configuration mismatch" should have pointed us to check HA/hcc config, ncs.conf.Regards,Van