Ncs cli stucked while fetching ssh keys for devices: 1. ncs cli got stucked while executing fetch keys for any device. (Devices are added from DLM)2. Observing huge number of requests received from 2 CW nodes {1 worker and 1 hybrid where dlm pods are running}Attached nso_response.rtf file for more data Impact on CW :REST delete call to NSO is timing out and connect to devices is failing.
 Hello, please provide us your setup information so we can troubleshoot issue.Unicasted setup details in Webex to Janaki Duggirala.Note : We are running daily regression automation suites with this setup. Current run is in progress. we can not stop any regression runs and expecting no disruption to this nso.If any config changes needs be done, pl inform us.Customer has provided setup issue. Proceeding to request meeting to troubleshoot further since Issue is not with NSO.NSO CLI is able to connect to devices successfully. Fetch host keys on all devices is successful. NSO API works correctly as well.Forwaded mail conversation from crosswork team to Janakiaddition to below scenario seeing following msg on nSO config mode randomlynso https (rest from 2001:420:10e:200e:c000::1711) on since 2023-08-22 11:45:04 terminal modenso https (rest from 2001:420:10e:200e:c000::1711) on since 2023-08-22 11:44:33 terminal modenso https (rest from 2001:420:10e:200e:c000::1711) on since 2023-08-22 11:40:04 terminal modeOpenfiles must be min 64Konly 6 CPU's &#8211; min needed is 8Needed 64 GB RAM (also recommended 128 GB)From meeting, requesting customer to update setup to above minimum system specifications.Reinstalled NSO and observing same issue1. Upgraded Java versionadmin@eagle-nso:~$ java --versionopenjdk 17.0.7 2023-04-18OpenJDK Runtime Environment (build 17.0.7+7-Ubuntu-0ubuntu122.10.2)OpenJDK 64-Bit Server VM (build 17.0.7+7-Ubuntu-0ubuntu122.10.2, mixed mode, sharing)2. Changed file descriptoradmin@eagle-nso:~$ ulimit -areal-time non-blocking time (microseconds, -R) unlimitedcore file size       (blocks, -c) 0data seg size        (kbytes, -d) unlimitedscheduling priority         (-e) 0file size          (blocks, -f) unlimitedpending signals           (-i) 63470max locked memory      (kbytes, -l) 2046464max memory size       (kbytes, -m) unlimitedopen files             (-n) 65535pipe size        (512 bytes, -p) 8POSIX message queues     (bytes, -q) 819200real-time priority         (-r) 0stack size         (kbytes, -s) 8192cpu time          (seconds, -t) unlimitedmax user processes         (-u) 63470virtual memory       (kbytes, -v) unlimitedfile locks             (-x) unlimitedadmin@eagle-nso:~$3. Increased CPU : System resources: 8vCPUs, 16Gb Mem, 100Gb HDDadmin@eagle-nso:~$ lscpuArchitecture:      x86_64 CPU op-mode(s):    32-bit, 64-bit Address sizes:     45 bits physical, 48 bits virtual Byte Order:      Little EndianCPU(s):         8 On-line CPU(s) list:  0-7Vendor ID:        GenuineIntel Model name:      Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz  CPU family:     6  Model:        63  Thread(s) per core: 1  Core(s) per socket: 1  Socket(s):      8  Stepping:      2  BogoMIPS:      4588.94  Flags:        fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqd             q ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsave             opt arat md_clear flush_l1d arch_capabilitiesVirtualization features:  Hypervisor vendor:   VMware Virtualization type:  fullCaches (sum of all):    L1d:          256 KiB (8 instances) L1i:          256 KiB (8 instances) L2:          2 MiB (8 instances) L3:          240 MiB (8 instances)NUMA:           NUMA node(s):     1 NUMA node0 CPU(s):   0-7Vulnerabilities:      Itlb multihit:     KVM: Mitigation: VMX unsupported L1tf:         Mitigation; PTE Inversion Mds:          Mitigation; Clear CPU buffers; SMT Host state unknown Meltdown:       Mitigation; PTI Mmio stale data:    Mitigation; Clear CPU buffers; SMT Host state unknown Retbleed:       Mitigation; IBRS Spec store bypass:   Mitigation; Speculative Store Bypass disabled via prctl Spectre v1:      Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2:      Mitigation; IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected Srbds:         Not affected Tsx async abort:    Not affectedadmin@eagle-nso:~$ Thank you for posting the upgrade information. Please let us know the result for fetch host keys (original issue).Issue is still seen with 6.1.2CDET - CSCwh32161 for tracking this. nso/tm-tc dev team is working on this issue.Moved to platform team. RT filed