NSO LSA - CFS crashed while trying to scale up the services- NSO 6.1 , TSDN 5.0.0 M5+ patch Eng build 2: Hi,Setup: 172.29.106.131 ( CFS) 172.29.106.132 ( RFS1)172.29.106.133 ( RFS2)NSO LSA installed with NSO 6.1 , TSDN 5.0.0 M5+ patch Eng build 26.1_230109.073843810.4384ecc7db9anso-6.1_230109-tsdn-5.0.0-M5From CW proxy API, with 5 concurrent users, trying to scale services in 5 pair of different endpoint. It is simple l2vpn p2p service. afte some around 450 services, CFS crashed.nso@nso-172-29-106-131:~$ ncs_cliFailed to connect to servernso@nso-172-29-106-131:~$CFS ( NSO provider) went unreachable in CW. hence all my services provisioning started failing.Collected logs and logs present at CFS device 172.29.106.131/home/nso/data_011923_103547_dirsetup is in issue state.please ping me for setup details., i can unicase the credentials. please use it for Read-only.Thanks,Arun.M.G
Hi Arun,Please try the test using this build :Also please enabled coredump on all your nodes CFS and RFSs .1. ulimit -S -c unlimited#enable core dump permanently, add following line in /etc/security/limits.conf to update system limit #to verify ulimit -c ,close all sessions and open new then new limit will be shown.  2. soft core unlimited#settings up coredump path3. sudo sysctl <del>w kernel.core_pattern=/tmp/core</del>%e-%s-%u-%g-%p-%torsudo sysctl -w kernel.core_pattern="/tmp/%e_core_dump.%p"(based on requirement )#You can make core dump file path changes permanent, by adding following line in /etc/sysctl.conf4. kernel.core_pattern="/tmp/core-%e-%s-%u-%g-%p-%t"orkernel.core_pattern="/tmp/%e_core_dump.%p"Significance of format specifier for coredump:<ol>%e The process or thread's comm value, which typically is the  same as the executable filename (without path prefix, and  truncated to a maximum of 15 characters)%i TID of thread that triggered core dump, as seen in the PID  namespace in which the thread resides.%p PID of dumped process, as seen in the PID namespace in which  the process resides.%s Number of signal causing dump.</ol>#To test crash to generate coredump5. kill -11 $$#Verify if coredumps get generated :ls -lart/tmp/ | grep core<del>rw</del>------ 1 admin1 admin1 2080768 Oct 6 11:18 bash_core_dump.21904#for ncs process crashkill -s SIGSEGV $(pgrep ncs.smp)#to verify ulimit -c ,close all sessions and open new then new limit will be shown.Before next run please enable and verify if coredump crash are getting generated .After crash please share coredump crash it will help platform team to analyze the failure.Thanks PriyankaWorking on CDETS