too many open files - scale setup: We have LSA scale setup with 25K devices with 3 RFS, 1 CFS, with 160K VPN services. We are sending REST API to provision more VPN services. We have set the files limit as specified in the install instruction. However while creating services, we get this critical error message and NCS is stopped.ncs.log:<CRIT> 6-Jan-2023::03:23:23.995 scale-nso2-CFS-133 ncs: - Writing state file failed: /var/opt/ncs/state/global.data: too many open files (emfile)ncs.log:<CRIT> 6-Jan-2023::03:23:24.037 scale-nso2-CFS-133 ncs: - Writing state file failed: /var/opt/ncs/state/global.data: too many open files (emfile)For such a scale setup, what is the recommended no of files value that should be setup. We are facing this issue for the second time in 3 days.setup: 172.20.63.133cnctest/Ovatest123@
Hi Rama,Please update limits in  file  to :soft nproc 100000hard nproc 100000soft nofile 100000hard nofile 100000soft core unlimited- nofile 100000we have used above settings in our scale lab.ThanksPriyankaWe are hitting this issue more often. We are provisioning ELAN service with 5 end points. Even 100K limit is not enough. I suspect there could be memory leak.Can you check this setup. Out of file descriptors for accept() - process limit reached# lsof | wc -l209011# lsof | grep ncs | wc -l8522#<CRIT> 9-Jan-2023::12:48:49.653 scale-nso2-CFS-133 ncs: - Out of file descriptors for accept() - process limit reached<CRIT> 10-Jan-2023::05:33:29.469 scale-nso2-CFS-133 ncs: - Daemon ncs-dp-2730-ietf-l3vpn-nm:main:9 timed out<CRIT> 10-Jan-2023::06:17:47.598 scale-nso2-CFS-133 ncs: - Out of file descriptors for accept() - process limit reached<CRIT> 10-Jan-2023::06:17:47.599 scale-nso2-CFS-133 ncs: - Writing state file failed: /var/opt/ncs/state/global.data: too many open files (emfile)<CRIT> 10-Jan-2023::09:59:36.011 scale-nso2-CFS-133 ncs: - Out of file descriptors for accept() - process limit reached<CRIT> 10-Jan-2023::14:54:51.177 scale-nso2-CFS-133 ncs: - Out of file descriptors for accept() - process limit reached<CRIT> 11-Jan-2023::03:28:11.120 scale-nso2-CFS-133 ncs: - Out of file descriptors for accept() - process limit reached<CRIT> 11-Jan-2023::05:33:01.156 scale-nso2-CFS-133 ncs: - Out of file descriptors for accept() - process limit reachedHi Rama,I checked your scale NSO :Your max user process is 4096 :$ ulimit -a core file size     (blocks, -c) unlimiteddata seg size      (kbytes, -d) unlimitedscheduling priority       (-e) 0file size        (blocks, -f) unlimitedpending signals         (-i) 531115max locked memory    (kbytes, -l) 64max memory size     (kbytes, -m) unlimitedopen files           (-n) 200000pipe size      (512 bytes, -p) 8POSIX message queues   (bytes, -q) 819200real-time priority       (-r) 0stack size       (kbytes, -s) 8192cpu time        (seconds, -t) unlimited virtual memory     (kbytes, -v) unlimitedfile locks           (-x) unlimitedIn my NSO it is : admin1@NOS5-172-73:~$ admin1@NOS5-172-73:~$ ulimit -a core file size     (blocks, -c) 0data seg size      (kbytes, -d) unlimitedscheduling priority       (-e) 0file size        (blocks, -f) unlimitedpending signals         (-i) 1031601max locked memory    (kbytes, -l) 65536max memory size     (kbytes, -m) unlimitedopen files           (-n) 100000pipe size      (512 bytes, -p) 8POSIX message queues   (bytes, -q) 819200real-time priority       (-r) 0stack size       (kbytes, -s) 8192cpu time        (seconds, -t) unlimitedvirtual memory     (kbytes, -v) unlimitedfile locks           (-x) unlimitedPlease update that value as well.Do following steps : sudo vi /etc/sysctl.confkernel.pid_max = 4194303To effect changes to take place :sudo sysctl -pPlease try after above and let me know if you see still see issue.ThanksPriyankaHi Priyanka,Am seeing below error while trying these steps. Am i missing some thing ?$ sudo vi /etc/sysctl.conf$ sudo sysctl -pvm.swappiness = 10sysctl: cannot stat /proc/sys/vn/nfs_cache_pressure: No such file or directorykernel.pid_max = 4194303Thanks,Sreedhar.Hi Sreedhar ,I was successfully able to update the max processes limits on your NSO :$ $ ulimit -a core file size     (blocks, -c) 0data seg size      (kbytes, -d) unlimitedscheduling priority       (-e) 0file size        (blocks, -f) unlimitedpending signals         (-i) 531115max locked memory    (kbytes, -l) 64max memory size     (kbytes, -m) unlimitedopen files           (-n) 200000pipe size      (512 bytes, -p) 8POSIX message queues   (bytes, -q) 819200real-time priority       (-r) 0stack size       (kbytes, -s) 8192cpu time        (seconds, -t) unlimitedmax user processes       (-u) 200000virtual memory     (kbytes, -v) unlimitedfile locks           (-x) unlimited$ $ Please run your tests and let us know if you still see issue.ThanksPriyanka 