NSO not closing Restconf stream sessions: This is triaged from wider team.I see that each Restconf stream is opening a new session, and session remains opened even when the request is closed.Also double checked the NSO documentation, and dont see nothing mentioning any logout is needed for such requests, also no login cookie token involved here.This is easily reproduced from CURL:admin@ncs> show users | countCount: 45 linesnso5711@iorgad-exsi-vm1:~$ curl -kisu USER:PASS -X GET -H "Accept:text/event-stream" HTTP/1.1 200 OKDate: Tue, 26 Apr 2022 05:16:47 GMTCache-Control: private, no-cache, must-revalidate, proxy-revalidateContent-Type: text/event-streamTransfer-Encoding: chunkedPragma: no-cacheContent-Security-Policy: default-src 'self'; block-all-mixed-content; base-uri 'self'; frame-ancestors 'none';Strict-Transport-Security: max-age=15552000; includeSubDomainsX-Content-Type-Options: nosniffX-Frame-Options: DENYX-XSS-Protection: 1; mode=block^Cnso5711@iorgad-exsi-vm1:~$admin@ncs> show users | countCount: 46 lines
Hi Liran,The restconf command that you test is for stream notification subscription and it is designed to have a very long session timeout.If you test on a normal restconf command, NSO does terminate the session. Take authgroup restconf for exampleadmin@ncs> show users | countCount: 9 linescurl -is -u admin:Cisco123# -H "Accept: application/yang-data+xml" HTTP/1.1 200 OKDate: Tue, 26 Apr 2022 20:51:30 GMTLast-Modified: Sat, 23 Apr 2022 01:36:47 GMTCache-Control: private, no-cache, must-revalidate, proxy-revalidateEtag: "1650-677834-287817"Content-Type: application/yang-data+xmlTransfer-Encoding: chunkedPragma: no-cacheAccess-Control-Allow-Origin: *Content-Security-Policy: default-src 'self'; block-all-mixed-content; base-uri 'self'; frame-ancestors 'none';Strict-Transport-Security: max-age=15552000; includeSubDomainsX-Content-Type-Options: nosniffX-Frame-Options: DENYX-XSS-Protection: 1; mode=block<authgroups xmlns="http://tail-f.com/ns/ncs" xmlns:ncs="http://tail-f.com/ns/ncs"> <group>  <name>iosxe_authgroup</name>  <default-map>   <remote-name>admin</remote-name>   <remote-password>$9$gdKMS2xf8pAySVzPZFeL/JZwwo6KF59UeIbvjYi/6fM=</remote-password>   <remote-secondary-password>$9$2DS2LsLUzsfBSnKEf+NJ4bugTxihmHR94nDcuGFtX+A=</remote-secondary-password>  </default-map> </group> <group>  <name>iosxr_authgroup</name>  <default-map>   <remote-name>admin</remote-name>   <remote-password>$9$h2kB2zmyVMWY+ZCttz5hrk0erNZ0MKKwrubw0Pv04iU=</remote-password>   <remote-secondary-password>$9$DoPJbqjNbu3ZCNbnImg8F3UPZ5Oof7QcMUbtrOaG0Gk=</remote-secondary-password>  </default-map> </group> <group>  <name>vxr_iosxr_authgroup</name>  <default-map>   <remote-name>cisco</remote-name>   <remote-password>$9$gyy5jCiJ/zUjBygSj80ToSrb4XKWmRSZUyWb3xaxFug=</remote-password>   <remote-secondary-password>$9$XFRCcOBiq0dVnWwAlt7hImmJvJ3xrQToaaIEOya7h6Q=</remote-secondary-password>  </default-map> </group></authgroups>admin@ncs> show users | countCount: 9 linesadmin@ncs>Termination session also shows in audit.log<INFO> 26-Apr-2022::13:51:30.065 linh-nso-172-8 ncs: audit user: admin/182 assigned to groups: ncsadmin,ncsoper,admin,wheel<INFO> 26-Apr-2022::13:51:30.066 linh-nso-172-8 ncs: audit user: admin/182 created new session via rest from 10.155.144.11:65404 with http<INFO> 26-Apr-2022::13:51:30.066 linh-nso-172-8 ncs: audit user: admin/182 RESTCONF: request with http: GET /restconf/data/tailf-ncs:devices/authgroups HTTP/1.1<INFO> 26-Apr-2022::13:51:30.074 linh-nso-172-8 ncs: audit user: admin/182 RESTCONF: response with http: HTTP/1.1 /restconf/data/tailf-ncs:devices/authgroups 200 duration 196521 ms<INFO> 26-Apr-2022::13:51:30.074 linh-nso-172-8 ncs: audit user: admin/182 terminated session (reason: normal)So if content-Type: is event-stream, it has a very long session timeout. Again it is by design.Thanks,LinhHi Liran,I will close this ticket in next few days if I don't see any update on your side.Thanks,LinhThis is first time we are aware that NSO sessions remain open.This creates an issue with CNC, thus requesting here a solution from NSO team.To add some context, from past issues, it was observed that stream connections were idle and did not receive any notifications without any indication this is why daily stream refresh mechanism was added, by closing the request and open a new one. Without it, there were notifications loss. Without closing sessions of closed connections by NSO, it can lead to reaching sessions limit and possibly further implications. Please note, additional ticket was opened for NSO team for option to use keep-alive as default at CNC. If it was added, we may have consider reducing the need for daily refresh mechanism. The option is supported and there was a request to add it to CNC NSO installation official guidelines, but it was rejected by NSO team.This leads to a problem with all possible considered solutions:1.Daily refresh mechanism, current situation: sessions count keep increasing, it can lead to reaching sessions limit and possibly further implications.2.Remove daily refresh mechanism: notification loss situations observed.3.Remove daily refresh mechanism together with keep-alive: rejected by NSO team to document it as CNC installation guidelines, and was not included in testing.Adding:Even regardless of daily refresh mechanism, NSO sessions remain open is still an issue, as in multiple scenarios, like CNC cluster restart, VM restart, service restart, leader change, user remove/add of NSO provider, patches installation, version update and more, a new request is triggered, leading to leftover NSO session from previous request. Over time, NSO sessions count can keep increasing. Also got indications from CX that additional clients may open NSO notification stream requests, which can lead to additional leftover NSO sessions on each disconnection / request stop.This is getting escalated from the customer and project manager, and triaged also from wider team.The keep-alive ticket:HI Lin/Van and Luan/Sahithi,Do we need to open an RT for this?Thanks,NipaHi Liran,Thanks for sharing the previous JSD. I reverified the settings as described this ticket. Event-connection session timeout works as expect with 4.0 M5 also as long as keepalive flag is in ncs.conf. To reiterate configuration setting, this flag '<keepalive>true</keepalive>' must be configured in ncs.conf.if keepalive-timeout is provided, the session will be closed by this time valueif keepalive-timeout is not provided, the session will be close in 3600 sec.Again, this settings works fine in 4.0 M5 Thanks,LinhAs written, keep-alive is supported, but it was officially rejected by NSO team to document it as CNC installation guidelines, therefore not included in CNC and was not included in testing - therefore it is not the current solution. Team still blocked and waiting for a solution from NSO team for this issue.Hi Liran,ncs.conf file has many configurations and it is really part of the platform. It does not make sense for us/the CFP team to keep repeating the documentation.Thanks,NipaPlatform provide a keep-alive settings in ncs.conf to resolve this issue.Regards,Linh