Recently I was trying to restart my FMW stack when NodeManager stopped responding. Even after forcefully killing the NodeManager process (kill -9 <PID> ), the start nodemanager script was not able to start the process.
Upon further analysis i noticed that even after killing / stopping Nodemanager process the socket connection was in CLOSE_WAIT state. My NodeManager was running on Port 5556. You can check this by below command.
netstat -an | grep 5556
CLOSE_WAIT means your program is still running, and hasn’t closed the socket (and the kernel is waiting for it to do so).
So because of above issue even if I try to run a NodeManager program it hangs and does not starts successfully. So to resolve above issue Add
netstat to get the pid, and then kill it more forcefully (with
SIGKILL if needed).
netstat -anp | grep 5556
In my case it was one of the Managed Server process. Kill that Java process or shutdown it down and it should clearoff the
CLOSE_WAIT socket connections.
Please find here more NodeManager related articles.