1) Probably good to relook the Linux thread stack size. Use “ulimit -s <size in kb>” for setting slightly bigger value and observe the test case.2) Check also your ulimit on virtual size, i.e. ulimit -v. Increasing this limit should let you create more threads with the same stack size. 3) Also, lowering -Xmx can help by leaving more memory for thread stacks.

All these are like tuning options. Still would be good to observe the jstack once again in long running instance and will see the possibilities of leaks as Camille suggested?