> I don't think the garbage collector is to blame. During the worst times, when the JVM almost maxed out 90 CPUs, the GC was only using 3s of CPU time per minute.
The graph linked only shows 3s in young gen GC - you should check the time spent in the old gen GC too.
You can get loads of time spent in GC even without running out of memory - running out of other resources like file handles or network connections will also trigger a full GC in the hopes of freeing some up.
If you've got 1000 file handles available, one process that uses 100 per second and doesn't leak, and another that uses 1 per hour and leaks it, after 900 hours everything will look fine, then after 1000 hours you'll run out - and the symptoms will manifest in the first process, not the second.
Admittedly, there's a text file of jstats output linked which doesn't show any full GC happening, so maybe this is nothing...
G1GC should be completely suitable, you can try with ZGC or Shenondoah. Both have some memory "penalty", each object takes a bit more memory, so with change you will see 5-15% increase memory usage. This would be normal.
G1GC should be fine, so enable GC logging, analyse them using:
For the GC analysing you will be looking for tenured generation, so must add this flag: -XX:+PrintTenuringDistribution
You would be looking for GC major and GC evacuation times. Major GCs are STW and take more time, so overall the goal is to eliminate them as much as possible.
I usually find it very important to have charts of heap usage. Overall heap allocated (all regions) vs complete heap size. The same for non-heap area.
The graph linked only shows 3s in young gen GC - you should check the time spent in the old gen GC too.
You can get loads of time spent in GC even without running out of memory - running out of other resources like file handles or network connections will also trigger a full GC in the hopes of freeing some up.
If you've got 1000 file handles available, one process that uses 100 per second and doesn't leak, and another that uses 1 per hour and leaks it, after 900 hours everything will look fine, then after 1000 hours you'll run out - and the symptoms will manifest in the first process, not the second.
Admittedly, there's a text file of jstats output linked which doesn't show any full GC happening, so maybe this is nothing...