At a cursory glance of the thread dumps i see the time spent in the server compiler (C2) is very very high. It might be worth exploring if that’s expected?
Alternatively if you just want a quick way to rule it out you could turn off tiered compilation, i had to google the option: -XX:-TieredCompilation
C1 vs C2 is best shot here IMO. Except a normal memory leak of course.
Problem with JVM is these compilation stages are difficult to monitor and tune. They require logging, parsing logs and trial and error approach.
OP definitely worth logging tiered compilation `-XX:+PrintCompilation` and `-XX:+PrintInlining` and `-XX:+LogCompilation`. When this turns out to be filled, emptied and filled again, try increasing `ReservedCodeCacheSize`
not having the tiered compilation would switch off c1, not c2. Tiered compilation is mostly an issue if the code generated remains c1 (the dumb compiler) and never promotes to c2... or if there is an OSR (on stack replacement issue - bug)
flip note: "java -XX:+PrintFlagsFinal -version", to see all available flags and their values, included the ones based on ergonomics.
Alternatively if you just want a quick way to rule it out you could turn off tiered compilation, i had to google the option: -XX:-TieredCompilation