When OOMKilled Hits a Java App: What I Learned Fixing Our Kubernetes Memory Mess

It was a Tuesday afternoon in November 2024, and I was about two hours deep into matching purchase orders when my phone started buzzing. Not the usual “vendor invoice is late” buzz—the kind that means something technical is on fire. Our ops lead Tom had looped me into a thread with just two words.

“Java eating memory.”

That shouldn't have been my problem, technically. I’m the admin buyer—the person who handles office supplies, vendor contracts, and the occasional corporate event. I don’t write deployment YAML. But I’d been the one who approved the budget for our Kubernetes migration three years earlier, and I’d learned the hard way that in a 40-person company, “procurement” sometimes means “whoever can find the right vendor and figure out what the hell is going on.” (Still salty about that $2,400 rejected expense from a supplier who couldn't produce a proper invoice, by the way. Different story.)

So I pulled up the cluster dashboard Tom had shared. Our Java microservices—the ones running the order processing pipeline I actually depend on for my vendor payments—were getting OOMKilled. Not occasionally. Repeatedly. Like, four or five times a day.

Where It Started: The Background

Let me back up a bit.

We run about 18 microservices across a three-node Kubernetes cluster. Nothing fancy, but it handles our internal procurement portal, vendor onboarding, and the expense approval workflows. When we migrated to K8s in late 2022, the Java apps were my main concern. Not because I know Java deeply—I don’t—but because I remembered our legacy on-prem system crashing every time the finance team ran month-end reports. Java and memory were not a happy combination in my experience.

Our initial setup used the classic approach: each Java pod had a requests limit of 512Mi and a limits cap of 1Gi. The thinking was simple: keep apps lightweight, pack more pods per node, save on cloud costs. I approved that sizing because Tom said it was standard. In his defense, it was standard—for smaller JVM workloads. But we made an assumption that cost us days of debugging.

(Note to self: never approve infrastructure sizes based on “standard” without asking “standard for what?”)

The First Symptoms: Slow and Then Sudden

It started with warnings in early October. Memory usage graphs looked like a sawtooth—spiking to the limit, dropping, then climbing again. Tom initially attributed it to a memory leak in our notification service. We roped in a freelance Java consultant, who ran heap dumps and found nothing conclusive. Classic leak hunt, classic dead end.

Then the OOMKilled events started. The first one I really noticed was during our quarterly vendor review. I was in a meeting with a supplier (one of our better ones, actually—but their portal kept timing out, and the blame was “confusingly on our side”), and Tom pinged me: “Order service killed. Restarting.”

It didn’t crash our whole operation. Kubernetes did its job, restarted the pod, and we moved on. But the pattern worried me. Unlike the “one small vendor crashing a full expense report” problem, these restarts happened silently in the background. If I hadn’t been in that meeting, I’d have never known.

I don’t have hard data on how often this happens across “typical” K8s deployments, but based on our experience, my sense is that silent OOMKilled restarts are way more common than most teams admit.

The real moment that got my attention: a batch reconciliation job—the one that matches our POs to vendor invoices every night—kept getting OOMKilled around 2:00 AM. For three nights in a row, the job would die. The pod would restart, try again, and sometimes finish on the third attempt.

But “sometimes” isn’t a process. And for me, as the person who needs accurate reconciliations to file expense reports, “sometimes” was a deal-breaker.

Finding the Real Culprit (and a Red Flag I Missed)

Here’s where I have to give Tom credit. He didn’t just reallocate memory and hope. He made a list of every OOMKilled pod over a two-week period and looked for patterns. Found one:

• All affected services were running the same Java runtime version—Eclipse Temurin JDK 17.0.8.
• All had the same container memory limit (1Gi).
• None of them had explicit JVM heap settings configured in the Dockerfile or deployment manifest.

That last one was the red flag. I’d seen plenty of vendor documentation in my procurement world about Java apps needing -Xmx settings, but I’d always assumed the default JVM behavior would just work. It turns out—and this was the moment the penny dropped—that without explicit heap configuration, the JVM sizes its heap based on the host node’s total memory, not the container limit.

That’s right. Kubernetes says “you get 1Gi,” but the JVM sees a node with 32GB and says “I’ll take a quarter of that, thanks.” The container hits the cgroup limit, the OS allocates more memory than the pod is allowed, and bam—OOMKilled. Not because the app had a memory leak, but because the JVM genuinely had no idea it was running inside a constrained container.

I’ll be honest: I should’ve caught this logic sooner. The JVM isn’t magic. How would it know the boundary if we didn’t tell it? But we all assumed “modern Java is container-aware.” In some ways it was—it detected cgroups and used MaxRAMPercentage in newer versions. But that graceful behavior relies on explicit configuration or newer JDK defaults. Our images were built with an older template that, for the love of me, still left heap size to the JVM’s startup heuristics, which look at the host.

The Fix: It Wasn’t Just “Add More Memory”

You’d think the answer was simple: raise the memory limit to 2Gi, let the JVM use more, and call it a day.

We didn’t do that. Tom pushed back, and I’m glad he did. We had 18 services, and doubling limits for all of them would’ve meant either a bigger cloud bill or fewer replicas—less resilience. Instead, we did a combination of things.

First, we set explicit -XX:MaxRAMPercentage=75 in the JVM options, so every Java process would use at most 75% of the container limit (leaving headroom for off-heap, threads, and JVM overhead). That alone made a huge difference, because it aligned the JVM’s math with Kubernetes’.

Second, we standardized the container memory limits. Our smaller services got 1Gi. The batch job—the one that really needed to chew through invoice files—got 2Gi. But instead of just bumping it blindly, we also tuned the batch job to process invoices in smaller chunks, so it didn’t need to load the entire month’s payment history into heap at once.

Third—and this was the process lesson—we added a memory-setting review to our deployment checklist. No Java service gets deployed without a documented -Xmx or -XX:MaxRAMPercentage value, and the tuning gets reviewed whenever we change a pod’s limits.

As of late January 2025, we’ve seen zero OOMKilled restarts on our Java services across three rolling deployments. The batch reconciliation has run clean for six weeks. Our finance team can attest to that—they’ve stopped emailing me about missing expense matches. (Small win, but honestly, a huge one for my Sunday nights.)

What I Wish I’d Known Sooner

I’m not a Java expert by trade. But after this experience, I’ve got a few hard-won takeaways that I think are worth passing on—especially for teams that, like ours, have a lot of “best practice” templates but not enough scrutiny:

1. The JVM doesn’t read your Kubernetes limit unless you tell it to. If you’re running Java in containers without setting heap percentages, you’re playing roulette. Always use -XX:MaxRAMPercentage (or at least -Xmx). Don’t assume “modern Java” will save you.
2. OOMKilled doesn’t always mean memory leak. It can mean memory math mismatch. After we fixed our settings, the “leak” we chased for two weeks disappeared. It was misconfiguration, not code rot.
3. Small fixes are better than big limits. Our instinct was to throw more memory at it. But setting the heap to a sane percentage of the container cap fixed the problem without inflating our cloud bill or reducing pod density.
4. Track the pattern, not just the incident. The pattern of which services died, when, and with which JVM settings, was what led to the fix. A single OOMKilled is noise; five a day is a signal.

I’m not going to pretend I wrote the deployment manifests myself—Tom did the technical work. But I owned the budget, the process, and the follow-up, which is its own kind of accountability. Being the person who signs off on infrastructure projects means I should understand the failure modes better than “someone else will handle it.” I’ll admit, I didn’t know what OOMKilled meant last October. Now I do.

This configuration was accurate as of January 2025. Java and Kubernetes both evolve fast, so verify the current JVM container-awareness defaults—JDK 21 and newer have some helpful improvements, but honestly, the safest approach is still to be explicit about your heap sizing and never leave it to guesswork.

Bottom line: if your Java app in Kubernetes is getting OOMKilled and you haven’t checked -XX:MaxRAMPercentage or -Xmx, start there. It might not always be the issue, but it’s the first thing I’d look at now. And if you have a colleague who keeps saying “it’s a leak, let’s just add memory,” send them this story. The leak might be in the configuration, not the code.