"How did you go bankrupt? Two ways. Gradually, then suddenly." Hemingway understood that calamity is either under-observed or noticed when it's already too late. Servers are the same. The load average had been creeping up for an hour, higher than usual but not in the red, the kind of number that's easy to misread as good news. Maybe we've got traffic. Break for lunch, no second thought. By the time I sat down with the mug of afternoon brown, htop was a horror show. CPU pinned at 100%. Load average at 14 on a 2-core box. Something was eating it, and it wasn't me.
The Old Man and the C
htop told me what I already knew, then told me something I didn't. There was a process called fghgf chewing through a CPU core. fghgf. Deliberately named to make any developer think they'd shipped a careless local test, and that's what I would've thought if not for the config file parked next door in /tmp invoking a small grep loop scanning for rival miners, which is how I learned that the freeloader squatting in my tmp directory had competitors and was worried about them. This wasn't a symptom. This was an infection.
I killed it. Twelve seconds later something else came up on a different port with a different keyboard-mash for a name. Then another. I'd just entered a game of Hack-A-Mole.
The hunt
You stop swatting and start tracing. Where did it get in. What is it touching. What's it talking to and on what schedule. Cryptominers are not stupid. They have cron jobs, systemd units, and hidden directories in places that look enough like real system paths to slide past a tired eye. Persistence is the whole point: the miner you can see is the decoy for the three you can't.
The entry point turned out to be a self-hosted Next.js app running an outdated version, vulnerable to CVE-2025-29927, the middleware authorization bypass that Vercel disclosed in March 2025. CVSS 9.1. Critical. The flaw is, frankly, the kind of thing you'd be embarrassed to ship: Next.js used a header called x-middleware-subrequest internally to prevent infinite middleware loops, and then trusted that header without verifying its origin. Set it on an inbound request from the open internet and the framework cheerfully waves you through, past every auth check the developer wrote.
If you want the gory walkthroughs, Snyk and Datadog Security Labs both did a proper job. The short version: anything before 14.2.25 on the 14.x line, before 15.2.3 on 15.x, and a long tail of older releases. Vercel-hosted apps were patched at the platform level. Self-hosted ones were on their own. I had tried to have an "independent day", but invaders had decided everything needed to be blown up.
Server, sanitised
There's a particular satisfaction to a ruthless clinical nuke: taking back control through experience and permissions, both of which the intruder lacks and you don't. You wrote the access policy. You hold the keys. These assholes got in, but they aren't getting out.
Stop the infected app. Pull the disk for forensics if you're being a grown-up, otherwise back it up off-box. Scrub the persistence: cron tabs, systemd units, anything weird in /etc/rc.local, hidden dot-files in /root dressed up to look like real services. Kill the orphans. Bind anything internal-only back to 127.0.0.1 where it should have been from the start. Reboot. Then sit and watch htop and ss -tlnp for a stretch, ready to shoot from the hip.
Patched the framework. Redeployed clean from git. Wrote a detection script while it was all still fresh, because the next time someone tries this on one of my other boxes I'd rather find them in 30 seconds than three hours. The script lives in a private repo and gets a healthy run on every droplet I touch.
Sometimes a paper cut. Sometimes Pompeii.
The cryptominer was a paper cut. Annoying, expensive in time and dignity, but bounded. One box, one bad afternoon, no permanent damage. The thing about giving an attacker access to your infrastructure is that the floor is a paper cut. There is no ceiling.
A few years ago, a client's AWS account I was responsible for got popped. Not my account, not my money on the line, but my problem. The first sign was the smallest possible one: I went to log in one morning and the credentials didn't work. No usage spike yet, no scary numbers, just a door that used to open and now didn't. I opened a support ticket flagging that I'd been locked out and that I suspected a compromise.
AWS support shrugged. Probably a temporary glitch. Have you tried clearing your cookies. The tone was the polite, faintly bored one you reserve for someone overreacting to a minor inconvenience. I escalated. Same answer.
The key under the plant pot
What AWS didn't know, and what I'd half-forgotten myself, was that I had a backdoor into one of the running instances. Not a sinister one. A pragmatic one. The kind of out-of-band access an honest sysadmin leaves on a client's infrastructure at 2am on a Friday, six months earlier, on the off chance future-them is locked out of a console for a reason they can't yet imagine. Not a vulnerability. A lifeboat. The client didn't know about it. They wouldn't have understood why I wanted it. I left it anyway, because someone has to think about the morning when the door doesn't open, and on that engagement that someone was me.
I used it. Got onto the machine, found exactly what you'd expect: a cryptominer, the same shape of opportunist as the fghgf story above, just running on infrastructure with a much higher ceiling. Killed the process. Killed the instance. Pulled the credentials it had been using. The attackers were burning compute for Monero on AWS's dime, which would shortly be my client's dime, and I was now actively setting fire to their setup from inside while AWS support was still asking me whether I'd tried a different browser. None of this was in the contract. None of it would have been billable. It was a Saturday.
Three days later the bill arrived anyway, because the time between the lockout and me getting back in had been sufficient for the attackers to spin up enough hardware to render a Pixar film. Final bill landed around $189,000.
Then the structural insult. AWS still wouldn't give us proper access to the account "for security reasons", which is a brave principle to invoke after the security has, demonstrably, already left the building. They would however cheerfully send invoices. The position was, broadly, that we owed them six figures for a compute spree we'd reported as suspicious before it started and partly disrupted ourselves while their support team suggested cache-clearing.
What saved us was screenshots. Every ignored support reply from those first three days, timestamped and archived, the lockout flag clearly preceding the first malicious instance launch. A paper trail showing we'd flagged it as a compromise on day zero, they'd told us to relax, and the meter had run anyway. That trail eventually got the bill written off. Without it: ruinous. With it: a story you tell at parties for the rest of your life.
Two morals, and they're not the same one. The thieves wanted compute. Bounded goal, bounded damage, and once I got onto the box I could end them. The cloud provider's incentive runs the other way. Every additional minute of uncertainty is money on the meter, and the meter is theirs. The thieves break in. The platform charges you rent on the broken window and bills you for the draft. That's the part nobody puts in the brochure, and it's the part you have to architect around.
The other moral is quieter, and it's about the job. Past-me left a backdoor on a client's box because past-me thought ahead. Past-me took screenshots in real time because past-me was, frankly, terrified. Both of those happened because the account on the line wasn't mine. Working on someone else's infrastructure is a particular kind of responsibility. You're not just protecting servers. You're protecting the company that trusted you with them, and the people whose jobs are downstream of those servers staying up. Diligence is when past-you is kind to future-you. When the work is for a client, it's also kind to them. That's most of what the job is.
Receipts and rituals
There is a quiet skill, mostly invisible, that separates the engineers who survive their careers from the ones who get burned by them. It is not technical brilliance. It is not even particularly impressive. It is the muscle memory of leaving a paper trail. Screenshots before you click anything dangerous. Logs piped somewhere off-box. Process trees captured before you start killing things. A backup of the support ticket you're about to escalate. The boring stuff that feels paranoid until it's load-bearing.
You don't earn the trail in the moment. You earn it months earlier, in the small decisions nobody saw. The detection script you wrote on a Sunday because the previous Saturday was bad enough to want to never repeat it. The emergency-access path you set up on a client's infrastructure because at 3am six months earlier you thought what if. The version pin you held the line on when a junior dev wanted to bump it without reading the changelog.
Philip Larkin wrote about parents passing on their faults. He had a bleak view of the inheritance. Past-you and future-you have a kinder option. You can leave the right things behind on purpose. A note in the runbook. A comment in the config. A timestamp on a ticket. A patch you applied when it still felt optional.
The freeloader in /tmp is gone. The bill for the Pixar farm got written off. The detection script is still there, watching. So is the lifeboat on the client's box, which they still don't know about. That's the deal. You leave the next person breadcrumbs, even if the next person is yourself, even if nobody ever notices, because one day they'll need them and the alternative is a very long Saturday.
