Your Rate Limiter Is Counting the Wrong IP
I've now found the same bug in two of my own services, on two platforms, in two frameworks, and both times the limiter looked correct in the code and did nothing in production. I'm writing it down so the third time is faster. There will be a third time.
The First Time
The first was a public API endpoint on a small Go service, rate-limited to 60 requests per minute per client IP. The library keyed on the connection's remote address, which is the sensible default.
Behind the platform's edge proxy, the remote address isn't the client. It's an internal address in the carrier-grade NAT range, and it rotates per connection, so a client could get around the cap by opening a new connection for each request.
I didn't find that by reading. I found it by pressure-testing production. Eighty requests over separate connections: zero throttled, each one with a fresh bucket. Eighty requests over one reused connection: a clean sixty successes and then twenty rejections. Every rejection log showed the same internal edge address, not the client.
The fix anyone would write is to key on the leftmost entry of the forwarded-for header, and I wrote that. I also shipped a temporary diagnostic that logged every forwarding header the edge set, because I didn't want to rely on a trust model I hadn't seen.
Good thing, because the next pressure test showed that on this platform, the client can forge the leftmost forwarded-for entry. The edge adds the real client address after any forged value, so an attacker can put whatever they like at the front. A different header, the real-IP header, is edge-set and overwrites forged values. The limiter keys on that now, with the remote address as a fallback when the service runs somewhere without the edge. The diagnostic came out in the same commit.
So that's two fixes, and the first one was also a bug. Both were caught by sending traffic at the real deployment, not by reading the code or the platform docs.
The Second Time
The second was a different service, Node this time, on a different platform, with a documented login throttle of five attempts per fifteen minutes per IP. This one is the authentication hub that mints the session cookie eight of my apps trust, which is the last place you want a throttle that doesn't exist.
The framework had been told to trust the proxy. With that flag on, it took the client IP from the leftmost forwarded-for entry, which, as above, is client-supplied. Rotate the header and you get unlimited login attempts: twelve POSTs, zero rejections, unauthenticated, remote, with no prerequisites.
I found that one by writing the first tests the repo had ever had against its auth surface. The same test pass found three more bugs. One of them reduced a password check to comparing two empty strings under a malformed stored value, which returns true for any password. That's a separate post. The point here is that the most trusted component in the family was the least tested, and the bug class was a boring one.
The General Shape
Every framework's rate limiter has a "key by IP" mode, and every one of them is correct on a bare server. The moment there's a proxy in front, the question becomes which header the proxy sets that the client can't overwrite. The answer is different on every platform, and it's usually not the header the framework's convenience flag reaches for.
The forwarded-for header is a list, and most proxies append to it. So the leftmost entry is whatever the client sent, the rightmost entry is what the last proxy saw, and in general the safe answer is neither. It's whichever single-valued header your specific edge overwrites, or the rightmost forwarded-for entry counted back by the number of proxies you actually control. You can't know that from the framework. You can only know it by sending forged headers at your real deployment and reading what arrives.
So: forge the header, send eighty requests, count the rejections. Then key on what survived. And leave a test that does the forging, so the next platform move doesn't silently undo it.
Why I Missed It Twice
Because the limiter was there. It was in the middleware chain with a sensible number, and the logs showed it firing occasionally. Everything a code review would check was present. What was missing was any evidence that the number it was counting belonged to a client, and nothing in the code could tell me that, because the answer lived in the proxy.
The part that fails is the part you didn't instrument. A rate limiter's instrumentation is a load test from outside, which almost nobody runs on their own tiny service. I run it now. It takes about a minute.
-- Justin Higgins. Software Engineer, Midwest. Found the same rate-limit bypass twice and pressure-tested his way out both times.
Companion pieces: Verify the Artifact, Not the Process - loopback is not a boundary either. Spider-Man Is Not a Person - read what the far side received.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com