Your Rate Limiter Is Counting the Wrong IP
I have now found the same bug in two of my own services, on two platforms, in two frameworks, and both times the limiter looked correct in the code and did nothing in production. I am writing it down so the third time is faster. There will be a third time.
The First Time
A public API endpoint on a small Go service, rate-limited to 60 requests per minute per client IP. The library keyed on the connection's remote address, which is the sensible default.
Behind the platform's edge proxy, the remote address is not the client. It is an internal address in the carrier-grade NAT range, and it rotates per connection. So a client could get around the cap by opening a new connection for each request.
I did not find that by reading. I found it by pressure-testing production. Eighty requests over separate connections: zero throttled, each one got a fresh bucket. Eighty requests over one reused connection: a clean sixty successes and then twenty rejections. Every rejection log showed the same internal edge address, not the client.
The fix that anyone would write is to key on the leftmost entry of the forwarded-for header. I wrote that. I also shipped a temporary diagnostic that logged every forwarding header the edge set. I did not want to rely on a trust model I had not seen.
Good thing. The next pressure test showed that on this platform, the client can forge the leftmost forwarded-for entry. The edge adds the real client address after any forged value, so an attacker can put whatever they like at the front. A different header, the real-IP header, is edge-set and overwrites forged values. The limiter keys on that now, with the remote address as a fallback when the service runs somewhere without the edge. The diagnostic came out in the same commit.
Two fixes. The first one was also a bug. Both were caught by sending traffic at the real deployment, not by reading the code or the platform docs.
The Second Time
A different service, Node this time, on a different platform, with a documented login throttle of five attempts per fifteen minutes per IP. This one is the authentication hub that mints the session cookie eight of my apps trust, which is the last place you want a throttle that does not exist.
The framework had been told to trust the proxy. With that flag on, it took the client IP from the leftmost forwarded-for entry. Which, as above, is client-supplied. Rotate the header, get unlimited login attempts. Twelve POSTs, zero rejections. Unauthenticated, remote, no prerequisites.
I found that one by writing the first tests the repo had ever had against its auth surface. The same test pass found three more bugs. One of them reduced a password check to comparing two empty strings under a malformed stored value, which returns true for any password. That is a separate post. The point here is that the most trusted component in the family was the least tested, and the bug class was a boring one.
The General Shape
Every framework's rate limiter has a "key by IP" mode and every one of them is correct on a bare server. The moment there is a proxy in front, the question becomes which header the proxy sets that the client cannot overwrite. The answer is different on every platform, and it is usually not the header the framework's convenience flag reaches for.
The forwarded-for header is a list. Most proxies append to it. That means the leftmost entry is whatever the client sent, and the rightmost entry is what the last proxy saw, and the safe answer is neither of those in general. It is whichever single-valued header your specific edge overwrites, or the rightmost forwarded-for entry counted back by the number of proxies you actually control. You cannot know that from the framework. You can only know it by sending forged headers at your real deployment and reading what arrives.
So: forge the header, send eighty requests, count the rejections. Then key on what survived. And leave a test that does the forging, so the next platform move does not silently undo it.
Why I Missed It Twice
Because the limiter was there. It was in the middleware chain. It had a sensible number. The logs showed it firing occasionally. Everything a code review would check was present. What was missing was any evidence that the number it was counting belonged to a client. Nothing in the code could tell me that, because the answer lived in the proxy.
The part that fails is the part you did not instrument. A rate limiter's instrumentation is a load test from outside, which almost nobody runs on their own tiny service. I run it now. It takes about a minute.
-- Justin Higgins. Software Engineer, Midwest. Found the same rate-limit bypass twice and pressure-tested his way out both times.
Companion pieces: Verify the Artifact, Not the Process - loopback is not a boundary either. Spider-Man Is Not a Person - read what the far side received.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com