The Same Alert Meant Two Different Failures

On the evening of November 4, my Debian box sent my Mac one line: "scheduled job failed: daily exited 1." It sent the same line on November 6, 8, 10, 11, 12, 13 and 14. Only the date changed.

I run two machines. My Mac is where I develop. The Debian box, which I call the Halo, runs most of my scheduled jobs. They talk through an append-only message ledger I call telegraph, and every alert, question and answer between them lands there with a timestamp.

I moved my scheduled jobs from the Mac to the Halo on November 1. One of them builds daily fantasy football lineups. Each night it has Codex run a research pass on the week's players and write a structured file. The rest of the job treats that file as evidence: it ingests it, re-projects, and publishes.

At least two different causes were behind those eight alerts, and the text didn't tell them apart.

The first cause: retyping the answer

On November 10 the Mac asked the Halo to diagnose three of the failures, read-only. The answer came back with one shared cause. Codex was hitting the research step's 40-minute deadline.

The research step asked Codex to return the whole research file as its final message. That file ran 130 to 230 KB. Codex writes at about 13 KB a minute, so the retyping alone took 10 to 18 minutes. The prompt told it to stop researching 10 minutes before the hard stop. That's not enough for a 17-minute retype.

The failed runs had researched for 23 to 30 minutes. Two of them did about 260 searches, where 80 is normal. They left 9, 9 and 16 minutes for the retype. The passing runs had researched for 11 to 25 minutes.

It didn't do much damage. Each time, there was already a valid draft on disk. The runner salvaged it, ingest recorded the run as partial, and every later step ran, including the web publish.

The alert wasn't an accident. A game-day run on November 1 had hit the same deadline mid-way through its final message and published nothing. On November 2 the runner learned to salvage the draft Codex keeps in its workspace. The script still exits 1 after a salvage, and the code comment says why: "exit 1 so the step and its notification show it."

So in that stretch, the alert meant "partial, published anyway."

The Halo proposed two fixes. One raised the reserve from 10 minutes to about 20 and capped searches. The other stopped the retyping: Codex writes the file, marks it complete and ends with a one-line message, and the runner publishes the file it's already checked. I gave the go on the second one only. A build session on the Halo had it committed that night on a branch, with one new regression test and 310 of 310 tests passing. It wasn't merged, it wasn't pushed, and it hadn't been tried on a live run.

The second cause: a dead login

On November 14 two more jobs exited 1, a Saturday recheck and the daily. The Mac asked why. This time Codex's error was different:

"refresh token was already used. Please log out and sign in again."

Research had failed with that error every night since the run at 01:01 UTC on November 13. That's 19:01 on the 12th, Central time. The last accepted run was the night before. So the November 12 alert, word for word the same as the ones before it, was already the second failure.

This one wasn't a partial. Research didn't produce anything. Every other step still passed, so lineups and picks still went out. The Halo's copy of the Codex login file hadn't been refreshed since November 2.

Where the login came from

On the night of the move, the Halo asked how it should get a Codex login. It gave three options: sign in on the Halo with a device code, copy my Mac's auth file over through the shared drive, or skip that night's run.

I picked the copy. At 19:12 the Mac posted the file to the share. A minute later the Halo had installed it, deleted the share copy and run a smoke test. By 19:33 the catch-up run had finished with all 13 steps ok. That login kept working until November 12.

The Halo's diagnosis on the 14th didn't overclaim. "Most likely" something else had used the same refresh token and rotated it. It named my Mac's copy as one candidate and a parallel refresh as another. A refresh token that can only be used once works for whichever machine uses it first. The other copy's dead from then on, and that machine finds out the next time it tries to refresh.

That fits what I've got. My Mac's Codex session folder has sessions on every day from November 1 to 16, so the Mac was using Codex the whole time. It doesn't prove anything. I haven't confirmed which machine refreshed when, and the logs I have can't show it. The shared token's the likely cause. It isn't a confirmed one.

What the alert couldn't tell me

The alert comes from a small filter on the Halo. It forwards any job notice matching a list of failure words, "exited N" among them. The line carries the job name, the exit code and the path to that night's summary file. Both causes produced the same three things.

Exit 1 meant two very different states. One was "research ran long, a partial was salvaged, everything published." The other was "research never ran, and won't until I sign in again." The first one's a nuisance. The second failed every night until someone signed in.

The only difference I can see in the ledger is the clock. The daily alerts on November 6, 8 and 10, the ones the diagnosis covered, all landed at 19:41. The November 11 alert, which nobody diagnosed on its own, landed at 19:36. The three daily alerts from the dead login landed at 19:01 or 19:02. The first of them hit the ledger 20 seconds into the minute its run started. A blown 40-minute deadline takes about 40 minutes to report. A dead login reports within about a minute.

The November 4 alert landed at 19:25. The ledger doesn't say what caused it.

What's still open

The retyping fix isn't on the pushed main. The runner there still asks Codex for the full answer as its final message, and the prompt's a version behind the fix. The repo's own roadmap, edited November 11, says the research run deadline is "already fixed" by that commit. The ledger doesn't say it was merged on the Halo.

The ledger doesn't record a new sign-in either. The job alerted again on November 15 for a game-day pass and on November 18 for the daily. The ledger doesn't diagnose either one. There were no daily alerts on November 15, 16 or 17, and the November 18 one landed at 19:41, the deadline's time. That reads like the login came back and the retyping failure didn't go away, but nothing in the ledger confirms it.

The Halo's fix for the login is one Codex login per machine, made on that machine. Until the alert says which failure it is, the timestamp is the only thing that tells them apart.

-- Justin Higgins. Software Engineer, Midwest. One alert string, two failures, and a login copied as a file.


Companion pieces: Spider-Man Is Not a Person - a health check that stayed green while the app was down. A Backup of a Lie - a backup that passed as valid and was empty.

Reactions, disagreements, war stories: jchigg2000.dev@gmail.com