The Probe Was the Bug
I am building a navigation app that knows every road you have driven and routes you down the ones you have not. It has a native front end that runs on the car's screen. For about a month it had a rendering bug: the route highlight came out oversized and torn. I added a probe to measure the defect.
The probe was what caused it.
The Crash Before the Glitch
One commit earlier, the same probe had been crashing the app on every drive. That is why the rendering glitch took a month to show up: the app rarely stayed up long enough to draw it.
The crash was a Swift initializer trap. When a subclass defines its own designated initializer in a certain shape, Swift does not inherit the superclass's designated initializer. It plants a trap in its place, an initializer whose entire body is a call to report an unimplemented initializer followed by a break instruction. The crash address was that function plus 40 bytes. I did not guess that offset. Compiling the same class shape with optimization puts the break at exactly plus 40 in that thunk.
Nothing in the repo could see it. The type checker skips Swift. No JavaScript test loads a native module. The Swift type-check compiles a planted trap without complaint, because the trap is code generation, not a type error. The scene type-check now has a second pass that looks for exactly that shape, because the bug type-checked perfectly.
The Glitch That Was the Instrument
With the crash fixed, the tearing appeared, and the probe was in place to measure it. What it measured was itself.
Eighty-three of eighty-six seconds contained a 16.7-millisecond frame where the renderer drew one tile at two or more zoom scales. The values stepped through powers of two. The probe's draw call was running once per tile per scale, and the overlay it drew to mark the route was the oversized, torn shape on screen.
The check that would have caught it sooner was written into the code and never run. The comment said: if the device log shows a call rate above zero, judge the tearing once with the probe disabled. The median call rate was 260 per second. Nobody had toggled it off and looked.
I did not take the cheap way out. Disabling the probe would have fixed the crash and thrown away the measurement the drive existed to produce. The fix was to make the probe draw once per frame instead of once per tile-scale, and keep it.
A Measurement Retired
Then the third turn. I had been measuring the route width in points from phone photographs of the car's screen, and using those numbers to tune the renderer.
An adversarial pass measured the photographs' own blur kernel from known-sharp interface edges. Twelve to twenty-eight pixels. The rim the width measurement depended on was five to forty-three pixels wide. The feature I was measuring was inside the camera's own blur, in exactly the rows I had leaned on hardest.
Do not quote a route width in points measured from a phone photo of a head unit. That is now a sentence in the repo's instructions, because the measurement had been treated as ground truth for weeks.
The Report Was the Opposite of the Cause
One more from the same project, because it is the same lesson from the user's side.
I reported, as the user, that the avatar tilts off course while tracking straight, and asked for some resistance to be tuned in. That is a perception, not a cause. The engineer's job is to measure the frames.
The avatar's screen position was exactly 602.5 device pixels in every frame, dead center of a 1,206-pixel screen, rendered upright in every frame. It never moved and never rotated. What moved was the route. The ribbon's centerline under the avatar swung 51 device pixels, about 16 meters of drift to the side, over 25 seconds.
The cause was in code, not perception. The camera's bearing came from a heading built out of two raw GPS fixes in a row, one second apart. A bearing from one pair of fixes is mostly sideways noise. At 30 miles per hour with three meters of noise, the pairwise bearing's spread is over ten degrees. The route tangent's is under one. Use the tangent. The avatar was never the problem.
Two more from the instruction file, both earned. Normalize photo orientation before making any geometry claim: two sessions in a row read un-rotated screenshots and reported a north-up camera and a rotating marker. The corrected crops showed a heading-up camera and a marker pointing the right way. The bug being chased did not exist. And the owner drives with the display's warm color shift permanently on. Under that, colors stop separating and only light against dark survives. So any palette decision that leans on color to separate two things is not a decision about what he will see.
The Shape
A probe that crashes the thing it measures. A probe that draws the defect it measures. A ruler blurrier than the thing it rules. A bug report that names the stationary object. Every one of those is the instrument, not the system.
I have written this lesson three or four times now in different places and I keep expecting to be done with it. The reason I do not is that the instrument is always the last thing anyone suspects, because it is the thing you built to be trusted. It should be the first.
-- Justin Higgins. Software Engineer, Midwest. Built a probe to measure a glitch, and the probe was the glitch.
Companion pieces: One Line Held the Lock - the logger that was the bottleneck. Three Bugs in the Instrument - the same shape in a retrieval system.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com