A bug fix can be written, reviewed, merged, tagged, and correct, and still never reach a single user. The fix for a real iOS connectivity bug had existed upstream for over a year while the people hitting the bug filed me too comments on an issue, because one git submodule pointer between the library and the client was never advanced. Merging changes the source of truth. It does not move the pointers, and in a dependency graph the last move is owned by no one.
Read More
In a single day of debugging, four different signals each looked like proof and each supported a smaller claim than I borrowed it for. A log line that printed exactly as designed while the process it was in died ten seconds later. An address that looked like proof of a cellular test and was identical on Wi-Fi. A fix that was clean and plausible and would have broken a whole class of users. A ten-minute hang with no error output at all. Evidence is not the problem. Using a true signal to support a claim it cannot carry is.
Read More
One app, one host, one cellular connection. The HTTP calls reach the server and the streaming connection times out with no error anywhere. Nothing is misconfigured. On an IPv6-only cellular network, getaddrinfo() on an IPv4 VPN literal returns a synthesized NAT64 address that routes straight out of the tunnel, and only the code path that resolves the address itself is affected. This is a field note on recognizing that failure, not discovering it.
Read More
When an agent reports an action it never actually completed, the lie almost never starts in the model. It starts one layer down, in a tool wrapper that returned 200 the moment the request was accepted rather than when the side effect was observed. This is the completion-ownership bug that has broken RPC systems for decades, wearing a tool schema. The fix is to put ownership in the wrapper: verify the effect before the word success ever reaches the model.
Read More
When the monthly LLM bill jumps several times over, the first instinct is that the model got more expensive or usage simply grew. It is almost always something else: a distributed systems failure mode, retry storms, fanout amplification, cache misses, unbounded conversation growth, that happens to be denominated in tokens instead of network calls. Debug the call graph, not the model price.
Read More
The most common production AI agent failure is treating validation as an assertion, a one-shot pass/fail check, instead of a loop that validates, scores, and decides whether to accept, retry, or escalate. Deterministic code can assert. Non-deterministic model output needs a closed loop, or your agent will report success while doing the wrong thing.
Read More
In a rules-first AI system, the deterministic checks get authority. They gate output and overrule the model judge. But a wrong check with authority is worse than no check at all. A first-principles standard for verifying the ruler before you trust it.
Read More
Every part of an AI system has exactly one right home: a rule, a state machine, or a bounded judge. Control belongs to the orchestrator, not to an autonomous loop. A first-principles boundary for deciding which technique owns which part.
Read More
The reliability of a probabilistic system is set by how much of it you let the model decide. A first-principles standard for shrinking the stochastic surface, on the generation side and the evaluation side.
Read More
Generative AI nails plausible shapes but misses constraints, composition, and verification. A first-principles read of the gap, and the architecture that closes it.
Read More