The bug was simple: I kept seeing buttons with no explanation above them.
Andy - my AI assistant - has a feature where it can offer choices at the end of a response. “Want me to do X or Y?” with clickable buttons. The system requires that the answer travels with the buttons in the same payload. Otherwise the answer gets lost somewhere in the pipeline between Claude and Discord.
The model kept forgetting to include the answer in the payload. Text would appear in the transcript - the full response existed - but by the time it reached me, I saw three buttons and zero context.
A detailed roofing analysis with material recommendations and permit guidance. Gone. Just buttons saying “Add permits section” and “Expand roofing details” with no explanation of what those meant.
The fix that felt clever
First attempt: add an optional field to the payload where the answer could live. Document it clearly. Tell the model to use it.
Models forget. That’s the nature of the problem. Documentation alone doesn’t enforce behavior.
Second attempt: build a hook that intercepts the button payload, reads the session transcript, finds the most recent substantial text block, and auto-injects it into the payload.
This felt clever. The model doesn’t need to remember anything. The system handles it automatically.
It worked in testing. Then it started grabbing the wrong content. Large JSON blocks. Code snippets. Things that looked like “substantial text” to the heuristics but weren’t the actual answer.
The hook was trying to solve a content-identification problem. What counts as “the real answer” versus “a code block the model outputted while working”? Every edge case required another filter. The filters had their own edge cases.
Four days of this.
What actually worked
Delete the hook. Make the script reject payloads without a sufficient answer field. Return a clear error message: “message field required, 100+ chars.”
Model gets an error. Model retries with the answer included.
One line of validation. No transcript parsing. No heuristics about what counts as “real” content. The boundary enforces the constraint.
The difference: instead of trying to fix bad input downstream, refuse to accept it in the first place.
What I’d tell someone building this
Reject at the boundary, don’t fix downstream. The hook tried to be clever about detecting “real” content versus code blocks. The simpler approach - require 100+ chars in the message field, reject otherwise - has no heuristics to fail. The constraint is binary.
Error messages work better than silent auto-correction. “Message field required, 100+ chars” gives the model clear feedback to retry correctly. The hook silently injected content - no learning signal, no feedback loop.
When you’re on the third variant of a fix, step back. We kept layering complexity: add field, add hook, fix hook edge cases, add more filters. The git history showed four days of chasing. The simple enforcement should have been the first attempt, not the fourth.
Test the failure mode, not just the happy path. We tested that the hook fired. We didn’t test what content it actually injected. The hook was “working” the entire time - it just wasn’t working correctly.
The uncomfortable pattern: every clever fix introduced a new category of failure. The boring fix - refuse bad input - closed the loop entirely.
Systems that try to repair malformed input create maintenance burdens that compound. Systems that reject malformed input stay simple. The boundary is the right place to enforce constraints, because that’s where you still have the leverage to say no.