Letting the LLM fix its own SQL — but only twice
The first time the agent fixed its own broken SQL, I was delighted. The query had failed on a type mismatch, I’d fed the error back to the model, and it came back with a corrected query that just worked. Self-healing! The third time it tried to “fix” the same query — producing a third distinct, confidently wrong variation — I realized I’d built an elegant way to set tokens on fire.
Back up. Even after grounding the model in real values and canonicalizing its filters, generated SQL still fails sometimes. A subtle type mismatch, a function used slightly wrong, an aggregation that doesn’t quite parse. The naïve response is to surface the database error to the user — but my users are non-technical restaurant operators, and Cannot GROUP BY an aggregate of type FLOAT64 means precisely nothing to them. A stack trace is not an answer.
The good response is self-correction. When a query fails, don’t give up — hand the model back its own query and the exact error the database returned, and ask it to fix it. This works far better than you’d expect, because the error message is genuinely informative: the model wrote bad SQL not knowing the column was a string, the database says so plainly, and the model corrects it. Most failures resolve on the first retry.
The trap is the word “retry,” with no number attached.
Because some queries don’t get fixed. The question is genuinely ambiguous, or the data can’t answer it, and the model doesn’t know that — so it keeps producing new wrong queries, each one different, each one looking like progress. That’s the insidious part: an unbounded correction loop doesn’t sit there obviously stuck. It looks busy. It’s generating, executing, failing, generating again, and every cycle costs another LLM call and another few seconds while the user stares at a spinner and the token meter ticks up. Left alone, it’s a loop that mistakes motion for progress and bills you for the privilege.
So the correction loop is bounded — a deliberately small number of attempts — and the bound is the whole point:
def run_with_correction(question, max_attempts=...): # deliberately low
sql = generate(question)
for attempt in range(max_attempts):
if estimate_cost(sql) > COST_CEILING: # dry-run BEFORE spending anything
return clarify("that looks very broad — can you narrow it down?")
ok, result, error = execute(sql)
if ok:
return result
sql = regenerate(question, failed_sql=sql, error=error) # feed the error back
return clarify("I couldn't turn that into a query I trust — did you mean X or Y?")Two guards, doing different jobs. The loop bound caps how many times the model is allowed to be wrong before the system stops and asks the user a clarifying question instead of spinning. And estimate_cost is a dry run — the warehouse will tell you how many bytes a query would process without executing it — so a pathological query that would scan an entire dataset gets caught and refused before it runs up a bill, not after. One guard bounds the model’s stubbornness; the other bounds its appetite.
Picking the bound was, like most of these constants, empirical — I watched what the retries actually did. What I found: if a query isn’t fixed within the first attempt or two, it’s almost never fixed by attempt five either — those later attempts are the model thrashing, not converging. So the ceiling is low, and crossing it isn’t treated as failure, it’s treated as a signal: this question can’t be answered as asked, so stop guessing and ask the human. Falling back to a clarifying question is a much better experience than either a stack trace or a four-second silence that ends in one anyway.
It holds for any agent with a feedback loop, not just this one: giving a model the right to correct itself is powerful, but “fix yourself” without “…up to N times” is an unbounded loop wearing a helpful smile. Any time you let an LLM react to its own output — retries, critiques, multi-step plans that revise themselves — you need a hard stop that doesn’t depend on the model deciding it’s done. The model is not a reliable judge of whether it’s making progress; that’s exactly the faculty that’s missing. So the bound lives in the harness, in plain Python, where it can’t be talked out of stopping.1
The honest open question: the right fallback when you hit the bound is probably smarter than “ask the user.” Sometimes the model’s second attempt was closer than its third, and I throw that away. A better system would keep the best partial result and offer it with a caveat, rather than discarding the whole loop. I haven’t built that yet — it’s on the list, somewhere below the things that were actually on fire.
From a consulting project building natural-language analytics for restaurant businesses. Customer details, schema, and constants are abstracted; the reasoning is as built. Code is illustrative.
Footnotes
Why feeding the error back works at all: the failure message is a high-signal, perfectly-targeted hint. The model didn’t write bad SQL out of stupidity — it wrote it missing one fact (a type, a column’s real name), and the database’s error supplies exactly that fact. It’s the cheapest fine-tuning signal you’ll ever get, available for the cost of catching an exception.↩︎