Give an agent a database tool and ask it about last quarter's revenue. Most of the time you get a good answer. Sometimes the tool comes back empty, or with something the agent doesn't quite understand, and the agent does what language models do best: it writes a plausible sentence with a plausible number in it.
Nothing errors. The number just isn't real.
This post builds a LangChain agent with one data tool that behaves differently. The tool returns either an answer with the query and rows behind it, or an explicit DECLINED, and the agent is told what to do with each. The data layer is FactQL, but the pattern (evidence in, declines out, a strict system prompt) works with any data tool you write.
About 60 lines of Python, LangChain v1.
pip install "langchain>=1" langchain-openai requests
export FACTQL_API_KEY=fl_live_... # free tier, no card: factql.ai
export OPENAI_API_KEY=... # or any chat model LangChain supports
Upload a CSV, or connect Postgres or MySQL in the FactQL app, so there's something to ask about.
import json, os, time
import requests
from langchain.tools import tool
RETRYABLE = {429, 500, 503}
@tool
def ask_factql(question: str) -> str:
"""Answer a question about the company's data (sales, customers, orders...).
Returns an answer with the query and rows behind it, or DECLINED if the
data can't support an answer."""
for attempt in range(3):
resp = requests.post(
"https://api.factql.ai/query",
headers={"X-API-Key": os.environ["FACTQL_API_KEY"]},
json={"q": question},
timeout=120,
)
if resp.status_code in RETRYABLE and attempt < 2:
time.sleep(int(resp.headers.get("Retry-After", 2 ** (attempt + 1))))
continue
resp.raise_for_status()
break
answer = resp.json()
if answer.get("result") != "success":
return "DECLINED: " + answer.get("explanation", "FactQL could not verify an answer.")
data = answer.get("data") or {}
return json.dumps({
"answer": answer.get("explanation"),
"query": answer.get("sql"),
"columns": data.get("columns"),
"rows": (data.get("rows") or [])[:10],
"lineage": (answer.get("lineage") or {}).get("text"),
})
Three details matter more than they look:
The docstring is the tool description. The model reads it to decide when to call the tool, and the last sentence tells it up front that DECLINED is a normal outcome, not a failure.
Return the evidence, not just the sentence. The query, a few rows and the lineage go back to the model with the answer. The agent can then cite where a number came from, and you can log it.
A decline is HTTP 200. When FactQL can't back an answer it still returns 200, with "result": "error" and an explanation written for the end user:
{
"result": "error",
"decline_reason": "no_safe_sql",
"explanation": "I couldn't find data that answers this. Try naming the table or column you mean."
}
If your tool only checks the status code, that response looks like success with an empty answer, which is exactly the gap a model fills with a guess. So the tool turns it into an unmistakable string: DECLINED: .... Retry only on 429, 500 and 503; retrying a decline gets the same decline.
SYSTEM = (
"You answer questions about the user's business data using the ask_factql tool. "
"Only state numbers that come back from the tool. If the tool returns DECLINED, "
"tell the user you can't verify that from their data and pass on the explanation. "
"Never estimate or fill in a number yourself."
)
Without the last sentence, a helpful model will often turn a decline into "it's probably around...". That's the one sentence you most want in there.
import sys
from langchain.agents import create_agent
agent = create_agent(
model=os.environ.get("AGENT_MODEL", "openai:gpt-5.4-mini"),
tools=[ask_factql],
system_prompt=SYSTEM,
)
if __name__ == "__main__":
question = " ".join(sys.argv[1:]) or "Which region had the highest revenue last month?"
result = agent.invoke({"messages": [{"role": "user", "content": question}]})
print(result["messages"][-1].content)
Still on LangChain 0.x? langgraph.prebuilt.create_react_agent(model, [ask_factql], prompt=SYSTEM) does the same job. In v1, create_agent replaces it and prompt became system_prompt.
Ask something your data covers:
python agent.py "Which region had the highest revenue last month?"
Then ask something it doesn't. On a sales-only dataset, ad spend is the easiest:
python agent.py "What was our ad spend last quarter?"
The second answer should say it can't verify that and pass on FactQL's explanation. If it gives you a number, your system prompt isn't strict enough, or the tool isn't returning the DECLINED: prefix.
You can check the agent's wiring offline with a fake chat model that calls the tool, and a mocked HTTP response:
from unittest import mock
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage
class ToolFake(GenericFakeChatModel):
def bind_tools(self, tools, **kw):
return self
model = ToolFake(messages=iter([
AIMessage(content="", tool_calls=[{"name": "ask_factql", "args": {"question": "ad spend?"}, "id": "1"}]),
AIMessage(content="I can't verify that from your data."),
]))
test_agent = create_agent(model=model, tools=[ask_factql], system_prompt=SYSTEM)
declined = mock.Mock(status_code=200, headers={})
declined.json.return_value = {"result": "error", "explanation": "I couldn't find data that answers this."}
with mock.patch("requests.post", return_value=declined):
out = test_agent.invoke({"messages": [{"role": "user", "content": "ad spend?"}]})
tool_reply = next(m for m in out["messages"] if m.type == "tool")
assert tool_reply.content.startswith("DECLINED:")
Put that in CI and a refactor can't quietly break your decline handling.
/query, and a $10 monthly spend cap (calls return 402 at the cap; you're never invoiced).DECLINED is a question your users asked that your data couldn't answer. Grouped weekly, that's your list of tables and columns to add next. More on that in If your agent or editor speaks MCP, you can skip the custom tool: FactQL runs a hosted MCP server at https://api.factql.ai/mcp, and the open-source server is at factql/factql-mcp. Its ask tool returns the same answer-or-declined shape.
The three blocks above (tool, system prompt, agent) are the whole agent: paste them into one agent.py and run it. More examples, for OpenAI-style function calling, a Next.js route and an embedded chat, are in factql/factql-examples (MIT).
I'm on the team building FactQL, so I'm biased. The pattern works with any data tool that can say "I can't verify that".