The answer arrives clean, well-formatted, and certain. You run it. The config flag doesn't exist.
Nothing in the response told you the difference between the parts it was sure about and the parts it filled in. The syntax was right. The explanation was reasonable. The flag had a sensible name and a clear purpose. It just wasn't real.
This is a calibration problem, not a trust problem. "Don't trust AI" is useless advice. It tells you to doubt everything, which is the same as doubting nothing, because you can't actually work that way. What you need instead is a way to tell which parts of an answer are grounded and which were filled in. There are patterns to look for, and once you know them, they are hard to miss.
This isn't about intelligence. It's about experience.
An engineer with a few years behind them has already lost time to a function that didn't exist. They have wasted an afternoon on a config parameter that turned out to belong to a different engine. They have built up a collection of "that looked right but wasn't" memories, and those memories now activate automatically. A small feeling that says check this one before they run it.
Beginners haven't had those afternoons yet. The pattern recognition that comes from repeated failures doesn't exist, because the failures haven't happened. Everything arrives in the same confident tone, and without past mistakes to compare against, there is no internal signal separating a grounded answer from a made-up one.
That's not a character flaw. It's missing experience, and it can be partly replaced with three concrete things to look for.
These aren't perfect. They are a minimum standard, and they catch the common cases. I'll be clear about what they miss at the end.
When the answer includes a precise detail (a specific config flag, a particular function signature, an exact version number) and doesn't say where that detail comes from, that is the highest-risk part of the response.
Grounded answers tend to be general where the model is less certain and specific where it is more certain. Made-up answers are often specific everywhere, because there is no visible uncertainty signal in the output. A real Spark config parameter and an invented one arrive looking exactly the same.
The check: if a detail is specific enough to copy into code, verify it exists in the actual documentation for your version before you run it. Not whether the logic makes sense. Whether the thing itself is real.
A Spark config flag that sounds right, follows the naming convention, and controls exactly what you need. And doesn't exist. That's sign 1.
This is closely related to sign 1, but it appears differently.
You ask how to do something in Snowflake. The answer uses a function that does exactly what you need, with clean syntax and a sensible name. You check the Snowflake documentation. The function isn't there. You search further. It exists in BigQuery.
The function is real. The capability is real. It just belongs to a different engine. This is more dangerous than a completely invented function, because it almost checks out. You can find documentation for it, just not for your system.
The check: when a function is unfamiliar, verify it against your engine's documentation specifically, not a general web search. "Does this function exist" and "does this function exist in Snowflake" are different questions, and only the second one matters.
This is the least obvious sign and the most reliable one.
Try this: ask a question with a specific constraint. Get an answer. Then ask the same question again with the constraint removed. If the answer barely changes, the constraint wasn't actually being used, which means the model may have produced a generic answer and wrapped it in your details.
For example: you ask for a partition strategy for a table with 500 million rows and heavy read traffic on a date column. You get a detailed answer. Now ask for a partition strategy for "a large table." If the answer is mostly the same (same recommendation, same reasoning, just without your numbers) then the first answer was not shaped by your constraints. It was a generic answer with your context added on top.
The check: on anything important, remove a key constraint and ask again. If the answer stays the same, it probably wasn't using that constraint in the first place.
The strongest version of the pushback isn't "don't trust AI." It's this: these signs are rules of thumb, and a model can be confidently wrong while showing none of them.
That's true. There will be answers that pass all three checks and are still wrong. The function exists in your version, the answer changes when you remove context, the specific details are real, and the logic is still subtly flawed. A three-point checklist can create false confidence in exactly the cases it misses.
That's a real risk, and it's why the framing matters. These are a minimum standard, not a guarantee. They catch the common failures: functions that don't exist, answers from the wrong engine, generic advice dressed up as specific advice. What they don't do is confirm that everything else is correct. Anything that passes all three checks still needs your judgment. But anything that fails one of them can be caught in thirty seconds, before it costs you an afternoon.
The goal isn't eliminating every bad answer. It's catching the ones that follow a pattern, cheaply, so your actual thinking time goes to the ones that don't.
Three signs. Check before you run.
1. Very specific details with no source. If it's specific enough to paste into code, verify it exists in your version's documentation.
2. A function from the wrong tool. Check against your engine specifically, not a general search.
3. Answers that don't change when context is removed. Take out a key constraint and ask again. If the answer stays the same, it wasn't using your context.
These won't catch everything. They catch the failures that follow a pattern, and those are the ones costing you the most time right now.
This is post #2 in a series on AI practice for data engineers. Next up: The Most Underused Prompt in Data Engineering.