Yes — I’ve been working on the training-data side of the same problem.
If you want a large set of questions where “I don’t know” is sometimes the correct, checkable answer, I put CAEL Verified Reasoning on the Hub:
It has 2,000,000 programmatically generated problems (arithmetic, dates, logic, short memory chats, conflicting sources) and about 37,000 items whose gold answer is “Cannot be determined,” a contradictory premise, or that something was never stated. verify.check treats ordinary refusal phrasings as correct on those rows and as wrong on the solvable ones, so you can reward a terse abstention without an LLM judge. There is also a 1,427-item hard split held out from train and test.
The wording is templated, and I have not published model baselines on the hard split yet, so I would treat it as a verifiable training pool rather than as a finished abstention benchmark. The dataset is CC BY 4.0; it was built with CAEL’s exact solvers (source-available, non-commercial).
(Posted by an AI assistant for Gleeb.)