I pointed my scanner at Cal.com this week, fully expecting to write a post about everything it got wrong.
The most interesting result was the thing it refused to say.
Quick context: I build DevTime, a local-first CLI that scans a repository and helps it explain itself from evidence. It detects a closed set of concepts (authentication, billing webhooks, background jobs, data export, admin permissions, file uploads), links every claim to evidence files, and reports uncertainty when it cannot prove something. No cloud, no telemetry, no code execution during scan, no AI.
Cal.com is a well-known open-source scheduling platform. Big TypeScript monorepo, Next.js frontend, NestJS API, Stripe integration. If you wanted to stress a heuristic scanner, this is a good place to do it.
The run:
pipx install devtime-ei
git clone --depth 1 https://github.com/calcom/cal.com
cd cal.com
dtc init
dtc scan
The scan:
Scan complete. Scanned 5736 files, 24516 signals, 3 concepts in 4.2s.
Local, 4.2 seconds, nothing left the machine.
1. Authentication confidence: high debt: medium
2. Billing Webhooks confidence: low debt: high
3. Background Jobs confidence: low debt: high
Authentication was solid. Real evidence, real routes:
- Authentication is present and supported by behavior evidence.
confidence: 0.82
evidence: apps/web/app/api/auth/forgot-password/route.ts,
apps/web/app/api/auth/oauth/me/route.ts
But Billing Webhooks at LOW confidence? Cal.com is a scheduling SaaS with Stripe integration. Of course it has billing webhooks. I assumed I had found my first miss.
Here is what DevTime actually said:
- Possible Billing Webhooks signals detected. Billing Webhooks behavior
is not established from current evidence.
confidence: 0.45
Uncertainty:
- Only dependency or manifest evidence was found for Billing Webhooks;
presence is not confirmed behavior.
So I went to check manually. The repo has exactly the file you would expect:
apps/web/pages/api/stripe/webhook.ts
Case closed, right? A file named stripe/webhook.ts in a Stripe-integrated SaaS. Any developer grepping the repo would say "here is their webhook handler". Any AI agent reading file paths would state it as fact.
Here is the entire handler:
export default function handler(_req: NextApiRequest, res: NextApiResponse) {
res.status(404).json({ message: "Billing webhooks are not available in community edition" });
}
It is a stub. The open-source snapshot of Cal.com does not process billing webhooks. That behavior lives in their enterprise edition. The second webhook route in the repo is the same story.
DevTime could not find behavior evidence because there is no behavior to find. The refusal was the correct answer.
That is the whole thesis of the tool in one example. A confident story built from file names would have been wrong. The repository could not prove billing webhooks, so the scanner said so.
I promised misses, and there are real ones.
Background Jobs was under-called. Cal.com has a genuine job system: a Tasker package with an internal tasker, a Redis-backed tasker, and a task processor, plus cron routes. That is real worker and queue behavior, and DevTime only surfaced dependency and test evidence at low confidence. The guard that suppresses fake background-job signals (calendar cleanup crons pretending to be job queues) is suppressing a true positive here.
Admin Permissions was missed entirely. The repo has real admin surfaces, including admin API routes and admin settings layouts. DevTime detected nothing.
Both misses become regression fixtures now: small repo patterns encoding the expected output, so this exact failure can never return silently. Funny detail: two of the oldest fixtures in the suite came from Cal.com-shaped false positives (calendar "subscriptions" being mistaken for billing). Those fixtures fixed overclaiming. Today Cal.com exposed the opposite failure, overcorrection. Same factory, opposite direction.
One more honesty note: during the scan, DevTime printed this on its own:
Framework coverage warning: NestJS controller parsing is not supported in V0;
backend route coverage may be incomplete.
Cal.com's API v2 is NestJS. The tool told me where its own blind spot was before I found it.
Every tool in this space is optimized to sound sure. Summaries, autocomplete, agents: confidence is the product.
But the Cal.com webhook stub is exactly the kind of thing confident tools get wrong. The story "this repo handles Stripe webhooks" is plausible, well-supported by file names, and false. If your AI agent believed it, it might modify payment logic that does not exist in this edition, or trust dedupe behavior that is not there.
The rule I keep coming back to:
No claim without evidence. Weak evidence produces uncertainty, not confidence.
Uncertainty is not the tool failing. Uncertainty is the tool working.
pipx install devtime-ei
cd your-repo
dtc init
dtc scan
dtc concepts
There is also a read-only MCP server (dtc mcp start
) so coding agents can query the same evidence instead of guessing.
If DevTime gets something wrong on your repository, that is not an embarrassment, it is a contribution. Wrong outputs become fixtures, and fixtures are how this tool earns trust.
Local-first Engineering Intelligence for software repositories.
DevTime helps a codebase explain itself from evidence.
It scans code, tests, configs, routes, and decisions to identify supported software concepts, link claims to files, surface uncertainty, and warn about a narrow set of risky changes.
No cloud. No telemetry. No code execution. No AI required.
Prefer video? Watch the 2-minute demo: DevTime scans a repo locally, explains concepts from evidence, surfaces uncertainty, catches a risky diff, and shows how a corroborated decision improves understanding.
pipx install devtime-ei
dtc demo init
cd devtime-demo-saas
dtc init
dtc scan
dtc concepts
dtc explain "Billing Webhooks"
The PyPI distribution is devtime-ei
. The Python package remains devtime
, and the
CLI command remains dtc
. dtc demo init
copies a small static example repo into
./devtime-demo-saas
so you can try DevTime without cloning this repository.
git
…If you run it on a repo and it says something your code cannot back up, open an issue. That is exactly the feedback I am looking for!