{"slug": "i-scanned-cal-com-with-my-evidence-based-cli-the-best-result-was-what-it-refused", "title": "I scanned Cal.com with my evidence-based CLI. The best result was what it refused to say", "summary": "A developer scanned the open-source Cal.com repository with DevTime, a local-first CLI tool that analyzes codebases without AI. The tool correctly refused to claim billing webhooks existed, finding only a stub handler in the community edition. However, it missed background jobs and admin permissions, which will be added as regression fixtures.", "body_md": "I pointed my scanner at Cal.com this week, fully expecting to write a post about everything it got wrong.\n\nThe most interesting result was the thing it refused to say.\n\nQuick context: I build DevTime, a local-first CLI that scans a repository and helps it explain itself from evidence. It detects a closed set of concepts (authentication, billing webhooks, background jobs, data export, admin permissions, file uploads), links every claim to evidence files, and reports uncertainty when it cannot prove something. No cloud, no telemetry, no code execution during scan, no AI.\n\nCal.com is a well-known open-source scheduling platform. Big TypeScript monorepo, Next.js frontend, NestJS API, Stripe integration. If you wanted to stress a heuristic scanner, this is a good place to do it.\n\nThe run:\n\n```\npipx install devtime-ei\ngit clone --depth 1 https://github.com/calcom/cal.com\ncd cal.com\ndtc init\ndtc scan\n```\n\nThe scan:\n\n```\nScan complete. Scanned 5736 files, 24516 signals, 3 concepts in 4.2s.\n```\n\nLocal, 4.2 seconds, nothing left the machine.\n\n```\n1. Authentication      confidence: high   debt: medium\n2. Billing Webhooks    confidence: low    debt: high\n3. Background Jobs     confidence: low    debt: high\n```\n\nAuthentication was solid. Real evidence, real routes:\n\n```\n- Authentication is present and supported by behavior evidence.\n  confidence: 0.82\n  evidence: apps/web/app/api/auth/forgot-password/route.ts,\n            apps/web/app/api/auth/oauth/me/route.ts\n```\n\nBut Billing Webhooks at LOW confidence? Cal.com is a scheduling SaaS with Stripe integration. Of course it has billing webhooks. I assumed I had found my first miss.\n\nHere is what DevTime actually said:\n\n```\n- Possible Billing Webhooks signals detected. Billing Webhooks behavior\n  is not established from current evidence.\n  confidence: 0.45\n\nUncertainty:\n  - Only dependency or manifest evidence was found for Billing Webhooks;\n    presence is not confirmed behavior.\n```\n\nSo I went to check manually. The repo has exactly the file you would expect:\n\n```\napps/web/pages/api/stripe/webhook.ts\n```\n\nCase closed, right? A file named stripe/webhook.ts in a Stripe-integrated SaaS. Any developer grepping the repo would say \"here is their webhook handler\". Any AI agent reading file paths would state it as fact.\n\nHere is the entire handler:\n\n```\nexport default function handler(_req: NextApiRequest, res: NextApiResponse) {\n  res.status(404).json({ message: \"Billing webhooks are not available in community edition\" });\n}\n```\n\nIt is a stub. The open-source snapshot of Cal.com does not process billing webhooks. That behavior lives in their enterprise edition. The second webhook route in the repo is the same story.\n\nDevTime could not find behavior evidence because there is no behavior to find. The refusal was the correct answer.\n\nThat is the whole thesis of the tool in one example. A confident story built from file names would have been wrong. The repository could not prove billing webhooks, so the scanner said so.\n\nI promised misses, and there are real ones.\n\n**Background Jobs was under-called.** Cal.com has a genuine job system: a Tasker package with an internal tasker, a Redis-backed tasker, and a task processor, plus cron routes. That is real worker and queue behavior, and DevTime only surfaced dependency and test evidence at low confidence. The guard that suppresses fake background-job signals (calendar cleanup crons pretending to be job queues) is suppressing a true positive here.\n\n**Admin Permissions was missed entirely.** The repo has real admin surfaces, including admin API routes and admin settings layouts. DevTime detected nothing.\n\nBoth misses become regression fixtures now: small repo patterns encoding the expected output, so this exact failure can never return silently. Funny detail: two of the oldest fixtures in the suite came from Cal.com-shaped false positives (calendar \"subscriptions\" being mistaken for billing). Those fixtures fixed overclaiming. Today Cal.com exposed the opposite failure, overcorrection. Same factory, opposite direction.\n\nOne more honesty note: during the scan, DevTime printed this on its own:\n\n```\nFramework coverage warning: NestJS controller parsing is not supported in V0;\nbackend route coverage may be incomplete.\n```\n\nCal.com's API v2 is NestJS. The tool told me where its own blind spot was before I found it.\n\nEvery tool in this space is optimized to sound sure. Summaries, autocomplete, agents: confidence is the product.\n\nBut the Cal.com webhook stub is exactly the kind of thing confident tools get wrong. The story \"this repo handles Stripe webhooks\" is plausible, well-supported by file names, and false. If your AI agent believed it, it might modify payment logic that does not exist in this edition, or trust dedupe behavior that is not there.\n\nThe rule I keep coming back to:\n\n**No claim without evidence. Weak evidence produces uncertainty, not confidence.**\n\nUncertainty is not the tool failing. Uncertainty is the tool working.\n\n```\npipx install devtime-ei\ncd your-repo\ndtc init\ndtc scan\ndtc concepts\n```\n\nThere is also a read-only MCP server (`dtc mcp start`\n\n) so coding agents can query the same evidence instead of guessing.\n\nIf DevTime gets something wrong on your repository, that is not an embarrassment, it is a contribution. Wrong outputs become fixtures, and fixtures are how this tool earns trust.\n\n**Local-first Engineering Intelligence for software repositories.**\n\nDevTime helps a codebase explain itself from evidence.\n\nIt scans code, tests, configs, routes, and decisions to identify supported software concepts, link claims to files, surface uncertainty, and warn about a narrow set of risky changes.\n\nNo cloud. No telemetry. No code execution. No AI required.\n\nPrefer video? [Watch the 2-minute demo](https://youtu.be/1Hiu3Y9J_SI): DevTime scans\na repo locally, explains concepts from evidence, surfaces uncertainty, catches a\nrisky diff, and shows how a corroborated decision improves understanding.\n\n```\npipx install devtime-ei\ndtc demo init\ncd devtime-demo-saas\ndtc init\ndtc scan\ndtc concepts\ndtc explain \"Billing Webhooks\"\n```\n\nThe PyPI distribution is `devtime-ei`\n\n. The Python package remains `devtime`\n\n, and the\nCLI command remains `dtc`\n\n. `dtc demo init`\n\ncopies a small static example repo into\n`./devtime-demo-saas`\n\nso you can try DevTime without cloning this repository.\n\n```\ngit\n```\n\n…If you run it on a repo and it says something your code cannot back up, open an issue. That is exactly the feedback I am looking for!", "url": "https://wpnews.pro/news/i-scanned-cal-com-with-my-evidence-based-cli-the-best-result-was-what-it-refused", "canonical_source": "https://dev.to/shakargy/i-scanned-calcom-with-my-evidence-based-cli-the-best-result-was-what-it-refused-to-say-7gj", "published_at": "2026-07-08 21:54:36+00:00", "updated_at": "2026-07-08 22:11:08.559133+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Cal.com", "DevTime", "Stripe"], "alternates": {"html": "https://wpnews.pro/news/i-scanned-cal-com-with-my-evidence-based-cli-the-best-result-was-what-it-refused", "markdown": "https://wpnews.pro/news/i-scanned-cal-com-with-my-evidence-based-cli-the-best-result-was-what-it-refused.md", "text": "https://wpnews.pro/news/i-scanned-cal-com-with-my-evidence-based-cli-the-best-result-was-what-it-refused.txt", "jsonld": "https://wpnews.pro/news/i-scanned-cal-com-with-my-evidence-based-cli-the-best-result-was-what-it-refused.jsonld"}}