Nine of my API tests were getting 401 Unauthorized instead of 200. Keploy said the run passed. Exit code 0.
That was the last thing I found. Here's how I got there.
Honestly, the first reason is that I'm applying for the DevRel internship at Keploy. The second is that I wanted to see how API testing actually works on a real app.
TaskFlow is my MERN project management app. It has an eval harness for its AI features, but the Express API itself had no tests at all. So I pointed Keploy at it and wrote down everything that happened, including the parts that went wrong.
keploy record -c "node server.js"
Use node, not nodemon. A restarting process confuses the recorder.
My first attempt just hung for five and a half minutes with no output. It turned out my Desktop folder syncs to iCloud, and most of node_modules had been offloaded, so Node was waiting on iCloud to download files one by one. That's not Keploy's fault. But Keploy also never told me that nothing was listening.
When I stopped recording, Keploy immediately replayed what it had just recorded. I didn't ask it to; that's the default. 7 of 12 tests failed.
I did not expect 7 of my endpoints to fail. Then I read the diffs, and none of them were bugs. They were values that are different every time:
_id s.Etag headers.joinedAt. globalNoise in keploy.yml:
test:
globalNoise:
global:
body:
project._id: ['^[0-9a-f]{24}$']
task._id: ['^[0-9a-f]{24}$']
workspace._id: ['^[0-9a-f]{24}$']
workspace.members._id: ['^[0-9a-f]{24}$']
user.id: ['^[0-9a-f]{24}$']
workspace.members.joinedAt: ['^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$']
accessToken: ['^eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$']
header:
Etag: ['^W/"[0-9a-f]+-[A-Za-z0-9+/]{27}"$']
Set-Cookie: ['^refreshToken=eyJ[A-Za-z0-9_.-]+; Max-Age=604800; Path=/; Expires=[^;]+; HttpOnly; SameSite=Lax$']
I used regex patterns instead of empty arrays on purpose. An empty array ignores the field completely. A pattern still checks that the new value looks right, so a null or a malformed ID would still fail.
This was the part I didn't expect. Keploy records every database and network call, so reading the mocks felt like reading my backend's diary.
1. A background write I'd forgotten about. When a task is created, TaskFlow computes an embedding and saves it to MongoDB, but it doesn't wait for that before responding. Keploy caught these writes landing in the next test's time window. When they failed, my app just logged embedding failed and moved on. No error, nothing visible to the user.
2. PUT /api/tasks/:id makes 33 database calls. The next-highest endpoint makes 9. I have no idea why yet, and I'm going to find out.
3. A 37 MB download hiding at runtime. The first time a task is created, TaskFlow downloads an embedding model from HuggingFace. Keploy captured all of it, and that one download made up about 98% of mocks.yaml. After I pre-downloaded the model outside Keploy, mocks.yaml went from 37.7 MB to 286 KB.
With the noise config and the model cached, I re-recorded and ran:
keploy test -c "node server.js" --delay 10
12 out of 12 passed.
The part I'd actually use every day: the Groq call is mocked too. The AI endpoint test replays the recorded LLM response instead of calling Groq. So it's deterministic, it's free, and it doesn't break because the model phrased something differently today.
One honest caveat: the background embedding write from point 1 is still a race. It only lined up on this run because the cached model made everything faster. The test is green, but the race is still there.
My access tokens expire after 15 minutes. Keploy replays the exact token it recorded. So I waited and ran the tests again, 16 minutes after recording.
Nine tests got 401 Not authorized instead of the recorded 200/ 201. Here's what Keploy reported:
)
Default run, 16 minutes after recording.
The other nine were marked obsolete, not failed. The report said PASSED, and the process exited with code 0. In CI, that's a green check on top of nine broken endpoints.
Keploy's summary said the mocks were probably stale and suggested re-recording, or running with --update-test-mapping. But the real cause was the 401. Updating the mappings would have quietly rewritten the tests to match the broken behavior.
Here's why it happens:
--help text for --assert-dependencies says such a test is "demoted to OBSOLETE today and the run still exits 0." I just hadn't read it until I went looking.
That's what stuck with me: not all bugs are easy to see, and they can still be fatal. A test suite that can't go red isn't protecting you.
What about --freezeTime? It's meant for exactly this: it pins the app's clock to the recording time. But the docs list it as an Enterprise feature for Linux, WSL and Docker. On native macOS, Keploy tried to inject a Linux library, logged an error saying time freezing couldn't be verified, and ran the tests anyway. Same result.
Same recording, same expired tokens:
| Run | Passed | Failed | Obsolete | Report | Exit code |
|---|---|---|---|---|---|
| Default | 3 | 0 | 9 | PASSED | 0 |
--strict-failure |
3 | 9 | 0 | FAILED | 1 |
--assert-dependencies |
3 | 9 | 0 | FAILED | 1 |
)
Same tests, same expired tokens, with --strict-failure.
--strict-failure fails a test when its expected=200 got=401) instead of blaming stale mocks.
If you run Keploy in CI, add one of these. I'd use both.
A short checklist, because I learned this the hard way:
mocks.yaml in plain text. Check your mocks before you commit or share anything.test.disableMockUpload: true as well..gitignore will skip the one file most likely to contain your keys.
I rotated my Groq key. Twice.
Yes, to check your code's reliability before it hits production. Recording real traffic and getting tests plus mocks back, including for LLM calls, is genuinely useful, and the logs explained almost everything I ran into.
My one warning: don't trust the default green. Run with --strict-failure, and read your mocks for secrets.
My last big bug was a token cap silently truncating JSON in 26% of TaskFlow's AI requests, and the lesson here was the same: a green check isn't proof that things work. I wrote about that one here.