Last week I wrote about tests that pass on your machine and fail in CI. I listed five usual suspects: speed, leftover state, environment, the network path and parallel runs.
Then the comments arrived, here and on Reddit, LinkedIn and the Ministry of Testing Club. Testers shared failures that took them a day, a week, and in one case over a year to solve. Most of them weren't on my list.
So this is part 2: the causes I learned from you, each with the fix that worked, and then a small CI setup that makes the next mystery quicker to solve.
One reader had tests that always passed locally and always failed in CI. The logs and screenshots showed an empty field. After a day of debugging: the mouse pointer.
On the CI runner, the pointer happened to sit over an element that opened a popup menu. The menu covered the field the test was filling, so the text went nowhere. The field was optional, so nothing failed there. The test blew up much later, at a step that had nothing to do with the cause.
Two habits would have saved most of that day:
// Make the starting state explicit instead of "wherever the pointer is"
await page.mouse.move(0, 0);
await page.getByLabel('Reference').fill('ABC-123');
// Assert right after you act, so the failure points at the real step
await expect(page.getByLabel('Reference')).toHaveValue('ABC-123');
Lesson: assert where you act. A failure three steps later sends you looking in the wrong place.
fs.readdir, glob and friends don't promise any order. The order you get depends on the filesystem. Locally it looks alphabetical every time, so nobody notices that the test depends on it. On the runner, it isn't.
Any test that takes "the first file" or compares a list without sorting is waiting to break.
const files = (await fs.promises.readdir(dir)).sort();
The same applies to database queries without an ORDER BY, API results without an explicit sort order, and rows in a table that your test reads as a list. Locally they usually come back in insertion order by coincidence; a fresh CI database with a different disk layout or query plan can flip it.
Most CI checkouts fetch only the latest commit and no tags. Anything that calls git describe, reads the last tag for a version number, or diffs against main works on your machine and fails, or quietly returns nonsense, in CI.
In GitHub Actions, fetch full history for the one job that needs it, not every job:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # full history and tags
This one was the reverse of the usual story. The build succeeded locally and in CI, but one component broke at runtime on a developer's machine.
The cause: CI couldn't read the lockfile in the repo, so it regenerated it on every run. CI and local ended up with different versions of a worker package. The team followed what CI did step by step and saw no difference, because the difference was in what got installed, not in the steps.
Two fixes make this kind of drift fail loudly at install time instead of at runtime:
.nvmrc or the engines field of package.json
npm ci in CI, not npm install. It installs exactly what's in the lockfile and fails if the lockfile is out of sync with A reader writing unit tests for kernel flags had dockerized builds and still got caught. Containers share the host's kernel. The newer kernel on their laptop supported flags that the CI runner's kernel didn't.
uname -r at the start of the CI job, so the kernel version is in every log
I wrote last week that tests sharing data can fail when run in parallel. A reader on X pointed out the scarier half: they can also pass when they shouldn't.
Two workers share a basket. One test passes because it finds the item the other test added. A false green is worse than a failure, because nobody looks at it.
test('adds an item to the basket', async ({ page }, testInfo) => {
// Each test gets its own basket, so nothing else can make it pass
const basketName = `basket-${testInfo.testId}`;
// ...create basketName, add the item, then assert on that basket only
});
Assert on the specific record you created, not on "an item exists".
Timezone was in part 1, but readers added the sneakiest version: a test that fails for only an hour or two each day, when local time and UTC fall on different dates. Everyone assumes it's flaky until someone notices it always fails late in the evening.
Set the timezone explicitly, and control the clock in tests that care about dates:
// playwright.config.ts
use: { timezoneId: 'Europe/London', locale: 'en-GB' }
// in a test that depends on "today"
await page.clock.setFixedTime(new Date('2026-11-09T23:30:00Z'));
The same goes for localeCompare and number or date formatting. Minimal CI images often have different locale data, so pass an explicit locale in tests.
Several replies came back to the same idea, and @alexshev put it best: CI failures are often environment mismatches disguised as code failures. The fix is evidence, not guesswork.
1. Print the environment at the start of every job. It costs seconds, and it means local versus CI becomes two lists you can compare instead of two memories:
- name: Environment
run: |
node -v
npx playwright --version
uname -r
date
echo "TZ=$TZ LANG=$LANG"
2. Keep the evidence when a run fails. Save the Playwright report and traces as artifacts, even when the tests fail:
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report
path: playwright-report/
retention-days: 14
3. Save the generated input, not just the seed. This was @alexshev's point, and it changed how I think about test data. He records the seed, generator version, locale, timezone and fixture version on every run. For a failed CI run, he also keeps the actual generated input (scrubbed if needed) alongside the trace for a short time. In his words, a seed alone is not enough once a dependency, clock, locale or generator behaviour changes. Once the bug is fixed, that exact input becomes a regression fixture.
In Playwright, attaching the input to the failing test takes one line:
await testInfo.attach('generated-input', {
body: JSON.stringify(input, null, 2),
contentType: 'application/json',
});
4. Decide who looks first. A question I got on Reddit: when tests break after a merge, is it the tester's job or the SDETs'? It depends on the team, but a common default works well. If the tests fail on your pull request, you look first. If something breaks after merge, whoever's change caused it investigates, and whoever owns the pipeline handles runner and config problems. The worst option is a red build that everyone assumes someone else is fixing.
Same view as part 1, sharpened by the comments. AI assistants are good at reading a long CI log, explaining an unfamiliar error and suggesting theories. Give one the environment printout from a passing local run and a failing CI run, and it will often spot the difference faster than you.
What it can't do is see your runner. Nobody's AI would have guessed the mouse pointer or the kernel version without that information in front of it. The evidence from the setup above is what makes its suggestions worth anything.
Twelve causes so far between the two posts, and I suspect we're nowhere near the end. What's yours? I'll keep collecting them, and I plan to turn the best ones into hands-on practice challenges.
Thank you to everyone who shared a story, and to @alexshev for letting me quote him.
I'm building Locator League, browser-based Playwright challenges where you write the test and it runs for real. I plan to turn some of the stories above into a "Works on my machine" practice track.