Stop waiting 10 minutes for CI to rebuild everything when you change one line of code. Here's what's actually breaking your Docker layer cache." #
This post is about Docker layer caching and why your builds probably take way longer than they should. If you've ever sat there watching Docker reinstall npm packages for the 10th time today after changing one line, yeah this is for you.
The Pain #
Okay so. You know that feeling, right? You changed literally ONE line of code. Fixed a typo in a comment or whatever. Push it up, CI kicks off, and now you're sitting there watching Docker reinstall 400 npm packages. Again. For the third time today.
And you're like "why is this happening to me"
Here's the thing - your build isn't slow because your code sucks or because AWS is being slow. It's slow because you wrote your Dockerfile wrong. We all do it.
I'm willing to bet you've got something like:
FROM node:18
COPY . .
RUN npm install
Yeah. That's the problem right there. Every code change = full npm install. Docker's layer cache? Gone. Destroyed. RIP.
After explaining this to coworkers (and myself) like 50 times, I finally got fed up and built LayerLint. It's basically a tool that looks at your Dockerfile and tells you when you messed up.
(Also it was a good excuse to write some Go code. But mainly the other thing.)
How Docker Caching Actually Works (The 5-Minute Version) #
So nobody really explains this when you're learning Docker. They just tell you to write a Dockerfile and good luck. But here's what's actually happening: every line in your Dockerfile creates a layer. These layers get cached. Which is great! Except the cache breaks super easily.
Best way I can explain it - imagine a stack of pancakes. (I'm hungry, don't judge.) Each line (FROM
, RUN
, COPY
, whatever) is one pancake. Docker caches each pancake. So far so good.
But then. Here's where it gets annoying: if you change one pancake, Docker throws away that pancake plus every single pancake above it.
So like, if you change pancake 3, pancakes 3, 4, 5, 6, and 7 all get tossed. Only pancakes 1 and 2 stay cached.
This whole thing is called the "invalidation chain" and it's literally why your builds take forever.
The Cache-Killer Pattern
Here's the classic mistake everyone makes (including me for like 2 years):
FROM node:18
WORKDIR /app
COPY . . # Layer 1: Copy everything
RUN npm install # Layer 2: Install dependencies
RUN npm run build # Layer 3: Build
CMD ["node", "dist/server.js"]
Okay so what happens when you change src/index.js
?
- Layer 1 (
COPY . .
) sees something changed ❌ - Cache = broken
- Layer 2 (
npm install
) has to run again ❌ - Layer 3 (
npm run build
) runs again too ❌
You literally just reinstalled 400 packages because you changed one file. And Docker's sitting there like "yep, seems right".
The Fix
Alright here's the actual correct way:
FROM node:18
WORKDIR /app
COPY package.json package-lock.json ./ # Layer 1: Copy deps manifest
RUN npm install # Layer 2: Install (cached!)
COPY . . # Layer 3: Copy source
RUN npm run build # Layer 4: Build
CMD ["node", "dist/server.js"]
Now when you change src/index.js
:
- Layer 1 (
package*.json
) - didn't change ✅ (uses cache) - Layer 2 (
npm install
) - didn't change ✅ (uses cache) - Layer 3 (
COPY . .
) - okay this changed, rebuild - Layer 4 (
npm run build
) - gotta rebuild this too
But look - layers 1 and 2 stayed cached! You just saved like 2-8 minutes every single build.
This pattern works everywhere btw:
- Node →
package.json
+package-lock.json
- Go →
go.mod
+go.sum
- Python →
requirements.txt
orpoetry.lock
- Rust →
Cargo.toml
+Cargo.lock
Same idea. Copy the dependency files first, install them, THEN copy your actual code.
Meet LayerLint: The Dockerfile Linter You Didn't Know You Needed #
So I've seen this same mistake approximately 47 times. Someone (usually me) puts COPY . .
before npm install
and then wonders why builds take forever. Eventually I just got annoyed enough to build something about it.
That's LayerLint.
What is it?
It's a static analysis tool I wrote in Go. You point it at your Dockerfile, it reads through it and goes "yo, this is gonna be slow". Doesn't even need to build the image. Just looks at the file.
Why not use hadolint or whatever?
Hadolint is great for syntax stuff and general best practices. LayerLint is specifically focused on layer caching anti-patterns. The stuff that makes your builds slow. Different problem.
Think of it like having that one DevOps person who's always grumpy about Dockerfiles, except it runs instantly and won't send you passive-aggressive Slack messages.
The Walkthrough: Let's Break Some Stuff #
Installation (Pick Your Poison)
There's like 3 different ways to install it depending on how paranoid you are:
The lazy way (this is what I use):
curl -sSL https://raw.githubusercontent.com/vviveksharma/layerLint/main/install.sh | sh
The "I don't trust random install scripts" way (fair enough):
wget https://github.com/vviveksharma/layerLint/releases/latest/download/layerLint_Linux_x86_64.tar.gz
tar -xzf layerLint_Linux_x86_64.tar.gz
chmod +x layerlint
The "I'm gonna build it from source" way (respect):
git clone https://github.com/vviveksharma/layerLint
cd layerLint
make generate-build
The Scan
Let me use that bad Dockerfile from before:
FROM golang:1.22
WORKDIR /app
COPY . .
RUN go mod download
RUN go build -o server ./cmd/server
Now run LayerLint on it:
./layerlint scan --dockerfile Dockerfile
The Output
╔══════════════════════════════════════════════════════════════════╗
║ LayerLint Report ║
╚══════════════════════════════════════════════════════════════════╝
RuleID: dockerfile/broad-copy-before-deps
Severity: high
File: Dockerfile
Line: 3
Title: Dependency install runs after broad source copy
Message: This dependency step runs after a broad COPY/ADD, so source
changes can invalidate the dependency cache.
Suggestion: Copy dependency manifests first (go.mod, go.sum), install
dependencies, then copy the rest of the source.
Example Fix:
COPY go.mod go.sum ./
RUN go mod download
COPY . .
═══════════════════════════════════════════════════════════════════
Found 1 violation:
- High: 1
- Medium: 0
- Low: 0
Pretty nice right? It basically tells you:
- What you did wrong (broad copy before deps)
- Where exactly it is (line 3, can't miss it)
- Why this is bad (breaks the cache)
- How to actually fix it (copy manifests first)
No guessing. No googling "why is my docker build slow reddit". Just tells you straight up.
Other Rules It Catches
LayerLint checks for a bunch of other stuff too that'll eventually bite you:
| Rule | Severity | Why You Should Care |
|---|---|---|
unpinned-base-image-tag |
||
| Medium | ||
:latest means your builds aren't reproducible (bad) |
||
copying-secrets-into-image |
||
| High | Secrets live forever in layer history even if you delete them | |
run-as-root |
||
| High | Security issue, containers shouldn't run as root | |
missing-dockerignore |
||
| Medium | Slow builds + might leak secrets | |
apt-update-without-install |
||
| Medium | Package cache goes stale | |
build-without-cache-mount |
||
| Low | Missing BuildKit cache mounts (free speed) |
There's more. Check the full rules docs if you're curious.
The Real Win: Automation with GitHub Actions #
Okay so finding issues is one thing. But the real win? Making it so nobody (including yourself in 3 months when you forget all this) can merge a bad Dockerfile.
Here's what you do. Add this to .github/workflows/docker-lint.yml
:
name: Lint Dockerfile
on: [push, pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install LayerLint
run: |
curl -sSL https://raw.githubusercontent.com/vviveksharma/layerLint/main/install.sh | sh
- name: Scan Dockerfile
run: ./layerlint scan --dockerfile ./Dockerfile --fail-on-severity high
And boom. Now every PR gets checked automatically. Someone tries to merge a slow Dockerfile? CI says no. PR blocked. They gotta fix it first.
(Saved me from myself more times than I can count.)
Oh and LayerLint can output different formats if you need:
--format json
if you're doing scripting stuff -
--format sarif
if you want it in GitHub's Security tab -
--format html
for when you need to show management something pretty
Real-World Example Workflows
I put some example workflows in the repo that you can just copy:
Basically just copy the yaml, change the paths to match your repo, commit it. Done.
The Before & After: Show Me the Numbers #
Alright let me show you actual numbers from a real project I fixed.
Setup: Node.js app, 342 npm packages (yeah it's a lot don't judge), changed one line in a component
Before (the bad way):
FROM node:18
COPY . .
RUN npm ci
RUN npm run build
Build time: 8 minutes 32 seconds
Every. Single. Time.
After (fixed it):
FROM node:18
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
Build time: 1 minute 12 seconds
That's 7 minutes saved per build. I push like 10 times a day sometimes (don't we all?), so that's an hour saved. Every day. Just from fixing the Dockerfile.
Do the yearly math on that and it's honestly kinda crazy. That's like... multiple work weeks just sitting there waiting for npm install.
And this is actual time. Not theoretical. Real minutes you get back to:
- Actually write code
- Read docs (lol who am I kidding)
- Get coffee
- Scroll through Twitter/X or whatever we're calling it now
- Take a walk
- Pet your dog
- Literally anything that isn't watching a progress bar slowly fill up
Other Platforms (Because Not Everyone Uses GitHub) #
LayerLint works on whatever CI system you're using:
GitLab CI:
docker-lint:
script:
- curl -sSL https://raw.githubusercontent.com/vviveksharma/layerLint/main/install.sh | sh
- ./layerlint scan --dockerfile Dockerfile
CircleCI:
- run:
name: Lint Dockerfile
command: |
curl -sSL https://raw.githubusercontent.com/vviveksharma/layerLint/main/install.sh | sh
./layerlint scan --dockerfile Dockerfile
Jenkins:
sh 'curl -sSL https://raw.githubusercontent.com/vviveksharma/layerLint/main/install.sh | sh'
sh './layerlint scan --dockerfile Dockerfile'
More platforms in the CI/CD guide.
Pre-commit hook (catch it before you even commit):
Add this to .pre-commit-config.yaml
:
repos:
- repo: local
hooks:
- id: layerlint
name: LayerLint
entry: layerlint scan --dockerfile
language: system
files: Dockerfile.*
Now it runs every time you commit. Catches mistakes before they even get pushed.
Pro-Tips for Maximum Speed #
1. Use .dockerignore
(seriously please)
I cannot stress this enough. Create a .dockerignore
file. Just do it:
node_modules/
.git/
dist/
build/
*.log
.env*
*.md
.github/
tests/
Docker won't send all that junk to the build context. Faster builds, smaller images, and you won't accidentally leak your .env
file into production.
(I may or may not have done that once. Not fun. Learn from my mistakes.)
2. Enable BuildKit Cache Mounts
If you're not using BuildKit yet... start. And then use cache mounts:
RUN --mount=type=cache,target=/root/.npm \
npm ci
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download
RUN --mount=type=cache,target=/root/.cache/pip \
pip install -r requirements.txt
This caches package downloads between builds. Not just within one build - between ALL of them. Absolute game changer.
First time I set this up I thought something was broken because the build finished so fast. Nope, just working correctly for once.
3. Multi-Stage Builds for Production
Also if you're shipping to prod, use multi-stage builds:
FROM node:18 AS builder
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM node:18-alpine
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
CMD ["node", "dist/server.js"]
Your final image ends up way smaller, and deploys are faster. Win-win.
Contributing (Or: "I Found a Bug / Have an Idea") #
Code's pretty straightforward if you wanna contribute. Each rule is just its own file in internal/rules/
. Wanna add a new rule?
-
Create
internal/rules/your_rule.go -
Implement the
Rule
interface:
type Rule interface {
ID() string
Check(dockerfile *parser.Result) []Finding
}
-
Register it in
internal/scanner/scanner.go -
Add a test case in
testFiles/
Look at broad_copy_before_deps.go to see how it's done.
PRs welcome. If you find a caching anti-pattern that LayerLint misses, definitely add it. I'm sure there's stuff I haven't thought of. The Go parser stuff is pretty straightforward once you get into it.
(First few times looking at the Dockerfile parser I was confused but it makes sense eventually.)
Why I Built This #
Honestly? I got tired of:
- Sitting around waiting for builds (so much time wasted)
- Explaining the same Docker caching stuff over and over
- Forgetting these rules myself and having to relearn them every few months
- Watching our CI bill go up because of dumb mistakes (my manager wasn't happy about that one)
So yeah. Built a tool that explains it for me. Now when someone asks I just go "run layerlint" and we're good. Plus I can share the repo instead of typing the same explanation in Slack for the 100th time.
If this saves you 5 minutes a day, cool. If it saves your whole team hours every week? Even better. That's the goal.
The Bottom Line #
Before:
- Builds take forever
- Burning through CI minutes (and money)
- Everyone's annoyed
- "Why is this so slow?" (nobody knows)
After:
- Builds are actually fast
- Cache works like it's supposed to
- CI catches bad Dockerfiles automatically
- You get your time back
The tool's free, it's open source, and it literally takes 30 seconds to run. So like... just try it? Worst case you wasted 30 seconds. Best case you save hours.
I mean you've read this far, might as well give it a shot right?
Get Started #
Alright so if you wanna try it:
curl -sSL https://raw.githubusercontent.com/vviveksharma/layerLint/main/install.sh | sh
./layerlint scan --dockerfile Dockerfile
Useful links:
GitHub:github.com/vviveksharma/layerLint - All the rules:docs/rules.md - CI setup guide:docs/ci-cd-integration.md - Download page:github.com/vviveksharma/layerLint/releases
If it helps, star the repo maybe? If it doesn't help, open an issue and tell me what's broken.
Anyway. Go fix your Dockerfiles. Future you will be grateful.
MIT License. Built it because I got lazy and tired of explaining Docker caching.
Comments Section Starter #
What's your worst Dockerfile horror story? I wanna hear it. Drop it in the comments.
Bonus points if it involved npm install
in production or a 2GB Docker image for a hello world app.