Search for a command to run...
Yusuf Adeyemo is a DevOps Engineer from Nigeria. He loves helping startups deliver better software and provide more control over their environment and software development process with the help of modern tools and automation.
No comments yet. Be the first to comment.
We’ve all been there. You spend hours perfectly tuning your OpenWrt configuration, setting up your captive portals, and hardening your firewall. Then, six months later, the system hangs. Why? Because
This article started from a tweet. Someone on Twitter said they "lowkey want to understand the technology behind Google Authenticator" and I dropped a quick reply - explaining that it's basically TOTP: your device and the server share a secret key, b...
After you complete this article, you will have a solid understanding of: Why deploying Next.js SSR on Cloudflare is different from traditional hosting The real difference between Cloudflare Pages with next-on-pages and OpenNext adapter How the fre...
After you complete this article, you will have a solid understanding of: What L1, L2, and L3 constructs actually are and when to use each Why AWS created three different abstraction levels (and the hidden benefits) How to avoid the most common CDK...
Have you ever sat there waiting for your CodeBuild project to rebuild your entire Docker image... again? Even though you only changed a single line of code?
Yeah, me too. And it's frustrating.
Today, I'm going to show you how I reduced our Docker build times from ~7 minutes down to ~5 minutes (that's about 25-30% faster!) by implementing Amazon ECR as a persistent cache backend. This is based on an official AWS blog post, but I'll walk you through the practical implementation.
Here's the thing about AWS CodeBuild: every build runs in a completely fresh, isolated environment. That means:
No build artifacts carry over between builds
Every build starts from scratch
CodeBuild's "local cache" is temporary and unreliable (works on a "best-effort" basis)
If your builds happen at different times throughout the day, the local cache probably isn't helping you
So even if you only changed one line in your code, CodeBuild rebuilds every single Docker layer. Every. Single. Time.
The solution is surprisingly elegant: store your Docker layer cache persistently in Amazon ECR (Elastic Container Registry). Think of it as a separate "cache image" that lives alongside your actual application image.
Here's how it works:
First Build: Build from scratch, then export the cache to ECR as a separate image
Subsequent Builds: Import the cache from ECR, rebuild only what changed, export the updated cache back
The beauty? Your cache is always available, no matter when you trigger a build.
Before we start, make sure you have:
An existing AWS CodeBuild project that builds Docker images
An ECR repository where your images are stored
IAM permissions for your CodeBuild role to push/pull from ECR (if you can already push images, you're good!)
About 10 minutes to implement this
Your current buildspec probably looks something like this:
version: 0.2
phases:
install:
commands:
- aws ecr get-login-password | docker login ...
build:
commands:
- docker build -t myapp:latest .
- docker tag myapp:latest $ECR_REPO:latest
post_build:
commands:
- docker push $ECR_REPO:latest
This is the "basic" approach. Every build starts from zero.
[📸 IMAGE SUGGESTION: Split screen showing "Basic Build" vs "Cached Build" with layer rebuilding visualization]
First, let's define a separate tag for our cache image. In your install phase, add:
install:
commands:
- CACHE_TAG=dev-cache # or prod-cache, staging-cache, etc.
- IMAGE_TAG=latest # your actual app image tag
This creates a separate cache image (e.g., myapp:dev-cache) that's distinct from your application image (myapp:latest).
Here's the key part: Docker's default builder doesn't support registry cache backends. We need to create a new builder using buildx with the containerd driver.
Add this to your install phase:
install:
commands:
# ... your existing commands ...
- docker buildx create --name containerd --driver=docker-container --driver-opt default-load=true --use || docker buildx use containerd
What's happening here?
docker buildx create: Creates a new builder instance
--driver=docker-container: Uses containerd (required for registry cache)
--driver-opt default-load=true: Loads built images into local Docker (important!)
|| docker buildx use containerd: If the builder already exists, just switch to it
Now replace your regular docker build command with the new buildx version:
build:
commands:
- |
docker buildx build \
--builder=containerd \
--cache-from type=registry,ref=$ECR_REPO:$CACHE_TAG \
--cache-to type=registry,ref=$ECR_REPO:$CACHE_TAG,mode=max,image-manifest=true \
-t $ECR_REPO:$IMAGE_TAG \
--load \
.
Let me break down what each flag does:
--builder=containerd: Use the builder we just created
--cache-from type=registry,ref=$ECR_REPO:$CACHE_TAG: Import cache from ECR
--cache-to type=registry,ref=$ECR_REPO:$CACHE_TAG,mode=max,image-manifest=true: Export cache back to ECR
mode=max: Export all layers (recommended for best caching)
image-manifest=true: Required for ECR storage
-t $ECR_REPO:$IMAGE_TAG: Tag your final image as usual
--load: Load the built image into local Docker (so you can run it in post_build)
.: Your Dockerfile location
Here's what a complete, production-ready buildspec looks like:
version: 0.2
phases:
install:
commands:
- echo Logging in to Amazon ECR
- aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
- CACHE_TAG=dev-cache
- IMAGE_TAG=latest
- ECR_REPO=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/myapp
- docker buildx create --name containerd --driver=docker-container --driver-opt default-load=true --use || docker buildx use containerd
build:
commands:
- echo Build started on `date`
- |
docker buildx build \
--builder=containerd \
--cache-from type=registry,ref=$ECR_REPO:$CACHE_TAG \
--cache-to type=registry,ref=$ECR_REPO:$CACHE_TAG,mode=max,image-manifest=true \
-t $ECR_REPO:$IMAGE_TAG \
--load \
.
- docker tag $ECR_REPO:$IMAGE_TAG $ECR_REPO:latest
post_build:
commands:
- echo Build completed on `date`
- docker push $ECR_REPO:$IMAGE_TAG
- docker push $ECR_REPO:latest
artifacts:
files:
- imageDefinitions.json
You can update your buildspec in two ways:
Option 1: If your buildspec is in your repo Just commit the changes and push. CodeBuild will pick up the new buildspec automatically.
Option 2: If your buildspec is defined in CodeBuild Use the AWS CLI:
aws codebuild update-project --name your-project-name --cli-input-json file://buildspec.json
Or update it through the AWS Console: CodeBuild → Your Project → Edit → Buildspec
Your first build after implementing this will actually take slightly longer (maybe 30-60 seconds more). Don't panic! This is normal.
Here's what's happening:
Creating the buildx builder (~5-10 seconds)
Attempting to import cache (fails - no cache exists yet)
Building all layers from scratch
Exporting the cache to ECR (new step, adds ~20-40 seconds)
You'll see messages like:
=> importing cache manifest from $ECR_REPO:dev-cache
=> error: not found
This is expected! The cache doesn't exist yet.
This is where the magic happens. Your next builds will:
Successfully import the cache from ECR
Identify which layers haven't changed
Reuse cached layers (fast!)
Rebuild only the changed layers
Export the updated cache
Expected time savings:
Before: 6-7 minutes (full rebuild every time)
After: 5-5.5 minutes (25-30% faster!)
Savings: 1-2 minutes per build
If you're doing 10 builds a day, that's 10-20 minutes saved daily. Over a month? That's 5-10 hours of compute time and costs saved.
After your first build completes, check your ECR repository. You should now see two image tags:
Your application image (e.g., latest)
Your cache image (e.g., dev-cache)
The cache image will be roughly the same size as your application image - this is normal! It's storing all the layer information.
Solution: Update your CodeBuild image to a newer version. Use aws/codebuild/standard:7.0 or later (or the ARM equivalent).
Solution: Check your IAM permissions. Your CodeBuild role needs:
ecr:BatchGetImage
ecr:GetDownloadUrlForLayer
ecr:BatchCheckLayerAvailability
ecr:PutImage
ecr:InitiateLayerUpload
ecr:UploadLayerPart
ecr:CompleteLayerUpload
Solution: Make sure privilegedMode: true is enabled in your CodeBuild environment settings. This is required for Docker-in-Docker operations.
If you have multiple environments (dev, staging, prod), use different cache tags for each:
- CACHE_TAG=${ENVIRONMENT}-cache # Results in: dev-cache, staging-cache, prod-cache
This way:
Dev builds don't invalidate staging cache
Each environment maintains its own optimized cache
You can still share a base cache if needed
Storage Cost: You're now storing an additional cache image in ECR. At roughly the same size as your app image, this might add $0.10-0.50/month per repository depending on image size.
Compute Savings: Faster builds = less compute time. If you're saving 1-2 minutes per build and doing 10 builds/day, that's roughly 10-20 fewer compute hours per month. At ~$0.005/minute for BUILD_GENERAL1_SMALL, you could save $3-6/month.
Net Result: Typically a small net savings, plus the huge developer experience win of faster feedback loops.
By implementing ECR as a remote cache backend for your CodeBuild Docker builds, you get:
✅ Faster build times
✅ Persistent, reliable caching across all builds
✅ Better layer reuse with intelligent cache management
✅ Minimal code changes (just updating your buildspec)
✅ Cost savings from reduced compute time
The implementation is straightforward, and the benefits are immediate (after the first build). Give it a try on your next project!
Got questions or run into issues? Drop a comment below - I'd love to hear about your experience implementing this!