Articles on DevOps, infrastructure engineering, CI/CD, and production systems by Yasas Banuka. https://www.iamyasasbanuka.me/blog en-US Sun, 07 Jun 2026 11:11:50 GMT ybanuka2003@gmail.com (Yasas Banuka) ybanuka2003@gmail.com (Yasas Banuka) https://www.iamyasasbanuka.me/blog/docker-zero-to-production-part-3-images-layers-builds https://www.iamyasasbanuka.me/blog/docker-zero-to-production-part-3-images-layers-builds Wed, 20 May 2026 00:00:00 GMT ybanuka2003@gmail.com (Yasas Banuka) docker dockerfile devops containers builds Watch what happens with this typical Dockerfile: ```dockerfile FROM node:22-alpine COPY . . # copies ALL your code RUN npm install # installs all dependencies ``` You change one line in your `index.js`. Just one line. What happens? - `COPY . .` detects that files changed, so the layer is invalidated - `RUN npm install` comes after it, so it also runs from scratch - npm downloads and installs every dependency again - Build time: 3 to 4 minutes. For one line change. Now look at this version: ```dockerfile FROM node:22-alpine COPY package.json . # copies only the dependency list RUN npm install # installs dependencies COPY . . # copies your actual code last ``` Same one-line change to `index.js`. What happens now? - `COPY package.json .` has not changed, so it is a cache hit - `RUN npm install` has nothing changed above it, so it is also a cache hit - `COPY . .` sees the code change and rebuilds, but this is nearly instant - Build time: 3 to 4 seconds. Same app. Same dependencies. The only difference is the order of two lines. **Put what changes least at the top. Put what changes most at the bottom.** Your code changes dozens of times a day. Your dependencies change maybe once a week. So dependencies should always sit above your code copy in any Dockerfile you write, regardless of language: ```text Node.js -> COPY package.json first, then RUN npm install Python -> COPY requirements.txt first, then RUN pip install Java -> COPY pom.xml first, then RUN mvn dependency:go-offline Go -> COPY go.mod go.sum first, then RUN go mod download ``` Same principle. Every language. Every project. ## Try It Right Now If you have Docker installed, do this. It takes about three minutes and you will feel the difference directly. Create a `Dockerfile.slow`: ```dockerfile FROM node:22-alpine WORKDIR /app COPY . . RUN npm init -y && npm install express ``` And a `Dockerfile.fast`: ```dockerfile FROM node:22-alpine WORKDIR /app COPY package.json* ./ RUN npm init -y && npm install express COPY . . ``` Build both. Note the time. Then make a tiny change to any file, even just add a comment. Build both again. The first rebuild takes almost as long as the initial build. The second rebuild is nearly instant. Same result, completely different experience. That is the cache working for you instead of against you. ## The Hidden Truth About Image Layers Here is the lesser-known angle that changes how you think about Docker images entirely. Docker images are internally structured almost identically to Git. In Git, every commit is identified by a SHA256 hash of its contents. Change one character anywhere and you get a completely different hash. Branches and tags are just human-readable pointers to specific commit hashes. The actual data is content-addressable, stored and retrieved by what it contains, not by a name. Docker image layers work exactly the same way. Every layer is identified by the SHA256 hash of its contents. Pull the `nginx` image and you will see something like: ```text sha256:a1b2c3d4e5f6... <- not a name, a fingerprint of the actual content ``` This means two completely different images that happen to share the same base OS layer share that layer physically on disk. Docker stores it once. Pulls it once. Uses it everywhere. It also means Docker can detect tampering. If someone intercepts your image pull and modifies a layer, the hash will not match and Docker knows immediately. This is the same guarantee that makes Git commits trustworthy. And just like Git tags point to specific commits, Docker tags point to specific image manifests. When you push `myapp:latest`, you are creating a pointer. When you push `myapp:latest` again with a new build, the pointer moves. The old content is still there but the tag now points somewhere new. This is why version tags matter so much in production. `myapp:1.0.0` is an immutable pointer forever. `myapp:latest` is a moving target that means whatever was last pushed. ## The Factory Floor Problem Now let's talk about image size, because this is where most production images quietly go wrong. Think about a car factory. The factory floor is massive: heavy machinery, welding robots, painting rigs, assembly lines, quality control stations. Every tool you need to build a car is there. But when the car rolls off the line and goes to a customer, do they get the factory too? Obviously not. They get the car. Just the car. **Most Docker images ship the factory.** To *build* a Node.js app you need the Node.js runtime, npm, all your source code, the TypeScript compiler, ESLint, test frameworks, and all your dev dependencies. To *run* that same app in production you need the compiled output files, maybe a web server, and production dependencies only. That is it. A naive Dockerfile ships everything. The factory floor, the tools, the raw materials, the mess. The result is an image that is 900MB, sometimes 1GB or more. Slow to pull, slow to deploy, and a much larger attack surface. **Multi-stage builds** solve this. You use multiple `FROM` statements in one Dockerfile, where each one starts a completely fresh environment. The final stage copies only what it needs from the previous stages. ```dockerfile # Stage 1: The Factory FROM node:22-alpine AS builder WORKDIR /app COPY package.json . RUN npm install COPY . . RUN npm run build # Stage 2: The Car FROM nginx:alpine COPY --from=builder /dist /usr/share/nginx/html # nginx + your compiled files. Nothing else. ``` The final image contains zero trace of Node.js, npm, your source code, or your dev dependencies. Just nginx and the compiled output. The size difference is not subtle. Here are real numbers from production: | Language | Before Multi-Stage | After Multi-Stage | Reduction | |---|---|---|---| | Node.js | 1,100 MB | ~148 MB | ~87% | | Python | 920 MB | ~85 MB | ~91% | | Java | 700 MB | ~180 MB | ~74% | | Go | 850 MB | **~12 MB** | **~99%** | Go's 99% reduction deserves a special mention. Go compiles to a single static binary with no external dependencies. The entire runtime is baked in. So the final image can be just that one binary sitting on a minimal base with no OS tools, no package manager, nothing. 850MB of build environment produces a 12MB production image. ## Two Common Mistakes Worth Fixing Today **Mistake 1: Not using `.dockerignore`** When you run `docker build`, Docker sends your entire project directory to the daemon as the build context. Everything in the folder. Including `node_modules` (often hundreds of MB), your `.git` history, test fixtures, and log files. Even if you never explicitly `COPY` these files, they inflate your build context and slow down every single build. The fix is a `.dockerignore` file in your project root: ```text node_modules .git *.log .env dist coverage ``` Same concept as `.gitignore` but for Docker builds. It takes two minutes to add and can save tens of seconds per build. **Mistake 2: Splitting `apt-get update` and `apt-get install`** This one is subtle: ```dockerfile # Wrong: two separate layers RUN apt-get update RUN apt-get install -y curl # Right: one layer, always fresh RUN apt-get update && apt-get install -y curl \ && rm -rf /var/lib/apt/lists/* ``` When they are on separate lines, `apt-get update` can get cached from a week ago while `apt-get install` runs fresh, resulting in package version mismatches and mysterious build failures. Combining them ensures they always run together. The `rm -rf /var/lib/apt/lists/*` at the end cleans up the package index files so they do not bloat your layer. ## Three Things to Do Right Now Docker images are not zip files. They are stacks of layers, each identified by a fingerprint of its content, each cached independently. The layer cache is your most powerful build optimization, and the order of your Dockerfile instructions is the single biggest lever you have over it. Take any Dockerfile you are working on right now and check these three things: 1. **Move dependency installation above your code copy.** `package.json` first, then install, then `COPY . .` 2. **Add a `.dockerignore`** if you do not have one. It takes two minutes. 3. **Check if your production image needs multi-stage builds.** If it is over 500MB, it almost certainly does. ## What's Next In the next article we are going into Docker networking. Why does `localhost` break inside a container? How do your containers actually find and talk to each other by name? What is really happening when you do `-p 8080:80`? If you have ever seen `connection refused` between two containers that should be talking to each other, that article will make it completely clear. --- *What is the worst slow build you have ever sat through? How long before you found the Dockerfile ordering issue? Drop it in the comments.* ]]> https://www.iamyasasbanuka.me/blog/docker-zero-to-production-part-2-whats-happening https://www.iamyasasbanuka.me/blog/docker-zero-to-production-part-2-whats-happening Sun, 17 May 2026 00:00:00 GMT ybanuka2003@gmail.com (Yasas Banuka) docker linux containers architecture devops So your 10 Nginx containers don't store 10 × 200MB on disk. They share one 200MB set of read-only layers, and each has a tiny writable layer on top that only holds whatever that specific container has written. This is also why containers start so fast. There is nothing to copy. The filesystem is already there, shared and ready. The kernel just needs to create a new writable layer and attach it on top. ## Think About It You have a container running Nginx. Inside the container, you run: ```bash echo "hello" > /etc/nginx/test.txt ``` You just wrote a file to what appears to be the Nginx config directory. Now you stop and delete that container. Then you spin up a brand new container from the exact same Nginx image. **Is that `test.txt` file there?** No. It is gone completely. Because that file only existed in the container's writable layer, a thin overlay that gets thrown away when the container is removed. The shared read-only image layers underneath were never touched. This is why containers are called **ephemeral**; they are designed to be disposable. Any data you want to survive must be stored outside the container's writable layer, in a Docker volume or a bind mount. The container itself is meant to be replaceable at any moment. ## The Call Chain: From Your Keyboard to a Running Process Now let's connect all three features to what actually happens when you type `docker run nginx`. Most people think Docker does everything. It doesn't. Docker is more like a coordinator. It delegates the actual work down a chain of highly specialized tools: Each layer has one job and hands off to the next. Here is what each one actually does: **Docker CLI:** Just a client. When you type `docker run`, it sends an HTTP request to the Docker daemon over a Unix socket. The CLI does no container work itself. **dockerd:** The manager. It handles the big picture: is the image available locally? If not, pull it from Docker Hub. Then it hands the actual container creation over to containerd. **containerd:** The lifecycle manager. It prepares everything the container needs; the filesystem bundle, the network setup, the configuration, and then asks runc to actually create it. **runc:** This tiny binary makes a special system call called `clone()` with specific flags that tell the kernel to create all six namespaces at once. It sets up cgroups for resource limits. It mounts the OverlayFS layers. Then it executes your process inside that isolated environment. And then runc does something that surprises almost everyone who learns about it. **runc exits.** It doesn't stick around to monitor the container. Its job is to *create* the environment and *start* the process. Once that's done, it's gone. If you look for runc processes on a machine with 20 running containers, you will find none. So who keeps the containers alive? **The shim.** That `containerd-shim` sitting between containerd and runc. There is one per container, silently running the whole time. The shim holds the container's stdio open, reports the exit code when the container finishes, and most importantly, keeps the container alive even if containerd or dockerd crashes and restarts. This is why if the Docker daemon dies, your running containers survive. They aren't held by Docker. They are held by their shims. ## What This Means for You Practically - **"Why did my container crash with exit code 137?"** The kernel OOM-killed it. Your container hit its cgroup memory limit. Add a memory limit and watch `docker stats` you'll see it coming. - **"Why do containers start so fast?"** There is no OS to boot. The kernel is already running. OverlayFS just attaches a writable layer to shared read-only layers. Milliseconds, not minutes. - **"If the Docker daemon crashes, do my containers die?"** No. The shim keeps them alive. You just can't manage them until dockerd restarts. - **"Why can't I find my file after the container restarts?"** Container writable layers are ephemeral. Use volumes for anything you want to persist. ## The Three Pillars, One Sentence Each After all of this, here is the clean summary you can carry in your head: - **Namespaces** give each container it's own private view of the world, its own processes, network, and filesystem. - **cgroups** enforce hard limits on how much of the machine each container can consume. - **OverlayFS** makes images efficient by sharing read-only layers and only copying files when they're actually written. Docker didn't invent any of these. It built a beautiful, developer-friendly UX on top of them. And now that you know what's *underneath*. ## What's Next In the next article, we are going deep into **Docker images and Dockerfiles**,how images are actually built layer by layer, why the order of your instructions matters for build speed, and the multi-stage build pattern that can take your image from 900MB down to 25MB. If you've ever copied a Dockerfile from Stack Overflow without fully understanding why it was structured that way, that article is for you. --- *Which of these three; namespaces, cgroups, or OverlayFS surprised you the most? And did you know runc exits immediately after starting a container? Drop your reaction in the comments.* ]]> https://www.iamyasasbanuka.me/blog/docker-zero-to-production-part-1-why-docker-exists https://www.iamyasasbanuka.me/blog/docker-zero-to-production-part-1-why-docker-exists Fri, 15 May 2026 00:00:00 GMT ybanuka2003@gmail.com (Yasas Banuka) docker containers devops linux history ## The Problem Every Developer Knows You've felt this. Even if you didn't have a name for it. You spend days building a feature on your laptop. The tests pass. The app compiles. Everything works flawlessly. You push it to the staging server, and it instantly crashes. Wrong version of Node. Missing C++ library. Different Linux distribution. A ghost environment variable that exists on your machine but nowhere else. The error stack trace makes absolutely no sense. You spend two days debugging infrastructure instead of writing code. **"Works on my machine."** This phrase became the running joke of a generation of software engineers. It was funny because it was universally, agonizingly true. Now, multiply that single failure across an entire organization: - **5 developers** with slightly different local setups - **A staging environment** that drifted away from production months ago - **A production server** that gets its packages updated independently - **Three different microservices**, each demanding conflicting runtime versions Every deployment turned into an archaeology project. You weren't just shipping code anymore - you were shipping a fragile, invisible web of assumptions. ## The First Attempt: Virtual Machines The industry actually had a solution for this long before Docker arrived. **Virtual Machines (VMs).** In theory, the idea was brilliant: instead of hoping the server's environment matches your code's expectations, just ship the entire environment. Pack up an operating system, your runtime, your libraries, your code - all of it - into one massive file. Run it anywhere. And it worked. Mostly. But VMs came with a hidden tax that developers hated. A VM isn't just a box your app lives in. It is a **full computer simulation**. It has its own operating system kernel, its own virtualized memory management, and its own fake hardware. All of that has to run on top of the real machine's hardware through a hypervisor. > Imagine you want to move to a new city for work. A VM is like picking up your entire house - foundations, plumbing, wiring, roof - and physically moving it to the new city. It works, but the truck is enormous, moving day takes hours, and you're paying to transport a lot of stuff you could have just found in the new city. The numbers tell the real story. On a standard server with 128GB of RAM: - You could comfortably run maybe **10-15 virtual machines**. - Each VM eats **600MB+** just to boot its guest OS - before your app even starts. - Startup time is measured in **minutes**, not seconds. And the problem wasn't just resource bloat. It was **rigidity**. Once you configured a VM, it was heavy, slow to spin up, and painful to replicate perfectly. Developers still had to write lengthy, brittle bash scripts and pray that what worked on the staging VM matched the production VM. The "works on my machine" problem hadn't been solved. It had just moved one layer deeper. ## The Insight That Changed Everything Here's the thing nobody initially noticed about VMs: most of that weight was completely unnecessary. Every single VM was carrying its own operating system kernel. But here is the question that unlocked the container revolution: **Why does every app need its own kernel?** The kernel - the core of the OS that manages hardware, memory, and CPU processes - is already running on the host machine. It's shared infrastructure, like the plumbing in an apartment building. Why does every tenant need to install their own water main? What if you could keep the isolation (each app stays in its own private box) but ditch the duplication (no separate OS per box)? Docker's answer: a container doesn't carry a kernel. It **borrows the host's**. The container gets total isolation - its own filesystem, its own network stack, its own process space, but the heavy lifting is handled by the existing host kernel. > Now, instead of dragging your whole house to the new city, you just **pack a suitcase**. You bring your clothes and your laptop. The city already has buildings, plumbing, and electricity. You only bring what is uniquely yours. The suitcase is your container. The city's infrastructure is the host kernel. ## The Efficiency Leap Let's make this concrete. Imagine running three apps on one server: a Node.js API, a Python data processor, and an Nginx web server. **With VMs:** You are running three separate virtual computers. Three full OS installations. Your server is burning a huge chunk of its RAM and CPU just keeping those three mini-computers alive, before your apps process a single user request. **With Containers:** You have three isolated boxes sharing one kernel. The server's power goes almost entirely to your actual apps. And because there's no bulky OS to boot up, each container starts in **milliseconds**. On that same 128GB server where you could only run 15 VMs, you can now comfortably run **400–500 containers**. That isn't just an efficiency gain. It's a fundamentally different way to compute. A container starts instantly because there is no OS to boot. It's just a new, isolated process using the kernel that's already running. Think of it as the difference between booting up a whole new laptop (VM) versus simply opening a new tab in your browser (Container). ## What Docker Actually Did Here is the most fascinating part of the origin story: **Docker didn't invent containers**. The underlying Linux features that make containers possible - namespaces for isolation, cgroups for resource limits, union filesystems for layered images - had existed in the Linux kernel for years. Google had been using similar technology internally since the mid-2000s. But nobody outside of hardcore systems experts could use it. It required deep Linux kernel knowledge, complex configuration files, and a whole lot of duct tape. What Solomon Hykes and the Docker team did was make it accessible. They built a clean, developer-friendly interface on top of these powerful but complex Linux features. A simple CLI. A standard format for packaging apps (the image). A registry for sharing them (Docker Hub). They didn't build the engine. They built the steering wheel, the pedals, and the dashboard. As Solomon himself put it: *"Docker happened to see the opportunity and shift the right tools at the right time... It was going to happen anyway, with or without Docker."* ## The Dockerfile That Didn't Exist Here is a lesser-known fact that perfectly captures Docker's explosive growth: the **Dockerfile** wasn't even in the original launch. The file we now use to define exactly how our app's environment gets built was added later as a *"simple experiment."* As Solomon admitted: *"It was taped together and not in the original version. It became popular, so we had to fix it."* The community grabbed it and ran. Teams started writing Dockerfiles before Docker's own engineers had even stabilized the feature. The tool got pulled forward by the sheer desperation of the people using it. Any rough edges were instantly forgiven because the core value was so obvious. ## Where This Leaves Us Let's step back and look at the arc. The problem was old and painful: **software is fragile because environments differ**. Your app isn't just your code; it's your code plus every assumption it makes about the world around it. VMs tried to solve this by shipping the whole world. It worked, but it was expensive, slow, and bloated. Docker solved it by realizing you only need to ship **your app's specific world**. Just the filesystem, the libraries, and the runtime. The kernel is already there. The result? Portable, lightweight, fast, and mathematically consistent environments that you can run anywhere and share with anyone. And it all started with a 5-minute talk that got cut off. ## What's Next This was the "why." In the next article, we are going to go one level deeper, into what is actually happening inside your machine when you type `docker run`. Spoiler: it involves three Linux kernel features that existed long before Docker, doing some genuinely clever things. Once you see how they work, containers will never feel like magic again. They'll just feel obvious. --- *Did you have a "works on my machine" moment that cost you real time? Drop it in the comments — I'd love to hear the most painful ones.* ]]>