Skip to main content

CPR for a Docker container

Background

Picture this: You're experimenting with a Docker container, enable a feature, and suddenly your container is stuck in an endless restart loop. In a panic, you run docker compose down and watch everything disappear.

What You Should Do Instead

Don't lose the container, no matter how bad it is. Create a new image from the container using docker commit: docker commit <container-id> broken-image (you get container id using docker ps -a command)

From here there are a couple of things you could do. I was using Dockerfile to build and setup things. Modify it to use broken image as base and install the missing dependencies:

FROM broken-image:latest
RUN pip install <package-name>

And with this in place I can restart the docker compose stack.

Complicated changes

In case damage to your container is more severe there will be additional steps to revive the container.

  1. Create image from your broken container: docker commit <container-id> broken-image
  2. Create a new container from this broken image with a different entrypoint command (default command would restart the broken stack): docker run -it --entrypoint=/bin/bash broken-image:latest
  3. Adjust or remove your changes, install missing dependencies, run migrations, what have you
  4. Create a new image from this container that has all the fixes: docker commit <container-id-with-bash-running> fixed-image
  5. Start a new container using this fixed-image and default command that brings up your original stack. In my case I removed build step from my docker-compose.yml file and pointed it to the new image:

       app:
         image: fixed-image:latest
    
  6. Be careful with running faulty containers. I was really confused balancing docker compose commands and direct docker commands. I noticed that docker compose (re)start command would also (re)start all broken/existing containers and I had to manually remove them to be able to get the complete stack running again.

With this approach I was no longer building a new Docker image. Instead I used docker compose to start container using fixed-image and default command.

Summary:

  • Always test changes in development first
  • Use docker commit with your container before making risky changes