Skip to content

docker

Docker uses containers, and is probably at the exact level I want to use.

note above this line was moved to /ops/virtualization/docker.md

debugging

The easiest way to look around in an image that is already started is :

execute commands inside the container
1
2
docker exec [visit](container) ls
docker exec -it [visit](container) sh

Note that sh is a command that works for both debian/bash and alpine/ash. For a non-running container, like builder containers etc, you use run :

execute commands inside the container
1
2
docker run [visit](container) ls
docker run -it [visit](container) sh

registry

To be able to deploy from the internet you can use the docker hub : visit

There you can maintain repositories, and when you click on them it show what the format is that you should push and pull from.

for example, for the klopt repository :

push an image to the repository
1
docker push kees/klopt:tagname

However you should first associate a docker image with that tagname. I had made a Dockerfile for the doc project and named it 'docker-doc'

If you list it, you will see it is now the repository name :

list
1
2
REPOSITORY                TAG                 IMAGE ID            SIZE
docker-doc                latest              d08ddebe8868        190MB

Now first give the docker-doc an tagname on the repository, and then push it.

tag image
1
2
docker image tag docker-doc kees/klopt:doc
docker push kees/klopt:doc

Now the repository is also available as kees/klopt:doc at the hub.

list images
1
2
3
4
docker image ls
REPOSITORY                TAG                 IMAGE ID            SIZE
kees/klopt                doc                 d08ddebe8868        190MB
docker-doc                latest              d08ddebe8868        190MB

So on another (deployment) machine, you can now :

login to repo
1
2
3
docker login
docker pull kees/klopt:doc
docker image run --publish 8888:80 kees/klopt:doc

The publish redirects all html traffic to port 8888 because 80 was already taken on the servert in question. Browser to visit and the complete web app runs !!

docker login

A word on docker login, note that it creates a file called ~/.docker/config.json

docker login cat ~/.docker/config.json # see whats in it echo "somebase64digits==" | base64 -d

If you replace somebase64digits== with the value of the auth: line you will see that it is the plaintext username:password.

So this could be done more secure, with credential helpers. The ones suggested only leave :

  • pass
  • D-Bus Secret Service: visit

I know pass, so i m starting with that. The helper for pass is here : visit And it should be in the clients PATH for docker to find.

credential helper
1
2
3
4
5
6
cd ~/Install
wget https://github.com/docker/docker-credential-helpers/releases/download/v0.6.3/docker-credential-pass-v0.6.3-amd64.tar.gz
chmoid +x docker-credential-pass
sudo mv docker-credential-pass /usr/local/bin
docker-credential-pass
Usage: docker-credential-pass [visit](store|get|erase|list|version)

Now point docker to this helper in ~/.docker/config.json. Just remove the content or 'docker logout' and put this :

~/.docker/config.json
1
2
3
{
    "credsStore": "pass"
}

The format used is in the format docker-credential-<credsStore>. So this would suffice for docker-credential-pass. Now if you log in the content will also change but now it becomes :

~/.docker/config.json
1
2
3
4
5
6
{
    "auths": {
        "https://index.docker.io/v1/": {}
    },
    "credsStore": "pass"
}

No more clear credentials, they are now in pass :

pass tree
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
> pass
Password Store
├── docker-credential-helpers
│   └── aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEv
│       └── kees
├── main
│   ├── hoek
│   └── servert1
└── web
    └── itaq

The docker-credential-helpers has been added to the entries I already added. View them with :

edit passwords
1
pass edit docker-credential-helpers/aHR0cHM6Ly9pbmRleC5kb2NrZXIuaW8vdjEv/kees

jenkins

Note that this invalidates the jenkins setup needed to push/pull from docker-hub. For now i prefer running devspace manually, but here is a guide on how it could be setup with another helper docker-credential-env : visit

Not. tried yet.

chaining builds

You can create images for building an application and then copy that over into another container like this :

chaining
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
FROM alpine as builder
RUN echo -e '@edge http://nl.alpinelinux.org/alpine/edge/main' >> /etc/apk/repositories
RUN echo -e '@testing http://nl.alpinelinux.org/alpine/edge/testing' >> /etc/apk/repositories
RUN echo -e '@community http://nl.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories
RUN apk add build-base cmake boost-dev libtbb@testing libtbb-dev@testing
RUN apk add expat-dev bzip2-dev lua5.2-dev zlib-dev
RUN apk add readline-dev util-linux-dev cunit-dev ctags fcgi-dev
RUN apk add nlohmann-json@community confuse-dev gtest-dev sqlite-dev
RUN apk add libpq autoconf bison flex

# copy our static linked executable
COPY 3pty 3pty
COPY lkh lkh
COPY backend backend
COPY makefile makefile
RUN make

FROM alpine as runner
RUN mkdir -p /etc/klopt
COPY backend/properties.conf /etc/klopt/properties.conf
COPY --from=builder /backend/release/backend backend
CMD ["./backend"]

This was the first draft of the backend Dockerfile before we split it into parts. Look at the highlighted lines the first one names the first images as builder, the second one specifies that it starts a new clean image again based on alpine which has not got all the software needed to build the builder image. The third line then specifies that it copies the executable from the builder into the clean image.

Needless to say the final image is much smaller. When you tag it this last imaged will receive the tag, the other one will be a nameless image.

build from tag
1
docker build -t runner .

splitting stages

Later on I split these two stages because the 3pty code almost never changed but still took very long to compile. This not only meant splitting the Dockerfile, but also the Jenkinsfile and the repository !! This is because Jenkins rebuilds are triggered by git commits and that would also mean that the 3pty code gets rebuilt at every change. Now the first part gets a separate tag :

Dockerfile
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
FROM alpine as builder
RUN echo -e '@edge http://nl.alpinelinux.org/alpine/edge/main' >> /etc/apk/repositories
RUN echo -e '@testing http://nl.alpinelinux.org/alpine/edge/testing' >> /etc/apk/repositories
RUN echo -e '@community http://nl.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories
RUN apk add build-base cmake boost-dev libtbb@testing libtbb-dev@testing
RUN apk add expat-dev bzip2-dev lua5.2-dev zlib-dev
RUN apk add readline-dev util-linux-dev cunit-dev ctags fcgi-dev
RUN apk add nlohmann-json@community confuse-dev gtest-dev sqlite-dev
RUN apk add libpq autoconf bison flex

# copy our static linked executable
COPY 3pty 3pty
RUN make
build from tag
1
docker build -t kees/klopt:backend-builder .

And the second one becomes :

Dockerfile
1
2
3
4
5
6
7
FROM kees/klopt:backend-builder
COPY lkh lkh
COPY backend backend
COPY makefile makefile
RUN make
RUN mkdir -p /etc/klopt
CMD ["/backend/release/backend"]

ansible

Ansible has commands for using docker. But to install it there are some problems if you don't have the correct version of pip :

ansible
1
2
3
sudo pip install "ansible-container[docker]"
# if that fails try to install this pip version
sudo pip install --upgrade pip==9.0.3

You might actually want to restore pip again after this because it is already at version 18! Anyway this works for installing ansible-container

Note that the next command will overwrite ansible.cfg if it is present !!

initialize
1
ansible-container init

Also the correct docker version should be installed .. sigh !!

correct docker version
1
2
3
pip install docker==2.7.0
ansible-container build
ansible-container run

Further reading:

visit

troubleshooting

Got permission denied while trying to connect to the Docker daemon socket

add to docker group
1
sudo useradd -aG docker kees

Cannot create an item in a locked collection

install gnupg2
1
sudo apt-get install gnupg2 pass

failed to build: failed to start shim:

When running docker-compose. No explanation.. but last time this helped.

restart docker
1
2
sudo service docker restart
docker-compose # error is gone !

Failed COPY command

When using COPY and it fails with a message like 'not a directory'. For instance

file existing
1
COPY backend backend

In this case, there was a file called 'backend' in the container.

cleanup all containers and images

If you want to start all over :

cleanup and make some space
1
docker system prune -a

docker-proxy hogging ports

You have stopped a container, even a complete systemprune (see prev chapter) Still you can not get the ports you want.

If you use ps you can see that docker-proxy is still running :

ps -elf
1
2
3
4
5
ps -elf | grep docker
4 S root        8789       1  5  80   0 - 1359133 -    08:42 ?        00:06:36 /usr/sbin/dockerd -H fd://
4 S root        9368    8789  0  80   0 - 287122 -     08:42 ?        00:00:00 /usr/sbin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 5409 -container-ip 172.31.0.2 -container-port 5432
4 S root        9381    8789  0  80   0 - 305555 -     08:42 ?        00:00:00 /usr/sbin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 8009 -container-ip 172.31.0.2 -container-port 80
0 S kees       93012    8559  0  80   0 -  1565 -      10:40 pts/3    00:00:00 grep docker

To fix this, this works

fresh restart
1
2
3
sudo service docker stop
sudo rm -rf /var/lib/docker/network/files/local-kv.db
sudo service docker start