Docker

From Mosuma
Jump to: navigation, search

Docker-toolbox vs Vagrant (Docker on OSX)

I have used predominantly Vagrant, and tried a little bit of Docker-toolbox (formerly boot2docker), here are my initial understanding of the pros/cons:

Pros of Vagrant

  1. Host OS (coreOS) is exactly the same as your target deployment cloud instance (e.g., Amazon EC2 and Digital Ocean)
    • allows you to run the full suite of etcd/fleetd coreOS cluster tools
  2. Matured GUI control/configuration/tweaking of your virtual machine using Virtualbox

Pros of Docker-toolbox

  1. Host OS is a stripped down version of Linux, may differ from your target deployment cloud instance
  2. Speed (the docker machine runs in RAM???)
  3. Seamless integration with OSX. Each Console/Terminal is a VM. You just need to run "eval $(docker-machine env YOUR_VM_NAME)", then you can directly run all the docker commands within the VM. No need to ssh into the VM first.

However, once it comes to mounting files/volumes from Mac to docker host, I prefer vagrant's approach, since the target docker host runs the same OS as deployment/production environment, I can write scripts that runs on both CoreOS and OSX. Whereas for Docker-toolbox, pathnames, etc. are different.


Installing on MacOS

refer to https://coreos.com/docs/running-coreos/platforms/vagrant/

  1. install vagrant
    • this install a command line tool vagrant
    • at this point you cannot run vagrant *yet*, proceed to the next step
  2. install virtualbox version>=4.3.10
  3. clone coreos
    cd ~/gits
    git clone https://github.com/coreos/coreos-vagrant.git
    cd coreos-vagrant
  4. create/modify
    ~/gits/coreos-vagrant/user-data
  5. create/modify
    ~/gits/coreos-vagrant/config.rb
  6. start the system
    vagrant up
    • three 1G virtualbox instances will be started, you can check status via
      vagrant status
    • the following is an example output
guiyu@apple:~/gits/coreos-vagrant$ vagrant status
Current machine states:

core-01                   running (virtualbox)
core-02                   running (virtualbox)
core-03                   running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
guiyu@apple:~/gits/coreos-vagrant$ 

ssh to CoreOS machines

  • connecting from macOS terminal
vagrant ssh core-01 -- -A
  • -- : allows additional ssh arguments for the ssh command, not the vagrant command
  • -A : (this is an ssh parameter, not vagrant parameter) enables ssh forwarding

Within CoreOS

Installing swap on Digital Ocean CoreOS

Referenced from https://gist.github.com/philips/7310435

1. Create the swapfile

fallocate -l 2048m /root/2GiB.swap
chmod 600 /root/2GiB.swap
mkswap /root/2GiB.swap

2. Create the following file: /root/swapon.service

[Unit]
Description=Turn on swap

[Service]
Type=oneshot
Environment="SWAPFILE=/root/2GiB.swap"
RemainAfterExit=true
ExecStartPre=/usr/sbin/losetup -f ${SWAPFILE}
ExecStart=/usr/bin/sh -c "/sbin/swapon $(/usr/sbin/losetup -j ${SWAPFILE} | /usr
/bin/cut -d : -f 1)"
ExecStop=/usr/bin/sh -c "/sbin/swapoff $(/usr/sbin/losetup -j ${SWAPFILE} | /usr
/bin/cut -d : -f 1)"
ExecStopPost=/usr/bin/sh -c "/usr/sbin/losetup -d $(/usr/sbin/losetup -j ${SWAPF
ILE} | /usr/bin/cut -d : -f 1)"

[Install]
WantedBy=local.target

3. Enable this so that swap will be started automatically:

systemctl enable --runtime /root/swapon.service
systemctl start swapon

4. Swap is enabled, check by typing 'top'

top - 12:59:46 up 2 days, 16:33,  1 user,  load average: 0.00, 0.01, 0.05
Tasks:  80 total,   1 running,  79 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    505940 total,   379268 used,   126672 free,     4584 buffers
KiB Swap:  2097148 total,        0 used,  2097148 free.   290044 cached Mem
Swap on Digital Ocean is a BAD long term solution

2015.05.12

I have seen quite a number digital ocean linux crashes when swap utilization goes beyond a certain point, e.g., 50%.

Thus it is better to upgrade to a larger memory instance versus using swap.

I have since disabled swaps on all my machines except the database servers, which have not yet breached their memory limits. The swap is there just as a guard.

connecting to docker VMs
docker run busybox /bin/echo hello world


docker run -t -i debian /bin/bash
  • -i : keeps stdin open even if unattached
  • -t : allocate pseudo terminal


Starting a shell in a running docker

From docker 1.3 onwards you can:

docker exec -it {name or id of running container} bash
Running docker in detached mode
  • -d : run docker detached with no screen
docker run -dit --entrypoint /usr/sbin/nginx debian
  • Alternatively, you can use the amphesand:
docker run -it --entrypoint /usr/sbin/nginx debian &

The above requires you to add the following line to /etc/nginx/nginx.conf:

daemon off;

Typical Flow

  1. docker run debian bash
  2. apt-get install this and that
  3. docker commit <containerid> <imagename>
  4. docker run <imagename> bash
  5. git clone git://.../mycode
  6. pip install -r requireiments.txt
  7. exit
  8. docker commit <containerid> <imagename>
  9. repeat steps 4-8 as necessary
  10. docker tag <imagename> <user/image>
  11. docker push <user/image>

</pre>


Private Docker Registry

Server (hosting the Docker Registry)

Docker Registry Server

Requirements
  • can be any linux machine with python/pip installed, because docker-registry is a python program
  • need NOT need docker installed
  • does NOT need a webserver, i.e., no nginx needed, no apache needed
  • can be running any linux kernel because docker is not needed
Installation
  • install docker-registry using pip
  • install nginx
  • follow docker-registry instructions https://github.com/docker/docker-registry/blob/master/README.md
  • generate CA (certificate authority)
    • must generate the CA and save it
  • generate self-signed certificate
  • copy ca.crt to client (who wishes to read/write from this private docker registry)

Client (running coreOS)

  • copy ca.crt to /etc/ssl/certs/ca.pem
    • must have suffix .pem
  • Run
update-ca-certificates
  • restart docker daemon (IMPORTANT so that new ca.crt is read) via rebooting or typing:
sudo systemctl restart docker
  • test by typing
    docker login https://yourprivatedockerrepository/

Client (docker-toolbox on OSX)

Accessing Private Docker Registry

To access private docker registry from an existing docker machine on OSX, the following finally worked for me. I am running

docker-machine version 0.6.0, build e27fb87
Docker version 1.10.3, build 20f81dd

Suppose your private docker registry has the domain name:

mydockerrepo.mydomain.com

and you have copied the registry's private key as MY_PRIVATE_REGISTRY_CA to your OSX computer.

Then just following the steps below:

scp MY_PRIVATE_REGISTRY_CA docker@##.##.##.##
  • ##.##.##.## is the IP address of your docker machine, shown in 'docker-machine ls'
  • default login/password is docker/tcuser
  • This will copy the CA.CRT to /home/docker/MY_PRIVATE_REGISTRY_CA
ssh docker@##.##.##.##
  • password is as above
sudo mkdir -p /etc/docker/certs.d/mydockerrepo.mydomain.com/
sudo mv ~/MY_PRIVATE_REGISTRY_CA /etc/docker/certs.d/mydockerrepo.mydomain.com/ca.crt
  • Next, you NEED NOT restart your docker-machine if everything is alright, i.e., you need NOT run the following
docker-machine restart YOUR_DOCKER_MACHINE

You should be able to access your private repository


References:

http://stackoverflow.com/questions/30654306/allow-insecure-registry-in-host-provisioned-with-docker-machine

1 modify existing docker VM (NOT WORKING)

NOT WORKING FOR ME

References:

https://akrambenaissi.com/2015/11/17/addingediting-insecure-registry-to-docker-machine-afterwards/

For https private docker registry you need to add (UNTESTED) the following line to .docker/machine/machines/YOUR_OLD_MACHINE :

"HostOptions": {

   "EngineOptions": {
      "InsecureRegistry": [
        "https://docker.mosuma.net"
      ],
   }

}

copying the config.json from a brand NEW VM to the OLD VM with relevant entries updated does NOT work either.

NOT WORKING

2 create new docker VM (NOT WORKING)
docker-machine create --driver virtualbox --virtualbox-memory "1024" \
    --engine-insecure-registry \
    https://YOUR_PRIVATE_REGISTRY any_machine_name
Firing up the new/old VM with insecure registry cert added
  • each shell/terminal window talks to ONE VM
  • Run this command to configure your CURRENT terminal/shell window to use any_machine_name:
eval $(docker-machine env any_machine_name)

pushing

clementi certs # docker images                                        
REPOSITORY                TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
debian                    test4               69e277397157        3 days ago          204.3 MB
debian                    test3               2855d0517a07        3 days ago          204.3 MB
samalba/docker-registry   latest              6b86e5be37f9        3 weeks ago         424 MB
debian                    7.5                 06af7ad6cff1        11 weeks ago        85.18 MB
clementi certs # docker tag 69e277397157 private-docker-repo.com/debian
clementi certs # docker push private-docker-repo.com/debian
The push refers to a repository [private-docker-repo.com/debian] (len: 1)
Sending image list
Pushing repository private-docker-repo.com (1 tags)
511136ea3c5a: Image successfully pushed 
...
69e277397157: Image successfully pushed 
Pushing tag for rev [69e277397157] on {https://private-docker-repo.com/v1/repositories/debian/tags/latest}


pulling

clementi certs # docker pull private-docker-repo.com/debian
Pulling repository private-docker-repo.com/debian
69e277397157: Download complete 
...
5a2b404ecbe0: Download complete 
clementi certs # 
clementi certs # docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
debian                     test4               69e277397157        3 days ago          204.3 MB
private-docker-repo.com/debian   latest              69e277397157        3 days ago          204.3 MB
debian                     test3               2855d0517a07        3 days ago          204.3 MB
samalba/docker-registry    latest              6b86e5be37f9        3 weeks ago         424 MB
debian                     7.5                 06af7ad6cff1        11 weeks ago        85.18 MB
clementi certs #   

tags

clementi certs # docker tag 69e277397157 private-docker-repo.com/debian:test4
clementi certs # docker push private-docker-repo.com/debian:test4
The push refers to a repository [private-docker-repo.com/debian] (len: 1)
Sending image list
Pushing repository private-docker-repo.com/debian (1 tags)
Image 511136ea3c5a already pushed, skipping
...
Image 69e277397157 already pushed, skipping
Pushing tag for rev [69e277397157] on {https://private-docker-repo.com/v1/repositories/debian/tags/test4}
clementi certs # docker pull private-docker-repo.com/debian:test4
Pulling repository private-docker-repo.com/debian
....
5a2b404ecbe0: Download complete 
clementi certs # docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
private-docker-repo.com/debian   latest              69e277397157        3 days ago          204.3 MB
private-docker-repo.com/debian   test4               69e277397157        3 days ago          204.3 MB
debian                           test4               69e277397157        3 days ago          204.3 MB
debian                           test3               2855d0517a07        3 days ago          204.3 MB
samalba/docker-registry          latest              6b86e5be37f9        3 weeks ago         424 MB
debian                           7.5                 06af7ad6cff1        11 weeks ago        85.18 MB
clementi certs #

FAQ

/etc/init.d/uwsgi starts in docker but complains about Failure

Symptoms
  • /etc/init.d/uwsgi start works (really does start the uwsgi) but complains about failure
  • subsequent execution of /etc/init.d/uwsgi start will continue to start new/additional uwsgi processes since start-stop-daemon cannot figure out if any existing uwsgi is running
  • /etc/init.d/uwsgi stop works properly provided that your configuration.ini file does not have a specified line
pidfile = some_path_to_pidfile
Cause
  • after starting the uwsgi daemon, start-stop-daemon will check for existence (success) of a running uwsgi process in /proc/##PID##/exe. However, /proc/##PID##/exe is broken due to docker security measures, so start-stop-daemon thinks that the process was not started properly, but in fact it was started and running fine, just do a
    ps aux |grep uwsgi
    and you will see the processes running nicely.
  • This is because docker security policies (see https://github.com/docker/docker/issues/6607) prevents the start-stop-daemon from writing to /proc/[PID]/{pid,cwd,root} "correctly for programs that involves scripts (e.g., python scripts)"[*]. For binary-only programs like nginx, /etc/init.d/nginx start/stop works fine under docker.

[*] This is just a conjecture of the cause, read up more on https://chris-lamb.co.uk/posts/start-stop-daemon-exec-vs-startas

Solution 1

For Docker version 1.2.0, build fa7b24f

  • stop the docker container
  • optionally commit/save the docker image of your stopped docker container
  • remove the stopped container via
    docker rm dockercontainer_id
    (now that you have an image of it)
  • restart the docker container using your saved image with the docker run option --cap-add SYS_PTRACE, e.g,
docker run  -p 80:80 --cap-add SYS_PTRACE --name kuiyu -i -t yourprivatedockeregistry.com/debian:7.5
Solution 2

For Debian 7, uwsgi version 1.2.3+dfsg-5+deb7u1

Reference: https://chris-lamb.co.uk/posts/start-stop-daemon-exec-vs-startas

If you are unable to add the --cap-add SYS_PTRACE option to your docker invocation, you can

1. modify the do_start_specific_daemon() method in the /usr/share/uwsgi/init/specific_daemon script by changing from line 60 onwards:

#/usr/share/uwsgi/init/specific_daemon
# @ line 60

  start-stop-daemon --start --quiet \
    --pidfile "$PIDFILE" \
    --exec "$DAEMON" \
    --test > /dev/null \
    && return 2

  return 0
}

to

#/usr/share/uwsgi/init/specific_daemon
# @ line 60 (modified to work with docker)

  start-stop-daemon --start --quiet \
    --pidfile "$PIDFILE" \
    --startas /usr/bin/uwsgi \
    --test > /dev/null \
    && return 2

  return 0
}

2. In addition, you must modify your corresponding /etc/uwsgi/apps-enabled/yourapp.ini to create the pid file, which is in the path /var/run/uwsgi/app/yourapp/pid, by adding the following line in your yourapp.ini:

# /etc/uwsgi/apps-enabled/yourapp.ini
pidfile = /var/run/uwsgi/app/yourapp/pid

supervisorctl complains about unix:///var/run/supervisor.sock refused connection

Symptoms
supervisord -c /etc/supervisor/supervisord.conf
  • Started supervisorctl with absolute path to /etc/supervisor/supervisord.conf as follows:
supervisorctl -c /etc/supervisor/supervisord.conf
  • Fails for *some* docker containers
  • Works for *others*
  • This is a strange error
Solution

If you have a solution, please drop me a message @ zhangguiyu AT mosuma.com