How to Run Docker Container on CoreOS etcd Cluster

January 5, 2017 | By in CONTAINERS
| Reply More

CoreOS is a new exciting operating system, designed with cluster concepts in mind, mainly, security and high-availability needs. Actually, CoreOS means three things, is the name of the company behind the OS, the name of the OS itself and the name of the cluster platform.

CoreOS is lightweight linux based operating system it comes with full support of docker. CoreOS allows us to easily run an entire cluster platform, to this end, CoreOS is based on other components mainly, etcd, docker and fleet.

Docker

CoreOS uses docker containers to ensure an isolated runtime for its underlying services and applications. It brings also docker out of the box for deploying and running applications in a CoreOS cluster.

Etcd

Etcd is a distributed key-value store (kind of key-value database) designed for high availability and mainly to store settings for running application in the cluster. CoreOS comes with etcdctl which is a command line tool to interact with etcd. Roughly speaking etcd is a service running on each machine in the cluster to handle coordination between software running on the cluster.

Fleet

Fleet is a kind of services manager for CoreOS cluster. With fleet we can manage our cluster as simple as if we manage a single machine. For instance, imaging that we need to start a specific service for an application, fleet handle our service by submitting it to the cluster and decide the host machine for our service. Besides that, with fleet we can define a set of policies for our services etc. Fleet comes also with fleetctl a command line tool to interact with it.

Set up a cluster under CoreOS

I should mention that CoreOS cluster could be run on different platform such as DigitalOcean, Amazon, Google Compute Engine etc. but, in this article I’ll be using vagrant and virtualbox to set up and run a local cluster. Therefore, vagrant and virtualbox are the two requirements for the current article, for sure, there are tons of good articles on how to install them on your machine.

Once you have installed vagrant and virtualbox, we need to download a repository maintained with CoreOS team, so go ahead and download this repo https://github.com/coreos/coreos-vagrant as illustrated in the image below otherwise you can clone it if you are familiar with git.

With git clone:

➜  coreos-vagrant git:(master) git clone https://github.com/coreos/coreos-vagrant.git

Once downloaded unzip the archive and open up the folder with your preferred text editor. Let’s walk throughout this repository and make some changes to fire our cluster.

etcd discovery token

to rapidly bootstrap a CoreOS cluster etcd needs a discovery service to connect instances together, by storing a list of peer addresses, metadata and the initial size of the cluster etc. CoreOS provides a free service to handle this need. Let’s generate a token for a cluster of size 4:

➜  coreos-vagrant git:(master)  curl -w "\n" 'https://discovery.etcd.io/new?size=4'

https://discovery.etcd.io/c8ac80bef794eff84e4d52c5af310db0

cloud-config

The above discovery URL can be provided to each CoreOS machine via cloud-config, a minimal config tool that's designed to get a machine connected to the network and join the cluster.

The downloaded repository provides a blueprint file for cloud-config named user-data.sample. So, we start by renaming this file to user-data and changing the discovery tag to our generate discovery URL, which is in my case: https://discovery.etcd.io/c8ac80bef794eff84e4d52c5af310db0

➜  coreos-vagrant git:(master) ✗ cat user-data

#cloud-config

coreos:

etcd2:

#generate a new token for each unique cluster from https://discovery.etcd.io/new

discovery: https://discovery.etcd.io/c8ac80bef794eff84e4d52c5af310db0

# multi-region and multi-cloud deployments need to use $public_ipv4

advertise-client-urls: http://$public_ipv4:2379

initial-advertise-peer-urls: http://$private_ipv4:2380

# listen on both the official ports and the legacy ports

# legacy ports can be omitted if your application doesn't depend on them

listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001

listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001

fleet:

public-ip: $public_ipv4

flannel:

interface: $public_ipv4

units:

- name: etcd2.service

command: start

- name: fleet.service

command: start

- name: flanneld.service

drop-ins:

- name: 50-network-config.conf

content: |

[Service]

ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'

command: start

- name: docker-tcp.socket

command: start

enable: true

content: |

[Unit]

Description=Docker Socket for the API

[Socket]

ListenStream=2375

Service=docker.service

BindIPv6Only=both

[Install]

WantedBy=sockets.target

Vagrantfile

Again we need to change change $num_instances in this file by assigning the value 5.

Start the cluster

To start the cluster just open a terminal and point out into the downloaded repository and run:

➜  coreos-vagrant git:(master) ✗ vagrant up

Bringing machine 'core-01' up with 'virtualbox' provider...

Bringing machine 'core-02' up with 'virtualbox'

...

Now we can ssh the created machines, like below:

➜  coreos-vagrant git:(master) ✗ vagrant ssh core-02

Last login: Fri Dec 23 17:22:51 UTC 2016 from 10.0.2.2 on ssh

CoreOS stable (1185.5.0)

core@core-03 ~ $

Now we can check the number of instances of our cluster:

core@core-02 ~ $ etcdctl member list

cce1ee23391bec4: name=397f492f266c4acab61e83ecc2adb388 peerURLs=http://172.17.8.103:2380 clientURLs=http://172.17.8.103:2379 isLeader=false

3e5d3e6bfe52303b: name=fb1c3d855e764520bb675ecf1fcbad22 peerURLs=http://172.17.8.102:2380 clientURLs=http://172.17.8.102:2379 isLeader=false

47f62622394dbef2: name=43bc777f8b654d19abddc2978a74cd0c peerURLs=http://172.17.8.101:2380 clientURLs=http://172.17.8.101:2379 isLeader=true

a8b3dab66f0a5fc5: name=57fcfc8813bd4bf49947e660574b525f peerURLs=http://172.17.8.104:2380 clientURLs=http://172.17.8.104:2379 isLeader=false

core@core-02 ~ $

We can also check the machines within the cluster:

core@core-02 ~ $ fleetctl list-machines --full=true

MACHINE                  IP     METADATA

397f492f266c4acab61e83ecc2adb388   172.17.8.103   -

43bc777f8b654d19abddc2978a74cd0c   172.17.8.101   -

57fcfc8813bd4bf49947e660574b525f   172.17.8.104   -

c773afd96d37462288c97c1b8695fad2   172.17.8.105   -

fb1c3d855e764520bb675ecf1fcbad22   172.17.8.102   -

Sample Node js app

Now I’m going to create a sample nodejs app to demonstrate how we can run docker containers inside our cluster.

// file - app.js

➜  sampleNodeJsApp git:(master) ✗ cat app.js

var express = require('express');

var app = express();

var PORT = 3000;

app.get('/ping', function (req, res) {

res.write('Hi there I\'m up and running!!');

res.end();

})

app.listen(PORT, function () {

console.log('Server listening on port: ', PORT);

})

// file – package.json

➜  sampleNodeJsApp git:(master) ✗ cat package.json

{

"name": "sampleNodeJsApp",

"version": "1.0.0",

"description": "",

"main": "index.js",

"scripts": {

"test": "echo \"Error: no test specified\" && exit 1"

},

"keywords": [],

"author": "",

"license": "ISC",

"dependencies": {

"express": "^4.14.0"

}

}

// file - Dockerfile

➜  sampleNodeJsApp git:(master) ✗ cat Dockerfile

FROM node:latest

RUN mkdir -p /usr/src/app

WORKDIR /usr/src/app

COPY package.json /usr/src/app/package.json

RUN npm i

COPY . /usr/src/app/

EXPOSE 8080

CMD [ "node", "app.js" ]

Building the docker image:

➜  sampleNodeJsApp git:(master) ✗ docker build -t linoxide/sample-nodejs-app:v0.0.1 .

Pushing the docker image to docker hub

➜  sampleNodeJsApp git:(master) ✗ docker push linoxide/sample-nodejs-app:v0.0.1

Unit files

Coreos deployment works using the called Unit files, therefore in order to be able to deploy the sample nodejs app already create, we need to create two files; the first file I’ll call it sample-nodejs-app-deployment@.service this file will pull our build and push docker image (see above) and fire a docker container; the second file I’ll call it sample-nodejs-app-service@.service this file expose our app deployment.

Please note these files need to be created inside one of our cluster machines:

// file - sample-nodejs-app-deployment@.service

core@core-02 ~ $ cat sample-nodejs-app-deployment\@.service

[Unit]

Description=Sample NodeJs app inside a docker container under CoreOS cluster

After=etcd2.service

After=docker.service

Requires=sample-nodejs-app-service@%i.service

[Service]

TimeoutStartSec=0

KillMode=none

EnvironmentFile=/etc/environment

ExecStartPre=-/usr/bin/docker kill sample-nodejs-app-deployment%i

ExecStartPre=-/usr/bin/docker rm sample-nodejs-app-deployment%i

ExecStartPre=/usr/bin/docker pull linoxide/sample-nodejs-app:v0.0.1

ExecStart=/usr/bin/docker run --name sample-nodejs-app-deployment%i -p %i:3000 \

    -P -e COREOS_PRIVATE_IPV4=${COREOS_PRIVATE_IPV4}        \

    linoxide/sample-nodejs-app:v0.0.1

ExecStop=/usr/bin/docker stop sample-nodejs-app-deployment%i

[X-Fleet]

Conflicts=sample-nodejs-app-deployment@*.service

// file - sample-nodejs-app-service@.service

core@core-02 ~ $ cat sample-nodejs-app-service\@.service

[Unit]

Description=Announce sample-nodejs-app-deployment@%i service

BindsTo=sample-nodejs-app-deployment@%i.service

[Service]

EnvironmentFile=/etc/environment

ExecStart=/bin/sh -c "while true; do etcdctl set /announce/services/sample-nodejs-app-deployment%i ${COREOS_PUBLIC_IPV4}:%i --ttl 60; sleep 45; done"

ExecStop=/usr/bin/etcdctl rm /announce/services/sample-nodejs-app-deployment%i

[X-Fleet]

MachineOf=sample-nodejs-app-deployment@%i.service

Submit sample-nodejs-app-deployment@.service and sample-nodejs-app-service@.service

core@core-02 ~ $ fleetctl submit sample-nodejs-app-deployment\@.service sample-nodejs-app-service\@.service

Unit sample-nodejs-app-deployment@.service inactive

Unit sample-nodejs-app-service@.service inactive

List all the services:

core@core-02 ~ $ fleetctl list-unit-files

UNIT                 HASH   DSTATE     STATE      TARGET

sample-nodejs-app-deployment@.service  8b9612b    inactive inactive -

sample-nodejs-app-service@.service d929f3f   inactive inactive -

Now both services are available inside the cluster. But, they are as you may notice in an inactive state. Let’s load the both services. We are going to pass the port 3000, within fleetctl in the name of the service as illustrated below, this port will be used to run docker container with port forward on this port.

core@core-02 ~ $ fleetctl load sample-nodejs-app-deployment@3000.service

WARNING: Unit sample-nodejs-app-deployment@.service in registry differs from local template unit file sample-nodejs-app-deployment@.service

Unit sample-nodejs-app-deployment@3000.service inactive

Unit sample-nodejs-app-deployment@3000.service loaded on 397f492f.../172.17.8.103

core@core-02 ~ $ fleetctl load sample-nodejs-app-service@3000.service

Unit sample-nodejs-app-service@3000.service inactive

Unit sample-nodejs-app-service@3000.service loaded on 397f492f.../172.17.8.103

Once the services are loaded, we can start them using fleetctl start:

core@core-02 ~ $ fleetctl start sample-nodejs-app-deployment@3000.service

WARNING: Unit sample-nodejs-app-deployment@3000.service in registry differs from local template unit file sample-nodejs-app-deployment@.service

Unit sample-nodejs-app-deployment@3000.service launched on 397f492f.../172.17.8.103

core@core-02 ~ $ fleetctl start sample-nodejs-app-service@3000.service

Unit sample-nodejs-app-service@3000.service launched on 397f492f.../172.17.8.103

Notice the IP address 172.17.8.103 where the node js app is hosted. So, let’s try that out, open up your browser and try to hit the ‘/ping’ endpoint: 172.17.8.103:3000/ping

browser 'ping' endpoint

Everything is working fine.

Conclusion

In this article, we have seen first, how to get a local CoreOS cluster up and running using vagrant and virtualbox, second we have seen how to run a web application in docker container inside our cluster.

Filed Under : CLUSTER, CONTAINERS

Tagged With : , ,

Free Linux Ebook to Download

Leave a Reply

Commenting Policy:
Promotion of your products ? Comment gets deleted.
All comments are subject to moderation.