Rethink Swarm Mode

So we need another “web site & database”, right? Of course. Always! But this time, we want it to be solid. Very solid.

So we create a stateless application server. Maybe I’ll write about that later.

So we look for a clustered database that scales easily. Very easily. Enter RethinkDB.

So we need it to run somewhere. Somewhere as in: I don’t care, as long as it’s pretty stable, and connected to Internet. Somewhere as in: on a laptop for the developers in exactly the same way as in the cloud for the end users. To minimise the unexpected. Enter Docker.

So we need a cluster of Docker things running a Rethink database together. Enter Docker Swarm Mode. Let me show you how I spin up any number of previously nonexistent machines to flock into a swarm and serve us a highly available, fault tolerant clustered database, at the touch of a button, anywhere I want it.

1. Install Docker

Docker lets you package your and others’ software into “images” that run as systems of their own, and are transferable between environments – so you can develop and test with exactly the same image on your laptop as the one that will end up serving the end users in the clouds.

Start on that developer laptop. Install Docker if it isn’t there yet – Mac Windows Linux.

By the way, if you are on Windows, you’ll need to find a way to run Bash to be able to run the commands listed in this post (as well as the scripts that we link to later on) – if you’re looking for a solution, try the Git BASH that’s included in Git for Windows.

2. Install Docker Machine

Docker Machine lets you create and manage… machines. Virtual machines to run Docker images. On your laptop, on your server, or in the cloud. It supports quite a few common cloud providers right out of the box.

But wait! If you installed Docker on your Mac or Windows PC, Machine is already there. Otherwise, read the instructions to get it.

3. Install VirtualBox

Oracle’s VirtualBox lets you create… virtual boxes. Virtual machines, that is. Docker Machine uses VirtualBox to run machines locally. On your developer laptop, for instance.

But wait! If you installed Docker on your Mac or Windows PC, VirtualBox is already there. Otherwise, download the sweetness.

4. Spin up some nodes

Open a terminal and create a local machine that will act as the “manager node” in our “development swarm”:

$ docker-machine create --driver virtualbox manager

Add a node that will act as a worker in our swarm:

$ docker-machine create --driver virtualbox worker1

And one more worker to top it off:

$ docker-machine create --driver virtualbox worker2

Now, wasn’t that easy?

5. Swarm it together

Lookup the IP address Docker Machine made up for our manager node:

$ MANAGER_IP=$(docker-machine ip manager)

Now to let there be a swarm, we use docker-machine to SSH into the manager node, and initialize the beast there. We need to feed it the IP address we found:

$ docker-machine ssh manager \
docker swarm init --advertise-addr $MANAGER_IP

It will tell us we need some token in order to get the workers to join the swarm as well. Let’s just fetch that thing once now, and keep it handy:

$ TOKEN=$(docker-machine ssh manager \
docker swarm join-token --quiet worker)

Now extend the swarm to include the two worker nodes:

$ docker-machine ssh worker1 \
docker swarm join --token $TOKEN $MANAGER_IP:2377
$ docker-machine ssh worker2 \
docker swarm join --token $TOKEN $MANAGER_IP:2377

There you go.

By the way: note that we’re using the newish Docker-native Swarm Mode here. Docker Machine provides some swarm-related options, but we don’t use those, since they’re for the “legacy” swarm feature, not for Swarm Mode.

6. Rethink all the nodes!

Now that we have this swarm of three, let’s put a network on it for our little database. We do this on the swarm’s manager node, and tell it to use the overlay driver to get it accessible swarm-wide, and call it “dbnet” – since names should make sense.

$ docker-machine ssh manager \
docker network create \
--driver overlay \
dbnet

Also, we need some storage for the data files:

$ docker-machine ssh manager \
docker volume create \
--name dbdata

Now, let’s get that server running:

$ docker-machine ssh manager \
docker service create \
--name db \
--replicas 1 \
--network dbnet \
--mount src=dbdata,dst=/data \
--publish 8080:8080 \
rethinkdb

We’re creating a “service” for it on the swarm, and we call it “db”, use our swarm-wide “dbnet” network, put its data files on the “dbdata” volume, let us reach the administrative web application on port 8080 from outside the swarm, and use the “rethinkdb” image that it’ll download from the Docker Hub. All nice and clean.

But hey, what is this “–replicas 1” sitting there? Are we starting just one instance of the server? Hardly a cluster then, right?

It’s true. The thing is: in order to form a cluster, we need to tell all subsequent servers, on starting them, to join the first one. And when we’re the first one, trying to join any other server would just fail miserably.

So let’s get some more to join the club. But first, we need some storage for those as well:

$ docker-machine ssh manager \
docker volume create \
--name db1data

Okay, now we go:

$ docker-machine ssh manager \
docker service create \
--name db1 \
--mode global \
--network dbnet \
--mount src=db1data,dst=/data \
rethinkdb \
rethinkdb --join db --bind all

So there we have our actually substantial “db1” service. Because of the “global” mode it’ll run three servers – one on each node: manager, worker1, and worker2. If we would have multiple server instances on a single node, they would clash with their respective data files on the “db1data” volume. Note that while the volume is managed on swarm level, its instances on each node are all separate, thus available exclusively to that node’s server. Should we want multiple servers per node, we could just add another global service “db2” and volume “db2data” in exactly the same way – no limits there, though I’m not really sure about the practical value of having more than one per node.

By the way, the first “rethinkdb” in the command line is the image name, the second is the command that starts the server – we need to override the default command that we relied on earlier, to get the instruction in for joining the cluster. It uses the service name “db” to reach the first server.

7. Check it out

Time to see what we have now. To have a consistent entry point for the web admin, create an SSH tunnel to it like this:

$ docker-machine ssh manager \
-fNL 8080:localhost:8080

Then, go for it:

schermafbeelding-2016-09-08-om-17-00-44
RethinkDB Web Admin

Sir, 4 servers connected, Sir! Gotta love this, don’t you?

8. Use it

Any clients should connect to port 28015 on the “db1” service. While the “db” service will work as well, you wouldn’t want to depend on the availability of that single replica, would you?

We could publish port 28015 to access it from outside the swarm, but why not create an application service running inside of it?

For instance, in go, we could try the Hello world example of gorethink, spraying some service-worthy behaviour on it by wrapping it in a canonical http server example:

package main

import (
  "fmt"
  r "gopkg.in/dancannon/gorethink.v2"
  "html"
  "log"
  "net/http"
)

func main() {

  var url = "db1:28015"

  session, err := r.Connect(r.ConnectOpts{
    Address: url,
  })
  if err != nil {
    log.Fatalln(err)
  }

  http.HandleFunc("/bar", func(w http.ResponseWriter, req *http.Request) {

    res, err := r.Expr("Hello from Rethink").Run(session)
    if err != nil {
      log.Fatalln(err)
    }

    var response string
    err = res.One(&response)
    if err != nil {
      log.Fatalln(err)
    }

    fmt.Fprintf(w, "Hello, %q 0.1\n", html.EscapeString(req.URL.Path))
    fmt.Fprintf(w, response+"\n")
  })

  log.Fatal(http.ListenAndServe(":9090", nil))
}

To package that, let’s follow Kelsey Hightower’s approach for assembling a completely dependency-free binary, that can run in the tiniest of tiny images.

If you’re not into go, and don’t feel like getting into it, you can skip over the next bit, and just pull my image from the Docker Hub. Otherwise:

Install go (locally) if you haven’t got it yet.

Create a new directory “rethinkswarmmode”, with a new file “foo.go”, and paste in the go code from above.

Navigate to the “rethinkswarmmode” directory, and run the formatter:

$ go fmt

Fetch the one source dependency (the gorethink driver):

$ go get gopkg.in/dancannon/gorethink.v2

Compile the code:

$ CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags '-w' .

Now, to build a Docker image, we need a Dockerfile:

$ echo "FROM scratch" > ./Dockerfile
$ echo "ADD rethinkswarmmode rethinkswarmmode" >> ./Dockerfile
$ echo "EXPOSE 9090" >> ./Dockerfile
$ echo "ENTRYPOINT [\"/rethinkswarmmode\"]" >> ./Dockerfile

That’s right: from scratch! Like I said: no dependencies  🙂

You could build the image “remotely”, on each consecutive swarm node…

$ docker-machine ssh manager \
docker build -t yourname/rethinkswarmmode:0.1 $PWD
$ docker-machine ssh worker1 \
docker build -t yourname/rethinkswarmmode:0.1 $PWD
$ docker-machine ssh worker2 \
docker build -t yourname/rethinkswarmmode:0.1 $PWD

…or you could build it locally, then push it to a shared repository (i.e. Docker Hub). That’s much prettier, but also slower, and requires you to have an account for the repository, and being logged in ($ docker login –username yourname –email youraddress, then type your password):

$ docker build -t yourname/rethinkswarmmode:0.1 .
$ docker push yourname/rethinkswarmmode:0.1

Either way… now it’s run time! (Just replace “yourname” with “wscherphof” if you skipped the go compiling and image building)

$ docker-machine ssh manager \
docker service create \
--name rethinkswarmmode \
--replicas 6 \
--network dbnet \
--publish 9090:9090 \
yourname/rethinkswarmmode:0.1
$ docker-machine ssh manager -fNL 9090:localhost:9090
$ curl http://localhost:9090/bar
Hello, "/bar" 0.1
Hello from Rethink

There you go!

$ docker service ps rethinkswarmmode
ID                        NAME               IMAGE                         NODE    DESIRED STATE CURRENT STATE          ERROR
4itnyefnkfp8v10zwu2ksx9cd rethinkswarmmode.1 yourname/rethinkswarmmode:0.1 manager Running       Running 21 seconds ago
dk76qyhlowrz1niiuc4q23f2d rethinkswarmmode.2 yourname/rethinkswarmmode:0.1 worker1 Running       Running 20 seconds ago
0het5jrtldneddkludyf1ahn1 rethinkswarmmode.3 yourname/rethinkswarmmode:0.1 worker1 Running       Running 20 seconds ago
emounxbjcuzo7sfe8siwydg3z rethinkswarmmode.4 yourname/rethinkswarmmode:0.1 worker1 Running       Running 20 seconds ago
a0f6qqfw3dcof39t77w7gm850 rethinkswarmmode.5 yourname/rethinkswarmmode:0.1 worker2 Running       Running 21 seconds ago
d4iasxlxj39kqrmxwz4hv64z7 rethinkswarmmode.6 yourname/rethinkswarmmode:0.1 worker2 Running       Running 21 seconds ago

Pure satisfaction, right? Come on; admit it!

9. Cloudification time

All good and well, but it’s about time to get this whole thing to the cloud, isn’t it? There’s actually quite a few clouds that Docker Machine supports right out of the box. Let’s pick DigitalOcean. Don’t ask me why – probably because they say it’s “designed for developers”, whatever that may mean. So get an account there. It’s not going to cost you much; just remember to not only stop, but actually remove your machines if you’re not using them. To just try some things out, it won’t cost you more than 1 or 2 dollars. Your account comes with an “access token”, and we need that one to create our new machines. Keep it somewhere safe and secret.

Now, to save you from going through all of our command line fiddling again from the start, I might as well confess to you now that… it was all scripted! Find the repo on GitHub, and download, clone, or fork it.

The scripts are designed to operate on a swarm for a conceptual “environment”, e.g. “dev” for your local development laptop, “tst” for the testers, “acc” for user acceptance, and “prd” for production (the end user environment), but you’re free to choose your own names.

Running the “nodes” command with just “dev” as the environment argument, will create the nodes “dev-manager-1”, “dev-worker-1”, and “dev-worker-2”, and swarm them up together. What we’ve been so painstakingly creating above, we could recreate from the ground up, with a snap of the fingers, like this:

$ ./nodes -m 1 -w 2 create dev
$ ./rethinkdb/r create dev
$ ./go/build -p 9090 ./rethinkdb/go/rethinkswarmmode \
yourname/rethinkswarmmode:0.1 dev
$ ./app -t 9090 -r 6 rethinkswarmmode \
yourname/rethinkswarmmode:0.1 dev

Local (on VirtualBox) is the default destination – to get the nodes up in the cloud, save your DigitalOcean access token in an environment variable:

$ export DIGITALOCEAN_ACCESS_TOKEN="945g4976gfg497456g4976g3t47634g9478gf480g408fg420f8g2408g08g4204"

Now you could pull a three-node “tst” swarm up in the cloud, like this…

$ ./nodes -m 1 -w 2 -d digitalocean -F tst

…but a swarm with all nodes just sitting in the same place together, isn’t nearly the most fail-safe of all, is it? Let’s fix that. DigitalOcean has separate regions (note that while most are reported “available”, some others aren’t), enabling us to swarm around the world in 80 nodes (or 3):

Start with a clean slate:

$ ./nodes rm tst

Create “tst-manager-1” in Amsterdam:

$ export DIGITALOCEAN_REGION="ams3";
$ ./nodes -m 1 -d digitalocean -F tst

Create “tst-worker-1” in Singapore:

$ export DIGITALOCEAN_REGION="sgp1"
$ ./nodes -w 1 -d digitalocean -F tst

Create “tst-worker-2” in New York:

$ export DIGITALOCEAN_REGION="nyc3"
$ ./nodes -w 1 -d digitalocean -F tst

When done, you should see the new nodes listed as “droplets” in your DigitalOcean account.

Now we can spin up the RethinkDB cluster on the “tst” swarm:

$ ./rethink/r tst create
* removing db0...
* removing db1...
* removing dbnet...
* creating dbnet...
cwy5m6uzmjrldcm30fmd887rv
* creating db0data...
db0data
* creating db0...
e4ipgrq5jddn6z0oyup4aw7r6
* creating db1data...
db1data
* creating db1...
5mvead8sckw9lxubzqiwk9oop
* connecting...
localhost:8081 -> tst:8080

It’ll open the RethinkDB web admin again, showing the cluster with  4 connected servers. Each swarm/environment gets its own tunnel with its own port number on your local machine.

Build a Docker image for the go application server (or skip it, and test with mine from Docker Hub, by just specifying “wscherphof” instead of “yourname” in the ./app command below – Docker knows where to find it then):

$ ./go/build -p 9090 ./rethinkdb/go/rethinkswarmmode \
yourname/rethinkswarmmode:0.1
* formatting source code...
* compiling...
* building image...
Sending build context to Docker daemon 5.965 MB
Step 1 : FROM scratch
 --->
Step 2 : ADD rethinkswarmmode rethinkswarmmode
 ---> 53b7d3aef48e
Removing intermediate container d664d1f2fb96
Step 3 : EXPOSE 9090
 ---> Running in 198a861bcb43
 ---> 7441c635d4ff
Removing intermediate container 198a861bcb43
Step 4 : ENTRYPOINT /rethinkswarmmode
 ---> Running in a44cb324d142
 ---> ef12312ecc18
Removing intermediate container a44cb324d142
Successfully built ef12312ecc18
* pushing image...
The push refers to a repository [docker.io/yourname/rethinkswarmmode]
ae96e9f40d95: Pushed
0.1: digest: sha256:b474e5e6014c7f4929fb4f746f0b29948278fe33c2850a423e8da41ca721b8a3 size: 528

Lastly, run that stuff:

$ ./app -t 9090 -r 6 rethinkswarmmode \
yourname/rethinkswarmmode:0.1 tst
* creating appdata...
appdata
* starting service...
7bjdr6xxl3nm7u7612gm3anuu
* connecting...
localhost:9091 -> tst:9090

Open your web browser at http://localhost:9091/bar, and you should find it showing that lovely little message again:

Hello, "/bar" 0.1
Hello from Rethink

Remember that droplets get billed even when turned off. So when you’re done, get rid of them:

$ ./nodes rm tst

10. But, but, but, …

…What if that precious single db replica goes down, the root of our cluster?

Well, let’s try:

$ docker-machine ssh tst-manager-1 docker service rm db0
db0
$ curl http://localhost:9091/bar
Hello, "/bar" 0.1
Hello from Rethink
$ curl http://localhost:8081
curl: (52) Empty reply from server

So it’s not so much of a root of the cluster then, is it? The cluster keeps running without it, and the application keeps safely connected to the redundant “db1” service. But we did lose our gateway to the Rethink web admin tool.

Let’s pull it back up then:

$ docker-machine ssh tst-manager-1 \
docker service create \
--name db0 \
--replicas 1 \
--network dbnet \
--mount src=db0data,dst=/data \
--publish 8080:8080 \
rethinkdb
ad5fjbxo0mwnnu515d6exnwxf
$ ./rethink/r tst

And… we’re back! It’ll take a minute, or two or three, before it’s reconnected to all of the other servers, but it’ll be all figured out by itself.

…What about other cloud providers?

There’s actually quite a few that Docker Machine supports. You can use any of them, by first ensuring an account, and then just setting the proper environment variables, and pass “-d drivername” to the “nodes” command. I couldn’t login to “azure”, but have played for some time with “google” and “amazonec2”. Both proved quite a bit more complex than digitalocean; you’ll need to develop a fair amount of very specific knowledge about their security groups and network settings and stuff to get it running smoothly. I’m very interested though, to get a swarm to run on nodes that are hosted not merely in different regions, but on totally different cloud providers. Should be possible, shouldn’t it? For now, I’ll leave it as an exercise to the reader!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s