Glide Cache and Docker

Intro

Dependency management in Golang got a lot better since go1.5, however, we still need tools to manage it.

For a long time, I used Godep which was working great, but handles everything based on your local GOPATH which result in massive change sets in git each time a different team member updates them, which makes the code review difficult.

Here comes Glide. A “newcomer” which uses a yaml config in order to explicitly set the dependency version needed. It is based on semver and allow for automatic update of path/minor version, in a similar way as npm.

Once the initial config is set (and glide allows to automatically generate it), glide up will generate a .lock file with the expected commit, based on the remote version specified in the yaml config.

Docker

For many reasons, Docker is a great tool and is a time saver. However, when it comes to develop in Go in a Docker environment, things quickly become slow, especially when using a lot of dependencies.

Let’s take a naive Dockerfile:

FROM       golang:1.7
ENV        APP_DIR $GOPATH/src/github.com/org/myapp
WORKDIR    $APP_DIR
ENTRYPOINT ["myapp"]
ADD        . $APP_DIR
RUN        go install

Each time something changes in the local directory, the ADD instruction will have its cache invalidated, resulting in the following go install to recompile the whole code, including all dependencies.

This is a major inconvenience when actively developing when we need to often recompile and/or run the tests, especially when dealing with statically linked, CGO disabled program.

Godep

With Godep, in go1.4, a simple solution is to add the Godeps directory first, compile it and then add the rest of the app.
In order to do that, we iterate over the dependency list and install them. As Godep uses json, we’ll need jq, an awesome tool in order to play with json in the shell.

FROM       golang:1.4
# Install jq and Godep.
RUN        apt-get update && apt-get install -y jq && go get github.com/tools/godep
ENV        APP_DIR $GOPATH/src/github.com/org/myapp
WORKDIR    $APP_DIR
ENTRYPOINT ["myapp"]
# Add Godeps and precompile.
ADD        Godeps/ $APP_DIR/Godeps
RUN        for pkg in $(cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath'); do \
             godep go install $pkg; \
           done
# Add App and install.
ADD        . $APP_DIR
RUN        godep go install

This is nice and saves up quite a lot of time, however, since go1.5, the vendor model changed and the imported packages are now scoped within the package itself instead of using the GOPATH one, which make this method obsolete.

If you are curious about the magic line for pkg in $(cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath'); do godep go install -ldflags -d $pkg; done, here is what it does:
Godep stores the known dependencies in the json file Godeps/Godeps.json which contains a json object with a Deps key which contains an array of dependencies. Each of which are a json object with the key ImportPath which is the value that interest us.
cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath' returns a list of values from the json file, which we iterate on via the for loop and then install the dependency.

Glide

With glide, in go1.5 and up, we need to rethink a bit the process. It will be similar, however, the first issue is that glide uses a yaml config. How to extract the values from a shell command?

yaml2json

I looked for tools similar to jq for yaml but didn’t find much so I built yaml2json which is a small go util which simply translate yaml to json using github.com/ghodss/yaml.

It can be installed via the go toolchain:

go get github.com/creack/yaml2json
yaml2json < glide.yaml > glide.json

or via Docker:

alias yaml2json='docker run -i --rm creack/yaml2json'
yaml2json < glide.yaml > glide.json

FYI, this Docker image contains only the statically linked, stripped down binary and weight only 3Mb!

$> docker images creack/yaml2json
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
creack/yaml2json    latest              a8e3e1fff7bb        3 weeks ago         3.007 MB

Caching

Now that we can have a json version of the yaml config, we can simply use jq in order to play with it.

With the new vendor model, the dependencies are now install as: $APP_DIR/vendor/$DEP_PATH rather than in the GOPATH directly.

Example: yaml2json is in github.com/creack/yaml2json and depends on github.com/ghodss/yaml so it will be installed as github.com/creack/yaml2json/vendor/github.com/ghodss/yaml

Another difficulty resides with the sub-packages, they are glide lists them as directory names under the parent’s imports section. We need to use a bit more advanced jq query to construct the full list to be installed.

Let’s see:

FROM       golang:1.7
# Install yaml2json and jq.
RUN        apt-get update && apt-get install -y jq && go get github.com/creack/yaml2json
ENV        APP_DIR  github.com/org/myapp
ENV        APP_PATH $GOPATH/src/$APP_DIR
WORKDIR    $APP_PATH
ENTRYPOINT ["myapp"]
# Add glide lock file and precompile.
ADD        glide.lock $APP_PATH/glide.lock
ADD        vendor     $APP_PATH/vendor
RUN        yaml2json < glide.lock | \
           jq -r -c '.imports[], .testImports[] | {name: .name, subpackages: (.subpackages + [""])}' | \
           jq -r -c '.name as $name | .subpackages[] | [$name, .] | join("/")' | sed 's|/$||' | \
           while read pkg; do \
             echo "$pkg...";  \
             go install $APP_DIR/vendor/$pkg 2> /dev/null; \
           done

# Add App and install.
ADD        . $APP_PATH
RUN        go install

First, we convert the lock file to json using yaml2json, then we extract the main import list as well as the test import list from which we need the name and the sub-packages if any.
As some dependencies will not have sub-package, we manually add + [""] to facilitate the next step.
Now that we have this list, we forge the full package names from the package list: we keep the “main” name and join it with the list of sub-packages (and "" for the main package itself).
Finally, we trim down the trailing / if any and install each dependency.

Alternative

Alternatively, instead of trying to pre-compile the dependencies, one could use the mount-bind feature of Docker in order to mount the local directory in a long running container and run the build/test there, which would allow to have the native caching of the go toolchain, but looses the reproducibility warranty of Docker.

Conclusion

This method might not be the most “straight forward” one, but gives us the ability to quickly iterate over our code without worrying about the toolchain.

Bonus: the actual Dockerfile that I use at Agrarian Labs for all our micro services:

FROM            golang:1.7
MAINTAINER      Guillaume J. Charmes <guillaume@leaf.ag>

# Install linters, coverage tools and test formatters.
RUN             go get github.com/alecthomas/gometalinter && gometalinter -i && \
                go get github.com/axw/gocov/... \
                       github.com/AlekSi/gocov-xml \
                       github.com/jstemmer/go-junit-report \
                       github.com/matm/gocov-html

# Disable CGO and recompile the stdlib.
ENV             CGO_ENABLED 0
RUN             go install -a -ldflags -d std

# Install jq and yaml2json for parsing glide.lock to precompile.
RUN             apt-get update && apt-get install -y jq
RUN             go get github.com/creack/yaml2json

ARG             APP_DIR

ENV             APP_PATH $GOPATH/src/$APP_DIR

WORKDIR         $APP_PATH

# Precompile deps.
ADD             glide.lock $APP_PATH/glide.lock
ADD             vendor     $APP_PATH/vendor
RUN             yaml2json < glide.lock | \
                jq -r -c '.imports[], .testImports[] | {name: .name, subpackages: (.subpackages + [""])}' | \
                jq -r -c '.name as $name | .subpackages[] | [$name, .] | join("/")' | sed 's|/$||' | \
                while read pkg; do \
                  echo "$pkg...";  \
                  go install -ldflags -d $APP_DIR/vendor/$pkg 2> /dev/null; \
                done

ADD             .          $APP_PATH

RUN             make install

Release Go code (and others) via Docker using Makefile

In this article, I’ll demonstrate how to leverage Makefile in order to release lightweight Docker image for production.

Sample Application code

package main

func main() {
        println("hello world")
}

Of course, do not forget to vendor your dependencies: godep save.

In order to create a release image, we first need to build the binary. We will have 2 Dockerfiles.

Main Dockerfile

The main Dockerfile is the “classic” one. For our example:

# Using google/golang as base image
FROM            google/golang:stable
# Install Godep for vendoring
RUN             go get github.com/tools/godep
# Recompile the standard library without CGO
RUN             CGO_ENABLED=0 go install -a std
# Declare the maintainer
MAINTAINER      Guillaume J. Charmes <guillaume@charmes.net>

# For convenience, set an env variable with the path of the code
ENV             APP_DIR         $GOPATH/src/example

# Set the entrypoint as the binary, so `docker run <image>` will behave as the binary
ENTRYPOINT      ["/example"]
# Add the sources to the APP_DIR
ADD             .       $APP_DIR
# Compile the binary and statically link
RUN             cd $APP_DIR && CGO_ENABLED=0 godep go build -o /example -ldflags '-d -w -s'

Note that the godep install and the CGO disabled std rebuild are done before the maintainer, this allows to keep the cache for this part when the maintainer changes.
In a future article, I’ll talk more in depth about static linking in Go (the -ldflags and go install std thingy)

Release Dockerfile

In order to release, we need a second Dockerfile. As Docker builds directories, we need to create a subdirectory release.

The release Makefile is very straight forward and look like this:

# Use "scratch" as base: it is an empty image.
FROM            scratch
# Set the entrypoint as the binary, so `docker run <image>` will behave as the binary
ENTRYPOINT      ["/example"]
# Add the binary. As it is statically linked, no need to add libc or anything else.
ADD             example /

Now, we need a Makefile to automate the process

Makefile

The Makefile allow for easy dependency scripting.

The goal is to push a “release” image. In order to do this, we need to have that image built.
In order to build this image, we need the binary, In order to get the binary, we need to build.
In order to build, we need the source. For convenience, we monitor . instead of each individual go files.

The default rule is all which triggers build

build depends on .build which is a file. If it exists then move on to the next dependency, otherwise, execute the rule. .build depend on .. This means that if anything changes in the current directoty, the cache gets invalidated and the rule will be executed.
In order to build, we call docker build and create the .build file.
The result is a full blown image roughly 600MB. release will help with this.
The first step is to extract the binary and /etc/ssl. The /etc/ssl is mandatory only if you plan on using SSL (otherwise, Go will complain it does not find the certificates). Once extracted in a tarball, we build the release Dockerfile.

NAME            = example
DOCKER_IMAGE    = 127.0.0.1:5000/$(NAME)

all             : build

.build          : .
                docker build -t $(NAME) .
                docker inspect -f '{{.Id}}' $(NAME) > .build

build           : .build

release/$(NAME) : build
                docker run --rm --entrypoint /bin/sh $(NAME) -c 'tar cf - /$(NAME) /etc/ssl' > $@ || (rm -f $@; false)
                docker build --rm -t $(DOCKER_IMAGE) release

release         : release/$(NAME)

push            : release
                docker push $(DOCKER_IMAGE)

clean           :
                $(RM) .build release/$(NAME)

.PHONY          : push release build all clean

Directory tree

In the end, the directory tree should look like this:

$> tree
.
├── Dockerfile
├── Godeps
│   ├── Godeps.json
│   ├── Readme
│   └── _workspace
├── Makefile
├── main.go
└── release
    └── Dockerfile

3 directories, 6 files

Conclusion

As we saw, the release process is a dependency cascade, which make the Makefile very useful.
The advantages of using something like this is easily shown by docker images:

Original image

$> docker images example
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
example             latest              e68fcc5482a8        10 seconds ago          605.3 MB

Release image

$> docker images 127.0.0.1:5000/example
REPOSITORY               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
127.0.0.1:5000/example   latest              6e9e455bf60d        4 seconds ago          617.6 kB

Orchestrate Docker locally with Makefiles: Example with Interactive Brokers in Go.

Docker Orchestration

In this article, I’ll talk about basic orchestration using names and links, it is mainly for local testing, for production use, the strategy would be different.

Names and Links

The names and links within Docker are very useful. When starting a container with --name, you can then manipulate it via that name instead of the ID or the generated name. It is particulary useful when you need to play with that container in the future. (An alternative would be to inject the ID in a variable, but this is very limiting).
The links allow you to “inject” all the information from a given container into an other. This includes exposed ports and environment variables.

Naming Convention / Good practice

As a convention, to avoid name collision, I always name my container with the _c suffix. I do not use any prefix/suffix on images.

Example

Start a dummy container with a name, an exposed port and an environment variable then link it to an other one.

#!/bin/sh
# make sure the container name is free, remove it if needed
docker rm -f mycontainer_c
docker run -d -p 4242 -e foo=bar --name mycontainer_c ubuntu:14.04 sleep 100
docker run --link mycontainer_c:mcc ubuntu:14.04 env
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> HOSTNAME=912b106bf5eb
> MCC_PORT=tcp://172.17.0.55:4242
> MCC_PORT_4242_TCP=tcp://172.17.0.55:4242
> MCC_PORT_4242_TCP_ADDR=172.17.0.55
> MCC_PORT_4242_TCP_PORT=4242
> MCC_PORT_4242_TCP_PROTO=tcp
> MCC_NAME=/naughty_mayer/mcc
> MCC_ENV_foo=bar
> HOME=/root

Docker build

In order to generate a usable image, Docker provides an incredible tool: Dockerfiles. It allows you to “describe” your image.

Example with Go

main.go

package main

func main() {
    println("hello world")
}

Dockerfile

FROM    google/golang:stable
ADD     .    /src
CMD     /src/a.out
RUN     cd /src && go build -o a.out .

Issues

While this is very powerful, in “real life”, it is often not really useful as is. What about your Database, caching, queuing or any other services your application interacts with?
This is where the names, links and Makefile come to play.

Makefiles

Makefiles are a powerful tool, even though not designed for this usecase, we can leverage its features to accomplish easy local orchestration.

Rules

The feature that interest us the most are the Makefile “rules”, or dependencies. It allows you to write easily a shell script with dependencies management.

Caching

One of the original purpose of the Makefile is to provide compilation caching. I.e., not recompile the whole project when you change one file.
As we are using Docker for this, we will not use the caching to that level. However, we will use it to cache already built images.

Conventions

For consistency, all my Makefile look alike:

• Uppercase variables
  
  • NAME
  • SERVICE_IMAGE
  • SERVICE_CONT
• all, clean, re rules (Yes, from Epitech)
• Often test rule is present
• all depends on build and build services
• Images and container names usually contain $(NAME)
• All dummy files for caching are part of .gitignore and .dockerignore

Example

• Without Docker - Watch whole directory for changes

NAME   = myproject

all    : build

.built : .
        go build .
        @touch .built

build  : .built

clean  :
        @$(RM) .built

re     : clean all

.PHONY : all build clean re

Calling make will invoque the all rule. all depends on build by convention.
build depends on .built, that rule is a bit particular as it actually represent a file on disc. It is used as placeholder to know if we need to redo the build. In a “regular” Makefile scenario, we would have source file instead.
If .built exists and didn’t change (mtime), then we do nothing. Otherwise, executre the rule.
.built depends on ., meaning that if anything (mtime) changes in the local directory, the rule will be reexecuted (upon next make call).
The .built rule, when finish creates the .built file. So next time make is called, nothing will happen unless something changed within the directory.
This is not really useful as is, but now, let’s take a look at the same thing with Docker.

• With Docker

NAME   = myproject

all    : build

.built : .
        docker build -t $(NAME) .
        @docker inspect -f '{{.Id}}' $(NAME) > .built

build  : .built

clean  :
        @$(RM) .built

re     : clean all

.PHONY : all build clean re

Very straight forward, we do the same thing expect we store the ID of the resulting image in the .built file.
Now that we know how to play with Makefile and Docker, let’s see an actual use case with some Orchestration

Interactive Brokers

Interactive Brokers (IB) is a brokerage company that offers extremely low fees. Where “big” companies offers things like $8 a trade or when you are lucky $5 a trade, IB is often under $0.80 per trade.
It also has the advantage to come with an API so you can do everything programatically. However, it requires their gateway (java and GUI) to be up and running.
Someone wrote a pure go library that interacts with the IB API: http://github.com/gofinance/ib, however, the tests requires to have Java and for some reason, I did not manage to have them work on my machine.
This is where the dockerize all the things come in place: let’s Dockerize it, so anyone can use it. Without Java and without hassle.

Dockerize

Main repository

It is incredibly complex to Dockerize something, especially with Go:

FROM            google/golang:stable
MAINTAINER      Guillaume J. Charmes <guillaume@charmes.net>

ADD             .         /src

That’s it. We are dealing with a library, so nothing will get executed. We will call the go test from the Makefile as they have some dependencies and can’t be ran stand alone.

Test server

The tests depends on the test server to be running, so let’s Dockerize it as well.
First let’s take a look at the original script to spawn the server:

#!/bin/bash

rm -rf tws
mkdir tws
pushd tws
jar -xf ../unixmacosx-943.2a.jar
pushd IBJts
java -cp ../../ibcontroller-2.12.0.jar:jts.jar:total.2013.jar -Xmx512M -XX:MaxPermSize=128M ibcontroller.IBGatewayController ../../ibcontroller-2.12.0.ini &
popd
popd

In order to make it work, I had to replace jar -xf by unzip and in order to avoid the Gateway to complain about missing X11, I run it within xvfb. My new script “docker ready” look like this:

#!/bin/bash

rm -rf tws
mkdir tws
cd tws
unzip ../unixmacosx-943.2a.jar
cd IBJts
socat TCP-LISTEN:4003,fork TCP:127.0.0.1:4002&
xvfb-run java -cp ../../ibcontroller-2.12.0.jar:jts.jar:total.2013.jar -Xmx512M -XX:MaxPermSize=128M ibcontroller.IBGatewayController ../../ibcontroller-2.12.0.ini

It is very similar. You will note the socat hack. For some reason, I did not manage to have the gateway listen on the outside (even when setting it to 0.0.0.0). So I create a simple proxy on an other port.
The Dockerfile then look like this:

FROM            ubuntu:14.04
MAINTAINER      Guillaume J. Charmes <guillaume@charmes.net>

RUN             apt-get update
RUN             apt-get install -y unzip socat xvfb gsettings-desktop-schemas openjdk-7-jre && rm -rf /var/lib/apt/lists/*
ENV             JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64

EXPOSE          4003

ADD             .         /src

CMD             cd /src && ./ibgwdocker

Now that we have both the main code and the test server dockerize, we need to orchestrate.

Makefile

The Makefile rule system makes a perfect orchestration tool. In order to run tests, we first need to have the code dokerize (i.e. build the Dockerfile) as well as the test server. We then need the test server to be up and running and only then we can run the tests.

NAME            =       ib
GATEWAY_IMAGE   =       $(NAME)_gateway_test
GATEWAY_CONT    =       $(GATEWAY_IMAGE)_c
TEST_CONT       =       $(NAME)_test_c

all             :       test

.build_gw_id    :       testserver
                docker build -t $(GATEWAY_IMAGE) testserver
                @docker inspect -f '{{.Id}}' $(GATEWAY_IMAGE) > .build_gw_id

.gateway_id     :       .build_gw_id
                -@docker rm -f $(GATEWAY_CONT) > /dev/null 2> /dev/null || true
                -@docker rm -f $(GATEWAY_CONT)_tmp > /dev/null 2> /dev/null || true
                docker run --name $(GATEWAY_CONT) -d $(GATEWAY_IMAGE)
                @echo Wait for Gateway to be started
                @sleep 1
                @docker run --link $(GATEWAY_CONT):gw --rm --name $(GATEWAY_CONT)_tmp -t ubuntu:14.04 \
                        bash -c 'for i in {1..60}; do \
                                        echo | nc $$GW_PORT_4003_TCP_ADDR 4002 && exit 0 || (echo -n ..; sleep 1); \
                                done; \
                                echo; \
                                echo Waiting for Gateway timed out; exit 1'
                @echo
                @docker inspect -f '{{.Id}}' $(GATEWAY_IMAGE) > .gateway_id

gateway         :       .gateway_id

.build_id       :       .
                docker build -t $(NAME) .
                @docker inspect -f '{{.Id}}' $(GATEWAY_IMAGE) > .build_id

build           :       .build_id

test            :       gateway build
                -@docker rm -f $(TEST_CONT) > /dev/null 2> /dev/null || true
                docker run --link $(GATEWAY_CONT):gw --name $(TEST_CONT) -t $(NAME) bash -c 'cd /src && go test $(TESTFLAGS) -gw $$GW_PORT_4003_TCP_ADDR:4003'

clean           :
                -@docker rm -f $(GATEWAY_CONT) > /dev/null 2> /dev/null || true
                -@docker rm -f $(GATEWAY_CONT)_tmp > /dev/null 2> /dev/null || true
                -@docker rm -f $(TEST_CONT) > /dev/null 2> /dev/null || true

clean_all       :       clean
                -@rm -f .build_id .build_gw_id .gateway_id

re              :       clean_all all


.PHONY          :       all gateway buld test clean clean_all re

That is maybe a bit much at once, let’s decompose.
The goal here is to run the tests, so I made all to depend on test instead of build because there is no binary to build.
The test rule depend on the repository to be build and the gateway (testserver) to be up and running. It depends on the build and gateway rules.
gateway depends on .gateway_id, which mean that when the file .gateway_id exists, the rule will not be executed. If it does not exists, then call the .gateway_id rule, which depends on .build_gw_id. This means that gateway represent the test server runtime, but in order to run, it needs to be build before hand.
.build_gw_id depends on testserver. So if anything changes within testserver, then the rule will be reexecuted.
That rule effectively build the testserver image. Once built, then the .gateway_id rule is executed. This actually starts the test server. You will not the small shell hack that tried to connect to the port 4002. When in succeed, it means the gateway is up and running.
We run the test server with: docker run --name $(GATEWAY_CONT) -d $(GATEWAY_IMAGE), so we give it a fixed name.
Once the rule finishes, we have a test server built and up and running, now the build rule gets executed.
This one is pretty straight forward: it uses .build_id has caching file, when it does not exists, it builds the repository image.
When both the rules are satisfied (i.e. cached or executed), then the test rule starts:

docker run --link $(GATEWAY_CONT):gw --name $(TEST_CONT) -t $(NAME) bash -c 'cd /src && go test $(TESTFLAGS) -gw $$GW_PORT_4003_TCP_ADDR:4003'

We link the container with the test server one, so inside we will have access to the $GW_PORT_4003_TCP_ADDR, which is the actual private ip of the test server container. We give -t only for interactive purpose: it allows for ctrl-c to interrupt.
At this point, we have a container with the ip of the running test server, so we start the tests and give the test server IP as parameter.
See https://github.com/gofinance/ib/pull/4 for the full Pull-Request which includes the Makefile, the Dockerfiles and the code changes to accept variable test server IP.

Other use cases

This is very useful for lot of situation, for example:

• Redis
• Postgres, Mongo (or any database)
• Elastic Search

If you need multiple services, simply add more rules. The makefile will allow you to easily describe your cascade dependencies and cache everything.