Glide Cache and Docker

Intro

Dependency management in Golang got a lot better since go1.5, however, we still need tools to manage it.

For a long time, I used Godep which was working great, but handles everything based on your local GOPATH which result in massive change sets in git each time a different team member updates them, which makes the code review difficult.

Here comes Glide. A “newcomer” which uses a yaml config in order to explicitly set the dependency version needed. It is based on semver and allow for automatic update of path/minor version, in a similar way as npm.

Once the initial config is set (and glide allows to automatically generate it), glide up will generate a .lock file with the expected commit, based on the remote version specified in the yaml config.

Docker

For many reasons, Docker is a great tool and is a time saver. However, when it comes to develop in Go in a Docker environment, things quickly become slow, especially when using a lot of dependencies.

Let’s take a naive Dockerfile:

FROM       golang:1.7
ENV        APP_DIR $GOPATH/src/github.com/org/myapp
WORKDIR    $APP_DIR
ENTRYPOINT ["myapp"]
ADD        . $APP_DIR
RUN        go install

Each time something changes in the local directory, the ADD instruction will have its cache invalidated, resulting in the following go install to recompile the whole code, including all dependencies.

This is a major inconvenience when actively developing when we need to often recompile and/or run the tests, especially when dealing with statically linked, CGO disabled program.

Godep

With Godep, in go1.4, a simple solution is to add the Godeps directory first, compile it and then add the rest of the app.
In order to do that, we iterate over the dependency list and install them. As Godep uses json, we’ll need jq, an awesome tool in order to play with json in the shell.

FROM       golang:1.4
# Install jq and Godep.
RUN        apt-get update && apt-get install -y jq && go get github.com/tools/godep
ENV        APP_DIR $GOPATH/src/github.com/org/myapp
WORKDIR    $APP_DIR
ENTRYPOINT ["myapp"]
# Add Godeps and precompile.
ADD        Godeps/ $APP_DIR/Godeps
RUN        for pkg in $(cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath'); do \
             godep go install $pkg; \
           done
# Add App and install.
ADD        . $APP_DIR
RUN        godep go install

This is nice and saves up quite a lot of time, however, since go1.5, the vendor model changed and the imported packages are now scoped within the package itself instead of using the GOPATH one, which make this method obsolete.

If you are curious about the magic line for pkg in $(cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath'); do godep go install -ldflags -d $pkg; done, here is what it does:
Godep stores the known dependencies in the json file Godeps/Godeps.json which contains a json object with a Deps key which contains an array of dependencies. Each of which are a json object with the key ImportPath which is the value that interest us.
cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath' returns a list of values from the json file, which we iterate on via the for loop and then install the dependency.

Glide

With glide, in go1.5 and up, we need to rethink a bit the process. It will be similar, however, the first issue is that glide uses a yaml config. How to extract the values from a shell command?

yaml2json

I looked for tools similar to jq for yaml but didn’t find much so I built yaml2json which is a small go util which simply translate yaml to json using github.com/ghodss/yaml.

It can be installed via the go toolchain:

go get github.com/creack/yaml2json
yaml2json < glide.yaml > glide.json

or via Docker:

alias yaml2json='docker run -i --rm creack/yaml2json'
yaml2json < glide.yaml > glide.json

FYI, this Docker image contains only the statically linked, stripped down binary and weight only 3Mb!

$> docker images creack/yaml2json
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
creack/yaml2json    latest              a8e3e1fff7bb        3 weeks ago         3.007 MB

Caching

Now that we can have a json version of the yaml config, we can simply use jq in order to play with it.

With the new vendor model, the dependencies are now install as: $APP_DIR/vendor/$DEP_PATH rather than in the GOPATH directly.

Example: yaml2json is in github.com/creack/yaml2json and depends on github.com/ghodss/yaml so it will be installed as github.com/creack/yaml2json/vendor/github.com/ghodss/yaml

Another difficulty resides with the sub-packages, they are glide lists them as directory names under the parent’s imports section. We need to use a bit more advanced jq query to construct the full list to be installed.

Let’s see:

FROM       golang:1.7
# Install yaml2json and jq.
RUN        apt-get update && apt-get install -y jq && go get github.com/creack/yaml2json
ENV        APP_DIR  github.com/org/myapp
ENV        APP_PATH $GOPATH/src/$APP_DIR
WORKDIR    $APP_PATH
ENTRYPOINT ["myapp"]
# Add glide lock file and precompile.
ADD        glide.lock $APP_PATH/glide.lock
ADD        vendor     $APP_PATH/vendor
RUN        yaml2json < glide.lock | \
           jq -r -c '.imports[], .testImports[] | {name: .name, subpackages: (.subpackages + [""])}' | \
           jq -r -c '.name as $name | .subpackages[] | [$name, .] | join("/")' | sed 's|/$||' | \
           while read pkg; do \
             echo "$pkg...";  \
             go install $APP_DIR/vendor/$pkg 2> /dev/null; \
           done

# Add App and install.
ADD        . $APP_PATH
RUN        go install

First, we convert the lock file to json using yaml2json, then we extract the main import list as well as the test import list from which we need the name and the sub-packages if any.
As some dependencies will not have sub-package, we manually add + [""] to facilitate the next step.
Now that we have this list, we forge the full package names from the package list: we keep the “main” name and join it with the list of sub-packages (and "" for the main package itself).
Finally, we trim down the trailing / if any and install each dependency.

Alternative

Alternatively, instead of trying to pre-compile the dependencies, one could use the mount-bind feature of Docker in order to mount the local directory in a long running container and run the build/test there, which would allow to have the native caching of the go toolchain, but looses the reproducibility warranty of Docker.

Conclusion

This method might not be the most “straight forward” one, but gives us the ability to quickly iterate over our code without worrying about the toolchain.

Bonus: the actual Dockerfile that I use at Agrarian Labs for all our micro services:

FROM            golang:1.7
MAINTAINER      Guillaume J. Charmes <guillaume@leaf.ag>

# Install linters, coverage tools and test formatters.
RUN             go get github.com/alecthomas/gometalinter && gometalinter -i && \
                go get github.com/axw/gocov/... \
                       github.com/AlekSi/gocov-xml \
                       github.com/jstemmer/go-junit-report \
                       github.com/matm/gocov-html

# Disable CGO and recompile the stdlib.
ENV             CGO_ENABLED 0
RUN             go install -a -ldflags -d std

# Install jq and yaml2json for parsing glide.lock to precompile.
RUN             apt-get update && apt-get install -y jq
RUN             go get github.com/creack/yaml2json

ARG             APP_DIR

ENV             APP_PATH $GOPATH/src/$APP_DIR

WORKDIR         $APP_PATH

# Precompile deps.
ADD             glide.lock $APP_PATH/glide.lock
ADD             vendor     $APP_PATH/vendor
RUN             yaml2json < glide.lock | \
                jq -r -c '.imports[], .testImports[] | {name: .name, subpackages: (.subpackages + [""])}' | \
                jq -r -c '.name as $name | .subpackages[] | [$name, .] | join("/")' | sed 's|/$||' | \
                while read pkg; do \
                  echo "$pkg...";  \
                  go install -ldflags -d $APP_DIR/vendor/$pkg 2> /dev/null; \
                done

ADD             .          $APP_PATH

RUN             make install

JSON Date management in Golang

TL;DR

Arbitrary date unmarshal support + easily set marshal date format for both json and bson.
The code and examples can be found here: https://github.com/simplereach/timeutils.

Small example

package main

import (
        "encoding/json"
        "fmt"
        "os"

        "github.com/simplereach/timeutils"
)

type data struct {
        Time timeutils.Time `json:"time"`
}

func main() {
        var d data
        jStr := `{"time":"09:51:20.939152pm 2014-31-12"}`
        _ = json.Unmarshal([]byte(jStr), &d)
        fmt.Println(d.Time)

        d = data{}
        jStr = `{"time":1438947306}`
        _ = json.Unmarshal([]byte(jStr), &d)
        fmt.Println(d.Time)

        d.Time = d.Time.FormatMode(timeutils.RFC1123)
        _ = json.NewEncoder(os.Stdout).Encode(d)
}

The Standard Library

Go provide an extensive support for dates/time in the standard library with the package time.

This allows to easily deal with dates, compare them or make operations on them as well as moving from a timezone to an other.

Example

package main

import (
    "fmt"
    "time"
)

func main() {
    fmt.Printf("%s\n", time.Now().UTC().Add(-1 * time.Day))
}

Formating

Within the time.Time object, there are easy ways to format the date:

package main

import (
    "fmt"
    "time"
)

func main() {
    now := time.Now().UTC()
    // Display the time as RFC3339
    fmt.Printf("%s\n", now.Format(time.RFC3339))
    // Display the timestamp
    fmt.Printf("%s\n", now.Unix())
    // Display only the hour/minute
    fmt.Printf("%s\n", now.Format("3:04PM"))
}

Parsing

When it comes to parsing, once again, the standard library offers tools.

Parsing date string

package main

import (
    "log"
    "fmt"
    "time"
)

func main() {
    t, err := time.Parse(time.RFC3339, "2006-01-02T15:04:05-07:00")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s\n", t)
}

“Parsing” timestamp

package main

import (
    "fmt"
    "time"
)

func main() {
    now := time.Unix(1438947306, 0).UTC()
    fmt.Printf("%s\n", now)
}

This is great, but what if we don’t know what time format we are expecting? i.e. user input or 3rd part API.

A solution would be to iterate through the available time formats until we succeed, but this is often cumbersome and unreliable.

Approxidate

The git library has this Approxidate component that parses arbitrary date format and there is a Golang binding so we can use it!

http://godoc.org/github.com/simplereach/timeutils#ParseDateString

This expects a string as input and will do everything it can to properly yield a time object.

package main

import (
        "fmt"
        "log"

        "github.com/simplereach/timeutils"
)

func main() {
        t, err := timeutils.ParseDateString("09:51:20.939152pm 2014-31-12")
        if err != nil {
                log.Fatal(err)
        }
        fmt.Println(t)
}

Case of JSON Marshal/Unmarshal

Unmarshal

Let’s start with the unmarshal. What if we don’t want to parse the time manually and let json.Unmarshal handle it? Let’s try:

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "time"
)

func main() {
    var t time.Time

    str := fmt.Sprintf("%q", time.Unix(1438947306, 123).Format(time.RFC3339))
    fmt.Printf("json string: %s\n", str)
    if err := json.Unmarshal([]byte(str), &t); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("result: %s\n", t.Format(time.RFC3339))
}

Magically, it works fine! This is great, isn’t it?
But wait, the specs require us to send the date as RFC1123, is this going to work?
Let’s try as well!

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "time"
)

func main() {
    var t time.Time

    str := fmt.Sprintf("%q", time.Unix(1438947306, 123).Format(time.RFC1123))
    fmt.Printf("json string: %s\n", str)
    if err := json.Unmarshal([]byte(str), &t); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("result: %s\n", t.Format(time.RFC1123))
}

2009/11/10 23:00:00 parsing time ""Fri, 07 Aug 2015 11:35:06 UTC"" as ""2006-01-02T15:04:05Z07:00"": cannot parse "Fri, 07 Aug 2015 11:35:06 UTC"" as "2006"

Oups.

So it does not work, how can we work around this?

A solution would be to implement the json.Unmarshaler interface and handle our own parsing format, but we’ll get to this.

Marshal

Ok, we have our time object, and we want to send it as json. Nothing easier:

package main

import (
    "encoding/json"
    "os"
    "time"
)

func main() {
    _ = json.NewEncoder(os.Stdout).Encode(time.Unix(1438947306, 0).UTC())
}

It works fine :) However, the client expects times as RFC1123, how can we set the format to json.Marhsal?

A way to do so would be to implement the json.Marshaler interface and handling our own formatting.

Custom Marshal/Unmarshal

In order to tell Go to use a custom method for json marshal/unmarshal, one needs to implement the json.Marshaler and json.Unmarshaler interfaces.
As we can’t do that on imported type time.Time, we need to create a custom type.

Custom type

In order to create a custom type in Go, we simply do:

type myTime time.Time

However, doing so “hides” all members and methods so we can’t do things like this:

var t myTime
t.UTC()

Which is pretty annoying as our goal is simply to override the JSON behavior. We still want our full blown object.
To do so, we’ll use a struct with an anynomous member:

type myTime struct {
    time.Time
}

This way, we can access all the methods of the nested time object.

Unmarshal RFC1123

As we expect RFC1123, we need a custom parsing, so le’ts implement json.Unmarshaler.
Let’s take our first RFC1123 example and improve it:

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "strings"
    "time"
)

type myTime struct {
    time.Time
}

func (t *myTime) UnmarshalJSON(buf []byte) error {
    tt, err := time.Parse(time.RFC1123, strings.Trim(string(buf), `"`))
    if err != nil {
        return err
    }
    t.Time = tt
    return nil
}

func main() {
    var t myTime

    str := fmt.Sprintf("%q", time.Unix(1438947306, 123).Format(time.RFC1123))
    fmt.Printf("json string: %s\n", str)
    if err := json.Unmarshal([]byte(str), &t); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("result: %s\n", t.Format(time.RFC1123))
}

And now it works! We have a json unmarshal that supports RFC1123 instead of RFC3339!

To implment the json.Unmarshaler interface, we need to write the func (t *myTime) UnmarshalJSON(buf []byte) error method.

This receives the json buffer and return an error. It is expected to set the parsed value to the receiver so it is important that the receiver is a pointer.

The first step, has we expect valid json is to trim down the " from the string, then we call the time.Parse and finally set the result to our object.

Marshal RFC1123

Instead of the default RFC3339, let’s have json encode our time as RFC1123:

package main

import (
    "encoding/json"
    "os"
    "time"
)

type myTime struct {
    time.Time
}

func (t myTime) MarshalJSON() ([]byte, error) {
    return []byte(`"` + t.Time.Format(time.RFC1123) + `"`), nil
}

func main() {
    now := myTime{time.Unix(1438947306, 123)}
    _ = json.NewEncoder(os.Stdout).Encode(now)
}

Same idea as unmarshal. Here we only dump data so we don’t want the receiver to be a pointer and we make sure that we return valid json wrapped in ".

Going further

Changing the time format is great, but what if we need to move around dates as a timestamp integer? Or as a nanosecond timestamp? Or if we expect arbitrary format?

What if we have a REST API that need to move date between json and bson?

The timeutils library (http://github.com/simplereach/timeutils) offers a Time type that supports arbitrary time format via aproxidate as well as Timestamp and nanosecond precision both for marshal/unmarshal in json and bson.

Parsing HTTP Query String in Go

TL;DR

Code and examples can be found here: https://github.com/creack/httpreq

HTTP Server

Go provides a very easy way to create a http server:

package main

import (
        "fmt"
        "log"
        "net/http"
)

func handler(w http.ResponseWriter, req *http.Request) {
        fmt.Fprintf(w, "hello world\n")
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}
// curl http://localhost:8080

But how to deal with data?

JSON body

A common way to pass data is via a json encoded body:

package main

import (
        "encoding/json"
        "fmt"
        "log"
        "net/http"
)

type query struct {
        Name string
}

func handler(w http.ResponseWriter, req *http.Request) {
        q := &query{}
        if err := json.NewDecoder(req.Body).Decode(q); err != nil {
                log.Printf("Error decoding body: %s", err)
                return
        }
        fmt.Fprintf(w, "hello %s\n", q.Name)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}
// curl -d '{"Name": "Guillaume"}' http://localhost:8080/

Query String

But what if we want to pass data via query string? Typically, pagination and extra data.

Go, once again, expose everything necessary:

package main

import (
        "fmt"
        "log"
        "net/http"
        "strconv"
)

func handler(w http.ResponseWriter, req *http.Request) {
        if err := req.ParseForm(); err != nil {
                log.Printf("Error parsing form: %s", err)
                return
        }
        l := req.Form.Get("limit")
        limit, err := strconv.Atoi(l)
        if err != nil {
                log.Printf("Error parsing limit: %s", err)
                return
        }

        dr := req.Form.Get("dryrun")
        dryRun, _ := strconv.ParseBool(dr)
        fmt.Fprintf(w, "hello world. Limit: %d, Dryrun: %t\n", limit, dryRun)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}
// curl 'http://localhost:8080?limit=42&dryrun=true'

As we can see, it works as expected, however, if we add more and more fields to our query string, the type conversions quickly become cumbersome.

A Better Query String management

We know how to convert any string to any type.
We know what data we are expecting.
We should be able to do something similar to json.Unmarshal.

Conversion functions

Let’s start with our previous example: we need an int and a bool. However, the strconv functions have different prototypes and return a value.

It would be interesting to write a small helper that will set a value instead of returning it. That way, we could instantiate our query object and pass the fields to be set. In order to do so, we need to use pointers.

package main

import (
        "fmt"
        "log"
        "net/http"
        "strconv"
)

type query struct {
        Limit  int
        DryRun bool
}

func parseBool(s string, dest *bool) error {
        // assume error = false
        *dest, _ = strconv.ParseBool(s)
        return nil
}

func parseInt(s string, dest *int) error {
        n, err := strconv.Atoi(s)
        if err != nil {
                return err
        }
        *dest = n
        return nil
}

func handler(w http.ResponseWriter, req *http.Request) {
        if err := req.ParseForm(); err != nil {
                log.Printf("Error parsing form: %s", err)
                return
        }
        q := &query{}
        if err := parseBool(req.Form.Get("dryrun"), &q.DryRun); err != nil {
                log.Printf("Error parsing dryrun: %s", err)
                return
        }
        if err := parseInt(req.Form.Get("limit"), &q.Limit); err != nil {
                log.Printf("Error parsing limit: %s", err)
                return
        }
        fmt.Fprintf(w, "hello world. Limit: %d, Dryrun: %t\n", q.Limit, q.DryRun)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}

Make it generic

It is a bit better, but still could be improved. What if we’d like to have this in a generic way?

As we can see, the conversion helpers have a very similar prototype, let’s make it the same using interface{}

package main

import (
        "fmt"
        "log"
        "net/http"
        "strconv"
)

type query struct {
        Limit  int
        DryRun bool
}

func parseBool(s string, dest interface{}) error {
        d, ok := dest.(*bool)
        if !ok {
                return fmt.Errorf("wrong type for parseBool: %T", dest)
        }
        // assume error = false
        *d, _ = strconv.ParseBool(s)
        return nil
}

func parseInt(s string, dest interface{}) error {
        d, ok := dest.(*int)
        if !ok {
                return fmt.Errorf("wrong type for parseInt: %T", dest)
        }
        n, err := strconv.Atoi(s)
        if err != nil {
                return err
        }
        *d = n
        return nil
}

func handler(w http.ResponseWriter, req *http.Request) {
        if err := req.ParseForm(); err != nil {
                log.Printf("Error parsing form: %s", err)
                return
        }
        q := &query{}
        if err := parseBool(req.Form.Get("dryrun"), &q.DryRun); err != nil {
                log.Printf("Error parsing dryrun: %s", err)
                return
        }
        if err := parseInt(req.Form.Get("limit"), &q.Limit); err != nil {
                log.Printf("Error parsing limit: %s", err)
                return
        }
        fmt.Fprintf(w, "hello world. Limit: %d, Dryrun: %t\n", q.Limit, q.DryRun)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}

Parsing object

Now that we have generic helpers, we can easily write a small object that will simplify the way we use it:

We need to store N parsing functions, so we’ll need a slice (or a map). In order to parse a field, we need the helper function, but we also need the original string and the destination.

We have our object!

type parsingMap []parsingMapElem

type parsingMapElem struct {
        Field string
        Fct   func(string, interface{}) error
        Dest  interface{}
}

Once our paringMap constructed, we then need to execute it, let’s write the loop logic:

func (p parsingMap) parse(form url.Values) error {
        for _, elem := range p {
                if err := elem.Fct(elem.Field, elem.Dest); err != nil {
                        return err
                }
        }
        return nil
}

We know can put everything together:

package main

import (
        "fmt"
        "log"
        "net/http"
        "net/url"
        "strconv"
)

// conversion helpers
func parseBool(s string, dest interface{}) error {
        d, ok := dest.(*bool)
        if !ok {
                return fmt.Errorf("wrong type for parseBool: %T", dest)
        }
        // assume error = false
        *d, _ = strconv.ParseBool(s)
        return nil
}

func parseInt(s string, dest interface{}) error {
        d, ok := dest.(*int)
        if !ok {
                return fmt.Errorf("wrong type for parseInt: %T", dest)
        }
        n, err := strconv.Atoi(s)
        if err != nil {
                return err
        }
        *d = n
        return nil
}

// parsingMap
type parsingMap []parsingMapElem

type parsingMapElem struct {
        Field string
        Fct   func(string, interface{}) error
        Dest  interface{}
}

func (p parsingMap) parse(form url.Values) error {
        for _, elem := range p {
                if err := elem.Fct(elem.Field, elem.Dest); err != nil {
                        return err
                }
        }
        return nil
}

// http server
type query struct {
        Limit  int
        DryRun bool
}

func handler(w http.ResponseWriter, req *http.Request) {
        if err := req.ParseForm(); err != nil {
                log.Printf("Error parsing form: %s", err)
                return
        }
        q := &query{}
        if err := (parsingMap{
                {"limit", parseInt, &q.Limit},
                {"dryrun", parseBool, &q.DryRun},
        }).parse(req.Form); err != nil {
                log.Printf("Error parsing query string: %s", err)
                return
        }

        fmt.Fprintf(w, "hello world. Limit: %d, Dryrun: %t\n", q.Limit, q.DryRun)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}

Going Further

I wrote this small library: https://github.com/creack/httpreq which provides more helpers and a cleaner API. It fits my current use case, but feel free to add any helper that can be missing :)

HTTP and Error management in Go

Go comes with a great standard library including net/http which allow a developer to create a reliable http server very easily.

TL;DR

Code and example can be found here: https://github.com/creack/ehttp

Basic http server

To provide context, let’s take a look at a basic http server in Go:

package main

import (
        "fmt"
        "log"
        "net/http"
)

func handler(w http.ResponseWriter, req *http.Request) {
        fmt.Fprintln(w, "hello world")
}

func main() {
        http.HandleFunc("/", handler)
        log.Fatal(http.ListenAndServe(":8080", nil))
}

The first step is to define a handler. In order to be understood by the Go’s http library, the handler needs to follow this specific prototype: func(http.ResponseWriter, *http.Request) which is defined as the http.HandlerFunc type.
Once we have our handler, we can register it on a specific route and then start the server.

This is great! In very few lines of code, we have a working web server ready to go!

Error management

You might have noticed: our handler does not return an error… But luckily, the net/http package exposes the http.Error function in order to report an error to the client. This will set the Content-Type, send a custom http status header and write the error as the body.

Small example:

func handler(w http.ResponseWriter, req *http.Request) {
        if err := doSomething(); err != nil {
                http.Error(w, err.Error(), http.StatusInternalServerError)
                return
        }
        fmt.Fprintln(w, "hello world")
}

As you can imagine, this can become cumbersome pretty fast. We could imagine writing a wrapper for http.Error which is going to log the error, send instrumentations and then call http.Error, but when doing a lot in a handler, we always need to have that call + return.
A nice way to go would be to consider the handler as a simple entrypoint and avoid doing any logic directly in the handler. This is a good approach, especially when you don’t want to be tight to http and able to switch to other protocols.

Custom Handler

A solution is to create a custom handler, let’s try:

type HandlerFunc func(http.ResponseWriter, *http.Request) error

Pretty straight forward for now, but wait, http.HandleFunc expects a different prototype, so we won’t be able to use it anymore, right?
Kind of.. Right in a sense that we can’t use it directly, but we always can work around anything ;)

http.HandlerFunc

In order to “convert” our custom handler to the native http one, we need to write a middleware and we are going to use that to handle our error management.

So, what is a middleware? It a simple “layer” that comes in between the client’s request and our final handler.

// MWError is the main middleware. When an error is returned, it send
// the data to the client if the header hasn't been sent yet, otherwise, log them.
func MWError(hdlr HandlerFunc) http.HandlerFunc {
        return func(w http.ResponseWriter, req *http.Request) {
                if err := hdlr(w, req); err != nil {
                        http.Error(w, err.Error(), http.StatusInternalError)
                        return
                }
        }
}

So, we have a function that takes our custom handler function as a parameter and return a native http HandlerFunc.

router.HandleFunc("/", MWError(hdlr))

When the client calls our server, it ends up in that newly generated native http Handler which in turn calls our custom one and then handle the error.

http.Handler

Alternatively, we can implement the http.Handler interface on our custom type so it can be used by all the http functions expecting that interface (I am thinking mainly about http.Handle and http.ListenAndServe

type HandlerFunc func(http.ResponseWriter, *http.Request) error

func (hdlr HandlerFunc) ServeHTTP(w http.ResponseWriter, req *http.Request) {
        if err := hdlr(w, req); err != nil {
                http.Error(w, err.Error(), http.StatusInternalError)
                return
        }
}

Going further

Now that we have a custom http handler and we can return error, we can think about improvements:
- smarter error management
- custom error type holding the HTTP Status
- panic recovery
- adaptors for non-standard library routers (gorilla, httprouter, etc)
- headers detection (we can’t send error headers if they have already been sent)

I started a small library that implements all this: https://github.com/creack/ehttp, it is very very simple and close to the standard library.

Reverse Proxy in Go

TL;DR

The final code can be found here: https://github.com/creack/goproxy

Goal

In this article, we are going to dive into the standard library's Reverse Proxy and see how to use it as a load balancer with persistent connections that doesn't lose any requests!

Here is our example setup:

• Service One - version 1 running on http://localhost:9091/ and http://localhost:9092/
• Reverse Proxy on http://localhost:9090/< service name>/< service version>/

When calling http://localhost:9090/serviceone/v1/, we want the proxy to balance between
http://localhost:9091/ and http://localhost:9092/ without loosing any request if one of the hosts goes down.

Standard Library Example

Let’s start with the doc: http://godoc.org/net/http/httputil#ReverseProxy.
We can see that the ReverseProxy structure has the ServerHTTP method, which means that we can use it as HTTP router directly with http.ListenAndServe.
There is also NewSingleHostReverseProxy, which sound great: we have an example on how to instantiate a ReverseProxy that works with a single host! So let’s see what it looks like:

// NewSingleHostReverseProxy returns a new ReverseProxy that rewrites
// URLs to the scheme, host, and base path provided in target. If the
// target's path is "/base" and the incoming request was for "/dir",
// the target request will be for /base/dir.
func NewSingleHostReverseProxy(target *url.URL) *ReverseProxy {
        targetQuery := target.RawQuery
        director := func(req *http.Request) {
                req.URL.Scheme = target.Scheme
                req.URL.Host = target.Host
                req.URL.Path = singleJoiningSlash(target.Path, req.URL.Path)
                if targetQuery == "" || req.URL.RawQuery == "" {
                        req.URL.RawQuery = targetQuery + req.URL.RawQuery
                } else {
                        req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
                }
        }
        return &ReverseProxy{Director: director}
}

The function takes a target as a parameter. This is going to be our target host URL.
Let’s skip the RawQuery part, it is simply used to forward properly the query string arguments.
Then we have director which we then give to the ReverseProxy object. This is what defines the behavior of our reverse proxy.
That director function takes the destination query as a parameter and needs to update it with the expected parameter. First, we need to set the request’s URL, the important parts are the Scheme and Host. The Path and RawQuery are used to manipulate the HTTP route.

So let’s try!

First, let’s write a small http server which is going to be our target server:

package main

import (
        "log"
        "net/http"
        "os"
        "strconv"
)

func main() {
        if len(os.Args) != 2 {
                log.Fatalf("Usage: %s <port>", os.Args[0])
        }
        if _, err := strconv.Atoi(os.Args[1]); err != nil {
                log.Fatalf("Invalid port: %s (%s)\n", os.Args[1], err)
        }

        http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
                println("--->", os.Args[1], req.URL.String())
        })
        http.ListenAndServe(":"+os.Args[1], nil)
}

This small http server listens on the first command line argument port and when called, displays the port and the http request url.

Now, let’s write a small reverse proxy:

package main

import (
        "net/http"
        "net/http/httputil"
        "net/url"
)

func main() {
        proxy := httputil.NewSingleHostReverseProxy(&url.URL{
                Scheme: "http",
                Host:   "localhost:9091",
        })
        http.ListenAndServe(":9090", proxy)
}

The code is straight forward: We create a new single host reverse proxy that targets http://localhost:9091/ and listens on 9090.

Try it! It works fine. curl http://localhost:9090 forwards properly to our http server running on 9091.

Multiple hosts case

The example we saw is working great and is very simple, but not really useful in production. What if we want to have more than one host?

Director

As we saw earlier, the main logic of the reverse proxy resides in the Director member. So let’s try to create our own ReverseProxy object.
We are going to copy/paste the httputil.NewSingleHostReverseProxy code and change the prototype to take a slice of url so we can balance between given hosts and alter the code to use a random host from the given ones.

package main

import (
        "log"
        "math/rand"
        "net/http"
        "net/http/httputil"
        "net/url"
)

// NewMultipleHostReverseProxy creates a reverse proxy that will randomly
// select a host from the passed `targets`
func NewMultipleHostReverseProxy(targets []*url.URL) *httputil.ReverseProxy {
        director := func(req *http.Request) {
                target := targets[rand.Int()%len(targets)]
                req.URL.Scheme = target.Scheme
                req.URL.Host = target.Host
                req.URL.Path = target.Path
        }
        return &httputil.ReverseProxy{Director: director}
}

func main() {
        proxy := NewMultipleHostReverseProxy([]*url.URL{
                {
                        Scheme: "http",
                        Host:   "localhost:9091",
                },
                {
                        Scheme: "http",
                        Host:   "localhost:9092",
                },
        })
        log.Fatal(http.ListenAndServe(":9090", proxy))
}

Demo time

Caveat

At the end of the previous demo, I kill one of the http server and we can see that the reverse proxy yield errors when hitting that host. This result in request loss, which is not ideal. Having a host going down happens, it should be the role of our proxy to make sure the client’s request reaches the expected target.

In order to understand what is going on, let’s dive in the ServerHTTP method. We can see at the beginning:

        transport := p.Transport
        if transport == nil {
                transport = http.DefaultTransport
        }

This means that because we didn’t provide a Transport object, the reverse proxy will use the default one.
Now let’s take a look at the default Transport:

var DefaultTransport RoundTripper = &Transport{
        Proxy: ProxyFromEnvironment,
        Dial: (&net.Dialer{
                Timeout:   30 * time.Second,
                KeepAlive: 30 * time.Second,
        }).Dial,
        TLSHandshakeTimeout: 10 * time.Second,
}

Proxy is a function that will apply the proxy settings, by default, it looks up the env HTTP_PROXY and co.
The next one is more interesting: Dial. It defines how to establish the connection to the target host. The default Transport uses the Dialer from net with some timeouts/keepalive settings.

The error yielded by the reverse proxy when one host went down is: http: proxy error: dial tcp 127.0.0.1:9091: getsockopt: connection refused. It is pretty clear: the issue comes from Dial.

To understand the behavior, let’s extend a bit our code to add some output so we can see exactly what gets called and when.

package main

import (
        "log"
        "math/rand"
        "net"
        "net/http"
        "net/http/httputil"
        "net/url"
        "time"
)

// NewMultipleHostReverseProxy creates a reverse proxy that will randomly
// select a host from the passed `targets`
func NewMultipleHostReverseProxy(targets []*url.URL) *httputil.ReverseProxy {
        director := func(req *http.Request) {
                println("CALLING DIRECTOR")
                target := targets[rand.Int()%len(targets)]
                req.URL.Scheme = target.Scheme
                req.URL.Host = target.Host
                req.URL.Path = target.Path
        }
        return &httputil.ReverseProxy{
                Director: director,
                Transport: &http.Transport{
                        Proxy: func(req *http.Request) (*url.URL, error) {
                                println("CALLING PROXY")
                                return http.ProxyFromEnvironment(req)
                        },
                        Dial: func(network, addr string) (net.Conn, error) {
                                println("CALLING DIAL")
                                conn, err := (&net.Dialer{
                                        Timeout:   30 * time.Second,
                                        KeepAlive: 30 * time.Second,
                                }).Dial(network, addr)
                                if err != nil {
                                        println("Error during DIAL:", err.Error())
                                }
                                return conn, err
                        },
                        TLSHandshakeTimeout: 10 * time.Second,
                },
        }
}

func main() {
        proxy := NewMultipleHostReverseProxy([]*url.URL{
                {
                        Scheme: "http",
                        Host:   "localhost:9091",
                },
                {
                        Scheme: "http",
                        Host:   "localhost:9092",
                },
        })
        log.Fatal(http.ListenAndServe(":9090", proxy))
}

What did we do? We simply reused the code of http.DefaultTransport and add some logging.

More Verbose Demo

As we can see, Dial is called only the first time Director yields a host, after that it reuses the already existing connection in the internal’s pool of ReverseProxy
When one of the servers goes away, the ReverseProxy receives EOF and remove the connection from the pool resulting in a new call to Dial upon next request.

Routing

Let’s put the request loss on the side for the moment and address the routing based on the request’s path.

Service Registry

In order to easily lookup an endpoint for a given service, let’s create a small Registry type instead of using a slice of *url.URL:

type Registry map[string][]string

var ServiceRegistry = Registry{
    "serviceone/v1": {
        "localhost:9091",
        "localhost:9092",
    },
}

Extract Service and Version from Request

In order to know what service we are targeting, we use the /serviceName/serviceVersion/ prefix in the path.

func extractNameVersion(target *url.URL) (name, version string, err error) {
        path := target.Path
        // Trim the leading `/`
        if len(path) > 1 && path[0] == '/' {
                path = path[1:]
        }
        // Explode on `/` and make sure we have at least
        // 2 elements (service name and version)
        tmp := strings.Split(path, "/")
        if len(tmp) < 2 {
                return "", "", fmt.Errorf("Invalid path")
        }
        name, version = tmp[0], tmp[1]
        // Rewrite the request's path without the prefix.
        target.Path = "/" + strings.Join(tmp[2:], "/")
        return name, version, nil
}

It is pretty straightforwrd but wait, where does that target *url.URL comes from?
You might have guess, it is the req.URL from our Director.

Registry Example

Let’s put all this together based on our first multi host example:

package main

import (
        "log"
        "math/rand"
        "net"
        "net/http"
        "net/http/httputil"
        "net/url"
        "time"
)

type Registry map[string][]string

func extractNameVersion(target *url.URL) (name, version string, err error) {
        path := target.Path
        // Trim the leading `/`
        if len(path) > 1 && path[0] == '/' {
                path = path[1:]
        }
        // Explode on `/` and make sure we have at least
        // 2 elements (service name and version)
        tmp := strings.Split(path, "/")
        if len(tmp) < 2 {
                return "", "", fmt.Errorf("Invalid path")
        }
        name, version = tmp[0], tmp[1]
        // Rewrite the request's path without the prefix.
        target.Path = "/" + strings.Join(tmp[2:], "/")
        return name, version, nil
}

// NewMultipleHostReverseProxy creates a reverse proxy that will randomly
// select a host from the passed `targets`
func NewMultipleHostReverseProxy(reg Registry) *httputil.ReverseProxy {
        director := func(req *http.Request) {
                name, version, err := extractNameVersion(req.URL)
                if err != nil {
                    log.Print(err)
                    return
                }
                endpoints := reg[name+"/"+version]
                if len(endpoints) == 0 {
                        log.Printf("Service/Version not found")
                        return
                }
                req.URL.Scheme = "http"
                req.URL.Host = endpoints[rand.Int()%len(endpoints)]
        }
        return &httputil.ReverseProxy{
                Director: director,
        }
}

func main() {
        proxy := NewMultipleHostReverseProxy(Registry{
                        "serviceone/v1": {"localhost:9091"},
                        "serviceone/v2": {"localhost:9092"},
        })
        log.Fatal(http.ListenAndServe(":9090", proxy))
}

We now have a working load balancer!
But we still have an issue when a host goes down..

Avoid loosing request

So, what can we do? When a host is down, the error comes from Dial but our logic is in Director.
So let’s move the logic to Dial! Indeed, it would be great but there is one big issue:
Dial does not know anything about the request: we can’t lookup the target service endpoint list.
In order to work around this, we are going to do something a bit hackish: use the Request’s Host has a placeholder!
We are going to put serviceName/serviceVersion has a string inside the Request which later on will be passed on to Dial where we can lookup the endpoints for our services.

func NewMultipleHostReverseProxy(reg Registry) *httputil.ReverseProxy {
    director := func(req *http.Request) {
        name, version, err := extractNameVersion(req.URL)
        if err != nil {
            log.Print(err)
            return
        }
        req.URL.Scheme = "http"
        req.URL.Host = name + "/" + version
    }
    return &httputil.ReverseProxy{
        Director: director,
        Transport: &http.Transport{
            Proxy: http.ProxyFromEnvironment,
            Dial: func(network, addr string) (net.Conn, error) {
                // Trim the `:80` added by Scheme http.
                addr = strings.Split(addr, ":")[0]
                endpoints := reg[addr]
                if len(endpoints) == 0 {
                    return nil, fmt.Errorf("Service/Version not found")
                }
                return net.Dial(network, endpoints[rand.Int()%len(endpoints)])
            },
            TLSHandshakeTimeout: 10 * time.Second,
        },
    }
}

Going further

Registry

The github.com/creack/goproxy/registry package exposes a Registry interface:

// Registry is an interface used to lookup the target host
// for a given service name / version pair.
type Registry interface {
        Add(name, version, endpoint string)                // Add an endpoint to our registry
        Delete(name, version, endpoint string)             // Remove an endpoint to our registry
        Failure(name, version, endpoint string, err error) // Mark an endpoint as failed.
        Lookup(name, version string) ([]string, error)     // Return the endpoint list for the given service name/version
}

Add and Delete are used to control the content of our registry. We might want to call Add when a new host is available and Delete when one goes away.
Failure is called when Dial fails, which probably means the target is not available anymore. We can use that method to store how many time it fails and eventually call Delete to remove the faulty host.
It is a good place to put some logging and instrumentation.
Lookup is pretty straight forward, it returns the hosts list for the given service name/version.

This interface can be implemented using ZooKeeper, etcd, consul or any service you might be using. The default implementation is a naive map.

Load Balancer

The github.com/creack/goproxy package is basically our latest example hooked with the Registry interface.

In top of NewMultiplHostReverProxy, it also exposes two functions: ExtractNameVersion and LoadBalance. They are not exposed in order to be used, but in order to be overridden.

ExtractNameVersion can be replace by a custom one in order to have a different path model.
LoadBalance is the load balancer logic. It takes the target service name and version as well as the registry and yield a net.Conn. The default one is a random but can be replaced by a smarter one.

Scope and Shadowing in Go

Shadowing variable in Go

Variable shadowing can be confusing in Go, let’s try to clear it up.

Case of errors

Without even maybe knowing it, you have been playing with shadowing with your errors.
Consider the following code:

package main

import (
    "io/ioutil"
    "log"
)

func main() {
    f, err := ioutil.TempFile("", "")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    if _, err := f.Write([]byte("hello world\n")); err != nil {
        log.Fatal(err)
    }
}

Notice that we first create two variable: f and err from the TempFile function.
We then call Write discarding the number of bytes written. We make the function call it within the if statement.
Let’s compile, it work fine.

Now, the same code with the Write call outside the if:

package main

import (
    "io/ioutil"
    "log"
)

func main() {
    f, err := ioutil.TempFile("", "")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    _, err := f.Write([]byte("hello world\n"))
    if err != nil {
        log.Fatal(err)
    }
}

Now, compilation fails: main.go:15: no new variables on left side of :=

So what happened?

Note that we call Write with :=, which means that we create a new variable err. In the second example, it is pretty obvious, err already exists so we can’t redeclare it. But then why did it work the first time?
Because in Go, variables are local to their scope. In the first example, we actually shadowed err within the if scope.

Simple Demo

package main

func main() {
    var err error
    _ = err
    var err error
    _ = err
}

This will obviously fail, however, if we scope the second err, it will work!

package main

func main() {
    var err error
    _ = err
    {
        var err error
        _ = err
    }
}

Package Shadowing

Consider the following code:

package main

import "fmt"

func Debugf(fmt string, args ...interface{}) {
    fmt.Printf(fmt, args...)
}

At first, it looks decent. We call Printf from the fmt package and pass the fmt variable to it.

WRONG

the fmt string from the function declaration actually shadows the package and is now “just” a variable. The compiler will complain:
We need to use a different variable name conserve access to the fmt package.

Global scope

Something to take into consideration is that a function is already a “sub scope”, it is a scope within the global scope. This means that any variable you declare within a function can shadow something from the global scope.

Just as we saw before that a variable can shadow a package, the concept is the same for global variables and functions.

Type enforcement

Just like we can shadow a package with a variable or a function, we also can shadow a variable by a new variable of any type. Shadowed variables does not need to be from the same type. This example compiles just fine:

package main

func main() {
    var a string
    _ = a
    {
        var a int
        _ = a
    }
}

Closures

The scope is very important when using embeded functions. Any variable used in a function and not declared are references to the upper scope ones.
Well known example using goroutines:

package main

import (
    "fmt"
    "time"
)

func main() {
    for _, elem := range []byte{'a', 'b', 'c'} {
        go func() {
            fmt.Printf("%c\n", elem)
        }()
    }
    time.Sleep(1e9) // Sleeping to give time to the goroutines to be executed.
}

The result is:

c
c
c

Which is not really what we wanted.
This is because the range changes elem which is referenced in the goroutine, so on short lists, it will always display the last element.

To avoid this, there are two solutions:

• Passing variable to the function

package main

import (
    "fmt"
    "time"
)

func main() {
    for _, elem := range []byte{'a', 'b', 'c'} {
        go func(char byte) {
            fmt.Printf("%c\n", char)
        }(elem)
    }
    time.Sleep(1e9)
}

• Create a copy of the variable in the local scope

package main

import (
    "fmt"
    "time"
)

func main() {
    for _, elem := range []byte{'a', 'b', 'c'} {
        char := elem
        go func() {
            fmt.Printf("%c\n", char)
        }()
    }
    time.Sleep(1e9)
}

In both case we get our expected result:

a
b
c

When we pass the variable to the function, we actually send a copy of the variable to the function which receives it as char. Because every goroutines gets its own copy, there is no problem.
When we make a copy of the variable, we create a new variable and assigns the value of elem to it.
We do this at each iteration, which means that for each steps, we create a new variable which the goroutine get a reference to. Each goroutine has a reference to a different variable and it work fine as well.

Now, as we know that we can shadow variable, why bother change the name? We can simply use the same name knowing that it will shadow the upper scope:

package main

import (
    "fmt"
    "time"
)

func main() {
    for _, elem := range []byte{'a', 'b', 'c'} {
        go func(elem byte) {
            fmt.Printf("%c\n", elem)
        }(elem)
    }
    time.Sleep(1e9)
}

package main

import (
    "fmt"
    "time"
)

func main() {
    for _, elem := range []byte{'a', 'b', 'c'} {
        elem := elem
        go func() {
            fmt.Printf("%c\n", elem)
        }()
    }
    time.Sleep(1e9)
}

When we pass the variable to the function, the same thing happens, we pass a copy of the variable to the function which gets it with the name elem with the correct value. From this scope, because the variable is shadowed, we can’t impact the elem from the upper scope and any change made will be applied only within this scope.
When we make a copy of the variable, same as before: we create a new variable and assigns the value of elem to it. In this case, that new variable happens to have the same name as the other one but the idea stays the same: new variable + assign value. As we create a new variable within the scope with the same name we effectively shadow that variable while keeping it’s value.

Case of :=

When using := with multiple return functions (or type assertion, channel receive and map access), we can endup with 3 variables out of 2 statements:

package main

func main() {
    var iface interface{}

    str, ok := iface.(string)
    if ok {
        println(str)
    }
    buf, ok := iface.([]byte)
    if ok {
        println(string(buf))
    }
}

In this situation, ok does not get shadowed, it simply gets overridden. Which is why ok can’t change type.
Doing so in a scope, however, would shadow the variable and allow for a different type:

package main

func main() {
    var m = map[string]interface{}{}

    elem, ok := m["test"]
    if ok {
        str, ok := elem.(string)
        if ok {
            println(str)
        }
    }
}

Conclusion

Shadowing can be very useful but needs to be something to keep in mind to avoid unexpected behavior.
It is of course on a case basis, it often helps readability and safety, but can also reduce it.

In the example of the goroutines, because it is a trivial example, it is more readable to shadow, but in a more complex situation, it might be best to use different names to make sure what you are modifying.
In an other hand, however, especially for errors, it is a very powerful tool.
Going back to my first example:

package main

import (
    "io/ioutil"
    "log"
)

func main() {
    f, err := ioutil.TempFile("", "")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    if _, err := f.Write([]byte("hello world\n")); err != nil {
        err = nil
    }
    // err is still the one form TempFile
}

In this situation, shadowing err within the gives a warranty that previous errors will not be impacted whereas if with the same code we used = instead of := in the if, it would not have shadowed the variable but override the value of the error.

Privileged Listen in Go

Introduction

Go does not play well with Forks and User permission. The reason is because it is a threaded runtime and there is no mechanism for clean fork or clean setuid.

To work around the fork issue, Go exposes syscall.ForkExec() which perform the fork (locked) but always perform an Exec() in the forked process, resulting in the calling one to disappear (overridden).

The Issue.

https://github.com/golang/go/issues/1435

$> GOMAXPROCS=4 ./test 65534 65534

and note output:

goroutine 1: uid=0 euid=0 gid=0 egid=0
goroutine 2: uid=0 euid=0 gid=0 egid=0
goroutine 3: uid=0 euid=0 gid=0 egid=0
goroutine 4: uid=0 euid=0 gid=0 egid=0
goroutine 5: uid=0 euid=0 gid=0 egid=0
goroutine 6: uid=0 euid=0 gid=0 egid=0
goroutine 7: uid=0 euid=0 gid=0 egid=0
goroutine 8: uid=0 euid=0 gid=0 egid=0
goroutine 9: uid=0 euid=0 gid=0 egid=0
goroutine 0: uid=65534 euid=65534 gid=65534 egid=65534
goroutine 1: uid=0 euid=0 gid=0 egid=0
goroutine 2: uid=0 euid=0 gid=0 egid=0
goroutine 3: uid=0 euid=0 gid=0 egid=0
goroutine 4: uid=0 euid=0 gid=0 egid=0
goroutine 5: uid=0 euid=0 gid=0 egid=0
goroutine 6: uid=0 euid=0 gid=0 egid=0
goroutine 7: uid=0 euid=0 gid=0 egid=0
goroutine 8: uid=0 euid=0 gid=0 egid=0
goroutine 9: uid=0 euid=0 gid=0 egid=0
goroutine 0: uid=65534 euid=65534 gid=65534 egid=65534

It would be annoying if our http Handler was ran as root!

The solution

To solve this issue, I wrote a small util: https://github.com/creack/golisten.

The idea is simple: perform the listen as root and then override the whole process with yourself as non-root.

package main

import (
    "fmt"
    "log"
    "net/http"
    "os/user"

    "github.com/creack/golisten"
)

func handler(w http.ResponseWriter, req *http.Request) {
    u, err := user.Current()
    if err != nil {
        log.Printf("Error getting user: %s", err)
        return
    }
    fmt.Fprintf(w, "%s\n", u.Uid)
}

func main() {
    http.HandleFunc("/", handler)
    log.Fatal(golisten.ListenAndServe("guillaume", ":80", nil))
}

Note

golisten is intended to be used with the http lib, but the concept can be used for any privilege de-escalation.
It is safe because we override the whole process, so all our active thread are from the child process.

Caveat

As we re-exec ourself, we need to be careful with what is done prior the call the golisten.
Maybe a bit more cumbersome, but it is best to use golisten.Listen at the beginning and pass around the listener to http.Serve() later on.

package main

import (
        "fmt"
        "log"
        "net/http"
        "os/user"

        "github.com/creack/golisten"
)

func handler(w http.ResponseWriter, req *http.Request) {
        u, err := user.Current()
        if err != nil {
                log.Printf("Error getting user: %s", err)
                return
        }
        fmt.Fprintf(w, "%s\n", u.Uid)
}

func main() {
        ln, err := golisten.Listen("guillaume", "tcp", ":80")
        if err != nil {
                log.Fatal(err)
        }
        http.HandleFunc("/", handler)
        println("ready")
        log.Fatal(http.Serve(ln, nil))
}

Futures in Go

The Problem

You have a function that returns an error, you want to run it in a goroutine. How to retrieve the error?

The Solution

I’ll take the example of Docker utils package: (from master on 08/06/2014)
https://github.com/docker/docker/blob/66c8f87e89ba0dd824cf640a159210fbbb8019ec/utils/utils.go#L40

func Go(f func() error) chan error {
        ch := make(chan error, 1)
        go func() {
                ch <- f()
        }()
        return ch
}

This small function effectively solves the issue: it starts the given function in a goroutine and return a chan which can be read to retrieve the error.

An other problem

While this work in lot of cases, sometime, in top of retrieving the error, you might as well want to retrieve some data.
Docker’s utils.Go() is nice and generic but it is maybe too generic for some situation.

Let’s take the example of a crawler, I want to do N http request and display the content, but I want to do so concurrently and I still want all the responses and all the errors.

A way to achieve this would be to return a chan struct { Reponse, error} instead of chan error, but I don’t really like that way. I prefer to return a future.

Futures / Promises

Nothing works better than an example:

package main

import (
        "io/ioutil"
        "log"
        "net/http"
        "regexp"
)

// CrawlFuture represent the actual future.
// When called, it "promises" that the request has been done
// and returns the values as if it were synchronous.
type CrawlFuture func() (*http.Response, error)

// Crawl initiates the http request and return the future
func Crawl(url string) CrawlFuture {
        var (
                ch  = make(chan *http.Response)
                err error
        )

        go func(url string) {
                defer close(ch)

                req, e1 := http.Get(url)
                err = e1
                ch <- req
        }(url)

        return func() (*http.Response, error) {
                return <-ch, err
        }
}

var regexTitle = regexp.MustCompile("<title>(.*?)</title>")

func getTitle(resp *http.Response) string {
        body, err := ioutil.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil {
                return "error with response"
        }
        matches := regexTitle.FindSubmatch(body)
        if len(matches) < 2 {
                return "no title found"
        }
        return string(matches[1])
}

func main() {
        urls := []string{
                "http://google.com",
                "http://yandex.com",
                "http://www.baidu.com",
                "http://invalid",
        }
        futures := make([]CrawlFuture, 0, len(urls))
        for _, url := range urls {
                futures = append(futures, Crawl(url))
        }
        for _, future := range futures {
                resp, err := future()
                if err != nil {
                        log.Printf("Error: %s\n", err)
                        continue
                }
                if resp.StatusCode != 200 {
                        log.Printf("Invalid status: %s %d\n", resp.Status, resp.StatusCode)
                        continue
                }
                println(getTitle(resp))
        }
}

Instead of returning a chan, I create the chan internally and return a function that wait on the chan. It allows to start N request at the same time while being able to call the future later like if it were synchronous.

Bonus

We could pass a sync.WaitGroup to the Crawl function. That way, we can initiate the crawl, do processing on the side and have a channel signal when all requests are done so we can go check the result.

ackage main

import (
        "io/ioutil"
        "log"
        "net/http"
        "regexp"
        "sync"
        "time"
)

// CrawlFuture represent the actual future.
// When called, it "promises" that the request has been done
// and returns the values as if it were synchronous.
type CrawlFuture func() (*http.Response, error)

// Crawl initiates the http request and return the future
func Crawl(url string, wg *sync.WaitGroup) CrawlFuture {
        var (
                ch  = make(chan *http.Response)
                err error
        )

        wg.Add(1)
        go func(url string) {
                defer close(ch)

                req, e1 := http.Get(url)
                err = e1
                wg.Done()
                ch <- req
        }(url)

        return func() (*http.Response, error) {
                return <-ch, err
        }
}

var regexTitle = regexp.MustCompile("<title>(.*?)</title>")

func getTitle(resp *http.Response) string {
        body, err := ioutil.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil {
                return "error with response"
        }
        matches := regexTitle.FindSubmatch(body)
        if len(matches) < 2 {
                return "no title found"
        }
        return string(matches[1])
}

func main() {
        urls := []string{
                "http://google.com",
                "http://yandex.com",
                "http://www.baidu.com",
                "http://invalid",
        }
        wg := &sync.WaitGroup{}
        done := make(chan struct{})

        futures := make([]CrawlFuture, 0, len(urls))
        for _, url := range urls {
                futures = append(futures, Crawl(url, wg))
        }
        go func() { wg.Wait(); close(done) }()

        select {
        case <-time.After(5 * time.Second):
                log.Fatal("timeout")
        case <-done:
                for _, future := range futures {
                        resp, err := future()
                        if err != nil {
                                log.Printf("Error: %s\n", err)
                                continue
                        }
                        if resp.StatusCode != 200 {
                                log.Printf("Invalid status: %s %d\n", resp.Status, resp.StatusCode)
                                continue
                        }
                        println(getTitle(resp))
                }
        }
}

That way, you have the warranty that your loop in the done case will be non-blocking. People familiar with select(2) will be happy: don’t start something unless you know it is ready.

How to properly contribute to a Go project

Golang as a very strict directory hierarchy on which the builder (and more generally the Go tools) rely on.

I am often asked what is the proper way to contribute to a Go project.

Use cases

• Contributing to a random Go project
• Working as a team on a “central” repository
• Contributing to a Go project with sub-repositories

The problem

The issue is that when you go get a project, it ends up in (with github) $GOPATH/src/github.com/creack/termios.

If you fork the project, you will have github.com/<myuser>/termios.

If you go get that forked URL, it will work fine. However, if you are dealing with a library that needs to be imported, then you need to update the import paths. Even worse, if the project has sub-repositories, then all import paths will target the original project.

Solution

Use your fork only as remote placeholder, do not use it locally.
Use the remote feature of git.

Example for user gcharmes trying to contribute to creack/termios

1. Fork the project
2. get the original one go get github.com/creack/termios
3. cd $GOPATH/src/github.com/creack/termios
4. Add the new remote: git remote add gcharmes git@github.com:gcharmes/termios.git
5. Fetch everything from github git fetch --all
6. Checkout in a new branch git checkout -b mybranch
7. Contribute, as we are on the original checkout, all the paths are correct
8. Commit and push git commit -p && git push gcharmes mybranch
9. Go to github and create the pull request.

That’s it! This work if you want to fork a project and use it transparentely, it works when working on sub-packages, it works everywhere :)

Extra tip: use Godeps (https://github.com/tools/godep). When using external dependencies, you should vendor them.
It as also the advantage to backup your local version, so if you submitted a PR that didn’t get accepted yet, you still can commit to your own project using your commit for that dependency.

Half a decade with Go - User side

Yesterday was Go 5th anniversary. For fun, I looked for trace of activity around that time. Today is the 5th anniversary of the oldest trace I found of myself playing with Go:
http://go-lang.cat-v.org/irc-logs/old-go-nuts

(11/11/2009) 23:56 -!- creack [n=creack@ip-67.net-80-236-112.lhaylesroses.rev.numericable.fr]
has joined #go-nuts

I will not repeat what has been said in the official Go blog, but here is a small history of what I have been playing with in Go.

Timeline

• IRC bot (2009)
• Shell (2009/2010)
• FIX (4.4) Engine (2010)
• Trading platform (2010/2011)
• Crypto tools (2011)
• OpenCV (2011)
• “scripts” tools form platform / testing (2012)
• Docker (2013/2014)
• Redis Server (2013/2014)
• Ray Tracer (2014)
• Citrix Application Platform (2014)
• Interactive Brokers API (2014)

I’ll try to talk about my experience through all this in future posts.

IRC Bot

The first thing I did in order to try the language. It was fun and I ended up with a working bot pretty quickly. However, after couple of weeks, it was not compiling anymore and I never updated it.
An IRC bot is very easy to implement, you just need to play with network and strings. I think it is a nice exercice for anyone trying Go for the first time.

Shell

As a student, one of my first project in C was a Shell, so I started to do the same in Go for learning purpose.
One of the issues was the signal management. Go didn’t allow to catch SIGQUIT (C-\), so I had to call CGO for it. (Otherwise, when pressing C-\, even while running a process would kill the shell with a backtrace)
An other was the lack of fork. So I started to call the syscall with SYS_FORK. But then, I had issues with wait(): There is not WAIT_ANY macro. When using the common -1, I ended up with random behavior depending on the platform. Also, on darwin, W_UNTRACED didn’t seem to be working.
Then comes the termcaps. A very important part of a shell is the UI, i.e. the command line. Setting things like the raw mode and the ICANON mode have been quite a challenge.

FIX Engine

When I started to develop a trading platform, I needed to communicate with the broker using FIX4.4. I first looked at QuickFix, but it was C++ and I really wanted to use Go. So I started to rewrite an engine :)
Unfortunately, the language were still evolving very fast and I had to rewrite my code pretty often. Even though Go provided a tool to help the rewrite, it was often failing.
I also had issue with socket stability, I think related to some weird stuff done by the GC. In the end, I dropped Go and went back to C++.

Trading Platform

Even though I dropped Go for the FIX Engine, I still used it for some other components. My favorite was a live vizualiser of Stock/Forex prices. You could subscrube to any Ticker and have a the prive live in a webpage.
I achieved this using Go and socket.io.
The backtest engine was also written in Go.

Crypto Tools

In 2011, I played a lot with Cryptography. Even though it was already implemented in the standard library, I wrote an RSA and 3DES implementation.
I also wrote couple of tools to generate keys, try to “hack” an encrypted message, embed messages within image (steganography).

OpenCV

In 2011, I found an OpenCV binding for Go, so I had fun with it. I implemented a Canny edge detection over the webcam, which I presented at the university. Thanks to this demo, I have been proposed a PhD at the State university of St Petersburg, Russia by Dr., Professor Gennady Yanovsky

Scripts for Platform / Testing

In 2012, while I was working in PHP for PrestaShop, I didn’t get the chance to write much of Go. So each time I needed to write a script in order to test something, monitor a service, manage a server, instead of using Python or Bash, I always used Go :)

Docker

In 2013, I started to work on Docker, once the 1.0 have been released, I took distance from the project.
There, I had similar issues as when I was working on a Shell: Termcaps, raw mode, fork, syscalls, etc.. However, Go was already in 1.0 and I managed to do everything I needed to.
The main big issue that we still can see the trace of today was the lack of pointer to method. This is why Docker REST Api uses functions instead of method: at the time, we could not pass methods around!

Go Redis Server

While at Docker, I also worked on a Redis Server in Go. Something I really liked about this was the variable channel management. In Go, you can’t use select on an arbitrary chan, you need to specify which one to use. In order to work around this, I had to “rebuild” select using the relect package.
The performances are not (yet) there, but the idea is.

Ray Tracer

For quite some time, I wanted to write a Ray Tracer, so why not in Go? I found a X11 library in pure Go so I started.
I’ll be talking about this more in depth at the GopherCon in Bangaluru, India in February, 2015.

Citrix Application Platform

After Docker, I joined Citrix where I integrated the Application Platform Group. There I keep writing go on a daily basis. We migrate services from Ruby to Go and develop new ones.

Interactive Brokers API

I have been using Interactive Brokers for quick some time now. I love them, however, there “interactive” is pretty difficult to use. In order to use their API, you need to run a local Java client (GUI, of course). They provide a Java and C++ sdk. The Gofinance project brings the power of Go to IB. And with Docker, I completely removed the need for the local GUI or even Java.
/mylife

Release Go code (and others) via Docker using Makefile

In this article, I’ll demonstrate how to leverage Makefile in order to release lightweight Docker image for production.

Sample Application code

package main

func main() {
        println("hello world")
}

Of course, do not forget to vendor your dependencies: godep save.

In order to create a release image, we first need to build the binary. We will have 2 Dockerfiles.

Main Dockerfile

The main Dockerfile is the “classic” one. For our example:

# Using google/golang as base image
FROM            google/golang:stable
# Install Godep for vendoring
RUN             go get github.com/tools/godep
# Recompile the standard library without CGO
RUN             CGO_ENABLED=0 go install -a std
# Declare the maintainer
MAINTAINER      Guillaume J. Charmes <guillaume@charmes.net>

# For convenience, set an env variable with the path of the code
ENV             APP_DIR         $GOPATH/src/example

# Set the entrypoint as the binary, so `docker run <image>` will behave as the binary
ENTRYPOINT      ["/example"]
# Add the sources to the APP_DIR
ADD             .       $APP_DIR
# Compile the binary and statically link
RUN             cd $APP_DIR && CGO_ENABLED=0 godep go build -o /example -ldflags '-d -w -s'

Note that the godep install and the CGO disabled std rebuild are done before the maintainer, this allows to keep the cache for this part when the maintainer changes.
In a future article, I’ll talk more in depth about static linking in Go (the -ldflags and go install std thingy)

Release Dockerfile

In order to release, we need a second Dockerfile. As Docker builds directories, we need to create a subdirectory release.

The release Makefile is very straight forward and look like this:

# Use "scratch" as base: it is an empty image.
FROM            scratch
# Set the entrypoint as the binary, so `docker run <image>` will behave as the binary
ENTRYPOINT      ["/example"]
# Add the binary. As it is statically linked, no need to add libc or anything else.
ADD             example /

Now, we need a Makefile to automate the process

Makefile

The Makefile allow for easy dependency scripting.

The goal is to push a “release” image. In order to do this, we need to have that image built.
In order to build this image, we need the binary, In order to get the binary, we need to build.
In order to build, we need the source. For convenience, we monitor . instead of each individual go files.

The default rule is all which triggers build

build depends on .build which is a file. If it exists then move on to the next dependency, otherwise, execute the rule. .build depend on .. This means that if anything changes in the current directoty, the cache gets invalidated and the rule will be executed.
In order to build, we call docker build and create the .build file.
The result is a full blown image roughly 600MB. release will help with this.
The first step is to extract the binary and /etc/ssl. The /etc/ssl is mandatory only if you plan on using SSL (otherwise, Go will complain it does not find the certificates). Once extracted in a tarball, we build the release Dockerfile.

NAME            = example
DOCKER_IMAGE    = 127.0.0.1:5000/$(NAME)

all             : build

.build          : .
                docker build -t $(NAME) .
                docker inspect -f '{{.Id}}' $(NAME) > .build

build           : .build

release/$(NAME) : build
                docker run --rm --entrypoint /bin/sh $(NAME) -c 'tar cf - /$(NAME) /etc/ssl' > $@ || (rm -f $@; false)
                docker build --rm -t $(DOCKER_IMAGE) release

release         : release/$(NAME)

push            : release
                docker push $(DOCKER_IMAGE)

clean           :
                $(RM) .build release/$(NAME)

.PHONY          : push release build all clean

Directory tree

In the end, the directory tree should look like this:

$> tree
.
├── Dockerfile
├── Godeps
│   ├── Godeps.json
│   ├── Readme
│   └── _workspace
├── Makefile
├── main.go
└── release
    └── Dockerfile

3 directories, 6 files

Conclusion

As we saw, the release process is a dependency cascade, which make the Makefile very useful.
The advantages of using something like this is easily shown by docker images:

Original image

$> docker images example
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
example             latest              e68fcc5482a8        10 seconds ago          605.3 MB

Release image

$> docker images 127.0.0.1:5000/example
REPOSITORY               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
127.0.0.1:5000/example   latest              6e9e455bf60d        4 seconds ago          617.6 kB

Orchestrate Docker locally with Makefiles: Example with Interactive Brokers in Go.

Docker Orchestration

In this article, I’ll talk about basic orchestration using names and links, it is mainly for local testing, for production use, the strategy would be different.

Names and Links

The names and links within Docker are very useful. When starting a container with --name, you can then manipulate it via that name instead of the ID or the generated name. It is particulary useful when you need to play with that container in the future. (An alternative would be to inject the ID in a variable, but this is very limiting).
The links allow you to “inject” all the information from a given container into an other. This includes exposed ports and environment variables.

Naming Convention / Good practice

As a convention, to avoid name collision, I always name my container with the _c suffix. I do not use any prefix/suffix on images.

Example

Start a dummy container with a name, an exposed port and an environment variable then link it to an other one.

#!/bin/sh
# make sure the container name is free, remove it if needed
docker rm -f mycontainer_c
docker run -d -p 4242 -e foo=bar --name mycontainer_c ubuntu:14.04 sleep 100
docker run --link mycontainer_c:mcc ubuntu:14.04 env
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> HOSTNAME=912b106bf5eb
> MCC_PORT=tcp://172.17.0.55:4242
> MCC_PORT_4242_TCP=tcp://172.17.0.55:4242
> MCC_PORT_4242_TCP_ADDR=172.17.0.55
> MCC_PORT_4242_TCP_PORT=4242
> MCC_PORT_4242_TCP_PROTO=tcp
> MCC_NAME=/naughty_mayer/mcc
> MCC_ENV_foo=bar
> HOME=/root

Docker build

In order to generate a usable image, Docker provides an incredible tool: Dockerfiles. It allows you to “describe” your image.

Example with Go

main.go

package main

func main() {
    println("hello world")
}

Dockerfile

FROM    google/golang:stable
ADD     .    /src
CMD     /src/a.out
RUN     cd /src && go build -o a.out .

Issues

While this is very powerful, in “real life”, it is often not really useful as is. What about your Database, caching, queuing or any other services your application interacts with?
This is where the names, links and Makefile come to play.

Makefiles

Makefiles are a powerful tool, even though not designed for this usecase, we can leverage its features to accomplish easy local orchestration.

Rules

The feature that interest us the most are the Makefile “rules”, or dependencies. It allows you to write easily a shell script with dependencies management.

Caching

One of the original purpose of the Makefile is to provide compilation caching. I.e., not recompile the whole project when you change one file.
As we are using Docker for this, we will not use the caching to that level. However, we will use it to cache already built images.

Conventions

For consistency, all my Makefile look alike:

• Uppercase variables
  
  • NAME
  • SERVICE_IMAGE
  • SERVICE_CONT
• all, clean, re rules (Yes, from Epitech)
• Often test rule is present
• all depends on build and build services
• Images and container names usually contain $(NAME)
• All dummy files for caching are part of .gitignore and .dockerignore

Example

• Without Docker - Watch whole directory for changes

NAME   = myproject

all    : build

.built : .
        go build .
        @touch .built

build  : .built

clean  :
        @$(RM) .built

re     : clean all

.PHONY : all build clean re

Calling make will invoque the all rule. all depends on build by convention.
build depends on .built, that rule is a bit particular as it actually represent a file on disc. It is used as placeholder to know if we need to redo the build. In a “regular” Makefile scenario, we would have source file instead.
If .built exists and didn’t change (mtime), then we do nothing. Otherwise, executre the rule.
.built depends on ., meaning that if anything (mtime) changes in the local directory, the rule will be reexecuted (upon next make call).
The .built rule, when finish creates the .built file. So next time make is called, nothing will happen unless something changed within the directory.
This is not really useful as is, but now, let’s take a look at the same thing with Docker.

• With Docker

NAME   = myproject

all    : build

.built : .
        docker build -t $(NAME) .
        @docker inspect -f '{{.Id}}' $(NAME) > .built

build  : .built

clean  :
        @$(RM) .built

re     : clean all

.PHONY : all build clean re

Very straight forward, we do the same thing expect we store the ID of the resulting image in the .built file.
Now that we know how to play with Makefile and Docker, let’s see an actual use case with some Orchestration

Interactive Brokers

Interactive Brokers (IB) is a brokerage company that offers extremely low fees. Where “big” companies offers things like $8 a trade or when you are lucky $5 a trade, IB is often under $0.80 per trade.
It also has the advantage to come with an API so you can do everything programatically. However, it requires their gateway (java and GUI) to be up and running.
Someone wrote a pure go library that interacts with the IB API: http://github.com/gofinance/ib, however, the tests requires to have Java and for some reason, I did not manage to have them work on my machine.
This is where the dockerize all the things come in place: let’s Dockerize it, so anyone can use it. Without Java and without hassle.

Dockerize

Main repository

It is incredibly complex to Dockerize something, especially with Go:

FROM            google/golang:stable
MAINTAINER      Guillaume J. Charmes <guillaume@charmes.net>

ADD             .         /src

That’s it. We are dealing with a library, so nothing will get executed. We will call the go test from the Makefile as they have some dependencies and can’t be ran stand alone.

Test server

The tests depends on the test server to be running, so let’s Dockerize it as well.
First let’s take a look at the original script to spawn the server:

#!/bin/bash

rm -rf tws
mkdir tws
pushd tws
jar -xf ../unixmacosx-943.2a.jar
pushd IBJts
java -cp ../../ibcontroller-2.12.0.jar:jts.jar:total.2013.jar -Xmx512M -XX:MaxPermSize=128M ibcontroller.IBGatewayController ../../ibcontroller-2.12.0.ini &
popd
popd

In order to make it work, I had to replace jar -xf by unzip and in order to avoid the Gateway to complain about missing X11, I run it within xvfb. My new script “docker ready” look like this:

#!/bin/bash

rm -rf tws
mkdir tws
cd tws
unzip ../unixmacosx-943.2a.jar
cd IBJts
socat TCP-LISTEN:4003,fork TCP:127.0.0.1:4002&
xvfb-run java -cp ../../ibcontroller-2.12.0.jar:jts.jar:total.2013.jar -Xmx512M -XX:MaxPermSize=128M ibcontroller.IBGatewayController ../../ibcontroller-2.12.0.ini

It is very similar. You will note the socat hack. For some reason, I did not manage to have the gateway listen on the outside (even when setting it to 0.0.0.0). So I create a simple proxy on an other port.
The Dockerfile then look like this:

FROM            ubuntu:14.04
MAINTAINER      Guillaume J. Charmes <guillaume@charmes.net>

RUN             apt-get update
RUN             apt-get install -y unzip socat xvfb gsettings-desktop-schemas openjdk-7-jre && rm -rf /var/lib/apt/lists/*
ENV             JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64

EXPOSE          4003

ADD             .         /src

CMD             cd /src && ./ibgwdocker

Now that we have both the main code and the test server dockerize, we need to orchestrate.

Makefile

The Makefile rule system makes a perfect orchestration tool. In order to run tests, we first need to have the code dokerize (i.e. build the Dockerfile) as well as the test server. We then need the test server to be up and running and only then we can run the tests.

NAME            =       ib
GATEWAY_IMAGE   =       $(NAME)_gateway_test
GATEWAY_CONT    =       $(GATEWAY_IMAGE)_c
TEST_CONT       =       $(NAME)_test_c

all             :       test

.build_gw_id    :       testserver
                docker build -t $(GATEWAY_IMAGE) testserver
                @docker inspect -f '{{.Id}}' $(GATEWAY_IMAGE) > .build_gw_id

.gateway_id     :       .build_gw_id
                -@docker rm -f $(GATEWAY_CONT) > /dev/null 2> /dev/null || true
                -@docker rm -f $(GATEWAY_CONT)_tmp > /dev/null 2> /dev/null || true
                docker run --name $(GATEWAY_CONT) -d $(GATEWAY_IMAGE)
                @echo Wait for Gateway to be started
                @sleep 1
                @docker run --link $(GATEWAY_CONT):gw --rm --name $(GATEWAY_CONT)_tmp -t ubuntu:14.04 \
                        bash -c 'for i in {1..60}; do \
                                        echo | nc $$GW_PORT_4003_TCP_ADDR 4002 && exit 0 || (echo -n ..; sleep 1); \
                                done; \
                                echo; \
                                echo Waiting for Gateway timed out; exit 1'
                @echo
                @docker inspect -f '{{.Id}}' $(GATEWAY_IMAGE) > .gateway_id

gateway         :       .gateway_id

.build_id       :       .
                docker build -t $(NAME) .
                @docker inspect -f '{{.Id}}' $(GATEWAY_IMAGE) > .build_id

build           :       .build_id

test            :       gateway build
                -@docker rm -f $(TEST_CONT) > /dev/null 2> /dev/null || true
                docker run --link $(GATEWAY_CONT):gw --name $(TEST_CONT) -t $(NAME) bash -c 'cd /src && go test $(TESTFLAGS) -gw $$GW_PORT_4003_TCP_ADDR:4003'

clean           :
                -@docker rm -f $(GATEWAY_CONT) > /dev/null 2> /dev/null || true
                -@docker rm -f $(GATEWAY_CONT)_tmp > /dev/null 2> /dev/null || true
                -@docker rm -f $(TEST_CONT) > /dev/null 2> /dev/null || true

clean_all       :       clean
                -@rm -f .build_id .build_gw_id .gateway_id

re              :       clean_all all


.PHONY          :       all gateway buld test clean clean_all re

That is maybe a bit much at once, let’s decompose.
The goal here is to run the tests, so I made all to depend on test instead of build because there is no binary to build.
The test rule depend on the repository to be build and the gateway (testserver) to be up and running. It depends on the build and gateway rules.
gateway depends on .gateway_id, which mean that when the file .gateway_id exists, the rule will not be executed. If it does not exists, then call the .gateway_id rule, which depends on .build_gw_id. This means that gateway represent the test server runtime, but in order to run, it needs to be build before hand.
.build_gw_id depends on testserver. So if anything changes within testserver, then the rule will be reexecuted.
That rule effectively build the testserver image. Once built, then the .gateway_id rule is executed. This actually starts the test server. You will not the small shell hack that tried to connect to the port 4002. When in succeed, it means the gateway is up and running.
We run the test server with: docker run --name $(GATEWAY_CONT) -d $(GATEWAY_IMAGE), so we give it a fixed name.
Once the rule finishes, we have a test server built and up and running, now the build rule gets executed.
This one is pretty straight forward: it uses .build_id has caching file, when it does not exists, it builds the repository image.
When both the rules are satisfied (i.e. cached or executed), then the test rule starts:

docker run --link $(GATEWAY_CONT):gw --name $(TEST_CONT) -t $(NAME) bash -c 'cd /src && go test $(TESTFLAGS) -gw $$GW_PORT_4003_TCP_ADDR:4003'

We link the container with the test server one, so inside we will have access to the $GW_PORT_4003_TCP_ADDR, which is the actual private ip of the test server container. We give -t only for interactive purpose: it allows for ctrl-c to interrupt.
At this point, we have a container with the ip of the running test server, so we start the tests and give the test server IP as parameter.
See https://github.com/gofinance/ib/pull/4 for the full Pull-Request which includes the Makefile, the Dockerfiles and the code changes to accept variable test server IP.

Other use cases

This is very useful for lot of situation, for example:

• Redis
• Postgres, Mongo (or any database)
• Elastic Search

If you need multiple services, simply add more rules. The makefile will allow you to easily describe your cascade dependencies and cache everything.