JSON Date management in Golang

TL;DR

Arbitrary date unmarshal support + easily set marshal date format for both json and bson.
The code and examples can be found here: https://github.com/simplereach/timeutils.

Small example

package main

import (
        "encoding/json"
        "fmt"
        "os"

        "github.com/simplereach/timeutils"
)

type data struct {
        Time timeutils.Time `json:"time"`
}

func main() {
        var d data
        jStr := `{"time":"09:51:20.939152pm 2014-31-12"}`
        _ = json.Unmarshal([]byte(jStr), &d)
        fmt.Println(d.Time)

        d = data{}
        jStr = `{"time":1438947306}`
        _ = json.Unmarshal([]byte(jStr), &d)
        fmt.Println(d.Time)

        d.Time = d.Time.FormatMode(timeutils.RFC1123)
        _ = json.NewEncoder(os.Stdout).Encode(d)
}

The Standard Library

Go provide an extensive support for dates/time in the standard library with the package time.

This allows to easily deal with dates, compare them or make operations on them as well as moving from a timezone to an other.

Example

package main

import (
    "fmt"
    "time"
)

func main() {
    fmt.Printf("%s\n", time.Now().UTC().Add(-1 * time.Day))
}

Formating

Within the time.Time object, there are easy ways to format the date:

package main

import (
    "fmt"
    "time"
)

func main() {
    now := time.Now().UTC()
    // Display the time as RFC3339
    fmt.Printf("%s\n", now.Format(time.RFC3339))
    // Display the timestamp
    fmt.Printf("%s\n", now.Unix())
    // Display only the hour/minute
    fmt.Printf("%s\n", now.Format("3:04PM"))
}

Parsing

When it comes to parsing, once again, the standard library offers tools.

Parsing date string

package main

import (
    "log"
    "fmt"
    "time"
)

func main() {
    t, err := time.Parse(time.RFC3339, "2006-01-02T15:04:05-07:00")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s\n", t)
}

“Parsing” timestamp

package main

import (
    "fmt"
    "time"
)

func main() {
    now := time.Unix(1438947306, 0).UTC()
    fmt.Printf("%s\n", now)
}

This is great, but what if we don’t know what time format we are expecting? i.e. user input or 3rd part API.

A solution would be to iterate through the available time formats until we succeed, but this is often cumbersome and unreliable.

Approxidate

The git library has this Approxidate component that parses arbitrary date format and there is a Golang binding so we can use it!

http://godoc.org/github.com/simplereach/timeutils#ParseDateString

This expects a string as input and will do everything it can to properly yield a time object.

package main

import (
        "fmt"
        "log"

        "github.com/simplereach/timeutils"
)

func main() {
        t, err := timeutils.ParseDateString("09:51:20.939152pm 2014-31-12")
        if err != nil {
                log.Fatal(err)
        }
        fmt.Println(t)
}

Case of JSON Marshal/Unmarshal

Unmarshal

Let’s start with the unmarshal. What if we don’t want to parse the time manually and let json.Unmarshal handle it? Let’s try:

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "time"
)

func main() {
    var t time.Time

    str := fmt.Sprintf("%q", time.Unix(1438947306, 123).Format(time.RFC3339))
    fmt.Printf("json string: %s\n", str)
    if err := json.Unmarshal([]byte(str), &t); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("result: %s\n", t.Format(time.RFC3339))
}

Magically, it works fine! This is great, isn’t it?
But wait, the specs require us to send the date as RFC1123, is this going to work?
Let’s try as well!

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "time"
)

func main() {
    var t time.Time

    str := fmt.Sprintf("%q", time.Unix(1438947306, 123).Format(time.RFC1123))
    fmt.Printf("json string: %s\n", str)
    if err := json.Unmarshal([]byte(str), &t); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("result: %s\n", t.Format(time.RFC1123))
}

2009/11/10 23:00:00 parsing time ""Fri, 07 Aug 2015 11:35:06 UTC"" as ""2006-01-02T15:04:05Z07:00"": cannot parse "Fri, 07 Aug 2015 11:35:06 UTC"" as "2006"

Oups.

So it does not work, how can we work around this?

A solution would be to implement the json.Unmarshaler interface and handle our own parsing format, but we’ll get to this.

Marshal

Ok, we have our time object, and we want to send it as json. Nothing easier:

package main

import (
    "encoding/json"
    "os"
    "time"
)

func main() {
    _ = json.NewEncoder(os.Stdout).Encode(time.Unix(1438947306, 0).UTC())
}

It works fine :) However, the client expects times as RFC1123, how can we set the format to json.Marhsal?

A way to do so would be to implement the json.Marshaler interface and handling our own formatting.

Custom Marshal/Unmarshal

In order to tell Go to use a custom method for json marshal/unmarshal, one needs to implement the json.Marshaler and json.Unmarshaler interfaces.
As we can’t do that on imported type time.Time, we need to create a custom type.

Custom type

In order to create a custom type in Go, we simply do:

type myTime time.Time

However, doing so “hides” all members and methods so we can’t do things like this:

var t myTime
t.UTC()

Which is pretty annoying as our goal is simply to override the JSON behavior. We still want our full blown object.
To do so, we’ll use a struct with an anynomous member:

type myTime struct {
    time.Time
}

This way, we can access all the methods of the nested time object.

Unmarshal RFC1123

As we expect RFC1123, we need a custom parsing, so le’ts implement json.Unmarshaler.
Let’s take our first RFC1123 example and improve it:

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "strings"
    "time"
)

type myTime struct {
    time.Time
}

func (t *myTime) UnmarshalJSON(buf []byte) error {
    tt, err := time.Parse(time.RFC1123, strings.Trim(string(buf), `"`))
    if err != nil {
        return err
    }
    t.Time = tt
    return nil
}

func main() {
    var t myTime

    str := fmt.Sprintf("%q", time.Unix(1438947306, 123).Format(time.RFC1123))
    fmt.Printf("json string: %s\n", str)
    if err := json.Unmarshal([]byte(str), &t); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("result: %s\n", t.Format(time.RFC1123))
}

And now it works! We have a json unmarshal that supports RFC1123 instead of RFC3339!

To implment the json.Unmarshaler interface, we need to write the func (t *myTime) UnmarshalJSON(buf []byte) error method.

This receives the json buffer and return an error. It is expected to set the parsed value to the receiver so it is important that the receiver is a pointer.

The first step, has we expect valid json is to trim down the " from the string, then we call the time.Parse and finally set the result to our object.

Marshal RFC1123

Instead of the default RFC3339, let’s have json encode our time as RFC1123:

package main

import (
    "encoding/json"
    "os"
    "time"
)

type myTime struct {
    time.Time
}

func (t myTime) MarshalJSON() ([]byte, error) {
    return []byte(`"` + t.Time.Format(time.RFC1123) + `"`), nil
}

func main() {
    now := myTime{time.Unix(1438947306, 123)}
    _ = json.NewEncoder(os.Stdout).Encode(now)
}

Same idea as unmarshal. Here we only dump data so we don’t want the receiver to be a pointer and we make sure that we return valid json wrapped in ".

Going further

Changing the time format is great, but what if we need to move around dates as a timestamp integer? Or as a nanosecond timestamp? Or if we expect arbitrary format?

What if we have a REST API that need to move date between json and bson?

The timeutils library (http://github.com/simplereach/timeutils) offers a Time type that supports arbitrary time format via aproxidate as well as Timestamp and nanosecond precision both for marshal/unmarshal in json and bson.

Parsing HTTP Query String in Go

TL;DR

Code and examples can be found here: https://github.com/creack/httpreq

HTTP Server

Go provides a very easy way to create a http server:

package main

import (
        "fmt"
        "log"
        "net/http"
)

func handler(w http.ResponseWriter, req *http.Request) {
        fmt.Fprintf(w, "hello world\n")
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}
// curl http://localhost:8080

But how to deal with data?

JSON body

A common way to pass data is via a json encoded body:

package main

import (
        "encoding/json"
        "fmt"
        "log"
        "net/http"
)

type query struct {
        Name string
}

func handler(w http.ResponseWriter, req *http.Request) {
        q := &query{}
        if err := json.NewDecoder(req.Body).Decode(q); err != nil {
                log.Printf("Error decoding body: %s", err)
                return
        }
        fmt.Fprintf(w, "hello %s\n", q.Name)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}
// curl -d '{"Name": "Guillaume"}' http://localhost:8080/

Query String

But what if we want to pass data via query string? Typically, pagination and extra data.

Go, once again, expose everything necessary:

package main

import (
        "fmt"
        "log"
        "net/http"
        "strconv"
)

func handler(w http.ResponseWriter, req *http.Request) {
        if err := req.ParseForm(); err != nil {
                log.Printf("Error parsing form: %s", err)
                return
        }
        l := req.Form.Get("limit")
        limit, err := strconv.Atoi(l)
        if err != nil {
                log.Printf("Error parsing limit: %s", err)
                return
        }

        dr := req.Form.Get("dryrun")
        dryRun, _ := strconv.ParseBool(dr)
        fmt.Fprintf(w, "hello world. Limit: %d, Dryrun: %t\n", limit, dryRun)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}
// curl 'http://localhost:8080?limit=42&dryrun=true'

As we can see, it works as expected, however, if we add more and more fields to our query string, the type conversions quickly become cumbersome.

A Better Query String management

We know how to convert any string to any type.
We know what data we are expecting.
We should be able to do something similar to json.Unmarshal.

Conversion functions

Let’s start with our previous example: we need an int and a bool. However, the strconv functions have different prototypes and return a value.

It would be interesting to write a small helper that will set a value instead of returning it. That way, we could instantiate our query object and pass the fields to be set. In order to do so, we need to use pointers.

package main

import (
        "fmt"
        "log"
        "net/http"
        "strconv"
)

type query struct {
        Limit  int
        DryRun bool
}

func parseBool(s string, dest *bool) error {
        // assume error = false
        *dest, _ = strconv.ParseBool(s)
        return nil
}

func parseInt(s string, dest *int) error {
        n, err := strconv.Atoi(s)
        if err != nil {
                return err
        }
        *dest = n
        return nil
}

func handler(w http.ResponseWriter, req *http.Request) {
        if err := req.ParseForm(); err != nil {
                log.Printf("Error parsing form: %s", err)
                return
        }
        q := &query{}
        if err := parseBool(req.Form.Get("dryrun"), &q.DryRun); err != nil {
                log.Printf("Error parsing dryrun: %s", err)
                return
        }
        if err := parseInt(req.Form.Get("limit"), &q.Limit); err != nil {
                log.Printf("Error parsing limit: %s", err)
                return
        }
        fmt.Fprintf(w, "hello world. Limit: %d, Dryrun: %t\n", q.Limit, q.DryRun)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}

Make it generic

It is a bit better, but still could be improved. What if we’d like to have this in a generic way?

As we can see, the conversion helpers have a very similar prototype, let’s make it the same using interface{}

package main

import (
        "fmt"
        "log"
        "net/http"
        "strconv"
)

type query struct {
        Limit  int
        DryRun bool
}

func parseBool(s string, dest interface{}) error {
        d, ok := dest.(*bool)
        if !ok {
                return fmt.Errorf("wrong type for parseBool: %T", dest)
        }
        // assume error = false
        *d, _ = strconv.ParseBool(s)
        return nil
}

func parseInt(s string, dest interface{}) error {
        d, ok := dest.(*int)
        if !ok {
                return fmt.Errorf("wrong type for parseInt: %T", dest)
        }
        n, err := strconv.Atoi(s)
        if err != nil {
                return err
        }
        *d = n
        return nil
}

func handler(w http.ResponseWriter, req *http.Request) {
        if err := req.ParseForm(); err != nil {
                log.Printf("Error parsing form: %s", err)
                return
        }
        q := &query{}
        if err := parseBool(req.Form.Get("dryrun"), &q.DryRun); err != nil {
                log.Printf("Error parsing dryrun: %s", err)
                return
        }
        if err := parseInt(req.Form.Get("limit"), &q.Limit); err != nil {
                log.Printf("Error parsing limit: %s", err)
                return
        }
        fmt.Fprintf(w, "hello world. Limit: %d, Dryrun: %t\n", q.Limit, q.DryRun)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}

Parsing object

Now that we have generic helpers, we can easily write a small object that will simplify the way we use it:

We need to store N parsing functions, so we’ll need a slice (or a map). In order to parse a field, we need the helper function, but we also need the original string and the destination.

We have our object!

type parsingMap []parsingMapElem

type parsingMapElem struct {
        Field string
        Fct   func(string, interface{}) error
        Dest  interface{}
}

Once our paringMap constructed, we then need to execute it, let’s write the loop logic:

func (p parsingMap) parse(form url.Values) error {
        for _, elem := range p {
                if err := elem.Fct(elem.Field, elem.Dest); err != nil {
                        return err
                }
        }
        return nil
}

We know can put everything together:

package main

import (
        "fmt"
        "log"
        "net/http"
        "net/url"
        "strconv"
)

// conversion helpers
func parseBool(s string, dest interface{}) error {
        d, ok := dest.(*bool)
        if !ok {
                return fmt.Errorf("wrong type for parseBool: %T", dest)
        }
        // assume error = false
        *d, _ = strconv.ParseBool(s)
        return nil
}

func parseInt(s string, dest interface{}) error {
        d, ok := dest.(*int)
        if !ok {
                return fmt.Errorf("wrong type for parseInt: %T", dest)
        }
        n, err := strconv.Atoi(s)
        if err != nil {
                return err
        }
        *d = n
        return nil
}

// parsingMap
type parsingMap []parsingMapElem

type parsingMapElem struct {
        Field string
        Fct   func(string, interface{}) error
        Dest  interface{}
}

func (p parsingMap) parse(form url.Values) error {
        for _, elem := range p {
                if err := elem.Fct(elem.Field, elem.Dest); err != nil {
                        return err
                }
        }
        return nil
}

// http server
type query struct {
        Limit  int
        DryRun bool
}

func handler(w http.ResponseWriter, req *http.Request) {
        if err := req.ParseForm(); err != nil {
                log.Printf("Error parsing form: %s", err)
                return
        }
        q := &query{}
        if err := (parsingMap{
                {"limit", parseInt, &q.Limit},
                {"dryrun", parseBool, &q.DryRun},
        }).parse(req.Form); err != nil {
                log.Printf("Error parsing query string: %s", err)
                return
        }

        fmt.Fprintf(w, "hello world. Limit: %d, Dryrun: %t\n", q.Limit, q.DryRun)
}

func main() {
        log.Fatal(http.ListenAndServe(":8080", http.HandlerFunc(handler)))
}

Going Further

I wrote this small library: https://github.com/creack/httpreq which provides more helpers and a cleaner API. It fits my current use case, but feel free to add any helper that can be missing :)