Subscribe
By Sebastian Roy in Go —

Relaying a Microservice JSON Response to the Client by Unmarshalling Go Structs

In this post we will transform a JSON response from a microservice by adding metadata like a name, date or a unique identifier and relaying it to the user client. This is part of my journey in building AI SaaS app on iOS for researchers, as an example we will pretend to send data to an AI object ide

Relaying a Microservice JSON Response to the Client by Unmarshalling Go Structs
JSON response from the AI microservice (left) ist transformed into a response for the client (right)

The past week I have been working on connecting an image analysis microservice to the user frontend. We can send an image via post request  and retrieve an array of x and y-coordinates as well as a label for the identified object and a confidence parameter.

{"success": true,
 "predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Pinguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
}

In this post I will talk about transforming the JSON response and adding metadata like a name, date or a unique identifier. You could also convert the data, so instead of forwarding x_min and x_max coordinates, we could relay X-Min as coordinate and width instead of x_max. In order to distinguish on the client if an object was detected by an algorithm or a human we add a new field method. This could also be a more complicated object to store the version of the AI algorithm. The relayed response should look like this

{"name": "Zoo Image Analysis",
 "xid": "9m4e2mr0ui3e8a215n4g"
 "predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"x": 682,
			"y": 109,
			"width": 77, 
			"height": 25,
			"method": "AI"
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"x": 782,
			"y": 209,
			"width":  -13,
			"height": -65,
			"method": "AI"
		}]
}

To do this we will create a ResponseAdapter struct that mimics the response structure of the AI microservice as well as a PredictionAdapter. Then we create a Response and a Prediction struct that contains the fields and methods we want to relay to our client.

  1. Print unknown JSON data with an empty interface
  2. Convert JSON data into a usable struct to access data fields
  3. Rewrite into testable functions
  4. Implementing custom UnmarshalJSON
  5. Conclusion

When retrieving some JSON data from an API, we cannot call those properties like with data.name. Converting this data into structs takes some unmarshalling, mapping and interfacing. So let’s play with some JSON handling examples to step by step arrive at our goal, which is making data from an API call usable.

You can follow these steps using your local tooling in VS Code and go-tools or just use Go Playground without any installation.

Printing unknown JSON data with an empty interface

To have some sample data, we read some JSON data from an AI microservice as a multiline string into a byte array.

var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

Next we create a temporary empty interface

var temp map[string]interface{}

Then wir unmarshall the data into this variable temp. This requires an import of „encoding/json“.

json.Unmarshal(data, &temp)

Catch any unmarshalling errors

if err != nil {
		fmt.Println(err)
}

Then we can print out temp.

fmt.Println(temp)

The result should be

map[predictions:[map[confidence:0.69146144 label:Lion x_max:186 x_min:109 y_max:707 y_min:682] map[confidence:0.7 label:Penguin x_max:196 x_min:209 y_max:717 y_min:782]] success:true]

Great. If we want to access just the title

fmt.Println(temp.predictions[0])

we get an error.

./prog.go:40:19: temp.predictions undefined (type map[string]interface{} has no field or method predictions)

Ok, let’s take care about this next. The listing for this section.

package main

import (
	"encoding/json"
	"fmt"
)

func main() {

	var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

	var temp map[string]interface{}

	//var prediction PredictionsAdapter

	err := json.Unmarshal(data, &temp)

	if err != nil {
		fmt.Println(err)
	}

	//fmt.Println(temp.predictions[0])
}

Accessing data fields

Assuming we know the expected data structure in advance, lets create a struct.

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

Scroll up to the first JSON response to see where this is coming from. This is mirroring the response we get from the AI microservice. To be exported, the names must start with a capital letter. []PredictionsAdapter means that we expect multiple objects or an array of type PredictionsAdapter.

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}

We extend the PredictionsAdapter with two functions Width() and Height(), since we need this information later for our client response with Predictions.

func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}

Instead of unmarshalling it into an empty interface, we use a variable of type ResponseAdapter.

var responseAdapter ResponseAdapter

json.Unmarshall(data, &responseAdapter)

fmt.Println(responseAdapter.Predictions[0])

The & implies that we are referring to the address in memory. And thats how we can access JSON data from go as structs :).

{0.69146144 Lion 682 109 707 186}

The full listing

package main

import (
	"encoding/json"
	"fmt"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}

func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}

func main() {

	var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

	//var temp map[string]interface{}

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
}

Testing

I find it always motivating to write something that works and see some result first. To adapt good practices let’s now rewrite everything into a testable function and get rid of main(). If you use Go Playground, it will automatically run the test functions in absence of main().

First import "testing" and "github.com/google/go-cmp/cmp". It is a library to compare expectations want with actual result got of a function. It checks if both are equal with if !cmp.Equal(want, got) and returns an error with the differences otherwise.

The declaration of datawe plug out of the functional context such that it is accessible from each testing function.

Most of what has been happening in mainwe will rename to ParseImageAnalysisResponse which will receive the JSON content of type []byte and return a ResponseAdapter and an error.

The test function has the same name with prefix TestXxx and a reference to *testing.T which enables automated testing in go with go test. The first line then enables parallel test execution with t.Parallel().

Then we declare our expected result. This time we don’t use the JSON format, since we expect data as a go struct. Thats why we state the type before opening curly parenthesises want := ResponseAdapter{... .}

want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
}

Then we have to call the function with our data and catch possible errors. got will be of type ResponseAdapter, since this is what ParseImageAnalysisResponseAdapter(…) returns.

got, err := ParseImageAnalysisResponseAdapter(data)
if err != nil {
		t.Fatal(err)
}

And last we compare expectation and result.

if !cmp.Equal(want, got) {
	t.Error(cmp.Diff(want, got))
}

If everything is correct we can use Run on Go Playground or go test in the command line and the test should pass or return an error.

=== RUN   TestParseImageAnalysisResponse
=== PAUSE TestParseImageAnalysisResponse
=== CONT  TestParseImageAnalysisResponse
{0.69146144 Lion 682 109 707 186}
--- PASS: TestParseImageAnalysisResponse (0.00s)
PASS

Full listing

package main

import (
	"encoding/json"
	"fmt"
	"github.com/google/go-cmp/cmp"
	"testing"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}

var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

func ParseImageAnalysisResponseAdapter(data []byte) (ResponseAdapter, error) {

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
	
	return responseAdapter, nil
}

func TestParseImageAnalysisResponseAdapter(t *testing.T) {
	t.Parallel()

	want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
	}

	got, err := ParseImageAnalysisResponseAdapter(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

Now we have the data available in such a format that we can actually use it. We could even convert it to JSON and pass it to our user client. But sometimes the implementation logic requires that we add or remove some fields and provide the data in a different format. If that is what makes most sense, we should do it to maintain a well designed data workflow, even if it takes some additional work.

In the next section we will define the types Response and Predictions in a way we actually need them and transform them from our Adapter types using custom UnmarshallJSON. First we will add some custom metadata. Adding or changing fields in []PredictionsAdapter however requires looping through each element in the array.

Implementing custom UnmarshallJSON

Let’s create the Test for ParseImageAnalysisResponse that we are going to implement next (ignore any errors). Out want will be a Response that contains some metadata like name and Xid and the array of predictions. We use the data to pass it to ParseImageAnalysisResponse and than see if our expectation was met.

func TestParseImageAnalysisResponse(t *testing.T) {
	t.Parallel()

	want := Response{
		Name: "Test Image",
		Xid:  "9m4e2mr0ui3e8a215n4g",
		Predictions: []Prediction{
			{
				Confidence: 0.69146144,
				Label:      	"Lion",
				X:       		109,
				Y:       		682,
				Width:      	77,
				Height:     	25,
				Method:     	"AI",
			},
			{
				Confidence:	 	0.7,
				Label:      	"Penguin",
				X:       		209,
				Y:       		782,
				Width:          -13,
				Height:       	-65,
				Method:     	"AI",
			},		
		},
	}

	got, err := ParseImageAnalysisResponse(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

Next we define types that match the JSON API response we want to relay to our client, that could be a web app or a mobile app for Android or iOS.

type Prediction struct {
	Confidence  float64 `json:"confidence"`
	Label       string  `json:"label"`
	X       	int     `json:"x"`
	Y	        int     `json:"y"`
	Width       int     `json:"width"`
	Height      int     `json:"height"`
	Method      string  `json:"method"` // AI, Manual, Predicted(?)
}

The Response will contain many of those predictions, since many objects can be detected in an image. But further more it will include meta data like a Name and an Xid. A Xid is a unique identifier that is a related but shorter version of UUID. We might need to change the type later, but for now let’s use string.

type Response struct {
	Name        string       `json:"name"`
	Xid         string       `json:"xid"`
	Predictions []Prediction `json:"predictions"`
}

Now we will adapt the ParseImageAnalysisResponse function. It will still use the data of type []byte as an argument, but instead of a ResponseAdapter it will now return our new Response. Therefore we use a variable response of type Response and use this when unmarshalling. We then check for errors and return the response.

func ParseImageAnalysisResponse(data []byte) (Response, error) {

	var response Response
	// Use custom UnmarshalJSON method to obtain Response
	err := json.Unmarshal(data, &response)

	if err != nil {
		fmt.Println(err)
	}

	return response, nil

}

Since there is no easy 1-to-1 mapping from the ResponseAdapter to the Response, we need to do some manual work. Instead of the default UnmarshalJSON which we don’t have to specify, we will now use a customized version of it that explicitly states how we can derive our Response to the client from the data in the ResponseAdapter.

The data we send to ParseImageAnalysisResponse is the one we want to unmarshall. We know that we can unmarshall it into a ResponseAdapter, so thats what we will do first, but not into the pointer of the response object r, but into a temporary variable tmp.

var tmp ResponseAdapter
err := json.Unmarshal(data, &tmp)

if err != nil {
	fmt.Println(err)
}

This variable also has the Predictions in form of a PredictionsAdapter. We now cast them into a new Predictions-Array.

var p []Prediction

We do this by looping though each element PredictionsAdapter

for _, element := range tmp.Predictions {

and create a temporary corresponding predictions element.

This predictions element also defines the data for any added field or it could also leave out fields from PredictionsAdapter. Then we append the Predictions element to the Predictions Array p, that was defined before the loop. I did variable creation and appending kind of in one step here and closed the loop.

p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			XMin:       element.XMin,
			XMax:       element.XMax,
			YMin:       element.YMin,
			YMax:       element.YMax,
	})
}

Then we set the Response fields. That includes metadata like the name and Xid and setting the predictions array p.

r.Name = "Test Image"
r.Xid = "9m4e2mr0ui3e8a215n4g"
r.Predictions = p

And last since UnmarshalJSON requires us to return an error, we do that as well.

return err

Now lets see how this function looks in its full glory.

func (r *Response) UnmarshalJSON(data []byte) error {


	var tmp ResponseAdapter
	err := json.Unmarshal(data, &tmp)

	if err != nil {
		fmt.Println(err)
	}

	var p []Prediction
	for _, element := range tmp.Predictions {

		p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			X:	        element.XMin,
			Y:     	    element.YMin,
			Width:      element.Width(),
			Height:     element.Height(),
		})

	}

	r.Name = "Test Image"
	r.Xid = "9m4e2mr0ui3e8a215n4g"
	r.Predictions = p

	return err
}

Conclusion

Great. We can now take an image that was annotated with an AI microservice and make this analysis useful for relaying it the user client by enhancing it with more relevant data for a JSON response. On the networking side how to exactly send JSON responses from a HTTP request to a client was explained . I talked about how to send POST requests to the AI micrososervice in a previous post.

Last, let’s look at the full listing, which you can find also on Go Playground. The tests could also be written into a separate file of course.

package main

import (
	"encoding/json"
	"fmt"
	"github.com/google/go-cmp/cmp"
	"testing"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}


func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}


type Prediction struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	X          int     `json:"x"`
	Y          int     `json:"y"`
	Width      int     `json:"width"`
	Height     int     `json:"height"`
	Method     string  `json:"method"` // AI, Manual, Predicted(?)
}


type Response struct {
	Name        string       `json:"name"`
	Xid         string       `json:"xid"`
	Predictions []Prediction `json:"predictions"`
}


func (r *Response) UnmarshalJSON(data []byte) error {


	var tmp ResponseAdapter
	err := json.Unmarshal(data, &tmp)

	if err != nil {
		fmt.Println(err)
	}

	var p []Prediction
	for _, element := range tmp.Predictions {

		// fmt.Println("element", element, "at index", index)

		p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			X:          element.XMin,
			Y:          element.YMin,
			Width:      element.Width(),
			Height:     element.Height(),
		})

	}

	r.Name = "Test Image"
	r.Xid = "9m4e2mr0ui3e8a215n4g"
	r.Predictions = p

	return err
}

var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

func ParseImageAnalysisResponseAdapter(data []byte) (ResponseAdapter, error) {

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
	
	return responseAdapter, nil
}

func ParseImageAnalysisResponse(data []byte) (Response, error) {

	var response Response
	// Use custom UnmarshalJSON method to obtain Response
	err := json.Unmarshal(data, &response)

	if err != nil {
		fmt.Println(err)
	}
	// fmt.Println("Response")
	// fmt.Println(response)

	return response, nil

}


func TestParseImageAnalysisResponseAdapter(t *testing.T) {
	t.Parallel()

	want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
	}

	got, err := ParseImageAnalysisResponseAdapter(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

func TestParseImageAnalysisResponse(t *testing.T) {
	t.Parallel()

	want := Response{
		Name: "Test Image",
		Xid:  "9m4e2mr0ui3e8a215n4g",
		Predictions: []Prediction{
			{
				Confidence: 0.69146144,
				Label:      	"Lion",
				X:       		109,
				Y:       		682,
				Width:      	77,
				Height:     	25,
				Method:     	"AI",
			},
			{
				Confidence:	 0.7,
				Label:      	"Penguin",
				X:       		209,
				Y:       		782,
				Width:          -13,
				Height:       	-65,
				Method:     	"AI",
			},
		},
	}

	got, err := ParseImageAnalysisResponse(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

https://bitfieldconsulting.com/golang/map-string-interface

https://jhall.io/posts/go-json-tricks-array-as-structs/

https://stackoverflow.com/questions/42377989/unmarshal-json-array-of-arrays-in-go

Subscribe to sebastianroy.de

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe