In the interface will define URL routes (/v1/uploadImage) that the server can accept and HTTP types (GET, POST, PUT). Retrofit2 uses decorators (@…) to achieve this. It also specifies the response type. I use Call<ResponseBody> first to obtain a raw JSON (think curly brackets { …}), but you can also define a type in Kotlin and cast the response immediately using Call<List<MyImageObjects>>to retrieve an array of image objects (name, UUID, x-coordinate…).

Since we don’t want to send some user information via POST request, but one or several images, our request body will be a MultiPart Form.

When retrieving more complex objects with metadata and countless arrays, it is difficult to cast the JSON response immediately into a Kotlin object. The structure of this object has to match the JSON response in all types. In that case, it is easier to just display the raw JSON response first and write a custom deserialiser of this JSON response for easier debugging.

1. Add Dependencies
2. Add Permissions
3. Preparing the Interface
4. View Model
5. Main Activity
6. Catch Errors
7. Conclusion

Add Dependencies

First, let’s add the dependencies to gradle.build

implementation "com.squareup.retrofit2:retrofit:2.9.0"
implementation "com.squareup.retrofit2:converter-gson:2.6.0"

The version might change, you can look it up here for retrofit and gson.

To apply changes, hit Sync Now.

Add Permissions

To access the internet, the app needs permission.  The user can later see the required permissions in the app store.

Add Internet Permission to AndroidManifest.xml.

 <uses-permission android:name="android.permission.INTERNET" />

Failing to add this will lead the app to crash when sending an HTTP request.

Prepare the Interface

Creating an interface is not strictly necessary, but good practice for interoperability and testability. However, it does not implement the function specifically, meaning how you handle the response is defined later when creating a class that complies with this interface.

So let’s create a new interface file FileApi.kt.

The first line will be your package name, similar to this.

package com.uploadImage.test

When you use a keyword that hasn’t been imported, Android Studio will give you a hint to do so. But sometimes there are multiple sources for one keyword. The required imports for this interface will be

import okhttp3.MultipartBody
import okhttp3.OkHttpClient
import retrofit2.Call
import retrofit2.Retrofit
import retrofit2.converter.gson.GsonConverterFactory
import retrofit2.http.Multipart
import retrofit2.http.POST
import retrofit2.http.Part

The next line has already been created by the IDE

interface FileApi {
	...
}

Now we define the routes. We want to send a POST request (to send instead of GET something) to http://someurl.com/api/v1/uploadimage as a Multipart form (since we upload an image).

    @Multipart
    @POST("/api/v1/uploadimage")

Next, we use asynchronous coroutines for networking. We add a function, uploadImage, that can be suspended to the background until a result is received.

fun uploadImage(

We define now the @Part that makes up the @Multipart request.

	@Part image: MultipartBody.Part

The resulting raw JSON can be obtained as a ResponseBody.

 ): Call<ResponseBody>

Alternatively, the resulting JSON can be cast into a custom type, e.g., a list of Words. We will make use of this later by adding a second function.

 ): Call<List<Word>>

So, here we create a companion object, which means that we create only a single instance in the whole application. This is also called a singleton pattern in Kotlin.

// No need to create an instance of an object, just use this class method
companion object {
     ...   
}

A singleton is an object that has just one instance and can be accessed globally over the lifetime of the app.
Object is a way to create a singleton. If that object is part of a class or an interface, it is a companion object.

But a companion object of what? A companion object of the OkHttpClient.

// Create HTTP Client for network request
private val client = OkHttpClient.Builder().build()

 val instance by lazy {
	...
 }

Retrofit acts as a layer above a basic http client, and we could choose different ones, so here we specify the client we want to use for our HTTP requests and the baseUrl to the server we intend to send the image to.

Retrofit.Builder()
      .baseUrl("https://testapi.com/")
      .client(client) // add a client instance here, e.g. OkHttpClient
      .addConverterFactory(GsonConverterFactory.create())
      .build()
      .create(FileApi::class.java)

If you would like to use your own local server, you can’t use encryption with https because you need a certificate for this. It is easy to get a certificate by letsencrypt, but you require a domain name and not just a local IP.

But Android doesn’t allow unencrypted traffic per default, so we need to enable it in our testing environment. We do this by going to AndroidManifest.xml and add

 android:usesCleartextTraffic="true"

in the <application… tag.

Another way would be to deploy your server on a droplet on Digital Ocean and forward a subdomain like api.yourdomain.com to the IP of this droplet. Then you can enable encryption on your server. In go with the echo framework, you could do this by launching the server by using e.StartAutoTLS instead of e.Start and using port 443.

e.Logger.Fatal(e.StartAutoTLS(":443"))

This generates a Letsencrypt certificate automatically for you.

Now you should be able to access the routes encrypted from your server using https://api.yourdomain.com/v1/uploadImage.

The Class

We create the file FileRepository.kt. It will open an image and call the companion object instance defined above.  We will also handle the response, which could be either successful or not. If it is successful, we log the raw JSON response and return true, and if it isn’t, we deal with the errors and return false.

So let’s start with the imports for logging, the HTTP client okhttp3, retrofit2, file and Input/Output exception.

import android.util.Log
import okhttp3.MultipartBody
import okhttp3.RequestBody.Companion.asRequestBody
import retrofit2.Call
import retrofit2.Callback
import retrofit2.HttpException
import retrofit2.Response
import java.io.File
import java.io.IOException

Then we create a function uploadImage, that gets passed a file of type File which we want to upload.

It will add a skeleton for a class

class FileRepository {
	...
}

We add a function, uploadImage, that expects a Boolean (true or false) depending on the success of the request.

fun uploadImage(file: File): Boolean {
	...
}

Next, we set the return variable res to false. It will be switched to true later in the callback function when the request was successful.

	var res = false 

Then we access uploadImage function of the companion object in the FileApi interface we created earlier. We enclose it in a try block to catch errors.

try {
       FileApi.instance.uploadImage(
                ...
            )
    }

As defined in the interface, we use image as a MultipartBody.Part an argument to uploadImage(). The image is created from file that was passed to this method by the FileViewModel, which we haven’t created yet.

    image = MultipartBody.Part
             .createFormData(
                  "image",
                  file.name,
                  file.asRequestBody()
             )

Now, enqueue() (as opposed to dequeue()) adds an object to the task queue.

.enqueue(
  object : Call<ResponseBody> {
		...
	}
)
...

The response is handled by onFailure() and onResponse(). In case of a failure, we log an error.

    override fun onFailure(call: Call<ResponseBody>, t: Throwable) {
                        Log.e("API Request", "I got an error and i don't know why :(")
                    }

In case of success, we read the response.body() and log it as a string.

    override fun onResponse(call: Call<ResponseBody>, response: Call<ResponseBody>) {
                        val responseBody = response.body()
                        Log.e("API Request", responseBody.toString())
     }

Since the response has been handles successfully, we will set our success switch res to true to return and exit the function.

 res = true
 return res

Otherwise, if we have an IOException or a HttpException we print the error and return false, as res was initially set to false.

catch (e: IOException) {
            e.printStackTrace()
            return res
} catch (e: HttpException) {
            e.printStackTrace()
            return res
} finally {
            Log.e("API Request", res.toString())
}

View Model

Next, we create the view Model FileViewModel.kt.

These are the required imports

import androidx.lifecycle.ViewModel
import androidx.lifecycle.viewModelScope
import java.io.File
import kotlinx.coroutines.launch

In the class FileViewModel we add an instance repository of a FileRepository we defined in the previous section.

class FileViewModel(
	private val repository: FileRepository = FileRepository()): ViewModel() {
	...
	}

We launch an asynchronous call with viewModelScope.launch to upload the image.

fun uploadImage(file: File) {
	viewModelScope.launch {
		repository.uploadImage(file)
	}
}

MainActivity

Now, we add the viewModel of type viewModel<FileViewModel>() to MainActivity.kt (after your apps theme).

...
setContent {
   UploadImageTheme {
		 val viewModel = FileViewModel()
         // Dependency injection?
         // val viewModel = viewModel<FileViewModel>()
	...

Then we add a Box in which other UI elements are contained

Box(
    modifier = Modifier.fillMaxSize(),
    contentAlignment = Alignment.Center
) {
	...
}

In this case, we just add an upload button. When it is clicked, it will look for the file img.png in cacheDir which we added to the code repository at the beginning of this tutorial.

Button(onClick = {
  val file = File(cacheDir, "img.png")
  file.createNewFile()
  file.outputStream().use {
  assets.open("img.png").copyTo(it)
  }
  viewModel.uploadImage(file)
  }) {
  Text(text = "Upload image")
}

Check if file img.png exists in the assets folder.

Catch Errors

• API server is not available (retrofit2.HttpException: HTTP 403)
• javax.net.ssl.SSLException: Unable to parse TLS packet header

Conclusion

In this post, I described how to send an image as a POST request using retrofit2. First we introduced the dependencies, permissions and talked about pitfalls in encrypted HTTPS requests and Android. We created an interface where we defined the routes. Then we created a class that used this interface and handled the response with success and failure. The file was send from the view model and the view model was accessed after a click on a button in the UI.

Links

• https://www.geeksforgeeks.org/retrofit-with-kotlin-coroutine-in-android/
• https://stackoverflow.com/questions/74825563/send-an-image-to-an-ocr-api-with-rertofit-and-kotlin
• https://trendoceans.com/how-to-fix-cleartext-http-traffic-not-permitted-in-android/

We will just return file size, image width and height in pixels as a basic JSON object. If you want to learn how to cast more complex JSON objects into structs, checkout Relaying a Microservice JSON Response to the Client by Unmarshalling Go Structs.

First, we need to obtain the image we intend to send. This could be done by taking a photo using the camera, selecting an image from the library, or loading an image from a URL. Loading an image from a URL is the easiest way. Then we need to create the POST request, send it and catch any error if something went wrong.

1. Backend in Go 1.19
2. Returning a JSON with file name, size, width and height
3. Frontend in Swift 5
4. Basic Concepts
5. Code Skeleton
6. Selecting an Image
7. Creating and Sending the HTTP Request
8. Conclusion
9. Links

If we can’t accept POST requests that contain an image, any request will remain unanswered. So let’s create a server to do that.

Backend in Go 1.19

First initialise a new Go environment and install the echo framework.

$ mkdir ImageHandlerTutorial && cd ImageHandlerTutorial 
$ go mod init ImageHandlerTutorial 
$ go get github.com/labstack/echo/v4 
$ go get github.com/labstack/echo/v4/middleware  

Then create a ImageHandlerTutorial.go file with a basic skeleton for using echo and other imports for writing files or encoding JSON.

package main

import (
	"bytes"
	// Formating strings
	"fmt"

	"io"
	// Multipart POST request
	"mime/multipart"
	// Opening and closing files
	"os"
	"time"

	"image"
	// Register possible image formats
	_ "image/png"
	_ "image/jpeg"

	"encoding/json"
	"io/ioutil"
	"log"
	"net/http"

	// Echo framework
	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
)

func main() {
	e := echo.New()

	// Avoid CORS error when testing with redocly
	e.Use(middleware.CORS())

	// API Versioning
	//...

	// POST request handling
	//...

	e.Logger.Fatal(e.Start(":1324"))
}

To provide some structure for the URLs, we create two groups, such that we can create resources under 127.0.0.1:1324/api/v1. The IP will change if it is running on a real server instead of the same local machine the iOS Simulator will run on later.

	// API Versioning
	apiGroup := e.Group("/api")
	apiv1Group := apiGroup.Group("/v1")

Under this group, we now create an endpoint imageanalysis. It will handle the POST request 127.0.0.1:1324/api/v1/imageanalysis.

This also includes a function to handle the image file. We open the data in field file into the source src and write the file to dst on disk.

The way it is written here, different uploads will overwrite the same file name file. This is a bit sketchy, but also doesn’t use much space. Think about deleting the file after analysis, renaming it if you need it later, and what happens when multiple users call the function at the same time.

	apiv1Group.POST("/imageanalysis", func(c echo.Context) error {
		// TODO: Catch error when no file is send in POST Request
		// Source
		file, err := c.FormFile("file")
		if err != nil {
			return err
		}
		src, err := file.Open()
		if err != nil {
			return err
		}
		defer src.Close()

		// Destination
		dst, err := os.Create(file.Filename)
		if err != nil {
			return err
		}
		defer dst.Close()

		// Copy
		if _, err = io.Copy(dst, src); err != nil {
			return err
		}
		// TODO: Why do i need to open the file here again, instead of just passing the reference from os.Create()?
		// TODO: Maybe don't create the file on disk at all, just send it from memory
		buf, err := os.Open(dst.Name())
		if err != nil {
			log.Fatal(err)
		}
		defer buf.Close()

		return ImageRequestHandler(c, buf)
	})

To finish, we start the echo server (but that part is already in the skeleton above)

e.Logger.Fatal(e.Start(":1324"))

Save the file and to run the server open a terminal and execute.

$ go run ImageHandlerTutorial.go          

This should output

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.9.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:1324

Go Playground doesn’t support anything with networking, so if you want to try it, you need to install Go on your machine. You can see and download the full listing here.

Returning a JSON with file name, size, width and height

So for practice, let’s return a JSON with the file size in kb and image size in pixels. Here we could also send the image for analysis to another server to do object detection. We would need to perform another POST request, this time from Go to an object detection microservice, and I have written about this in Sending Images in POST request as MultiPart Form from Go to Microservice.

// Send an image to microservice for image analysis
func ImageRequestHandler(c echo.Context, img *os.File) error {
	// TODO: Use user image instead
	log.Print(img.Name())

	body := &bytes.Buffer{}
	writer := multipart.NewWriter(body)
	// TODO: Try to change image name here
	fw, err := writer.CreateFormFile("image", "image.jpg")

	_, err = io.Copy(fw, img)
	if err != nil {
		log.Fatal(err)
	}

	writer.Close()

	// Get filesize of image
	stat, err := img.Stat()
	log.Print("File size: ", stat.Size()/1024, " kb")

	// Get width and height in px for the image
	im, _, err := image.DecodeConfig(img)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s: %v\n", img.Name(), err)
	}
	log.Print(im.Width)

	// Form JSON response that contains name, size, height and width
	jsonStr := fmt.Sprintf(`{ "name": "%s",
				 "size": %d,
				 "width": %d,
				 "height": %d
				}`, img.Name(), stat.Size(), im.Width, im.Height)
	log.Print(jsonStr)

	c.Response().Header().Set("Content-Type", "application/json")
	//TODO: Avoid double casting []byte string?!
	c.Response().Write([]byte(string(jsonStr)))

	// TODO: Figure out how to avoid return and just return with c.Response() similar to w.Write
	return err
}

Basic concepts

Multiform/data http requests

A POST request is one type of HTTP request like GET and PUT. There are many types of POST request, which we can specify in the header. There are image/png or text for example. For flexibility, however, we will use multiformat/data. This offers multiple data fields.

HTTP requests have a boundary. It tells the parser that interesting data is coming and when it sees the same boundary again that this is the end of that data batch. We will define it using a random uuidString. This is just a common way to generate random text to uniquely identify something and can look like 633a76d8-87a3-494c-bd1c-7192985e01f5.

Asynchronous tasks with DispatchQueues

Like any type of networking activity, it takes some time to execute and might even fail. We want to avoid blocking the user interface waiting for a response. That’s why multithreading is used. This is a way to make asynchronous networking requests. There are many ways to do this and it is a complex subject. Here we will use DispatchQueues. This is a way to schedule tasks with the first-in-first-out principle (FIFO) and we let iOS handle thread management.

Swift Code Skeleton

We will use two functions loadFrom and uploadImage and wrap those up in a class POSTImage. Then we create an object postRequest of our new type POSTImage and execute.

import UIKit

class POSTImage {
	var image = UIImage()
	
	// Load an image from URL 
	func loadFromURL(URLAddress: String) {
		 
	}

	// Load an image from Library (TODO)
	func loadImageFromLibrary() {}

	// Sending the image to the server
	func uploadImage(fileName: String, image: UIImage) {
	
	}
}


let postRequest = POSTImage()
postRequest.loadFromURL(URLAddress: "https://upload.wikimedia.org/wikipedia/commons/b/b9/Panther_chameleon_%28Furcifer_pardalis%29_male_Montagne_d%E2%80%99Ambre.jpg")

Selecting an Image

Load an image from URL

So, now let’s get an image by loading it from a URL with loadFromURL(...) with an URLAddress of type String as argument. We specified the URL in the last line of our skeleton above already. In a real scenario, this could be using a URL from a s3 object storage. We use guard to catch errors when parsing the URL.

 guard let url = URL(string: URLAddress) else {
     return
 }

Next we will load the image. Loading an image takes some time and can fail. We would rather not block the user interface, therefore handling the task over to a separate thread with DispatchQueue.main.async. First, we try to obtain the Data from the url into imageData. Then the imageData will be cast into loadedImage which can be used for display as UIImage.

Now we will upload the image. This is a provisional solution, but it works. The issue is that we loaded the image in a separate thread. We can only call uploadImage, once we already got the image. Therefore, our program has to wait or we need a callback function that is called after the image was loaded from URL. We will do that later, but for now, we just call uploadImage from the sub thread in DispatchQueue.

DispatchQueue.main.async { [weak self] in
    if let imageData = try? Data(contentsOf: url) {
    if let loadedImage = UIImage(data: imageData) {
       self?.image = loadedImage
       // TODO: Remove this and use completion handler instead
       self?.uploadImage(fileName: "file", image: loadedImage)
    }
}

The other issue is that we haven’t implemented uploadImage yet, which we will do next. Here is the full listing for the function selectImageFromURL.

func selectImageFromURL(URLAddress: String) {
     guard let url = URL(string: URLAddress) else {
         return
     }
        
     DispatchQueue.main.async { [weak self] in
        if let imageData = try? Data(contentsOf: url) {
    		if let loadedImage = UIImage(data: imageData) {
    		   self?.image = loadedImage
      		 // TODO: Remove this and use completion handler instead
    		   self?.uploadImage(fileName: "file", image: loadedImage)
    }
}

Create and send POST request

Now we create a URLSession named session. It will be sent to the url specified by URL(...). This must be an endpoint to your server, that will handle HTTP POST requests. I have written a previous post about how to do this also with URLSession and DispatchQueues at least with JSON objects (text, no images) in Communications from Go Backend to SwiftUI Frontend with JSON API.

First, we need to define the server URL and start a URLSession to handle networking.

let url = URL(string: "http://127.0.0.1:1324/api/v1/imageanalysis")

let session = URLSession.shared

Now we configure the request as POST request and making it a multipart/form-data request in the header, where we configure how the boundary looks like.

let boundary = UUID().uuidString
var urlRequest = URLRequest(url: url!)
urlRequest.httpMethod = "POST"
urlRequest.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")

After the header we need to set the actual data. We create data of type Data. Then we append the boundary and add the parameter ”file”. The Content-Type of this parameter is image/png. And finally we have to append the actual file data image.pngData. Finally, we close the requests boundary.

var data = Data()
        
// Add the image data to the raw http request data
data.append("\r\n--\(boundary)\r\n".data(using: .utf8)!)

let paramName = "file"
data.append("Content-Disposition: form-data; name=\"\(paramName)\"; filename=\"\(fileName)\"\r\n".data(using: .utf8)!)
data.append("Content-Type: image/png\r\n\r\n".data(using: .utf8)!)
data.append(image.pngData()!)
        
data.append("\r\n--\(boundary)--\r\n".data(using: .utf8)!)

Now that we defined the request, it is time to execute it. We use the session.uploadTask with our created urlRequest and data as arguments. Then we handle the response in the completionHandler. There we convert the JSON response into a Go struct.

session.uploadTask(with: urlRequest, from: data, completionHandler: { responseData, response, error in
                if error == nil {
                    let jsonData = try?JSONSerialization.jsonObject(with: responseData!, options: .allowFragments)
                    if let json = jsonData as? [String: Any] {
                        print(json)
                    }
                }
	}).resume()

Now let’s look at the full listing of the function uploadImage.

func uploadImage(fileName: String, image: UIImage) {
        let url = URL(string: "http://127.0.0.1:1324/api/v1/electrophoresis/imageanalysis")
        let boundary = UUID().uuidString

        let session = URLSession.shared

        var urlRequest = URLRequest(url: url!)
        urlRequest.httpMethod = "POST"

        urlRequest.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")

        var data = Data()
        
        // Add the image data to the raw http request data
        data.append("\r\n--\(boundary)\r\n".data(using: .utf8)!)
        let paramName = "file"
        data.append("Content-Disposition: form-data; name=\"\(paramName)\"; filename=\"\(fileName)\"\r\n".data(using: .utf8)!)
        data.append("Content-Type: image/png\r\n\r\n".data(using: .utf8)!)
        data.append(image.pngData()!)
        
        data.append("\r\n--\(boundary)--\r\n".data(using: .utf8)!)
        
        session.uploadTask(with: urlRequest, from: data, completionHandler: { responseData, response, error in
                if error == nil {
                    let jsonData = try? JSONSerialization.jsonObject(with: responseData!, options: .allowFragments)
                    if let json = jsonData as? [String: Any] {
                        print(json)
                    }
                }
            }).resume()
    }

Finally, we get to see the result of all the hard work so far:

["size": 5451466, "name": file, "height": 1396, "width": 2872]

This is the analysis done by the server, shown on the client. Real world analysis would be more complex and the client would display that data on the UI instead of just printing it to the console.

Bonus: Some curious errors

One issue I had was that data function wasn’t found. This was because I created my own data type that was called data. This has overridden the native function. So give your types new names that are not used in iOS. iOS libraries prefix any new class or struct to avoid it.

Full Listing

import UIKit


class POSTImage {
    var image = UIImage()
    
    func loadFrom(URLAddress: String) {
        guard let url = URL(string: URLAddress) else {
            return
        }
        
        DispatchQueue.main.async { [weak self] in
            if let imageData = try? Data(contentsOf: url) {
                if let loadedImage = UIImage(data: imageData) {
                     // Why is self optional here?
                     self?.image = loadedImage
                     self?.uploadImage(fileName: "file", image: loadedImage)
                }
            }
        }
    }
    
    func uploadImage(fileName: String, image: UIImage) {
        let url = URL(string: "http://127.0.0.1:1324/api/v1/imageanalysis")
        let boundary = UUID().uuidString

        let session = URLSession.shared

        var urlRequest = URLRequest(url: url!)
        urlRequest.httpMethod = "POST"

        urlRequest.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")

        var data = Data()
        
        // Add the image data to the raw http request data
        data.append("\r\n--\(boundary)\r\n".data(using: .utf8)!)
        let paramName = "file"
        data.append("Content-Disposition: form-data; name=\"\(paramName)\"; filename=\"\(fileName)\"\r\n".data(using: .utf8)!)
        data.append("Content-Type: image/png\r\n\r\n".data(using: .utf8)!)
        data.append(image.pngData()!)
        
        data.append("\r\n--\(boundary)--\r\n".data(using: .utf8)!)
        
        session.uploadTask(with: urlRequest, from: data, completionHandler: { responseData, response, error in
                if error == nil {
                    let jsonData = try? JSONSerialization.jsonObject(with: responseData!, options: .allowFragments)
                    if let json = jsonData as? [String: Any] {
                        print(json)
                    }
                }
            }).resume()
    }
}

let postRequest = POSTImage()
postRequest.loadFrom(URLAddress: "https://upload.wikimedia.org/wikipedia/commons/b/b9/Panther_chameleon_%28Furcifer_pardalis%29_male_Montagne_d%E2%80%99Ambre.jpg")

// TODO: wait for myimageview to load
// uploadImage(fileName: "file", image: myImageView.image)

Conclusion

To sum things up, we discussed the server and client side of sending images over HTTP with POST requests, doing simple analysis on the server and returning the result as JSON to the client. I hope I could cover this subject in a useful way with most of the puzzle pieces.

Links

https://orjpap.github.io/swift/http/ios/urlsession/2021/04/26/Multipart-Form-Requests.html

https://stackoverflow.com/questions/29623187/upload-image-with-multipart-form-data-ios-in-swift

https://www.hackingwithswift.com/books/ios-swiftui/sending-and-receiving-orders-over-the-internet

The past week I have been working on connecting an image analysis microservice to the user frontend. We can send an image via post request  and retrieve an array of x and y-coordinates as well as a label for the identified object and a confidence parameter.

{"success": true,
 "predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Pinguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
}

In this post I will talk about transforming the JSON response and adding metadata like a name, date or a unique identifier. You could also convert the data, so instead of forwarding x_min and x_max coordinates, we could relay X-Min as coordinate and width instead of x_max. In order to distinguish on the client if an object was detected by an algorithm or a human we add a new field method. This could also be a more complicated object to store the version of the AI algorithm. The relayed response should look like this

{"name": "Zoo Image Analysis",
 "xid": "9m4e2mr0ui3e8a215n4g"
 "predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"x": 682,
			"y": 109,
			"width": 77, 
			"height": 25,
			"method": "AI"
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"x": 782,
			"y": 209,
			"width":  -13,
			"height": -65,
			"method": "AI"
		}]
}

To do this we will create a ResponseAdapter struct that mimics the response structure of the AI microservice as well as a PredictionAdapter. Then we create a Response and a Prediction struct that contains the fields and methods we want to relay to our client.

1. Print unknown JSON data with an empty interface
2. Convert JSON data into a usable struct to access data fields
3. Rewrite into testable functions
4. Implementing custom UnmarshalJSON
5. Conclusion

When retrieving some JSON data from an API, we cannot call those properties like with data.name. Converting this data into structs takes some unmarshalling, mapping and interfacing. So let’s play with some JSON handling examples to step by step arrive at our goal, which is making data from an API call usable.

You can follow these steps using your local tooling in VS Code and go-tools or just use Go Playground without any installation.

Printing unknown JSON data with an empty interface

To have some sample data, we read some JSON data from an AI microservice as a multiline string into a byte array.

var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

Next we create a temporary empty interface

var temp map[string]interface{}

Then wir unmarshall the data into this variable temp. This requires an import of „encoding/json“.

json.Unmarshal(data, &temp)

Catch any unmarshalling errors

if err != nil {
		fmt.Println(err)
}

Then we can print out temp.

fmt.Println(temp)

The result should be

map[predictions:[map[confidence:0.69146144 label:Lion x_max:186 x_min:109 y_max:707 y_min:682] map[confidence:0.7 label:Penguin x_max:196 x_min:209 y_max:717 y_min:782]] success:true]

Great. If we want to access just the title

fmt.Println(temp.predictions[0])

we get an error.

./prog.go:40:19: temp.predictions undefined (type map[string]interface{} has no field or method predictions)

Ok, let’s take care about this next. The listing for this section.

package main

import (
	"encoding/json"
	"fmt"
)

func main() {

	var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

	var temp map[string]interface{}

	//var prediction PredictionsAdapter

	err := json.Unmarshal(data, &temp)

	if err != nil {
		fmt.Println(err)
	}

	//fmt.Println(temp.predictions[0])
}

Accessing data fields

Assuming we know the expected data structure in advance, lets create a struct.

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

Scroll up to the first JSON response to see where this is coming from. This is mirroring the response we get from the AI microservice. To be exported, the names must start with a capital letter. []PredictionsAdapter means that we expect multiple objects or an array of type PredictionsAdapter.

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}

We extend the PredictionsAdapter with two functions Width() and Height(), since we need this information later for our client response with Predictions.

func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}

Instead of unmarshalling it into an empty interface, we use a variable of type ResponseAdapter.

var responseAdapter ResponseAdapter

json.Unmarshall(data, &responseAdapter)

fmt.Println(responseAdapter.Predictions[0])

The & implies that we are referring to the address in memory. And thats how we can access JSON data from go as structs :).

{0.69146144 Lion 682 109 707 186}

The full listing

package main

import (
	"encoding/json"
	"fmt"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}

func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}

func main() {

	var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

	//var temp map[string]interface{}

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
}

Testing

I find it always motivating to write something that works and see some result first. To adapt good practices let’s now rewrite everything into a testable function and get rid of main(). If you use Go Playground, it will automatically run the test functions in absence of main().

First import "testing" and "github.com/google/go-cmp/cmp". It is a library to compare expectations want with actual result got of a function. It checks if both are equal with if !cmp.Equal(want, got) and returns an error with the differences otherwise.

The declaration of datawe plug out of the functional context such that it is accessible from each testing function.

Most of what has been happening in mainwe will rename to ParseImageAnalysisResponse which will receive the JSON content of type []byte and return a ResponseAdapter and an error.

The test function has the same name with prefix TestXxx and a reference to *testing.T which enables automated testing in go with go test. The first line then enables parallel test execution with t.Parallel().

Then we declare our expected result. This time we don’t use the JSON format, since we expect data as a go struct. Thats why we state the type before opening curly parenthesises want := ResponseAdapter{... .}

want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
}

Then we have to call the function with our data and catch possible errors. got will be of type ResponseAdapter, since this is what ParseImageAnalysisResponseAdapter(…) returns.

got, err := ParseImageAnalysisResponseAdapter(data)
if err != nil {
		t.Fatal(err)
}

And last we compare expectation and result.

if !cmp.Equal(want, got) {
	t.Error(cmp.Diff(want, got))
}

If everything is correct we can use Run on Go Playground or go test in the command line and the test should pass or return an error.

=== RUN   TestParseImageAnalysisResponse
=== PAUSE TestParseImageAnalysisResponse
=== CONT  TestParseImageAnalysisResponse
{0.69146144 Lion 682 109 707 186}
--- PASS: TestParseImageAnalysisResponse (0.00s)
PASS

Full listing

package main

import (
	"encoding/json"
	"fmt"
	"github.com/google/go-cmp/cmp"
	"testing"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}

var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

func ParseImageAnalysisResponseAdapter(data []byte) (ResponseAdapter, error) {

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
	
	return responseAdapter, nil
}

func TestParseImageAnalysisResponseAdapter(t *testing.T) {
	t.Parallel()

	want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
	}

	got, err := ParseImageAnalysisResponseAdapter(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

Now we have the data available in such a format that we can actually use it. We could even convert it to JSON and pass it to our user client. But sometimes the implementation logic requires that we add or remove some fields and provide the data in a different format. If that is what makes most sense, we should do it to maintain a well designed data workflow, even if it takes some additional work.

In the next section we will define the types Response and Predictions in a way we actually need them and transform them from our Adapter types using custom UnmarshallJSON. First we will add some custom metadata. Adding or changing fields in []PredictionsAdapter however requires looping through each element in the array.

Implementing custom UnmarshallJSON

Let’s create the Test for ParseImageAnalysisResponse that we are going to implement next (ignore any errors). Out want will be a Response that contains some metadata like name and Xid and the array of predictions. We use the data to pass it to ParseImageAnalysisResponse and than see if our expectation was met.

func TestParseImageAnalysisResponse(t *testing.T) {
	t.Parallel()

	want := Response{
		Name: "Test Image",
		Xid:  "9m4e2mr0ui3e8a215n4g",
		Predictions: []Prediction{
			{
				Confidence: 0.69146144,
				Label:      	"Lion",
				X:       		109,
				Y:       		682,
				Width:      	77,
				Height:     	25,
				Method:     	"AI",
			},
			{
				Confidence:	 	0.7,
				Label:      	"Penguin",
				X:       		209,
				Y:       		782,
				Width:          -13,
				Height:       	-65,
				Method:     	"AI",
			},		
		},
	}

	got, err := ParseImageAnalysisResponse(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

Next we define types that match the JSON API response we want to relay to our client, that could be a web app or a mobile app for Android or iOS.

type Prediction struct {
	Confidence  float64 `json:"confidence"`
	Label       string  `json:"label"`
	X       	int     `json:"x"`
	Y	        int     `json:"y"`
	Width       int     `json:"width"`
	Height      int     `json:"height"`
	Method      string  `json:"method"` // AI, Manual, Predicted(?)
}

The Response will contain many of those predictions, since many objects can be detected in an image. But further more it will include meta data like a Name and an Xid. A Xid is a unique identifier that is a related but shorter version of UUID. We might need to change the type later, but for now let’s use string.

type Response struct {
	Name        string       `json:"name"`
	Xid         string       `json:"xid"`
	Predictions []Prediction `json:"predictions"`
}

Now we will adapt the ParseImageAnalysisResponse function. It will still use the data of type []byte as an argument, but instead of a ResponseAdapter it will now return our new Response. Therefore we use a variable response of type Response and use this when unmarshalling. We then check for errors and return the response.

func ParseImageAnalysisResponse(data []byte) (Response, error) {

	var response Response
	// Use custom UnmarshalJSON method to obtain Response
	err := json.Unmarshal(data, &response)

	if err != nil {
		fmt.Println(err)
	}

	return response, nil

}

Since there is no easy 1-to-1 mapping from the ResponseAdapter to the Response, we need to do some manual work. Instead of the default UnmarshalJSON which we don’t have to specify, we will now use a customized version of it that explicitly states how we can derive our Response to the client from the data in the ResponseAdapter.

The data we send to ParseImageAnalysisResponse is the one we want to unmarshall. We know that we can unmarshall it into a ResponseAdapter, so thats what we will do first, but not into the pointer of the response object r, but into a temporary variable tmp.

var tmp ResponseAdapter
err := json.Unmarshal(data, &tmp)

if err != nil {
	fmt.Println(err)
}

This variable also has the Predictions in form of a PredictionsAdapter. We now cast them into a new Predictions-Array.

var p []Prediction

We do this by looping though each element PredictionsAdapter

for _, element := range tmp.Predictions {

and create a temporary corresponding predictions element.

This predictions element also defines the data for any added field or it could also leave out fields from PredictionsAdapter. Then we append the Predictions element to the Predictions Array p, that was defined before the loop. I did variable creation and appending kind of in one step here and closed the loop.

p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			XMin:       element.XMin,
			XMax:       element.XMax,
			YMin:       element.YMin,
			YMax:       element.YMax,
	})
}

Then we set the Response fields. That includes metadata like the name and Xid and setting the predictions array p.

r.Name = "Test Image"
r.Xid = "9m4e2mr0ui3e8a215n4g"
r.Predictions = p

And last since UnmarshalJSON requires us to return an error, we do that as well.

return err

Now lets see how this function looks in its full glory.

func (r *Response) UnmarshalJSON(data []byte) error {


	var tmp ResponseAdapter
	err := json.Unmarshal(data, &tmp)

	if err != nil {
		fmt.Println(err)
	}

	var p []Prediction
	for _, element := range tmp.Predictions {

		p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			X:	        element.XMin,
			Y:     	    element.YMin,
			Width:      element.Width(),
			Height:     element.Height(),
		})

	}

	r.Name = "Test Image"
	r.Xid = "9m4e2mr0ui3e8a215n4g"
	r.Predictions = p

	return err
}

Conclusion

Great. We can now take an image that was annotated with an AI microservice and make this analysis useful for relaying it the user client by enhancing it with more relevant data for a JSON response. On the networking side how to exactly send JSON responses from a HTTP request to a client was explained . I talked about how to send POST requests to the AI micrososervice in a previous post.

Last, let’s look at the full listing, which you can find also on Go Playground. The tests could also be written into a separate file of course.

package main

import (
	"encoding/json"
	"fmt"
	"github.com/google/go-cmp/cmp"
	"testing"
)

type ResponseAdapter struct {
	Success     bool                 `json:"success"`
	Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	YMin       int     `json:"y_min"`
	XMin       int     `json:"x_min"`
	YMax       int     `json:"y_max"`
	XMax       int     `json:"x_max"`
}


func (p PredictionsAdapter) Width() int {

	return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

	return p.YMax - p.YMin
}


type Prediction struct {
	Confidence float64 `json:"confidence"`
	Label      string  `json:"label"`
	X          int     `json:"x"`
	Y          int     `json:"y"`
	Width      int     `json:"width"`
	Height     int     `json:"height"`
	Method     string  `json:"method"` // AI, Manual, Predicted(?)
}


type Response struct {
	Name        string       `json:"name"`
	Xid         string       `json:"xid"`
	Predictions []Prediction `json:"predictions"`
}


func (r *Response) UnmarshalJSON(data []byte) error {


	var tmp ResponseAdapter
	err := json.Unmarshal(data, &tmp)

	if err != nil {
		fmt.Println(err)
	}

	var p []Prediction
	for _, element := range tmp.Predictions {

		// fmt.Println("element", element, "at index", index)

		p = append(p, Prediction{
			Confidence: element.Confidence,
			Method:     "AI",
			Label:      element.Label,
			X:          element.XMin,
			Y:          element.YMin,
			Width:      element.Width(),
			Height:     element.Height(),
		})

	}

	r.Name = "Test Image"
	r.Xid = "9m4e2mr0ui3e8a215n4g"
	r.Predictions = p

	return err
}

var data = []byte(`{"success": true,
	"predictions": [
		{
			"confidence": 0.69146144,
			"label": "Lion",
			"y_min": 682,
			"x_min": 109,
			"y_max": 707,
			"x_max": 186
		},
		{
			"confidence": 0.7,
			"label": "Penguin",
			"y_min": 782,
			"x_min": 209,
			"y_max": 717,
			"x_max": 196
		}]
	}`)

func ParseImageAnalysisResponseAdapter(data []byte) (ResponseAdapter, error) {

	var responseAdapter ResponseAdapter

	err := json.Unmarshal(data, &responseAdapter)

	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(responseAdapter.Predictions[0])
	
	return responseAdapter, nil
}

func ParseImageAnalysisResponse(data []byte) (Response, error) {

	var response Response
	// Use custom UnmarshalJSON method to obtain Response
	err := json.Unmarshal(data, &response)

	if err != nil {
		fmt.Println(err)
	}
	// fmt.Println("Response")
	// fmt.Println(response)

	return response, nil

}


func TestParseImageAnalysisResponseAdapter(t *testing.T) {
	t.Parallel()

	want := ResponseAdapter{
		Success: true,
		Predictions: []PredictionsAdapter{
			{
				Confidence: 0.69146144,
				Label:      "Lion",
				YMin:       682,
				XMin:       109,
				YMax:       707,
				XMax:       186,
			},
			{
				Confidence: 0.7,
				Label:      "Penguin",
				YMin:       782,
				XMin:       209,
				YMax:       717,
				XMax:       196,
			},
		},
	}

	got, err := ParseImageAnalysisResponseAdapter(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

func TestParseImageAnalysisResponse(t *testing.T) {
	t.Parallel()

	want := Response{
		Name: "Test Image",
		Xid:  "9m4e2mr0ui3e8a215n4g",
		Predictions: []Prediction{
			{
				Confidence: 0.69146144,
				Label:      	"Lion",
				X:       		109,
				Y:       		682,
				Width:      	77,
				Height:     	25,
				Method:     	"AI",
			},
			{
				Confidence:	 0.7,
				Label:      	"Penguin",
				X:       		209,
				Y:       		782,
				Width:          -13,
				Height:       	-65,
				Method:     	"AI",
			},
		},
	}

	got, err := ParseImageAnalysisResponse(data)
	if err != nil {
		t.Fatal(err)
	}

	if !cmp.Equal(want, got) {
		t.Error(cmp.Diff(want, got))
	}

}

Links

https://bitfieldconsulting.com/golang/map-string-interface

https://jhall.io/posts/go-json-tricks-array-as-structs/

https://stackoverflow.com/questions/42377989/unmarshal-json-array-of-arrays-in-go

This problem occurred as I was trying to detect objects in an image with AI. I was able to train a server side object detection algorithm. A client specific solution i.e. with Create ML on iOS would be premature optimisation and makes more sense once the use case is well established. For now I want to be able to update the object detection algorithm anytime with improved training data.

We will talk about

1. Skeleton and imports
2. Manipulate binary data from an image
3. Multipart Messages
4. HTTP Post request
5. HTTP Response
6. Conclusion
7. Full Listing

There are two ways to handle user images. The first is to upload the image to an image hoster like cloudinary or uploadcare and then pass just the URL to the server. I actually suggest to use this method and I have written a post about this earlier . But sometimes you may want to handle images at the server directly.

Simple text forms can be send as x-www-form-urlencoded. But we will create a multipart/form-data POST request in Go to forward images.

Skeleton and imports

Let’s start with a skeleton that contains imports and main function. The main function launches a server that listens on port 5000. It will call the function handleRequest once the url http://127.0.0.1:5000/image is called. This function will load an image file from disk in the same folder, send it to an analysis microservice and receives the result.

package main

import (
	"bufio"
	"bytes"
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"mime/multipart"
	"net/http"
	"net/url"
	"os"
	"time"
)

func main() {
	handler := http.HandlerFunc(handleRequest)
	http.Handle("/image", handler)
	fmt.Println("Server starts at port 5000")
	http.ListenAndServe(":5000", nil)
}

func handleRequest(w http.ResponseWriter, r *http.Request) { 

// TODO: send image to microservice

}

Manipulate binary data from an image

In contrast to Python is Go a language that makes memory management and pointer visible. That makes it performant, but also requires some down to the metal thinking. The & operator returns the memory address of a variable. We create a http.Client. Since we don’t want to stall our server, just because the microservice is not available, we set a Timeout.

When dealing with text we use strings to perform operations like split or search. When dealing with data e.g. from images, this is done in a buffer of &bytes, where the & operator refers again to the memory address of that buffer of image bytes.

client := &http.Client{
	Timeout: time.Second * 20,
}
body := &bytes.Buffer{}

Multipart Messages

Messages that contain multiple parts like a text message and an image attachment can be encoded with MIME/Multipart, which is also used in e-mails. In this case we will only use one part. We create a multipart writer. The image section of the message with key name „image“ and file name „image.jpg“ is stored in form writer fw. You can choose any name, we are not referring to the filename on disk yet.

writer := multipart.NewWriter(body)
fw, err := writer.CreateFormFile("image", "image.jpg")

Next we open the image from disk using the operating system package os. This file is then copied into the multipart message form writer fw. Any error is logged and then the writer needs to be closed.

buf, err := os.Open("image.jpg")
_, err = io.Copy(fw, buf)
if err != nil {
	log.Fatal(err)
}
writer.Close()

HTTP Post request

Next we create a new http POST request, which in contrast to GET is used to send images and not retrieve them. Here we pass a URL and the bytes buffer that contains the image now and set the appropriate content type header.

req, err := http.NewRequest("POST", "http://127.0.0.1:80/v1/vision/custom/testmodel", bytes.NewReader(body.Bytes()))
req.Header.Set("Content-Type", writer.FormDataContentType())

Then we use the client to perform the request, catch the http status code like 200 for OK or 404 or 400 when there is an issue and store the response in rsp.

HTTP Response

rsp, _ := client.Do(req)

if rsp.StatusCode != http.StatusOK {
	log.Printf("Request failed with response code: %d", rsp.StatusCode)
}
log.Print(rsp.StatusCode)

Then we read the body section of the response and for testing purposes print it to the command line on the server.

body, err := ioutil.ReadAll(rsp.Body)
if err != nil {
	log.Fatal(err)
}
log.Println(string(body))

To return data to the user as JSON we use http response which was passed as an argument in handleRequest function.

w.Header().Set("Content-Type", "JSON")
w.Write(body)

This is now the result of the analysis the microservice has performed on the image, like the name of the detected object and its boundaries.

Conclusion

In this post we loaded an image from disk, send it to a microservice for analysis and provided the response to the user. Missing is the part, where the user can actually send the image himself from the client to the server. If you like the content and want to be notified about any updates please subscribe to the email list.

Full Listing

Here is the full listing. You can also view the code as Go Playground, but unfortunately you can’t execute it (deadlock error). For it to work you need to actually launch the microservice, which we haven’t discussed here.

package main

import (
	"bytes"
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"mime/multipart"
	"net/http"
	"os"
	"time"
)

func main() {
	handler := http.HandlerFunc(handleRequest)
	http.Handle("/image", handler)
	fmt.Println("Server startet at port 8080")
	http.ListenAndServe(":5000", nil)
}

func handleRequest(w http.ResponseWriter, r *http.Request) {

	buf, err := os.Open("image.jpg")

	body := &bytes.Buffer{}
	writer := multipart.NewWriter(body)
	fw, err := writer.CreateFormFile("image", "image.jpg")

	_, err = io.Copy(fw, buf)
	if err != nil {
		log.Fatal(err)
	}
	writer.Close()

	req, err := http.NewRequest("POST", "http://127.0.0.1:80/v1/vision/custom/testmodel", bytes.NewReader(body.Bytes()))
	req.Header.Set("Content-Type", writer.FormDataContentType())
	client := &http.Client{
		// Set timeout to not be at mercy of microservice to respond and stall the server
		Timeout: time.Second * 20,
	}

	rsp, _ := client.Do(req)

	if rsp.StatusCode != http.StatusOK {
		log.Printf("Request failed with response code: %d", rsp.StatusCode)
	}
	log.Print(rsp.StatusCode)

	body2, err := ioutil.ReadAll(rsp.Body)
	if err != nil {
		log.Fatal(err)
	}
	log.Println(string(body2))

	w.Header().Set("Content-Type", "JSON")
	w.Write(body2)
}

Links

• https://golangbyexample.com/http-mutipart-form-body-golang/

This might seem overly complicated. So, why? In a real world applications not everyone should know about all the data, some calculations should be done on a server, some on a client, some data needs to be shared, other needs to be private.

This post will not limit itself to unwrapping JSON data in Swift and Go, but also discuss tools how to create a more complicated data structure. A founder might not have a team of engineers that each can focus on a single programming language. But it is not necessary to master a programming language with all its glory. It is fine if we can mange the parts we need and learn as we go.

This post will provide everything you need from planning the communication flow and get it going from backend to frontend. We will not deal with databases though.

1. Define JSON Structure
2. Create basic backend API server in Go
   1. Convert JSON Structure to Golang types
3. Test JSON API
4. Connect UI to Backend API
   1. Convert JSON Structure to Swift structs

We will use XCode 13.1, Swift 5 and iOS 15.1.

Define JSON Structure

The way we structure data needs to be consistent along database, backend, JSON API and the front end. Like using a software for UI mockups (Figma, Sketch), it is helpful to have tool to define data structures like jsoneditoronline.org. This will serve as a source of truth for the data structures. I use this because I can save structures. Later we will use quicktype to convert the JSON structure into structs in Swift and Go.

Create basic API server in Go

Install Go

Install Go from its website https://go.dev/doc/install.

Open the commandline and type go version to check if the installation worked.

$ go version
go version go1.18.2 darwin/amd64

I use Visual Studio Code with the Go extension as code editor and echo as a framework.

Create a new project by creating a folder and changing into the new directory. Then initialize the new app and fetch the echo framework from GitHub.

$ mkdir jsonserver && cd jsonserver
$ go mod init jsonserver
$ go get github.com/labstack/echo/v4

Hello World

Now write a short server in Go. It will display Hello World when called with a browser. In the next chapter we will output a JSON structure instead.

package main

import (
	"net/http"
	
	"github.com/labstack/echo/v4"
)

func main() {
	e := echo.New()
	e.GET("/", func(c echo.Context) error {
		return c.String(http.StatusOK, "Hello, World!")
	})
	e.Logger.Fatal(e.Start(":1323"))
}

The crucial function is e.GET(). It says wenn we call the root URL / it will call a function that returns a string. We will create a separate function to do this next.

Save this file as server.go and from the terminal run $ go run server.go.

$ go run server.go
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.7.2
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:1323

Success! Open a browser with the URL 127.0.0.1:1323/and you should see Hello, World! in the browser window.

Convert JSON Structure to Golang types

First we create a sample file groceries.json where we will coding books.

{
"title" : "Weekend groceries"
}

{ } in JSON indicates a dictionary and not a function like in Go or Swift. A dictionary consists of key and value pairs. Later we can access the title by printing print(groceries.title). Parentheseses ""contains text or a string. Numbers on the other hand don’t need parentheses and are of type Int. These are basic types. We can also create our own types or structs with multiple fields.

We will create a struct to load this structure into an object in Go. After the import segment insert the following lines to declare how the structure of an object of type groceries looks like.

Note: The purpose here is to make programming easier. The code editor will suggest existing fields and give an error when a field doesn’t exist. There are other languages like python that are untyped or where types are optional. Working without types, classes, structs and objects is easier to learn, but more error prone later.

type Groceries struct {
	title 	string 	'json:"title"'
}

Title is the field name, string is the type and ‘json:”title"’ contains the corresponding json field, which usually is the same, but it doesn’t have to be.

Next we change the URL from root / to /api/v1/groceries using groups calling the function readJSON() that reads and returns the groceries.json file. Add the following in the main() function after initialising the echo e framework.

apiGroup := e.Group("/api")
v1 := v1.Group("/v1")

v1.GET("/groceries", func(c echo.Context) error {
		books := readJSON()

		return c.JSON(http.StatusOK, groceries)
		}
	)

Return JSON File

We still need to define the function readJSON().

func readJSON() book {
	file, err := ioutil.ReadFile("./groceries.json")
	
	if err != nil {
		log.Fatal(err)
	}

	b := book{}
	err json.Unmarshal(file, &b)

	if err != nil {
		log.Fatal(err)
	}
	return b
}

Visual Studio Code automatically adds required imports, but if it doesn’t here is the full listing with the added imports.

package main

import (
	"net/http"
	
	"encoding/json"
	"io/ioutil"
	"log"

	"github.com/labstack/echo/v4"
)

type Book struct {
	title 	string 	'json:"title"'
}

func readJSON() book {
	file, err := ioutil.ReadFile("./groceries.json")
	
	if err != nil {
		log.Fatal(err)
	}

	b := book{}
	err json.Unmarshal(file, &b)

	if err != nil {
		log.Fatal(err)
	}
	return b
}

func main() {
	e := echo.New()
	e.GET("/", func(c echo.Context) error {
		return c.String(http.StatusOK, "Hello, World!")
	})

	apiGroup := e.Group("/api")
	v1 := v1.Group("/v1")

	v1.GET("/books", func(c echo.Context) error {
		books := readJSON()

		return c.JSON(http.StatusOK, books)
		}
	)
	e.Logger.Fatal(e.Start(":1323"))
}

We run this again with go run server.go and open a browser http://127.0.0.1:1323/api/v1/groceries.

Integrate an advanced JSON Model

If everything worked so far (and write a comment below if it doesn’t), we can move on to a more advanced data structure. The issue is that the JSON structure and the fields in the Go and Swift structs need to match. When fields are labelled wrong, are missing or have the wrong type the app crashes. Then it is important to have some kind of error message to tell you where to look for any issues.

Open JSONEditorOnline to create your own data structure or use the following grocery list.

{
  "list": {
       "title": "Dinner",
       "uid": "wq3413lkj"
  },
 "items": [{
    "name": "flour",
    "uid": "asd3lpr2j",
    "quantity": {
      "value": 500,
      "unit": "g"
    },
    "brand": "Natura",
    "expired": "12-12-2022",
    "image": "s3dasfsf"
     },
     
    {
    "name": "potatoes",
    "uid": "4h3q4erg",
    "quantity": {
      "value": 2,
      "unit": "kg" },
    "brand": "regio",
    "expired": "08-12-2022",
    "image": "s3/2343gvdv" }]
}

Also save this list as groceries.json in the Go folder. Before we will move to the front end, let me introduce a tool to test APIs. When APIs become more complex you don’t want to test each request by hand again and again.

Test JSON API

To test APIs I use Talend API Tester as a Chrome Browser extension, but there are others available.

Connect UI to Backend API

The backend now has the most basic functionality. You can expand it with user authentication, or attach other micro services. Now we will move to the front end . It will display the data encoded in JSON to the user. You will need XCode 13.1 or above, some functionality like async is not available in earlier versions and later versions don’t work on older macs.

Open Xcode to create a new project.

Create a new Folder Services and a SwiftUI File JSONService.swift.

Create a new struct JSONContentUI: View. To store the data we create a @StateObject and call it groceries. This will automatically update the UI when data is changed.

struct JSONContentUI: View {
@StateObject var groceries = GroceriesAPI()	
	var body: some View {
	
		List {
			ForEach(groceries.groceries, id: \.self) { grocery in
				HStack {
					Text(grocery.list.title)
			}
		}
		.onAppear {
			groceries.fetchJSON()
		}
	}

}

As you can see we use a class GroceriesAPI which has the function fechJSON(). Let’s implement this class next.

First we define the source URL. This might fail because of an invalid URL, so we need to guard it. We then open an URLSessionto obtain the JSON as a string and print it.

class GroceriesAPI: ObservableObject {
	@Published var groceries: [groceries] = []
	
	func fetchJSON() {
		guard let url = URL(string: "http://127.0.0.1:1323/api/v1/groceries") else { return }
		
		let task URLSession.shared.dataTask(with: url) { data, response, error in
				if let data = data, let string = String(data: data, encoding: .utf8) {
					print(string)
				}
	
		// Decode JSON data
	}

}

Convert JSON Structure to Swift structs

Then we add the structs to encode groceries. We paste the JSON structure on quicktype, name it Groceries and select Swift as a target language and copy the structs to the JSONService.swift file.

// MARK: - Welcome
struct Welcome: Codable {
    let list: List
    let items: [Item]
}

// MARK: - Item
struct Item: Codable {
    let name, uid: String
    let quantity: Quantity
    let brand, expired, image: String
}

// MARK: - Quantity
struct Quantity: Codable {
    let value: Int
    let unit: String
}

// MARK: - List
struct List: Codable {
    let title, uid: String
}

Decode JSON to struct

We then expand fetchJSON() to decode the JSON string.

let json = try JSONDecoder().decode([user].self, from: data)

To catch errors we wrap this in a do {} catch {} block. We then assign the encoded JSON groceries object to the groceries StateObject that is read by the UI.

do {
    let json = try JSONDecoder().decode([groceries].self, from: data)
    print(json[0])

    DispatchQueue.main.async {
        self.groceries = json
   	}
} catch {
	print("JSON Format does not comply with struct keys.")
}

Conclusion

So congratulations if you made it this far. There was a lot of stuff in here, I am writing these articles as I learn them myself, so please feel free to add any recommendations of issues you had while going through this.

Working on a business is similar to working out. Working on it daily leads to results over several months. And we want to get shredded. The best way to get shredded it to setup the environment to make working out easy and even fun.

Why Taking Notes Matters

A good tool has to grow with the project. It starts with a blank canvas. Then you add a few bullet points. After a while you see the patterns to group similar bullet points under headlines like “product features" or "marketing". But you just have a rough idea to use a blog as a marketing channel, so there is nothing else to write. You don’t want to make it a page yet, let alone introduce a new tool just for marketing. It has to grow organically. Especially when you add a team members who needs to dig through your thoughts.

What else is out there?

For some time I am missing a tool to launch new projects weather it is an NGO, an app, a video or a business. I have been using Smartsheet, Google Docs, Almanac and Notion, but haven’t been happy so far.

Don’t get me wrong, Smartsheet has amazing GANTT charts for very complex projects with lots of nested tasks and dependencies with multiple teams.

Google Docs is good for collaborating on simple projects in real time.

Notion has its database capabilities and I am sure you can organise your life with it, but I haven’t become cozy with it yet. Mainly because it is not a native app and therefore feels slow. Almanac is nice but it doesn’t have a mobile app at all.

Notes

I recently fell in love with Craft.do. The downside, it is iOS first. It has a WebApp, but as of May 2022 it is not nicely usable on an android devices (change to desktop mode on your browser in the craft web app). If you can live with that, it's a stellar experience so far. Bullet points grow into headlines, headlines can be turned into pages. And before you know, you have a business plan draft. Pages are linked and can be shared with advisors or new team members. They update themselves, so no need to send out a new version of final draft version 2. But most importantly it accommodates the thought process of bringing a new project to life through writing.

Writing means clarity of thought. It avoids repeatedly thinking about the same thing over and over again and getting ideas out of you brain to make room for new ones. It allows you to get back to an idea at any point and reevaluate their validity. This avoids shiny new object syndrome.

Keyboard

To work on the go, I have been using an iPad for ten years now. It has become a canvas, great for consuming video and for writing - with a pen. But it hasn’t replaced my Laptop so far mainly because of the keyboard. There are bluetooth keyboards, which is a second item to carry around, or Apples smart over, which has a terrible writing experience.

The Apple Magic Keyboard is really nice but 300 bucks for a keyboard which will probably not work for the next generation iPad. Sorry.

Luckily Logitech has come up with the Combo Touch Keyboard. Not only does it have a keyboard, it also has a trackpad and it is fully detachable when you want to watch a movie. It costs around 140 EUR but it turns a tablet into a laptop. This is highly valuable for anyone that needs to write long pages of text for a business plan or blog post like this one.

Conclusion

The Craft app (Free plan, 4 $ personal, 10 $ Business) together with the Logitech combo touch keyboard (140 EUR) turns the iPad from a consumption into a productivity device. Yes, it works as a laptop replacement if you don’t have special needs like virtual machines or XCode. It’s fast, it's quiet and it's light. And most of all, it's fun to use. You won’t be disappointed.

Update 05. December 2022: Added how to edit DNS records for custom email domains and updated labeling frameworks for object detection.
TODO: Elaborate on Step 11 and 12.

Ideas are cheap, execution is key - we know that. We also know that we should not build out all kinds of features that nobody wants, but launch a Minimum-Viable-Product (MVP) or better a Minimum-Sellable-Product (MSP) and get it into the hands of users as soon as possible. But at the idea stage, are you sure you can deliver on your promise? When should you build a landing page for customer validation? Here is my experience from the past few weeks.

In this series I will document my 12-step process as a solo low budget technical SaaS founder from idea to MVP using AI. But what if you are non-technical founder ? Even then you will be able to get to step 10 of 12 and gain credibility with investors, potential technical co-founders and save time when working with freelance developers. I will introduce some tools that I actually use like Mailerlite (Affiliate) for landing pages or Looka to create a logo with AI. The job of a founder is to stich these existing services together in an elegant way such that a product or service appears on the other end. Even as a technical founder, don’t waste your time reinventing the wheel.

It is of course possible to build anything from scratch using open source tools, you can host a website on your laptop and forward ports of your router if you are on a no budget, but you get distracted with maintenance and you loose mental energy from your core product.

The 12-step process will cover

1. Defining the Target Audience
2. Estimate the Market Size
3. Product Idea
4. Estimate Pricing
5. A Proof-of-Concept
6. Branding: Name, Logo, Color-Scheme
7. Customer Validation with Landing Pages, Email-Lists and Surveys
8. Feature Priorisation
9. Mockups and Prototype
10. User Testing
11. Minimal Viable Product
12. Beta Program on iOS

1. Defining the Target Audience

A good business solves a pain point of a large but specific group of people. Low hanging fruits, problems that basically everyones has, are already harvested, have large barriers to entry or are too competitive. How do you compete if someone is able to sell a product at cost or even at a loss? Therefore it is fine to address issues only a niche group of people has or you had personally in your business or private life.

By defining your target audience you learn where to find them. Who do they follow on Twitter, which groups do they join on WhatsApp, which conferences do they visit? If you are your target audience, then observe yourself, where do you spend your time on- and offline?

It further helps to distinguish between your product not being sophisticated enough for a customer and someone who simply doesn’t has any demand for your product at this point in time. Try to sell drugs to someone who isn’t into drugs, and no matter the quality of your product you will never convince them to buy anything. The same is true for someone who has already bought enough drugs or is currently out of money, no matter what you do, they will not be your customer, maybe later. You will be surprised how many businesses fail to understand this. We will validate assumptions made here in step 7 (don’t do drugs tough, they are only useful to illustrate the point).

Let’s pick people visiting the zoo with a smartphone as an example.

2. Estimate the Market Size

Starting out it is good to niche down, but here the market size is already determined and this decides if an idea is fundable. No matter how fancy the technology is, if the market is too small you might need to go without funding, also called bootstrapping or worse there is no business. I have worked at an angle funded startup that went out of business for that reason.

There are 318 zoos in Germany. To sell an app to each german zoo (which is already an illusion) for 10 $ a month, thats 3180 € a month or 38 160 € a year. You can’t even hire one engineer from this. BUT if you can run it as a side business, it might be viable.

Can you sell to each zoo visitor? Different story, 800 000 people visit the zoo in Frankfurt each year, well at least when there is no pandemic. So let’s say we target 10% of the zoos (sounds low? They won’t even respond to your email, I tried.) and 10% of the visitors (because the others don’t have a smartphone, were counted twice, are not our target demographic, poor students or simply not interested), that makes 31 zoos with 80 000 convertible visitors each (not every zoo is as large as Frankfurt, make a note, and refine later), so 2.4 million potential customers, if we can sell all of them something for 10 € a month, thats a whooping 288 million € a year. Here we have a business. This is our Total Addressable Market (TAM). If we multiply the estimated yearly revenue by 10, we get a market value of 2.8 billion €. Thats more than a billion and therefore a unicorn (= an investors dream).

The numbers don’t need to be accurate, the goal is to distinguish between business and no business using suitable orders of magnitude. Just don’t make the mistake to hire a team of ten people to then figure out later your market can only support one person part time or pitch a startup to investors that you need to bootstrap anyway.

Side note: The decision to accept venture capital directly leads to the need for an exit, so to sell the company to a larger company or go public via an IPO or recently with SPACs. Thats where investors and the founders get their payday.

Not everyone wants to sell their company though. Mark Zuckerberg didn’t sell Facebook to Yahoo for 1 billion $ in 2006, because he said it might be the only idea he will ever have. Brian Acton now works for and donated a billion $ to Signal after selling WhatsApp for 19 billion $ to Facebook. The founder of Wonderlist Christian Reber wanted to buy back his company after selling it to Microsoft for 250 million $ - who in turn shut it down. Now he is again building his second ToDo-App.

3. Product Idea

Let’s start with a machine learning app to identify large cats (panthera) in the zoo, so it should detect if an animal is a lion, a tiger or a leopard. Does this sound more like a feature or like a long term vision? I think it is good to have both. To just identify lions may be just a feature, but it will be the feature we will focus on till MVP stage.

The long term vision might be an Augmented Reality (AR) based zoo guide. Identifying lions will then only be a small, but killer feature. At this stage focus on the feature, but keep the vision in the back of you head, because that will be the business in the end.

4. Estimate Pricing

Let’s assume people value their time around 20 € an hour (got that number from a guy who works in neuroscience). A solution should deliver 10x more value than the customer has to pay. So if someone saves 2 hours a month using the app by not going to the library to identify how a lion looks like or employing a zoo guide, thats 40 €. A tenth of this would be 4 € a month.

We all undervalue our work, so always try to double the price once in a while, see if something bad happens. When charging too little, it is not possible to deploy enough resources to build a product that satisfies the customers. There is also more than productivity to pricing, like brand, convenience, competition and design.

5. A Proof-of-Concept

Ok, we have an idea, and it is a suitable business, but can we actually deliver? Of course you can skip this and go to customer validation right away. But personally I like to make sure I actually can build the product first. So even with any help available can you actually train and deploy an AI to identify images of lions?

First we need to label the images that we can then use to train an algorithm width. For labelling Label Studio provides a great suite and is free as well.

Then we need to train an algorithm. Since we will most likely have only few images it is best to use pre-trained models like YOLO.  DeepStack provides a Docker container to use labeled images for training an algorithm and then launch a server to make predictions.
To classify images Detecto is a good start as well. It is a python framework based on PyTorch, a widely used machine learning framework. The most important and labor intensive part is labelling the images.

Second you can try to find machine learning services like V7Labs. They are good, but cost around 150 € per month. They have a 14 day free trial though and good customer support. V7 is especially good for labelling (also in groups, so it can be outsourced), training and directly offering an API for your users to classify images. This is the fastest way to get going.

6. Branding: Name, Logo, Domain and Email

Now it is time to pick a name. There are awesome AI powered services like Namelix to create name ideas as well as Looka to create logo ideas (free to create, 80 $ for commercial use). From the logo generator I also try to get a color scheme to use with the landing page for a recognisable branding.

Then use the name to get a suitable domain on something like united-domains.com. .com or .org domains might be hard to get, but there are also new domains like .app or .io that are usually not that crowded (59 $ / year).

For email boxes like name@domain.app I use the independent swiss provider migadu.com (19 $ / year), they also charge based on number of emails send and not artificially limited based on number of domains registered or number of email boxes.

To do be able to send and receive emails from and @yourdomain.com address the records for the Domain-Name-System (DNS) need to be specified. It is responsible for converting an IP 97.120.21.123 into yourdomain.com. For the email service it requires copying entries from Migadu Setup Instruction page

to the DNS Entry of United Domains.

This process takes some time, first by setting it up but also till changes take effect, the DNS system is slow and it takes time till changes a synced over the internet. In a similar way you need to setup the DNS entries from your website, which talk about next.

7. Customer Validation: Landing pages and Surveys

Once we know it can be done and created our brand, it is time to test the market. A good way is to create a landing page with a concise description of our product and make user sign up for a mailing list to get early access. I prefer Mailerlite (Affiliate, free, 90 $ if you want your own domain) as an all in one solution with landing page, email list and analytics, but Webflow and Unbounce.com (90 $/month) and Mailchimp (Affiliate) are also common options.

Like in step 6. it requires copying DNS settings from Mailerlite (Account settings -> Domains -> View records) or Webflow into the DNS entries page of United Domains.

Once I created a nice landing page with a concise value proposition, I try to post the link in a group where I know there are interested customers, like a WhatsApp group.

When I posted the link in a WhatsApp group with 194 members, I got 28 visits and 8 of them subscribed to the newsletter. Thats a conversion rate of 28 %, of course we have small numbers here, but that is at least a signal that demand exists.

This step is not always easy, be sensible where posting links is appropriate and not spammy. We can also do targeted paid advertising on Facebook, Twitter and LinkedIn, a next step for me to try out. It is always important to target customers that might actually be interested, otherwise we get a lot of visits to our site with no signups and that ruins the conversion rate statistic on Mailerlite.

To learn more about your potential customers, like how much they would be willing to pay for your product, you can ask them to fill out a survey. Now as we all know surveys like Google Forms are an option but are often very annoying, it should be as convenient as possible to fill them out. For example you can ask different questions based on age group or role in a company or ask them if they are willing to pay 50 € a month and only if they answer „no“, you could again ask if they would be wiling to pay 25 € a month. This can be done by branching questions and only ask certain questions based on already given answers. The #1 tool to do this is Typeform.com, but their plan with 21 € per month is also very expensive. Fortunately there is a nice and for now free alternative called Tripetto.app. Check them out, you won’t be disappointed.

Example of how a branched questionnaire in Tripetto might look like.

8. Feature Priorisation

While getting excited about the first subscribers in this phase your mind will probably generate a ton of ideas what your app should be able to do. It could also detect apes or plants, maybe it should display more details about an animal and maybe a small video. Those ideas are helpful, if dealt with responsibly. Most of them should not make it into the MVP, but are ideas for further development later and maybe even much later.

But to prepare for this phase and maybe include one killer feature into the MVP, it is helpful to sort features by priority with the help of your new subscribers. You can even let them make suggestions for new features themselves. Services that can help with this are Productboard, Upvoty, Hellonext and Noorahq, which do more or less the same, so pick the one you like most. This will help you develop clarity about what your app should be able to do, which will be the guide for the next step, sketching out a prototype.

9. Mockup and Prototype

Now, we have the idea, we know it is potentially a business, the pain point we want to address and the most important features. Now the solution needs to be translated into text fields, images, boxes and buttons. As a mockup tool I use Proto.io (25 € / month), but Adobe XD (11,89 € / month), Sketch (9 $ / month) and Figma.com (free for three files) are other common tools. If you are not comfortable with technical tools, just use pen and paper and draw out the flow of your app.

Before you implement the app, you will need to hire an UX Designer who creates a high fidelity version of your prototype, but too get your idea from your head into the real world and to communicate it to users, investors and a potential co-founder being able to draft a prototype is an important skill for any SaaS founder.

10. User Testing

While landing pages and feature priorisation have to be done with your target audience, you don’t need them for user testing. Anyone can test your product. Here it is about, does a normal human use your product in a way you would expect them to do. Do they search for functions where you have hidden them? Just give them a task they should complete and observe how they interact with your app.

For more professional user testing there are also companies like Loop11 and Userbrain.com to help with this, but they seem not worth it at early stages. But at least now you know they exist, use them if needed.

11. Minimum Viable Product (MVP) with in App Customer Feedback

After 10 steps we are finally ready to get our hands dirty. We pretty much determined what we want to build, now it’s time to build it. You can do or learn this yourself if you want. I subscribed to raywenderlich.com (485 € incl VAT, but Black Friday discount available) for mobile frontend and usegolang.com (220 € life time access, but Black Friday discount available) for backend development, but there are also cheaper courses on Skillshare, Udemy and Youtube especially to just get started. If you need help or are stuck with an error, large companies have senior developers in different departments, but you still can get similar mentoring from experts in their field on codementor.io (20 $ per 15 minutes).

But learning to code while building a startup might not be possible. It is always good to engage a technical co-founder and this stage is perfect for this, since now we actually know what we want to build. If you contact a developer at an earlier stage they feel like you had an idea but want them to do all of the work. Developers develop what you ask them according to your specification. But it is not their fault if the specification is wrong in the first place.

If the app does not demand using specific technology like 3D, Apple Pencil, Augmented Reality or new UI elements it is good to use cross platform languages like Flutter, React Native or NativeScript Vue. This enables you to use one codebase for iOS and Android.

If you need core technologies it is better to develop the iOS version in SwiftUI and Android version in Kotlin. This way you can leverage the strength of each platform. I love NativeScript, it does actually give you access to native APIs, but then you add a new level of complexity, because you need to understand iOS and Android as well as NativeScript and you don’t get as many prebuilt libraries.

12. Beta Program with User Feedback and User Analytics

You can develop the MVP in the last step in the iOS Simulator for free, but if you want to test on device you need access to the Apple Developer Program, which is 99 $ per year. This also enables you to use Inflight for beta user testing. Now we actually need target customers unlike in step 10.

Since we collected a few email addresses using the landing page from step 7, this is the time we can leverage them. First contact individual subscribers directly, ask them to test the product and get feedback. Then reach out to groups of subscribers and so on.

The Bottom Line

So how much do we have to spend to get here?

So for 3085 € we can develop an MVP that leverages the power of machine learning in about a month, sure thats not cheap but definitely also not millions either. But you can also go for just the essentials with 275 € per year. So you are either convinced that your idea is that good to put your money and effort on the line - or you don’t and will never know. But let me tell you seeing other people succeed with ideas you never acted on is a painful experience.

V7 is the largest expense, you can avoid them by using PyTorch or Detecto or using their trial and then wait until you have actual customers waiting in line, but ultimately if your are not a computer vision expert I would not waste time with PyTorch and just use V7s service (while I have some affiliate links in this article, there is unfortunately none for them. V7 was just the best option available to develop a machine learning Proof-of-Concept in 3-4 days without any code or advanced machine learning knowledge.)

Conclusion

So there we have it, the hopefully complete step by step process including budget to develop an AI based MVP of a mobile app starting with just an idea. Which tools do you prefere to use? Any suggestions comment below.

This is the first article on my blog sebastianroy.de and Medium, so as as sign of interest I would be happy if you subscribe to my newsletter, share it with friends who might benefit from it and give me some feedback of the three items you liked and three items you disagree with. Let me know if this lengthy but conclusive style of article is something you like to see more of.

Sebastian Roy is a biophysicist from Frankfurt who co-founded the iGEM Team Frankfurt, the NGO Nepal Health e.V. and currently developing gelly.bio, an app to automatically annotate gel electrophoresis images.