Delighted To Resolve Unexpected Consequences In Go JSON Marshal

I ran into a bug in some code I wrote a few years back that was caused by some “invalid” JSON data values, and the way Go handles that scenario.

This was fairly early in my journey to be a better Go developer, and I remember being surprised that the marshaller didn’t just handle the inconsistency.

Data Incosistency

The data had elements for numeric values like budget, that were being returned as a JSON number in some cases, and as a string in others. On top of that, not all of the items containing the data had that element.

Explaining the problem

To explan the problem, I have a simplified example. The JSON below shows the problem:

{
  "Items": [
             {
               "id": 1,
               "name": "some string",
               "problem": 1
             },             
             {
               "id": 2,
               "name": "some string",
               "problem": "2"
             },             {
               "id": 3,
               "name": "some string",
             },
           ]
}

You can see from the above that, I added a field named problem. It has a number in the first instance, a string in the second, and doesn’t exist in the third.

In traditional Go fashion I would get that data, and put it in a structure based on the data I was expecting like illustrated below.

type ExampleInput struct {
	Items []struct {
		Id      int         `json:"id"`
		Name    string      `json:"name"`
		Problem interface{} `json:"problem,omitempty"`
	} `json:"Items"`

Unintended Consequences

Next I updated the struct and defined all the fields as strings. I was relatively new to GO. I thought I knew that by defining it as a string the data would be unmarshalled as an empty string. The new type looked like:

type ExampleInput struct {
	Items []struct {
		Id      string      `json:"id,omitempty"`
		Name    string      `json:"name,omitempty"`
		Problem string      `json:"problem,omitempty"`
	} `json:"Items"`

I ran my test again only to get the error below.

error: json: cannot unmarshal number into Go struct field .Items.problem of type string

I tried I making that field a number as well, only the message was:

error: json: cannot unmarshal string into Go struct field .Items.problem of type int

Data consistency

Continuing to get a consistent set of data, I modified the struct for the json.Unmarshal to usethe interface{} type.

I added a function to copy the data from there into the one with all strings, and made the assumption that I could simply use fmt.Sprintf(“%v”, inputStruct.problem) like this:

package main

import (
	"encoding/json"
	"fmt"
	"log"
)

type FromServer struct {
	Items []Item `json:"Items"`
}

type Item struct {
	Id      int         `json:"id,omitempty"`
	Name    string      `json:"name,omitempty"`
	Problem interface{} `json:"problem,omitempty"`
}

type DesiredDataStruct struct {
	Id      int    `json:"id,omitempty"`
	Name    string `json:"name,omitempty"`
	Problem string `json:"problem,omitempty"`
}

var myInput = `{
  "Items": [
             {
               "id": 1,
               "name": "some string",
               "problem": 1
             },             
             {
               "id": 2,
               "name": "some string",
               "problem": "2"
             },             
			{
               "id": 3,
               "name": "some string"
             }
           ]
}`

func main() {
	var myData FromServer
	err := json.Unmarshal([]byte(myInput), &myData)
	if err != nil {
		fmt.Println("unmarshal error:", err)
	}
	newData, err := ConvertDataInput(myData)
	if err != nil {
		fmt.Println("conversion error:", err)
	}
	log.Printf("newData: %+v", newData)
}

func ConvertDataInput(dataInput FromServer) (output []DesiredDataStruct, err error) {
	for _, item := range dataInput.Items {
		output = append(output, DesiredDataStruct{
			Id:      item.Id,
			Name:    item.Name,
			Problem: fmt.Sprintf("%v", item.Problem),
		})
	}
	return output, nil
}

That seemed to work. On closer inspection, the omitempty on the problem field was being set to <nil> . That was the unintended consequence of using the fmt.Sprintf

2024/01/29 22:02:29 newData: [{Id:1 Name:some string Problem:1} {Id:2 Name:some string Problem:2} {Id:3 Name:some string Problem:<nil>}]

For most things that wasn’t a problem as I could code around the value, and I hadn’t really gotten to where I was using that particular field in any significant way.

Unintended Consequence

When I needed to update the records in the database I saw the unintended consequences: this code caused the rows where the problem field was missing got a value of <nil>

[
             {
               "id": "1",
               "name": "some string",
               "problem": "1"
             },             
             {
               "id": "2",
               "name": "some string",
               "problem": "2"
             },             
			{
               "id": "1",
               "name": "some string"
               "problem": "<nil>"
             }
           ]

The Problem field was no longer throwing an error, it allowed me to read the JSON and use the data. The problem came up later when I went do store it in the database.

Custom MarshalJSON

t hadn’t really thought about why I’d use a custom marshaller. It turns out to be a perfect solution for handling the inconsistent data.

Never having written one, normally I would have Googled, read and figured it out. I have Github CoPilot in my IDE, I just asked it what to write and it gave me a pretty clear set of changes.

func (d DesiredDataStruct) MarshalJSON() ([]byte, error) {
	problem := ""
	switch v := d.Problem.(type) {
	case int, float64:
		problem = fmt.Sprintf("%v", v)
	case string:
		problem = v
	}

	return json.Marshal(&struct {
		Id      int    `json:"id,omitempty"`
		Name    string `json:"name,omitempty"`
		Problem string `json:"problem,omitempty"`
	}{
		Id:      d.Id,
		Name:    d.Name,
		Problem: problem,
	})
}

For an experienced developer, LLMs like CoPilot are a useful tool. They help you in places where you may not have the expertise and/or want to spend hours learning a subject. AI tools can hallucinate and provide incorrect answers. You do need to understand and verify the proposed solution.

That took care of the problem in my example Problem field, and I no longer got those <nil> values.

Handling all the fields

In my case, the actual structure had a number of fields that had “illegal” values in the JSON (ones that sometimes were strings and sometimes numbers). I asked Copilot to generalize this to handle all the fields in the structure.

func (d DesiredDataStruct) MarshalJSON() ([]byte, error) {
	type Alias DesiredDataStruct
	alias := Alias(d)

	val := reflect.ValueOf(&alias).Elem()
	typ := val.Type()

	for i := 0; i < val.NumField(); i++ {
		field := val.Field(i)
		switch field.Kind() {
		case reflect.Int, reflect.Float64:
			val.Field(i).SetString(fmt.Sprintf("%v", field.Interface()))
		case reflect.String:
			if field.IsNil() {
				val.Field(i).SetString("")
			}
		}
	}

	return json.Marshal(&struct {
		Alias
	}{
		Alias: alias,
	})
}

In the code above, reflection is used on the DesiredDataStruct to loop through all the fields in the struct. That way all the fields in the struct will function on the “illegal” data formats. We can add any number of new fields to the struct and have the same results.

In a real solution I may have forgone the reflection and only handled the fields I expected to have the problem. The above is provided as an example of what sort of code CoPilot can write for you.

To Conclude

I’m sure the new code has other unintended consequences that I may need to deal with. This did get me past my problem of populating the underlying fields in the MongoDB with <nil>.

I mentioned that Github CoPilot (or any similar AI LLM) can help find a quick solution to a problem if you ask it the right questions.

Hopefully it will help you out in your coding adventures.