Golang Unmarshal json without struct definition

When you start working with Golang, you might see a struct definition like the following.

type employee struct {
    Name string `json:"name"`
    Age  int    `json:"age,omitempty"`
}

It absolutely looks like JSON-related things but how it actually works?

If you need to parse or convert JSON because your application communicates with REST API or something similar, this post is for you.

How to convert from struct to json and vice versa

Golang offers a simple way to convert. Let’s define the following simple struct.

type product struct {
    Name  string
    Price int
}

struct to json

json.Marshal(json) can be the first option.

item := product{
    Name:  "Apple",
    Price: 55,
}
jsonBytes, err := json.Marshal(item)
if err != nil {
    fmt.Println(err)
}
fmt.Println(string(jsonBytes)) 
// {"Name":"Apple","Price":55}

The json.Marshal returns byte array. So it’s necessary to convert it to string.

json to struct with indent

Use json.MarshalIndent() instead if you want to make it more readable.

prefix := ""
indent := "\t"
jsonBytes2, err := json.MarshalIndent(item, prefix, indent)
fmt.Println(string(jsonBytes2))
// {
//         "Name": "Apple",
//         "Price": 55
// }

This is useful when you want to write the content of the struct to log.

json to struct

Use json.Unmarshal(jsonBytes, *dest) to convert JSON to struct.

var fromJson []product
myJson := []byte(`[
    {"Name": "Apple", "prICe": 11, "id": 3},
    {"Name": "Melon", "PricE": 22, "id": 4}
]`)
err = json.Unmarshal(myJson, &fromJson)
if err != nil {
    fmt.Println(err)
}
fmt.Printf("%+v\n", fromJson)
// [{Name:Apple Price:11} {Name:Melon Price:22}]

The second parameter needs to be a pointer.

As you can see, the undefined key value is not assigned to the struct. However, it is not case-sensitive. Even though one of the key letters’ cases is different, the value is correctly assigned.

Define the name usesd in the json

If you want to define the name used in JSON, you can define it on the same line as the variable in a struct. The format is the following.

json:"key_name"

Let’s see this example.

type employee struct {
    Name           string         "json:name"
    Age            int            `json:"age"`
    Gender         string         `json:"gender"`
    Job            string         `json:"role"`
    WithoutSchema  string
}

employee := employee{
    Name:   "Yuto",
    Age:    35,
    Gender: "Male",
    Job:    "Software Developer",
}
prefix := ""
indent := "\t"
jsonBytes2, _ := json.MarshalIndent(employee, prefix, indent)
fmt.Println(string(jsonBytes2))
// {
//         "Name": "Yuto",
//         "age": 35,
//         "gender": "Male",
//         "role": "Software Developer",
//         "WithoutSchema": ""
// }

The Name variable doesn’t have double quotes for the key itself. Therefore, the variable name is used in the JSON as it is.

Age and Gender have the expected format, so the first letter is lowercase. The name of Job in JSON is totally different but it’s also written as expected.

As you saw in the previous section, the variable name is used since it has no schema definition for WithoutSchema.

Is a key-value assigned to struct when the json name is different?

What happens if the two struct has the same variable name but a different JSON name? Let’s define the following struct. Name has all the lower cases but the same letter. Gender has totally different naming.

type person struct {
    Name   string `json:"name"`
    Age    int    `json:"age"`
    Gender string `json:"fooo"`
}

Then, let’s try to convert a JSON to struct.

employee := employee{
    Name:   "Yuto",
    Age:    35,
    Gender: "Male",
    Job:    "Software Developer",
}
jsonBytes, _ := json.Marshal(employee)

var yuto person
err := json.Unmarshal(jsonBytes, &yuto)
if err != nil {
    fmt.Println(err)
}
fmt.Printf("%+v\n", yuto)
// {Name:Yuto Age:35 Gender:}
fmt.Printf("Name: %s, Age: %d, Gender: %s\n", yuto.Name, yuto.Age, yuto.Gender)
// Name: Yuto, Age: 35, Gender:

The destination name is fooo while the JSON object has gender hence the key value is not assigned to the Gender property.

How to assign null to json

If the variable is defined as a non-pointer type, the default value is assigned when no data is specified.

If you want to assign null to the key, you need to define the variable as a pointer.

type jsonTestStruct struct {
    // add asterisk to the data type
    StringPointer0  *string        `json:"stringPointer0"`
    StringPointer1  *string        `json:"stringPointer1"`
}

str := ""
obj := jsonTestStruct{
    StringPointer0: &str,
    StringPointer1: nil,
}
jsonBytes, _ := json.MarshalIndent(obj, prefix, indent)
fmt.Println(string(jsonBytes))
// {
//     "stringPointer0": "",
//     "stringPointer1": null,
// }

You don’t have to explicitly set nil.

How to omit the key when the value is not set

There are some cases where the key should not appear in the JSON if the value is not specified. In this case, you can add omitempty keyword in the schema.

type jsonTestStruct struct {
    StringPointer0 *string        `json:"stringPointer0"`
    StringPointer1 *string        `json:"stringPointer1"`
    StringPointer2 *string        `json:"stringPointer2,omitempty"`
    StringValue    string         `json:"stringValue,omitempty"`
}

It must be in the double quotes. Don’t add a space after the comma.

str := ""
obj := jsonTestStruct{
    StringPointer2: &str,
    StringValue:    "",
}
jsonBytes, _ := json.MarshalIndent(obj, prefix, indent)
fmt.Println(string(jsonBytes))
// {
//     "stringPointer0": null,
//     "stringPointer1": null,
//     "stringPointer2": "",
// }

stringPointer2 has an empty string there because it is a pointer type. On the other hand, StringValue is omitted. According to the official documentation, if the value is the default value, the key is omitted.

The “omitempty” option specifies that the field should be omitted from the encoding if the field has an empty value, defined as false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string.
https://pkg.go.dev/encoding/json#Unmarshal

If you want to differentiate between a default value and null, you need to define the variable as a pointer type.

Let’s test a little bit more for those types described above.

type jsonTestDefault struct {
    IntValue  int            `json:"intValue,omitempty"`
    IntArray  []int          `json:"intArray,omitempty"`
    BoolValue bool           `json:"boolValue,omitempty"`
    MapValue  map[int]string `json:"mapValue,omitempty"`
}

obj := jsonTestDefault{
    IntValue:  0,
    IntArray:  []int{},
    BoolValue: false,
    MapValue:  map[int]string{},
}
jsonBytes, _ := json.MarshalIndent(obj, prefix, indent)
fmt.Println(string(jsonBytes))
// {}

The result is as expected.

obj := jsonTestDefault{
    IntValue:  -1,
    IntArray:  []int{0},
    BoolValue: true,
    MapValue:  map[int]string{0: ""},
}
jsonBytes, _ := json.MarshalIndent(obj, prefix, indent)
fmt.Println(string(jsonBytes))
// {
//         "intValue": -1,
//         "intArray": [
//                 0
//         ],
//         "boolValue": true,
//         "mapValue": {
//                 "0": ""
//         }
// }

When the value is set, all of them are set correctly.

How to ignore key

If there are some credentials in the struct, you might not want to add them to the JSON. In this case, define the name as dash “-“.

type jsonTestOther struct {
    AsIs    int    `json:""`
    Ignored string `json:"-"`
    Dash    string `json:"-,"`
}

obj := jsonTestOther{
    AsIs:    11,
    Ignored: "this is ignored",
    Dash:    "this is not ignored",
}
jsonBytes, _ := json.MarshalIndent(obj, prefix, indent)
fmt.Println(string(jsonBytes))
// {
//     "AsIs": 11,
//     "-": "this is not ignored"
// }

Note that the key is not ignored if you add a comma followed by a dash.

How to create nested json structure

You might already have the idea. You need to just define a struct where one of the keys has another struct.

type department struct {
    Name    string     `json:"name"`
    Members []employee `json:"members,omitempty`
}

type employee struct {
    Name          string "json:name"
    Age           int    `json:"age"`
    Gender        string `json:"gender"`
    Job           string `json:"role"`
    WithoutSchema string
}

Then, the rest is the same.

employees := []employee{
    {
        Name:   "Yuto",
        Age:    35,
        Gender: "Male",
        Job:    "Software Developer",
    },
    {
        Name:   "Daniel",
        Age:    40,
        Gender: "Male",
        Job:    "Software Tester",
    },
}

department := department{
    Name:    "Technical Feeder",
    Members: employees,
}
jsonBytes, _ := json.MarshalIndent(department, prefix, indent)
fmt.Println(string(jsonBytes))
// {
//     "name": "Technical Feeder",
//     "Members": [
//             {
//                     "Name": "Yuto",
//                     "age": 35,
//                     "gender": "Male",
//                     "role": "Software Developer",
//                     "WithoutSchema": ""
//             },
//             {
//                     "Name": "Daniel",
//                     "age": 40,
//                     "gender": "Male",
//                     "role": "Software Tester",
//                     "WithoutSchema": ""
//             }
//     ]
// }

How to Unmarshal without struct definition

There is a case where it’s impossible to define a struct in advance. How can we unmarshal the JSON data in this case? We can use map.

Let’s prepare a map object and the JSON data.

var objMap map[string]any

if err := json.Unmarshal([]byte(`{
    "name": "Yuto",
    "prop1": 12,
    "prop2": 2.2,
    "prop3": false,
    "prop4": null,
    "prop5": [1,2,3,"str"],
    "prop6": {
        "prop6-nested": 15
    }
    }`), &objMap); err != nil {
    fmt.Println(err)
    return
}

The JSON data contains all the possible data types. The key in the JSON data is assigned to the map key.

How to read literal values

The value for string, number, bool, and null can be easily read by specifying the key name in the map.

func showData(data any) {
    dataType := reflect.TypeOf(data).Kind().String()
    fmt.Printf("type: %s, value: %+v\n", dataType, data)
}

showData(objMap)                            // type: map, value: map[name:Yuto prop1:12 prop2:2.2 prop3:false prop4:<nil> prop5:[1 2 3 str] prop6:map[prop5-nested:15]]
showData(objMap["name"])                    // type: string, value: Yuto
showData(objMap["prop1"])                   // type: float64, value: 12
fmt.Println(objMap["prop1"] == 12)          // false
fmt.Println(objMap["prop1"] == float64(12)) // true
showData(objMap["prop2"])                   // type: float64, value: 2.2
showData(objMap["prop3"])                   // type: bool, value: false
fmt.Println(objMap["prop4"])                // <nil>

Note that all numbers are converted to float64. Both 12 and 2.2 are float64. Therefore, if it’s necessary to compare the value with int value, one of the values must be converted to the proper data type.

How to read an Array data

If the data type is Array, we need additional steps. It can’t be used in for with range keyword because the data type is any.

// cannot range over array (variable of type any)
for index, data := range array {}

It’s possible to read the whole data but not possible to read data with index array.

array := objMap["prop5"]
showData(array) // type: slice, value: [1 2 3,str]

// invalid operation: cannot index array (variable of type any)
showData(array[0])

If the specific data needs to be read, it has to be accessed in the following way.

valueOfArray := reflect.ValueOf(array)
showData(valueOfArray.Index(0).Interface())     // type: float64, value: 1
showData(valueOfArray.Index(0).Elem().Float())  // type: float64, value: 1
showData(valueOfArray.Index(1).Interface())     // type: float64, value: 2
showData(valueOfArray.Index(1).Elem().Float())  // type: float64, value: 2
showData(valueOfArray.Index(2).Interface())     // type: float64, value: 3
showData(valueOfArray.Index(2).Elem().Float())  // type: float64, value: 3
showData(valueOfArray.Index(3).Interface())     // type: string, value: str
showData(valueOfArray.Index(3).Elem().String()) // type: string, value: str

If you just need the value without the data type, use Interface(). If the value needs to be passed to a function that defines a concrete data type, the data should be read by Elem().Float() for example. It depends on the data which method should be used. If it is a string, use Elem().String() instead.

Check if the data is Slice and the length of the slice if Index() needs to be used. Otherwise, it panics.

fmt.Println(valueOfArray.Kind().String())         // slice
fmt.Println(valueOfArray.Kind() == reflect.Slice) // true
showData(valueOfArray.Len())                      // type: int, value: 4

If the array needs to be iterated over, use a switch case to check the data type. range can be used in the case statement.

func handleArrayAndMap(array any) {
    fmt.Println("--- handleArray start ---")
    switch v := array.(type) {
    case []any:
        for _, data := range v {
            showData(data)
        }
    case map[string]any:
        for _, data := range v {
            showData(data)
        }
    default:
        fmt.Println("not an array/map")
    }
    fmt.Println("--- handleArray end ---")
}

handleArrayAndMap(array)
// type: float64, value: 1
// type: float64, value: 2
// type: float64, value: 3
// type: string, value: str

The function can be used for map too.

How to read nested object data

If a JSON value is an object, it is mapped to map. prop6 is an object.

"prop6": {
    "prop6-nested": 15
}

The whole value can be easily read. If it’s necessary to iterate the items, we can write it in the same way as array. See the above about the detail for handleArrayAndMap.

nestedObj := objMap["prop6"]
showData(nestedObj)          // type: map, value: map[prop6-nested:15]
handleArrayAndMap(nestedObj) // key: prop6-nested, value: 15

If reflect.ValueOf() is used for the map value, it is converted to a struct. The value can be accessed via MapIndex() with reflect.ValueOf("key-name").

valueOfNestedObj := reflect.ValueOf(nestedObj)
showData(valueOfNestedObj)  // type: struct, value: map[prop6-nested:15]
showData(valueOfNestedObj.MapIndex(reflect.ValueOf("prop6-nested"))) // type: float64, value: 15

Another way to read the data is to cast it to map.

convertedMap, ok := nestedObj.(map[string]any)
if !ok {
    fmt.Println("conversion error for map")
} else {
    showData(convertedMap["prop6-nested"]) // type: float64, value: 15
}

The data can be read with the key name if the cast succeeds. Of course, it’s possible to loop the map in this way.