GSON to Moshi migration – the good, the bad and the deserializer

We at Wolt have recently migrated the default JSON parser in our Android application from GSON to Moshi. Read more as Khiem “Kimbo” Hang, our Android developer, recalls the fun but challenging undertaking.

Introduction: The problem with GSON

On a late morning on a beautiful April’s day, a ticket appeared in the Consumer Mobile Team Android kanban board. It read simply “Add strict data validation”, and requested that any JSON parsing error should crash the Debug version of our application. This was part of an ongoing effort to further stabilise the app, since some part of the legacy Python backend that it was communicating with didn’t have strict type checking and that led to unexpected results.

What a sensible ticket. It shouldn’t be too hard to implement and get immediate results. Or so we thought when we first saw the ticket. Unfortunately, this was one of those times reality didn’t match our expectations. GSON, the JSON parser that was chosen 5 years ago when the app was built, didn’t have validation capabilities unless we were to implement custom deserializers for each network type we have. Not a scalable solution.

In fact, some of the legacy network models were definitely written 5 years ago. They were in Java (the horror!), so there was actually no nullability for any fields in those classes. How were we supposed to know if a field is null or not, if both the client and the server had no concrete idea?

What’s more, upon further investigation, we found out that GSON was using the unsafe allocateInstance() method to initialize all the network classes. This implies that Kotlin constructor default values and nullability signatures have always been ignored. There were also complaints from the websocket guy that GSON didn’t come with a polymorphic type adapter that worked well with Retrofit. He had to download some “sketchy” RuntimeTypeAdapterFactory or make his own TypeAdapter. One could implement his own type adapter to solve this problem, but `TypeAdapterFactory` was quite verbose in GSON, and `JsonDeserializer` was not recommended in the GSON documentation.

And thus, we unanimously decided that it was time for us to replace our beloved JSON Parser with something new, something more kotlin compatible, something easier to work with, something that would open the pathway to migrate all the Java classes to Kotlin. Moshi entered the picture. 

“What we didn’t realise is how big an undertaking it would be to migrate to Moshi. This blog post is the account of the fun, wild, heart-aching, brain-shattering experience we had.”

(*) RuntimeTypeAdapterFactory or make his own TypeAdapter. One could implement his own type adapter to solve this problem, but `TypeAdapterFactory` was quite verbose in GSON, and `JsonDeserializer` was not recommended in the GSON documentation.

The journey: Initial migration.

A wise man once said: “Never break user space” and at Wolt, we take that to heart. One of the criteria of the migration was that it would not result in any visible change to the application. What that meant in practice was that we needed to have a transition phase between the two JSON parsers through feature flagging. When we would be confident enough that Moshi worked without a hitch, we could remove GSON out of our project.

What that also meant was, for a fleeting moment, we had to support the two JSON parsers side by side. This was the first challenge we encountered in our journey.

If you are familiar with GSON, you’ll find this annotation familiar. In short, this is used by retrofit to deserialize a JSON field to a Java / Kotlin field.

1
@SerializedName("title") val title1: String?

And if you are on the Moshi side of thing, this is how it is done:

1
@Json(name = "title") val title1: String?

What we wanted was the combination of both:

1
2
@Json(name = "title")
@SerializedName("title") val title1: String?, 

And we want it for every single value that is annotated in our network models. Like any medium large application, we have hundreds of network models, which would take days just to replace the field one by one. We were quite happy, because this was the first time we had an actual use case for Intellij’s powerful regex replacer. 

1
2
From: @SerializedName\(\"(.*?)\"\)
To: @SerializedName(name = "$1") @SerializedName\("$1"\)

A similar operation was done for the @JsonClass annotation but without the hassle of the regex.

And that took care of the Kotlin side of things. We’d still need to figure out what was what with the Java classes. As mentioned before, none of us in the team remembered or knew anymore what field should be null or not, since both Java and Python were built without this information. We were stumped, perhaps this migration was doomed before we even started.

But then, our lovely friends in the iOS department extended their hands. “We have some nullability information here”. Although the legacy Objective-C has always been daunting for us in the Android team, it did contain some of the most useful information, but not all. We also then journeyed to the depth of the TypeScript data model where we were finally 90% confident that the model should work. The last 10% could be picked up through rigorous testing.

With this, the initial migration was done, the app should build, the code should work and we should be able to order donuts to celebrate. The run button was pressed, fingers were crossed, and the build failed.

Moshi Gson shenanigans

Apparently Moshi had no idea how to deserialize this one JsonObject class. But of course it didn’t, since JsonObject is a GSON class. Apparently, the lack of a polymorphic type adapter led to some, let’s say interesting, manipulation from our side. For classes that need to deserialize polymorphically, the object was returned as a JsonObject, deserialized once to get the correct type, and then deserialized again to get the final typed object.

So for the time being, we needed to get Moshi to decode the JSON fields to a JsonObject too for backward compatibility.  Fortunately, in Moshi, it’s quite easy to create a type adapter for such a use case. We can easily register as well as deregister this in Moshi creation whenever we have finished with GSON.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 class MoshiGsonObjectAdapter(
      val gson: Gson
) {

  @ToJson
  fun toJson(jsonObject: JsonObject): String {
      // We don't use this method anyway
      return jsonObject.toString()
  }

  @FromJson
  fun fromJson(reader: JsonReader): JsonObject? {
      val string = reader.nextSource().use(BufferedSource::readUtf8)
      return gson.fromJson(string, JsonObject::class.java)
  }

} 

Unit tests were  also quickly written to confirm that the adapter was working. We added the adapter to the Moshi Builder, and everything worked as expected. Things were getting converted and we were ready to try out Moshi in practice. Moshi was enabled for internal usage.

Actually fixing the problem: Moshi Polymorphism

At the previous stage our code base started working, it compiles, runs, and orders can be made. But of course any software engineer with a good conscience cannot stop here. Not while JsonObject is still at large.

Before we dive into how we can fix the symptom, let’s take a look at what actually causes it. Let’s say we want to decode this list of objects:

1
2
3
4
5
6
7
8
9
10
[
   {
      "type":"dog",
      "bark":"woof"
   },
   {
      "type":"cat",
      "cry":"meow"
   }
]

And we want to decode them into the following classes:

1
2
3
4
class Animal(val type) {
    class Dog(val type, val bark: String)
    class Cat(val type, val cry: String)
}

What we used to do for GSON was to ask Retrofit for the list to be a List<JsonObject>, convert each item of that list into the Animal class through Gson.fromJson. Then based on the type field we call Gson.fromJson again, this time with the correct type. So we ended up doing the conversion twice. With Moshi converting the list to a JsonObject in the MoshiGsonObjectAdapter that would be 3x conversion.

Suffice to say, this was not the most performant conversion. Hence, we decided to eliminate the JsonObject in-between-state and convert directly from retrofit to classes. Fortunately in Moshi, there is a ready made solution for that: Moshi Polymorphic adapter.

With the Moshi polymorphic adapter, the previous conversion of the Dog and Cat type can be simplified into these short lines of code:

1
2
3
4
5
6
7
var moshi = Moshi.Builder()
  .add(
      PolymorphicJsonAdapterFactory.of(Animal::class.java, "")
          .withSubtype(Cat::class.java, "cat")
          .withSubtype(Dog::class.java, "dog")
  .add(KotlinJsonAdapterFactory())
  .build()

Was it worth it: The benefit

The most crucial benefit of the migration revealed itself to us almost immediately. Our test app started crashing at random places. This was due to certain fields that were supposed to be non-nullable, but were actually null in the data the backend sends. Many Pull Requests later, the most glaring NullPointerExceptions were fixed. Fixing these issues was exactly why we set out to use Moshi in the first place.

Another benefit of the migration is to finally convert the few notoriously hard-to-convert Java classes to Kotlin. Though part of the work – converting iOS and Web network models back to Kotlin – doesn’t involve too much Moshi, having Moshi to catch the exception that the conversion may produce provides great peace of mind. Finally, after 4 years of effort converting to Kotlin, we are now down to 6 utils classes in Java, and 0 classes in network models.

There are a lot of theoretical benchmarks online detailing how much faster Moshi is compared to GSON. However, for our case, it didn’t make too much of a difference. Our app startup, which relies on communicating with the server, feels the same. Our internal analytics reported that it took 300ms less to start up. The bottleneck here is most likely the network latency rather than the speed of parsing a few hundred lines of JSON. That being said, we are happy with any performance improvements we could get in the application.

What we did notice though, was the streamlined API that Moshi provided.

“Moshi was written with Kotlin in mind and that definitely shines when compared to GSON. “

Polymorphic Adapter, or even a simple Type adapter, is much easier to use and to write tests for. This gives us an opportunity to further improve the code quality of our projects. 

Conclusion

Converting from GSON to Moshi has been a fun undertaking. It was not easy, and we encountered one problem after another as discussed here. But it was very much worth the time and effort we spent, improving not only the code quality but also the workflow in our network layer.

To keep the blog post short, we didn’t detail all the issues with the Moshi migration, but for the sake of completeness, they are included here:

• Encoding problem with field variables in Moshi

• Moshi usage in websocket

• HashMap and ArrayList incompatibility

• Moshi reflection vs Moshi code gen

Finally, a few more resources are included for those who want to give Moshi a try as well.

• https://proandroiddev.com/goodbye-gson-hello-moshi-4e591116231e – Good Moshi and GSON comparision

• https://gist.github.com/ChainsDD/07c6b52837a2fa0622c93df07a92f48d – Moshi custom type adapter API

• https://proandroiddev.com/moshi-polymorphic-adapter-is-d25deebbd7c5 – Moshi polymorphic type adapter

• https://zacsweers.github.io/json-serialization-benchmarking/ – Some benchmark for Moshi and other json parser that you can run at home 

Want to join Kimbo and build great things at Wolt? 👋 Check all of our engineering roles!