A Tale of Converter Exceptions and Intercepted Hopes and Dreams 😮‍💨

Pesky Data Drift

On a system that I work on, we have a nice mechanism for modelling data that changes frequently or can have complex shapes.

Using postgresql and dotnet: we utilise jsonb columns, and in the application layer we deserialise that data, based on an embedded discriminator, into objects.

This works nicely, and makes working with this data mostly transparent.

[!NOTE] Other than having to set the RowState to EntityState.Modified manually when only updating the internals of the jsonb.

…looking at you Entity Framework 🤨

However! Sometimes data shapes drift, and this can cause issues when older data gets loaded.

For example, if a field was changed at some point from a string to a boolean, and not all of the existing data in the database was cleaned up, then our very happy jsonb deserialisation suddenly becomes 😰.

Exceptions start being thrown around. And before you know it things are getting hairy, no one’s sure exactly where the issues are coming from, and pasting the exception into ChatGPT seems to cause the LLM to go in circles.

On Circuitous LLM Conversations

We use entity framework converters in order to serialise and deserialise our jsonb data.

Something like this:

1// Only the finest value comparers, quite, quite 🧐
2var valueComparer = CreateAFunValueComparer<PropertyType>();  
3  
4return propertyBuilder.HasConversion(  
5    (value) => value != null ? JsonSerializer.Serialize(value, OurFunJsonSerializer<PropertyType>.JsonSerializerOptions) : "",  
6    (json) => DeserializeOurFunSerializedJson<PropertyType>(json),  
7    valueComparer);

So, in my infinite naivete, I thought to myself: “There must be a straight forward way to get the table name and id when deserialising fails, so that I can log which row failed to point developers towards the source of the error”.

Discussing this with a popular LLM confirmed my suspicions. I can add a materialisation interceptor that would intercept errors with the converter, and give me access to the Entity that was being deserialised, so that I could log its name and id to aid in debugging.

Oh, younger Barry, how naive you were.

Upon reading more about converters¹², I became suspicious that the LLM had given me… lets call it… less than reliable information 😜.

In a huff, I confronted the LLM with the fact that entity context is not available in value converters (at least not yet).³

Of course it responded “Yes, you’re absolutely right! I was wrong about using interceptors”.

Restarting the conversation from scratch played out the same rigamarole, where the LLM would suggest using interceptors, I would tell it that wouldn’t work, it would praise me and agree, and after a few more turns of conversation it would recommend using interceptors again.

Intercepting the Suggestion

After more research and turns with the LLM, I came to the following conclusion:

• In order to catch an error with a materialisation interceptor, the conversion itself also has to occur in the interceptor
  • This isn’t outside of the realm of possibility, but would go against the design intent of entity framework (which is to use converters for converting property values)
• If we’re sticking with the converter, then options include:
  • Pass in an entity type when defining the conversion
    • Not a bad idea, but I don’t like the boilerplate and it opens up room for human error
  • Wrap deserialisation converter with an error handler that:
    • Looks for a discriminator property in the json, and logs it if it exists
    • Rethrows the exception (wouldn’t want to swallow errors we should know about 🤢)

In the end I went with the last option, adding a last ditch attempt to log out the discriminator for the failing conversion, and then rethrowing the error.

References