Bsrgvty

Problem

Say we have a C# class with is serialized to JSON (currently via Newtonsoft's JSON.Net) and stored in a database:

public class User

 public string authInfo;

If the class definition changes, the old data will fail to load. Even if we try to update the database by hand, data may be lost unless we have server downtime during conversion.

public class User

 public string username;
 public string token;

Solution (my attempt)

We may use a callback which is run after deserialization that converts the old data to the new data format. (The attribute and parameters need to be adapted based on which serialization framework is being used.):

public class User

 public string username;
 public string token;
 [Obsolete] public string authInfo;

 [OnDeserialized]
 public void FixData()
 
 if (username == null)
 
 var parts = authInfo.Split("/");
 username = parts[0];
 token = parts[1];
 authInfo = null;
 
 

If a field's format needs to change from a list to an object (or number) or vice versa, the newer field should be called authInfo_2, and incremented when the type changes again.
If a field's format needs to change from a list of one type to a list of another type, a new field must also be created.

public class User

 [Obsolete] public List<string> address;
 public List<AddressLine> address_2;
 // FixData() will convert from address to address_2

Problem: If null is a valid value for the old or new data, we can't determine whether the data has been migrated to the newer format. The following is a workaround that will track whether new data has been added:

public class User

 [Obsolete] public List<string> name; // serialized old data
 private string _familyName; // serialized
 private bool _isFamilyNameSet; // serialized
 public string familyName get return _familyName; set _familyName = value; _isFamilyNameSet = true; // not serialized
 // FixData() will convert from name to familyName

Question

This procedure is a bunch of rules I made up, and I've probably missed something important. Is there an accepted best practice that deals with versioning in serialized data? (Including a version number seems like it would lead to a lot of problems.)

Problems

Generally speaking, handling different versions of the same data model in the same code results in extra unwanted complexity. Some common issues include:

• Fields renamed


• Data types changed


• Old fields removed


• New fields added


• Existing data refactored into multiple fields


• Existing data combined together into a single field


• Semantics of existing fields redefined

None of these are things which you want to have creeping in to your core/domain logic if you can at all help it.

Furthermore, if you have other version changes planned in the future, then by holding on to old formats, you're potentially looking at an explosion in complexity once you've been through multiple evolutions of the data format.

Ideally, migrate all old data into the new Format and disband the old format entirely

The ideal way to handle this scenario is to make sure that your domain logic is never bothered by different data formats in the first place. Every time you add a new format, complexity increases, but by migrating data it can be a 'one off' operation.

When performing Data Migration, it's important to create a 'rollback' path - i.e. put appropriate backup/restore procedures in place so that you can prevent data loss if anything goes wrong during the migration.

Also ensure that you have appropriate sanity checks and data verification in-place to make sure the data is in a good state following the migration.

Of course, this is not always an option. Multiple data versions are sometimes an unavoidable, necessary evil.

If Migration is not an option, separate your Persistence format away from your Domain Models

The logic would be the somewhat similar to the migration code, except it would occur at run-time instead, and the 'migration logic' would be sticking around long-term until the data is either fully migrated or retired, and extra care is needed to decouple it from the rest of the application.

Any concerns regarding different versions or variations in the shape of the same data stored in different formats within your database should be handled in one place away from the rest of your code; hidden behind a standard Data Layer interface which contains everything that the rest of the logic needs. This can minimise the complexity and impact of storing multiple data formats in your database.

Avoid exposing multiple different formats to your core logic wherever possible The rest of your code should be agnostic to the actual shape or format of your persistent data.

Internally to your data layer, keep different 'model' structures which you can use to deserialise into with JSON.NET. Have a look at AutoMapper for switching between your 'persistence' models and the domain model -- don't use these JSON serialiser models anywhere in your core logic because they represent knowledge of your persistence format.

Some form of versioning will be necessary for this - your repository/serialiser will need to know which internal JSON model format to deserialise into, so you'd probably need to store a version number within the database alongside the serialised data, or otherwise have some way of unambiguously distinguishing between different data versions.

Avoid using a boolean field to switch between your versions -- if your way of distinguishing data formats ends up being a true/false value such as "isNewVersion" then that'll be a problem if you ever happen to introduce version 3 in the future.

For example:

internal class MyDataModelVersion1 /* Old JSON Persistence Format POCO */ 

internal class MyDataModelVersion2 /* New JSON Persistence Format POCO */ 

public class MyStandardModel /* Common/Domain Model */ 

public class MyRepository 

 public MyStandardModel GetData(int id) 
 
 var row = ReadFromDatabase(id);
 MyStandardModel model = null;

 if (row.Version == 1)
 
 var data = Json.DeserializeObject<MyDataModelVersion1>(row.Json);
 model = Mapper.Map<MyStandardModel>(data);
 
 else if (row.Version == 2)
 
 var data = Json.DeserializeObject<MyDataModelVersion2>(row.Json);
 model = Mapper.Map<MyStandardModel>(data);
 
 else /* throw exception */ 

 return model;
 

The main reason for this approach is to ensure that the only part of your code which needs to change when you introduce a new data shape is in the Repository/Data layer -- the rest of your code shouldn't need to care.

It's still less ideal than migration but it encapsulates the data version switching into one place and avoids polluting your core logic.

I would avoid having the new class know about the old class.

If the class name changes you can have

OldRepository

 public List<OldUser> GetAll()


Converter

 public NewUser Convert(OldUser)


NewRepository

 public void Add(NewUser)

You can then convert the whole DB to the new format with a script, or do on the fly conversion without having a dependency on the old class in the new class.

Generally if you have to store serialised data in a DB like this, rather than splitting out the fields you should include some sort of data versioning, to allow you to know what version of the data is stored in a particular row.

As @Hans-Martin says below. Having multiple data versions hanging around for a long time can cause unforeseen issues. If you can do a clean break and upgrade all the data to the new structure thats a good thing.

The main problem is in handling the change over with zero downtime.