MessagePack for C# v3 Release with Source Generator Support

Last month, the MessagePack for C# project joined the .NET Foundation! I hope this will help users feel more confident about using the library with a stable perspective.

And now, after long development, the major version upgrade v3 has been released. While the core part remains mostly unchanged from v2, it fully incorporates Source Generator. Since IL dynamic generation still exists, it becomes a hybrid serializer with both IL dynamic generation and Source Generator. v3 comes with built-in Source Generator and Analyzer, and existing code will automatically be Source Generator-enabled just by compiling with v3. No additional code writing is required from users to support Source Generator when updating from v2 to v3!

Let’s look at the behavior in detail. For example, when you write code like:

[MessagePackObject]
public class MyTestClass
{
    [Key(0)]
    public int MyProperty { get; set; }
}

The following code is automatically generated internally by the Source Generator:

partial class GeneratedMessagePackResolver
{
    internal sealed class MyTestClassFormatter : IMessagePackFormatter<MyTestClass>
    {
        public void Serialize(ref MessagePackWriter writer, MyTestClass value, MessagePackSerializerOptions options)
        {
            if (value == null)
            {
                writer.WriteNil();
                return;
            }

            writer.WriteArrayHeader(1);
            writer.Write(value.MyProperty);
        }

        public MyTestClass Deserialize(ref MessagePackReader reader, MessagePackSerializerOptions options)
        {
            if (reader.TryReadNil())
            {
                return null;
            }

            options.Security.DepthStep(ref reader);
            var length = reader.ReadArrayHeader();
            var ____result = new MyTestClass();

            for (int i = 0; i < length; i++)
            {
                switch (i)
                {
                    case 0:
                        ____result.MyProperty = reader.ReadInt32();
                        break;
                    default:
                        reader.Skip();
                        break;
                }
            }

            reader.Depth--;
            return ____result;
        }
    }
}

Moreover, this GeneratedMessagePackResolver is already registered in the default options (like StandardResolver):

public static readonly IFormatterResolver[] DefaultResolvers = [
    BuiltinResolver.Instance,
    AttributeFormatterResolver.Instance,
    SourceGeneratedFormatterResolver.Instance, // here
    ImmutableCollection.ImmutableCollectionResolver.Instance,
    CompositeResolver.Create(ExpandoObjectFormatter.Instance),
    DynamicGenericResolver.Instance, // only enable for RuntimeFeature.IsDynamicCodeSupported
    DynamicUnionResolver.Instance];

Serialization target classes included in user code assemblies will prioritize using code generated by the Source Generator. GeneratedMessagePackResolver offers several customization points, such as changing the default namespace and names, or modifying generated formatters to be map-based. For more details, please check the new documentation. For those wanting to know the detailed changes from v2 to v3, please check the Migration Guide v2 -> v3.

For Unity, the installation method has significantly changed. The core library is now common with the .NET version and requires installation from NuGet. Additionally, you need to download Unity-specific additional code via UPM. For details, please check the MessagePack-CSharp#unity-support section.

The .unitypackage distribution has been discontinued. Also, mpc, which was required for IL2CPP support, is no longer needed. It has been completely migrated to Source Generator. Therefore, Unity support version starts from 2022.3.12f1. Regarding Source Generator, it is automatically enabled when installing the core library via NuGetForUnity, so no additional work is required.

History and Next

The original MessagePack for C# (v1) was released by me (Yoshifumi Kawai/@neuecc) in 2017. I created it as a performance-focused binary serializer because the existing (binary) serializers in 2016 couldn’t meet the performance requirements for solving issues in the game I was developing at the time. Along with it, I also released MagicOnion, a gRPC-based RPC framework created as a network system.

While v1 release only targeted byte[], .NET kept adding new I/O-related APIs like Span<T> and IBufferWriter<T>, so v2 introduced a new design focusing on these. This implementation was led and released by Microsoft Engineer Andrew Arnott / @AArnott.

Since then, it has continued under joint maintenance and moved from my personal repository (neuecc/MessagePack-CSharp) to an organization (MessagePack-CSharp/MessagePack-CSharp). It’s used in major Microsoft products like Visual Studio 2022, SignalR’s binary protocol, and Blazor Server protocol, and has gathered the most stars on GitHub among .NET binary serializers. It’s also recommended as one of the migration targets for BinaryFormatter, which is being deprecated in .NET 9.

With v3’s Source Generator support, we’ve taken the first step toward higher performance, flexibility, and AOT compatibility.

While I consider the MessagePack for C# project a great success, AArnott is currently starting development on his own new MessagePack project. During this time, I’ve also released MemoryPack, a serializer with a different format. Therefore, I think it’s necessary to explain somewhat about the future of MessagePack for C# and its characteristics.

I believe the maintenance system will continue with two people, but regarding active development, I might take the lead again. I operate with the understanding that MessagePack and MemoryPack have different characteristics as formats, and both are important. I like the original implementation of MessagePack for C#, and I think it’s still absolutely competitive even today.

AArnott’s different MessagePack serializer has slightly different fundamental philosophy. In that regard, I recognize it not as an improved serializer but as one with a different personality. Let me explain the differences.

Binary spec, default settings and performance

What’s important for serializer performance is both “specification and implementation.” For example, binary formats are generally faster than text formats like JSON. However, a well-implemented JSON serializer is faster than a mediocre binary serializer (I’ve demonstrated this by creating a serializer called Utf8Json). So, both specification and implementation are important. If you can achieve both, that becomes the best-performing serializer.

MessagePack’s binary specification is expressed as a binary version of JSON, as its motto “It’s like JSON, but fast and small” suggests. However, MessagePack for C#’s default doesn’t necessarily aim to be JSON-like.

[MessagePackObject]
public class MsgPackSchema
{
    [Key(0)]
    public bool Compact { get; set; }
    [Key(1)]
    public int Schema { get; set; }
}

When this class is serialized, it would be expressed in JSON as [true, 0]. This is because the object is serialized array-based, whereas if serialized map-based, it would be expressed as {"Compact":true,"Schema":0}.

The advantage of array-based serialization is, as you can see, it becomes more compact in binary size. Compact size means less processing, which positively affects serialization speed. Also, for deserialization, since there’s no need to search for properties to deserialize by comparing strings, faster deserialization speed can be expected.

Note that array-based serialization is also adopted by msgpack-java, the reference implementation by MessagePack specification creator Sadayuki Furuhashi, so it’s not an unorthodox approach.

In MessagePack-CSharp, if you want to serialize in a JSON-like map-based format, you can write [MessagePackObject(true)]. Also, with Source Generator, you can override at the Resolver level to force map-based serialization.

[MessagePackObject(keyAsPropertyName: true)]
public class MsgPackSchema
{
    public bool Compact { get; set; }
    public int Schema { get; set; }
}

The advantages of maps are enabling flexible schema evolution, easier communication when interfacing with other languages, and higher self-descriptiveness of the binary itself. The disadvantages are the impact on size and performance, especially in arrays of objects where property names are included for each element, which becomes quite wasteful.

The default is set to array for pursuing compactness and performance. I considered MessagePack as a binary specification capable of achieving high performance before being JSON-like. Of course, maps are important too, so I made it possible to easily achieve map mode by just adding (true) to the attribute.

In array mode, you need to attach the Key attribute to all properties. This is necessary, just as Protocol Buffers requires numeric tags, when you’re not using the property name itself as the key. Of course, automatic numbering in sequence is possible, but I’ve determined that implicit handling of binary format keys is too risky (binary compatibility would break just by manipulating the order). In other words, explicit is the default. In large project development, both senior and junior members will touch the code; not everyone touching the code understands everything. So, implicit behavior should be avoided, and things should be explicit — this strong conviction led to this design choice.

However, attaching Keys to all properties is very painful (I had painful experiences with DataContract and protobuf-net before developing MessagePack-CSharp). So, we provided a feature to automatically attach them through Analyzer + Code Fix. This alleviates the pain of being explicit while getting the best of both worlds.

The other MessagePack serializer’s default appears to be map-based. This is partly because it’s based on PolyType, an abstraction library for creating Source Generator-based libraries, and partly because it seems to be an explicit preference for that approach.

A library can only choose one “default.” Even if it can process in either mode, there can only be one “default.” To reiterate, I prefer and prioritize “compactness and performance” as a binary format.

You might be hearing about PolyType for the first time. I’m not very favorable towards PolyType. While I think it’s very convenient for creating small things, I believe its limitations as an abstraction layer are too significant when aiming for best performance or expressing the best ideas. Therefore, I won’t adopt it in MessagePack for C# or in creating anything else.

Unity(multiplatform) Support

MessagePack for C# has provided first-class support for the Unity game engine since v1. This is partly because I serve as CEO of Cysharp, an affiliated company of Cygames, a Japanese game company, and have deep connections with the video game industry. We’ve actually created and used things that run on Unity ourselves. Of course, we also use it for server-side and desktop applications.

Unity has its own AOT system called IL2CPP, which is essential especially for releasing on mobile platforms like iOS. Even before Source Generator existed, we created and provided mpc, a code generation tool using Roslyn. It’s no exaggeration to say that MessagePack being used in hundreds of mobile games is thanks to my passionate support. With v3 finally becoming Source Generator-based, the workflow will be greatly simplified!

Generally, Unity support has been quite undervalued in the .NET community. Also, from an outside perspective, Microsoft and Microsoft employees seem to share this attitude, with little interest in platforms other than their own. I don’t think this attitude is very favorable, and it’s also limiting the potential of .NET. I think Xamarin’s failure to achieve growth trajectory was also partly due to such cold regard from Microsoft itself.

I take care to ensure that the libraries I create can properly support Unity as much as possible (the latest being Cysharp/R3, a new Reactive Extensions library). As for the other MessagePack serializer, it doesn’t seem likely to have solid Unity support…

Beyond v3

v3’s Native AOT Support is not complete. It’s challenging that just making it Source Generator-based doesn’t result in complete Native AOT support. This is honestly perplexing given that it works perfectly with Unity’s AOT, IL2CPP, and I think it also shows Microsoft’s not-so-good habits. In other words, they’re providing something complex to achieve perfect support. That’s the current Native AOT. While I can understand some aspects of the complex and bizarre attributes and flows, I think they should have been simplified more. Well, it probably won’t be fixed anymore…

In terms of performance, there are also points that regressed from v1 to v2, so we need to make implementation improvements based on the latest insights. I’m particularly dissatisfied with how the wide use of ReadOnlySequence creates significant constraints.

Better asynchronous APIs due to the standardization of PipeReader/PipeWriter in .NET 9, and streaming support that achieves both performance might also become major topics.

Because MessagePack for C# is widely used, breaking changes are difficult to make, and maintaining compatibility is the most important topic. However, as the world changes, choosing not to evolve is choosing the path to extinction. I think there’s still a lot we can do, so I want to continue being the cutting-edge, best binary serializer in .NET (MemoryPack too…!)

First, please try v3’s Source Generator. I think one of the good things about OSS is that we can create better things with everyone’s power.