blog

A .NET coding puzzle: Can strings change?

The holiday season is upon us, and what better way to celebrate than with a festive coding challenge?
As part of this year’s “C# Advent” event, I’m sharing a tricky little puzzle that combines holiday cheer with some .NET and C# challenge. If you are a seasoned .NET developer and enjoy solving unique challenges, you might enjoy this one.

Disclaimer
This coding challenge is designed purely for fun and to explore some of .NET’s internal mechanics. It showcases advanced techniques that can be harmful if not handled carefully and you probably should never use this example in real-world production code.

The Challenge

In the spirit of the season, let’s take a look at the problem:

void Main()
{
    Initialize();

    string happyHolidays = "Merry Christmas";
 
    System.Console.WriteLine(happyHolidays); // should output: Advent of C#

    static void Initialize()
    {
        // Your implementation here
    }
}

Your task is to modify the “Initialize” method so that the program outputs exactly: Advent of C#

Here are the constraints:

  • You are only allowed to modify the implementation of the Initialize method.
  • You cannot change the method name, parameters, or signature.
  • You cannot alter any other parts of the program.
  • You cannot use System.Console.WriteLine (or similar) in the Initialize method.
  • You cannot change the default output stream.

This is one of my favorite challenges, it requires some out-of-the-box thinking and a deep knowledge in how Strings works in the .NET runtime.
Before diving into the solution, give yourself a minute to try to solve it!

Solution Breakdown

When solving this challenge, we need to remember that the primary task is to modify the Initialize method in a way that allows the happyHolidays variable to reflect the string “Advent of C#” instead of “Merry Christmas.” Since we’re restricted to modifying only the Initialize method, the solution lies in:

  1. Exploring how strings work in C#: Strings in .NET are immutable, which means they cannot be changed after they are created. However, their references can be manipulated under certain conditions.
  2. Using an internal implementation detail of the runtime and the C# compiler: Since the “Initialize” method is static we can’t directly access the “happyHolidays” variable, therefore we have to be clever and use knowledge of how both the C# compiler and the .NET runtime treat strings.

Solution

There are multiple ways to code the solution, some of which are more elegant that others, but all solutions uses the same tricks:

  1. The string “Merry Christmas” is defined as a constant, and the compiler optimizes for efficiency by eliminating duplicate string constants. To achieve this, it ensures that each constant string is interned.
    In .NET, the runtime uses a String Intern Pool, which acts as a cache to store a single instance of each string. This allows multiple references to point to the same string, saving memory. Since strings are immutable, this sharing doesn’t cause any issues.
    While strings are not automatically added to the intern pool at runtime and requires an explicit call to “String.Intern”, constant strings are interned by default by the compiler.
  2. In .NET, strings are immutable, but this immutability is enforced by the provided APIs. While there are no official public APIs to modify the value of a string instance, this doesn’t mean a string’s value is entirely unchangeable. Using C# features like unsafe code and pointers, it is possible to directly access and modify the string’s memory, bypassing the usual immutability constraints.
  3. Since “Merry Christmas” is longer than “Advent of C#”, we can reuse the existing memory allocated for “Merry Christmas” without needing to allocate new memory. Making a string shorter is easier than making it longer.

Solution #1 – not the most elegant solution

static void Initialize()
{
    string adventOfCsharp= "Advent of C#";
    string happyHolidays = "Merry Christmas";

    unsafe
    {
        fixed (char* happyHolidaysPointer = happyHolidays)
        {
            for (int i = 0; i < adventOfCsharp.Length; i++)
            {
                happyHolidaysPointer[i] = adventOfCsharp[i];
            }
        }
    }

    typeof(string)
        .GetField("_stringLength", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic)
        .SetValue(happyHolidays, adventOfCsharp.Length);
}

Solution explanation

This solution works by directly modifying the memory and metadata of the interned string “Merry Christmas”, replacing its content with “Advent of C#”.

  1. String Memory Modification:
    • The unsafe block and the fixed keyword pin the memory of the happyHolidays string, allowing direct pointer access.
    • A loop overwrites the characters in happyHolidays with the corresponding characters from adventOfCsharp
  2. Updating String Metadata:
      • The typeof(string).GetField() retrieves the private _stringLength field, which stores the length of the string.
      • Using reflection, the length of happyHolidays is updated to match the new content, ensuring that subsequent operations recognize the string’s new length.

     

Solution #2 – the elegant solution

static void Initialize()
{ 
    var source = "Merry Christmas";
    var target = "Advent of C#";
    ref var sourceRef = ref MemoryMarshal.GetReference(source.AsSpan());
    var sourceSpan = MemoryMarshal.CreateSpan(ref sourceRef, source.Length);
    target.AsSpan().CopyTo(sourceSpan);
    Unsafe.Subtract(ref Unsafe.As<char, int>(ref sourceRef), 1) = target.Length;
}

Solution explanation

This solution is safer since it doesn’t use “unsafe” and direct pointer access, instead it works by using the MemoryMarshal which is less error-prone than raw pointer manipulation.

  1. Memory Modification:
    • Uses MemoryMarshal.GetReference to retrieve a reference to the string’s memory.
    • Creates a mutable Span from the reference, allowing efficient memory modification without explicit pointer arithmetic.
  2. Length Update:
    • Uses Unsafe.As<char, int> and Unsafe.Subtract to directly modify the string’s length, as the length stored in memory just before the start of the string’s content
    • Avoids reflection entirely, making the solution faster and more resilient to changes in .NET internals.

Important
Although this solution seem safer since it avoids direct pointer manipulation and pointer arithmetic, using these APIs can still performs operations that violate the runtime’s assumptions, making it inherently risky despite being arguably more elegant.
Both solutions are not really safe to use if you are not careful, and depending on the situation, it is easy to make mistakes and introduce hard to track bugs or even interrupt with GC/JIT optimizations.
Use both of these APIs in the scenarios they were intended to be used and by being very careful.

Key Takeaways

This challenge highlights the importance of understanding how strings and memory work under the hood. While direct memory manipulation is rarely needed in day-to-day programming, it’s a valuable tool for understanding how things work behind the scenes.

And most importantly, this challenge showcases advanced .NET techniques for learning purposes only. You probably should not use these methods in production code 🙂

Boost your productivity with Visual Studio’s DebuggerAttributes

Visual Studio is a great IDE with a great Debugger. It currently provides one of the best debugging experiences out there. The extensibility and the customizability of Visual Studio make it even better. For example, it provides Attributes for customizing the display of objects during debugging sessions. By utilizing these attributes we can decrease debugging time.

Assuming we need to examine a collection of objects during a debugging session when the objects are of type Scientist. By hovering over the collection, Visual Studio will list the objects using DataTips in the following manner:
List of objects

These DataTips aren’t much help since we can’t see any information about the objects without expanding each one of them. This could increase debugging time if the collection is long.

One way to improve these DataTips is by overriding the ToString method of the Scientist class. For the following implementation of ToString:

public override string ToString() 
    => $"Name: {Name}, Birthday: {Birthday}";

This is how the collection will be displayed:

List of objects with ToString implementation

This is much better, but this isn’t always a viable option. For example, when we don’t want to override the ToString method only for debug purposes or when we completely can’t, for instance, when the class is defined in a third-party assembly.

Fortunately, Visual Studio provides ways to customize how objects are displayed during debug sessions by using Debug Attributes.

With DebuggerDisplayAttribute we can define how objects are displayed when hovering over a variable. The Attribute can be set either on the class level (Above a class) or on an assembly level for when we can’t modify the class.
The attributes expect a string similar to C#’s 6 String interpolation format and an optional TargetType that is only necessary when we define the attribute on an assembly level.

For our example, we can define the attribute directly over the class:
DebuggerDisplayAttribute on a class level

Or on an assembly level:
DebuggerDisplayAttribute on an assembly level

Note: Instead of using the Target parameter which accepts a Type, we could use TargetName which accepts a fully qualified name of a class. This is handy when the attribute is applied on an assembly level that don't reference the assembly defining the Type.

In both cases, the result will be the same:
List of objects with a DebuggerDisplay attribute

Defining attributes for each type we might encounter while debugging isn’t too hard, but sometimes it might harm productivity. You’ll have to stop a debug session just for placing a new attribute or for changing how DebuggerDisplayAttribute displays a type. In addition, this requires a code modification that you’ll have to decide whether to commit to your source control or not.

Fortunately, OzCode has a neat feature called Reveal that solves this problem. It can change how objects are displayed at runtime without interrupting the debug session and without modifying your code. Just press on the ‘Star’ icon of each property you want to display.
List of objects with OzCode's Reveal feature

Now, what about how each object is displayed when expanding it? By default Visual Studio will display all of its properties and fields:
How objects are displayed by default when expanded

This might be ok in most of the times, but sometimes either the object has too many fields or it might be missing fields or properties that might be helpful for debugging.

Luckily, Visual Studio provides a way to change completely how each object is displayed when it is expanded. This is done by using the DebuggerTypeProxyAttribute with a class that we will write to expose only the necessary properties and fields.

Assuming the Awards of each scientist aren’t necessary for our current debugging session and assuming we usually need to calculate how many years has passed since the current scientist was born, we implement the following class:

class ScientistTypeProxy
{
    private readonly Scientist _scientist;

    public ScientistTypeProxy(Scientist scientist)
    {
        _scientist = scientist;
    }

    public string Name => _scientist.Name;
    public string[] Fields => _scientist.Fields;
    public int YearsPassed => _scientist.Birthday.YearsPassedSince();
}

We can apply the DebuggerTypeProxyAttribute directly over the Scientist class or on the assembly level, similar to the DebuggerDisplayAttribute.
DebuggerTypeProxyAttribute Class Level

Or by the class Level:
DebuggerTypeProxy on the Assembly Level

This also might interrupt the debugging session and require code modifications. Fortunately, OzCode’s Custom Expression to the rescue! It provides the ability to add custom properties to be displayed when expanding an object during debugging. These Custom Expression’s won’t modify the actual classes, so no code modifications are necessary.
OzCode's Custom Expression Feature

This is already better, but we still need to expand the array in order to see the values, or at least add a property to the DebuggerTypeProxy’s class or a Custom Expression using Ozcode to aggregate the array to a string. This is acceptable, but Visual Studio provides the DebuggerBrowsableAttribute specifically for that. Using this attribute, we can hide members of a class or even expand arrays by default using the RootHidden argument.

By applying the following attribute with RootHidden over the Fields member in ScientistTypeProxy
DebuggerBrowsable over Fields with RootHidden

We receive the following result:
Result of DebuggerBrowsableAttribute with RootHidden over Fields

This is even better but we can go even further. For complex data, like collections, values for plotting graphs or for visualizing any complex model; properties and fields might not be the best way to debug the values of an object.
With the DebuggerVisualizerAttribute we can get total control on how objects are displayed. Actually, using a WinForms/WPF window, we can even plot graphs, display images and do whatever is needed to better debug such objects.

For example, the following class is a custom visualizer that can show the scientist’s picture if it is available

public class ScientistVisualizer : DialogDebuggerVisualizer
{
    public ScientistVisualizer()
    {
    }

    protected override void Show(IDialogVisualizerService windowService, IVisualizerObjectProvider objectProvider)
    {
        var scientist = (Scientist)objectProvider.GetObject();

        Window window = new Window()
        {
            Title = scientist.Name,
            Width = 400,
            Height = 300
        };

        var images = new Dictionary<string, string>;
        {
            ["Marie Curie"] = "Marie_Curie.jpg",
            ["Maria Goeppert-Mayer"] = "Maria_Goeppert-Mayer.jpg",
            ["Rosalind Franklin"] = "Rosalind_Franklin.jpg",
            ["Barbara McClintock"] = "Barbara_McClintock.jpg",
        };

        if (images.ContainsKey(scientist.Name))
        {
            string imageName = images[scientist.Name];

            window.Background = new ImageBrush(new BitmapImage(new Uri(string.Format("pack://application:,,,/{0};component/Images/{1}", typeof(ScientistVisualizer).Assembly.GetName().Name, imageName))));

            window.WindowStartupLocation = WindowStartupLocation.CenterScreen;
            window.ShowDialog();
        }
    }
}

We have to add the DebuggerVisualizerAttribute for the class we want to visualize. Similarly to the rest of the attributes, we can add it over the class or on the assembly level.

In addition, we have to add the SerializableAttribute over the class being visualized.
DebuggerVisualizerAttribute over class

Or by the assembly level
DebuggerVisualizerAttribute on the assembly level

As a result, by pressing on the “Magnify” icon next to each item, an image of the scientist appears:
DebuggerVisualizer Magnify icon

Examples:
Custom Visualizer visualizing Marie Curie

Custom Visualizer visualizing Barbara Mclintock

Because the custom visualizer has to extend the type DialogDebuggerVisualizer and because it opens a WPF window, we had to add references for:

  • Microsoft.VisualStudio.DebuggerVisualizers.dll
  • PresentationCore.dll
  • PresentationFramework.dll
  • System.Xaml.dll
  • and WindowsBase.dll

Because of this, it is usually better to define the Visualizer in a separate assembly.

This is great, but in case of a long collection, it might still be hard to find the items we are interested in. Luckily, OzCode has a collection filtering feature and a Search feature that are really handy in this situations.

You can find all of the classes above in this GitHub Gist or in this GitHub Repository.

When disaster strikes: the complete guide to Failover Appenders in Log4net

Log4Net is a cool, stable, fully featured, highly configurable, highly customizable and open source logging framework for .Net.

One of its powerful features is that it can be used to write logs to multiple targets, by using “Appenders”.
An Appender is a Log4Net component that handles log events; It receives a log event each time a new message is logged and it ‘handles’ the event. For example, a simple file appender will write the new log event to a local file.

Although there are a lot of Log4Net Appenders that are included in the Log4net framework, occasionally we won’t find one that fully satisfies our needs. During a project I was working on, I had to implement a failover mechanism for logging, where the app had to start logging to a remote service, and then failback to a local file system if that remote service wasn’t reachable anymore.

Fortunately, Log4Net allows us implement our own Custom Appenders.
The Appender had to start writing logs to a remote service, and fallback to a local disk file after the first failed attempt to send a log message to that service.

Implementing the Appender

To create a custom Appender we have to implement the IAppender interface. Although easy to implement, Log4Net makes it even simpler by providing the AppenderSkeleton abstract class, which implements IAppender and adds common functionalities on top of it.

public class FailoverAppender : AppenderSkeleton
{
    private AppenderSkeleton _primaryAppender;
    private AppenderSkeleton _failOverAppender;

    //Public setters are necessary for configuring
    //the appender using a config file
    public AppenderSkeleton PrimaryAppender 
    { 
        get { return _primaryAppender;} 
        set 
        { 
             _primaryAppender = value; 
             SetAppenderErrorHandler(value); 
        } 
    }

    public AppenderSkeleton FailOverAppender 
    { 
        get { return _failOverAppender; } 
        set 
        { 
            _failOverAppender = value; 
            SetAppenderErrorHandler(value); 
        } 
    }

    public IErrorHandler DefaultErrorHandler { get; set; }

    //Whether to use the failover Appender or not
    public bool LogToFailOverAppender { get; private set; }

    public FailoverAppender()
    {
        //The ErrorHandler property is defined in
        //AppenderSkeleton
        DefaultErrorHandler = ErrorHandler;
        ErrorHandler = new FailOverErrorHandler(this);
    }

    protected override void Append(LoggingEvent loggingEvent)
    {
        if (LogToFailOverAppender)
        {
            _failOverAppender?.DoAppend(loggingEvent);
        }
        else
        {
            try
            {
                _primaryAppender?.DoAppend(loggingEvent);
            }
            catch
            {
                ActivateFailOverMode();
                Append(loggingEvent);
            }
        }
    }

    private void SetAppenderErrorHandler(AppenderSkeleton appender)
        => appender.ErrorHandler = new PropogateErrorHandler();

    internal void ActivateFailOverMode()
    {
        ErrorHandler = DefaultErrorHandler;
        LogToFailOverAppender = true;
    }
}

The FailoverAppender above accepts two appenders; a primary appender and a failover appender.

By default it will propagate Log events only to the primary appender, but in case an exception is thrown from the primary appender during event logging , it will stop sending log events to that appender and instead it starts propagating log events only to the failover appender.

I’ve used AppenderSkeleton to reference both the primary and the failover appenders in order to utilize a functionality in the AppenderSkeleton class – in this case the ability to handle errors (a.k.a Exceptions) that were thrown during an appender attempt to log an event.
We can do so by assigning the ErrorHandler property defined in AppenderSkeleton an object.

I use the LogToFailOverAppender flag to determine whether we are in ‘normal’ mode or in ‘FailOver’ mode.

The actual logging logic exists in the overridden ‘Append’ method:

protected override void Append(LoggingEvent loggingEvent)
{
    if (LogToFailOverAppender)
    {
        _failOverAppender?.DoAppend(loggingEvent);
    }
    else
    {
        try
        {
            _primaryAppender?.DoAppend(loggingEvent);
        }
        catch
        {
            ActivateFailOverMode();
            Append(loggingEvent);
        }
    }
}

If the LogToFailOverAppender flag is active, it logs events using the failover appender, as it means an exception has been thrown already. Otherwise, it logs events using the primary appender, and it will activate the failover mode, if an exception is thrown during that time.

The following are the IErrorHandlers that I defined and used

/*
This is important. 
By default the AppenderSkeleton's ErrorHandler doesn't
propagate exceptions
*/
class PropogateErrorHandler : IErrorHandler
{
    public void Error(string message, Exception e, ErrorCode errorCode)
    {
        throw new AggregateException(message, e);
    }

    public void Error(string message, Exception e)
    {
        throw new AggregateException(message, e);
    }

    public void Error(string message)
    {
        throw new LogException($"Error logging an event: {message}");
    }
}
/*
This is just in case something bad happens. It signals 
the FailoverAppender to use the failback appender.
*/
class FailOverErrorHandler : IErrorHandler
{
    public FailOverAppender FailOverAppender { get; set; }
        
    public FailOverErrorHandler(FailOverAppender failOverAppender)
    {
        FailOverAppender = failOverAppender;
    }

    public void Error(string message, Exception e, ErrorCode errorCode)
        => FailOverAppender.ActivateFailOverMode();

    public void Error(string message, Exception e)
        => FailOverAppender.ActivateFailOverMode();

    public void Error(string message)
        => FailOverAppender.ActivateFailOverMode();
}

Testing the Appender

I’ve created a config file you can use to test the appender. These are the important bits:

<!--This custom appender handles failovers. If the first appender fails, it'll delegate the message to the back appender-->
<appender name="FailoverAppender" type="MoreAppenders.FailoverAppender">
    <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%thread] %-5level %logger - %message%newline"/>
    </layout>

    <!--This is a custom test appender that will always throw an exception -->
    <!--The first and the default appender that will be used.-->
    <PrimaryAppender type="MoreAppenders.ExceptionThrowerAppender" >
        <ThrowExceptionForCount value="1" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%date [%thread] %-5level %logger - %message%newline"/>
        </layout>        
    </PrimaryAppender>

    <!--This appender will be used only if the PrimaryAppender has failed-->
    <FailOverAppender type="log4net.Appender.RollingFileAppender">
        <file value="log.txt"/>
        <rollingStyle value="Size"/>
        <maxSizeRollBackups value="10"/>
        <maximumFileSize value="100mb"/>
        <appendToFile value="true"/>
        <staticLogFileName value="true"/>
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%date [%thread] %-5level %logger - %message%newline"/>
        </layout>
    </FailOverAppender>
</appender>

In this post I’ll discuss the parts that are relevant to the appender. You can find the full config file here. The rest of the config file are a regular Log4Net configurations, which you can read more about here and here.

Log4Net has a feature that give us an ability to instantiate and assign values to public properties of appenders in the config file using XML. I’m using this feature to instantiate and assign values to both the PrimaryAppender and the FailOverAppender properties.

In this section I’m instantiating the PrimaryAppender:

<PrimaryAppender type="MoreAppenders.ExceptionThrowerAppender" >
    <ThrowExceptionForCount value="1" />
    <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%thread] %-5level %logger - %message%newline"/>
    </layout>        
</PrimaryAppender>

The type attribute’s value is the fully qualified name of the appender’s class.
For our example, I’ve created the ExceptionThrowerAppender appender for testing purposes. It can be configured to throw exceptions once per a certain amount of log events.

In a similar manner, in the following XML I’ve instantiated and configured the FailOverApppender to be a regular RollingFileAppender

<FailOverAppender type="log4net.Appender.RollingFileAppender">
    <file value="log.txt"/>
    <rollingStyle value="Size"/>
    <maxSizeRollBackups value="10"/>
    <maximumFileSize value="100mb"/>
    <appendToFile value="true"/>
    <staticLogFileName value="true"/>
    <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%thread] %-5level %logger - %message%newline"/>
    </layout>
</FailOverAppender>

I used the following code to create log events:

class Program
{
    static void Main(string[] args)
    {
        XmlConfigurator.Configure();

        var logger = LogManager.GetLogger(typeof(Program));

        for (var index = 0; index < int.MaxValue; ++index)
        {
            logger.Debug($"This is a debug message number {index}");
        }

        Console.ReadLine();
    }
}

I started the program in debug mode and placed a breakpoint inside the ‘Append’ method:

First logging message - goes to the primary appender

Notice how OzCode’s Predict the Future feature marks the if-statement with an X and with a red background, telling us that the condition is evaluated to false. That means an exception wasn’t thrown yet from the primary appender.

In order to make figuring out the loggingEvent message value easier, I’ve used OzCode’s Magic Glance feature to view the necessary information in every LoggingEvent object.

Selecting the properties to show

The result:

Magic Glance feature

By continuing the program, the primary appender will handle the logging event, and it will throw an exception

Exception is thrown.

After that exception is propagated by the ErrorHandler, it will be handled by the catch-clause, which activates the FailOverAppender mode (notice how the log event is sent to the FailOverAppender as well) which would send future logging events only to the FailOverAppender

FailOverAppender mode is active

This time the if-statement is marked by a green ‘V’ . This tell us that the condition is evaluated to be true and that it will execute the if-statement body (sends the logging-event to the failover appender).

You can view and download the code by visiting this GitHub Repository.

Summary

Log4Net is a well-known logging framework for .Net. The framework comes with a list of out-of-the-box Appenders that we can use in our programs.
Although these Appenders are usually sufficient for most of us, sometimes you’ll need Appenders that are more customized for your needs.

We saw how we can use Log4Net’s extensibility features to implement our own Log4Net custom Appenders. In this example, we have created a Fail-over mechanism for Appenders that we can use to change the active Appender when it is failing to append log messages.

Log4Net is highly extensible and it has many more extensibility features that I encourage you to explore.

Note: this post is published also at OzCode’s blog.

Code Generation Chronicles #1 – Using Fody to Inject KnownType attributes

Code Generation Chronicles

As part of my new year resolutions, I’ve decided to put more effort in learning code generation techniques.

This is the first blog post in a series exploring code generation. Although I won’t always dive into the internals, I do promise that I’ll show examples of what we can achieve with each technique and when it is better to use each one.

Fody

Fody is an open source library for IL weaving. It can generate and manipulate IL code during build time.
Getting Fody is easy – it is available as a Nuget Package and doesn’t require any installation/modification to the build system.

One of the benefits of using Fody is that there is no footprint – since the IL weaving is done during build time, there aren’t any Fody-related assemblies needed at runtime, which could be a good thing if you worried about your project dependencies or the number of assemblies you ship with your product.
And on top of that Fody is highly extensible and there is an active open source community around it.

One example for how Fody can save developer’s time is such extension: “PropertyChanged” which works by placing the ImplementPropertyChanged attribute it provides on a class, it will generate all of the code necessary to fully implement the INotifyPropertyChanged interface in that class.
More extensions can be found here.

Using Fody to add WCF’s KnownType attributes

At work we’ve used a client which was implemented using .Net Remoting as the communication model for it services. At some point, we wanted to exchange .Net Remoting with the more modern WCF. The problem was that a lot of the code base was unmaintainable legacy code, and we’ve wanted to perform as little changes as possible, and reuse as much of the existing architecture as we could.
One of the challenges resided in a WCF quirk, were by default it doesn’t allow passing a derived type of the DataContract type that is defined in the OperationContract.
The way to make derived classes work is done in WCF by adding a KnownType attribute on the base type for each derived type we use.
As an example, if we have a class A and two derived classes B and C, we would have to add two KnownType attributes over class A; one for Class B and one for Class C.

[KnownType(typeof(B))]
[KnownType(typeof(C))]
class A
{
}

class B : A
{
}

class C : A
{
}

Since this scenario was common in the current architecture, we were looking for a way to solve this issue without manually adding KnownType attributes, since it would be easy to miss some or to forget adding the attribute over new derived classes in the future.

In the end we’ve managed to save time and replace the communication layer easily by using Fody for automatically adding KnownType attributes for all derived types on their base classes during build time.

Implementing a Fody extension

If you want to write Fody extension start by cloning the BasicFodyAddin repository. This repository is maintained by the open source community and it simplifies both the implementation and the deployment of the extension as a Nuget Package.
BasicFodyAddin source code contains a Solution with four C# projects:

  1. BasicFodyAddin which will contain the extension
  2. Nuget which is for deploying a Nuget package
  3. Tests for writing unit tests
  4. AssemblyToProcess that is used as a target assembly for testing your mew Fody extension.

We will focus on BasicFodyAddin.

In the BasicFodyAddin project there is a file called ModuleWeaver.cs Go ahead and replace it content with the following code:

using System;
using System.Linq;
using Mono.Cecil;
using Mono.Cecil.Rocks;

namespace KnownTypes.Fody
{
   public class ModuleWeaver
   {
      public ModuleDefinition ModuleDefinition { get; set; }
   
      public void Execute()
      {
      }

      void AddKnownTypeAttributes(ModuleDefinition module, TypeDefinition baseType)
      {
      }

      void RemoveKnowsDeriveTypesAttribute(TypeDefinition baseType)
      {
      }
   }
}

This is the class we will use to implement the extension. The ModuleDefinition property will be populated during build time by Fody and it will contain the target Module for the IL Weaving.
As you have probably noticed, the property is of type ModuleDefinition.
ModuleDefinition is a class representing a MSIL Module in Mono.Cecil which is a library Fody relies on to generate and manipulate IL code. In addition to ModuleDefinition we will use more types it defines such as TypeDefinition.

When using Fody your ModuleWeaver class must be implemented according the following:

  • Be public, instance and not abstract.
  • Have an empty constructor.
  • Have a ModuleDefinition property that will be populated during build.
  • Have an Execute method.

The KnownTypeAttributeWeaver class has three main methods:

  • Execute – Entry point. It calls other methods for performing the IL weaving.
  • AddKnownTypeAttributes – Decorates base types with KnownType attributes.
  • RemoveKnowsDeriveTypesAttribute – Removes KnowsDeriveTypes attributes from base types

Note: A few helper methods will be added down the road.

 

Now create another a new class called KnowsDeriveTypesAttribute.

[AttributeUsage(AttributeTargets.Class, Inherited = false)]
public class KnowsDeriveTypesAttribute : Attribute
{
}

This attribute is for explicitly specifying the base types we want to decorate with KnownType attributes. We could have added KnownType attributes over every base class in the assembly, but we wanted the extra control.

Implementing Execute

public void Execute()
{
   foreach (var type in ModuleDefinition.GetTypes().Where(HasKnownDeriveTypesAttribute))
   {
      AddKnownTypeAttributes(module, type);
      RemoveKnowsDeriveTypesAttribute(type);
   }
}

//A helper method 
//For filtering the types without the KnowsDeriveTypes Attribute.
bool HasKnownDeriveTypesAttribute(TypeDefinition type) => 
   type.CustomAttributes.Any(attr => attr.AttributeType.FullName == typeof(KnowsDerivativeTypesAttribute).FullName);

In Execute we locate all the types decorated with KnowsDeriveTypesAttribute, and we decorate them with KnownType attributes according to their sub types.
After that is done, we remove the KnowsDeriveTypesAttribute as it isn’t necessary anymore.

Implementing AddKnownTypeAttributes

void AddKnownTypeAttributes(ModuleDefinition module, TypeDefinition baseType)
{
   //Locate derived types
   var derivedTypes = GetDerivedTypes(module, baseType);

   //Gets a TypeDefinition representing the KnownTypeAttribute type.
   var knownTypeAttributeTypeDefinition = GetTypeDefinition(module, "System.Runtime.Serialization", "KnownTypeAttribute");

   //Gets the constructor for the KnownTypeAttribute type.
   var knownTypeConstrcutor = GetConstructorForKnownTypeAttribute(module, knownTypeAttributeTypeDefinition);

   //Adds KnownType attribute for each derive type
   foreach (var derivedType in derivedTypes)
   {
      var attribute = new CustomAttribute(constructorToUse);
      attribute.ConstructorArguments.Add(new CustomAttributeArgument(constructorToUse.Parameters.First().ParameterType, derivedType));

      baseType.CustomAttributes.Add(attribute);
   }
}

Let’s break it down.

//Locate derived types
var derivedTypes = GetDerivedTypes(module, baseType);

//Gets a TypeDefinition representing the KnownTypeAttribute type.
var knownTypeAttributeTypeDefinition = GetTypeDefinition(module, "System.Runtime.Serialization", "KnownTypeAttribute");

//Gets the constructor for the KnownTypeAttribute type.
var knownTypeConstrcutor = GetConstructorForKnownTypeAttribute(module, knownTypeAttributeTypeDefinition);

In the code above we are performing three steps:

  1. Finding and retrieving all derived types of the given base type.
  2. Creating a TypeDefinition instance for the KnownType attribute.
  3. Finding and retrieving a constructor for the KnownType attribute.

The first step is done using a helper method called GetDerivedTypes.

Given a module and a type it returns an array of TypeDefinitions, containing all of the types in the given module that derive from the given type

The second step is done using a helper method called GetTypeDefinition.

Given an assembly name and a type name, it returns a TypeDefinition for that type.

The third step is done using a helper method called GetConstructorForKnownTypeAttribute.

Given a ModuleDefinition and a TypeDefinition it returns a the first constructor for that type.

//A helper method. Given a module and a baes type:
//It returns all derived types of that base type.
TypeDefinition[] GetDerivedTypes(ModuleDefinition module, TypeDefinition baseType)
   => module.GetTypes()
         .Where(type => type.BaseType?.FullName == baseType.FullName)
         .ToArray();

//A helper method. Given an assembly and a type name:
//It returns a TypeDefinision for that type.
TypeDefinition GetTypeDefinition(ModuleDefinition module, string assemblyName, string typeName) 
   => module.AssemblyResolver
         .Resolve(assemblyName)
         .MainModule.Types.Single(type => type.Name == typeName);

//A helper method. Given a module and type definition for the KnownType Attribute:
//It returns the constructor for the attribute accepting a System.Type object.
MethodReference GetConstructorForKnownTypeAttribute(ModuleDefinition module, TypeDefinition knownTypeAttributeTypeDefinition)
{
   var constructorMethodToImport = knownTypeAttributeTypeDefinition
                                     .GetConstructors()
                                     .Single(ctor => 1 == ctor.Parameters.Count && "System.Type" == ctor.Parameters[0].ParameterType.FullName);

   return module.Import(constructorMethodToImport);
}

Let’s continue and break down the last part of AddKnownTypeAttributes.

//Adds KnownType attribute for each derive type
foreach (var derivedType in derivedTypes)
{
   var attribute = new CustomAttribute(constructorToUse);
   attribute.ConstructorArguments.Add(new CustomAttributeArgument(constructorToUse.Parameters.First().ParameterType, derivedType));

   baseType.CustomAttributes.Add(attribute);
}

In this part of the code we add a KnownType attribute to the base type for each derived type we previously found.
We are using the KnownType attribute’s constructor that expects the System.Type of the sub class.

Implementing RemoveKnowsDeriveTypesAttribute

This part is straight forward. We locate and then remove the KnowsDeriveTypes attribute from the base type.

void RemoveKnowsDeriveTypesAttribute(TypeDefinition baseType)
{
   var foundAttribute = baseType.CustomAttributes
         .Single(attribute => attribute.AttributeType.FullName == typeof(KnowsDeriveTypesAttribute).FullName);

   baseType.CustomAttributes.Remove(foundAttribute);
}

Unit Tests & Debugging

Unit Tests are always important, especially when writing Fody extensions. We had a good code coverage in the final product, but for this blog post I’ll use Unit Tests solely for debugging purposes, as I will be able to debug the extension by running the Unit Tests in debug mode.

For this post I’ve created two Unit Tests. One for testing the addition of the KnownType attributes and one for testing the removal of the KnowsDeriveTypes attribute. You can view the Unit Tests file here. I’ve also went ahead and added three classes to the AssemblyToProcess project for testing the IL Weaving process.

Even though Fody has a basic logging capability it isn’t always easy to find and view logs in the output window.
Fortunately, at debug time we can use OzCode’s Trace Points feature. I’ve added a trace point at the line which adds a KnownType attribute over base types.

//Trace point was added here in the AddKnownTypeAttributes method.
baseType.CustomAttributes.Add(attribute);

Read More

Using LINQ to abstract Bitwise operations

There are 10 types of developers; those who work in a higher level and depend on abstractions, and those who work with the bits and bytes alongside the bare metal.

As a C# developer, I am one of the former, and I usually try to abstract everything I do. A few weeks ago, I figured out that I can use some of these abstraction techniques to solve “low-level” challenges too. To make a long story short, I was able to utilize LINQ in order to eliminate a rather very complicated bitwise operation.

The mission

A framework for communicating with FPGA cards (Field-Programmable Gate Array), where it sends and receives messages to and from those cards.

One of the framework’s abilities is to serialize and deserialize messages that were sent or received from a FPGA card to and from byte arrays, according to an application-specific protocol.

The structure and the parts each message is composed of vary, and are only determined at runtime.

This looks easy enough

Each message is composed of different parts with different lengths.
By serializing a message, we concatenate each part’s value according to its length in bits.

The following is an example of a message that is composed of 3 parts.

Message
Part 0 Part 1 Part 2
001 1010 00111110000
3 Bits 4 Bits 11 Bits

In this example, “Part 0” has the value of 0x1, “Part 1” has the value of “0xA” and “Part 2” has the value of “0x1F0”.

The framework doesn’t really care about the content of the messages, but rather cares for their structure.

When deserializing a message, we only need the structure or the schema of the message. For each message-part we need to specify its name, index (its order in the message) and length in bits.

I have therefore introduced the following class:

public class LayoutPart
{
	public string Name { get; set; }
	public int Index { get; set; }
	public int BitCount { get; set; }

	public LayoutPart(string name, int index, int bitCount)
	{
		Name = name;
		Index = index;
		BitCount = bitCount;
	}
}

When serializing a message, each message-part has to specify four properties. The three properties of LayoutPart, in addition to the Value of that part.

Here are the classes I have added:

public class MessagePart : LayoutPart
{
     public int Value { get; set; }

     public MessagePart(string name, int index, int bitCount, int value)
          : base(name, index, bitCount)
     {
          Value = value
     }
}

In reality, the content’s type wasn’t known beforehand, and it surly wasn’t always an int, but I decided to use an int in all of my examples to keep them simple.

The relevant API is really simple, and it has two methods; Serialize and Deserialize:

public interface IMessageSerializer
{
    byte[] Serialize(IEnumerable<MessagePart> messageValues);
    IEnumerable<MessagePart> Deserialize(IEnumerabe<LayoutPart> layout,
                                         byte[] bytes);
}

Bitwise operations: This is really ugly

So far so good 😀

After defining the API, I went on and started implementing it.

The following is an example of serializing the message from the example above

Name BitCount Value Value in Bits Value in Bytes
Part 0 3 1 001 00000001
Part 1 4 0xA 1010 00001010
Part 2 11 0x1F0 00111110000 00000001 11110000

The byte array should look like this

Byte 2 Byte 1 Byte 0
00000000 11111000 01010001

The first byte is composed of Part 0 (3 bits), Part 1 (4 bits) and the first bit of Part 2 (11 bits). The second byte is composed of bits 1-8 of Part 2, and the third byte is composed of bits 9-10 of Part 2 with 6 more trailing zeros to complete a byte of 8 bits.

I approached it using the standard-naive way: bitwise operations.
I needed the ability to extract bits from different bytes and bit indexes, then concatenate them together for composing a byte array.

Assuming we are working with System.Int32, how would I extract the n first bits? Easy enough!

value & ((1 << n) - 1)

And to extract the n last bits?

(value >> (32 - n)) & ((1 << n) - 1)

What about extracting n bits that are in the “middle”? This is starting to be more complicated.

((value >> startIndex) & ((1 << n) - 1));

I did a little refactoring and defined them in methods.

long ExtractBits(uint value, int n, int startIndex) =>
                ((value >> startIndex) & ((1 << n) - 1));

long ExtractFirstBits(uint value, int n) => ExtractBits(value, n, 0);

long ExtractLastBits(uint value, int n) => ExtractBits(value, n, 32 - n);

This looks a little bit better. I had to use System.Int64 and system.UInt32 for overflow/underflow safety when performing bitwise operations.

Most importantly, this is an over simplified version of what I needed to do. I also had to consider:

  • Concatenating the results
  • Messaging parts of length exceeding 8 bytes (sizeof(long)). Current implementation won’t work
  • Transfer the results into a byte array
  • Implementing deserialization, which is even harder

I went on and finished the rest of the implementation. It took me a full day of work (implementing + testing), and it wasn’t easy, at least not as I thought it would be. I kept telling myself that there has to be a better way.

The code wasn’t maintainable, even with the 90% code coverage I had of tests. Not by the version of me from next month or even from next week, and of course not by another dev from my team. This code will become an unmaintainable, ‘don’t touch’, legacy code as soon as I’ll push the changes to the repo and mark the task as ‘Done’.

This code is ugly.

legacy code

Oh, much better

I took a step back and tackled the problem from a different angle. I figured that my goal is pretty simple; I have a collection of values and all I have to do for the Serialize method is to:

  1. Project each message part into bits
  2. Concatenate the bits of all values
  3. Project the result into a byte array

And all I have to do for the deserialize method is to:

  1. Project the byte array into bits.
  2. Slice them according to the length of each message part
  3. Project each slice into the corresponding message part object

I quickly thought about LINQ! Isn’t this one of the scenarios that are easily solved by LINQ? I went on and explored this idea further.

I decided to use the Type-Safe Enum Pattern to represent a Bit (Bit.cs). I also used some LINQ operations from Jon Skeet’s MoreLinq library.

For the Serialize method I started by writing the following extension methods:

//Projects a byte into IEnumerable<Bit>
public static IEnumerable<Bit> ToBits(this byte @byte)
{
    for(var index = 0; index < 8; ++index, @byte >>= 1)
    {
        yield return 1 == (@byte & 1);
    }
}

//Projects IEnumerable<Bit> into a byte
public static byte ToByte(this IEnumerable<Bit> bits) =>
    bits.Reverse().Aggregate((byte)0, (currentValue, bit) => 
        (byte)((currentValue << 1 | bit) ));

//Projects a byte[] into IEnumerable<Bit>
public static IEnumerable<Bit> ToBits(this IEnumerable<byte> bytes) 
    => bytes.SelectMany(ToBits);

The following are the steps for the Serialize method:

  1. Project each value into bits, according to the BitCount property of its message-part object
    1. Project the value into bytes
    2. Project the bytes into bits
  2. Group the bits into a group of 8-bit each
  3. Project each group into a byte
  4. Project the IEnumerable of bytes into a byte array
byte[] Serialize(IEnumerable<MessagePart> messageParts)
{
    return messageParts.OrderBy(part => part.Index)
        .SelectMany(part => 
            BitConverter.GetBytes(part.Value)
            .ToBits())
        .Batch(8)
        .Select(slice => slice.ToByte())
        .ToArray();
}

(Did you notice the bug? I’ll discuss it later)

For implementing the deserialize method, I created the following extension methods:

//Projects an array of 4 bytes into an int
public static int ToInt(this byte[] bytes) => 
                            BitConverter.ToInt32(bytes, 0);

//Projects a bunch of 8-bit IEnumerables into an IEnumerable of bytes
public static IEnumerable<byte> 
    ToBytes(this IEnumerable<IEnumerable<Bit>> bits) => bits.Select(ToByte);

The following are the steps for the deserialize method:

  1. Project the byte array into IEnumerable of bits
  2. Slice the bits according to each message-part length and order
  3. Project each slice into 4 bytes (sizeof(System.Int32))
  4. Project each 4 bytes into a System.Int32
IEnumerable<MessagePart> Deserialize(IEnumerable<LayoutPart> layout, 
                                     byte[] bytes)
{
    var bits = bytes.ToBits();
    
    return layout.OrderBy(part => part.Index).Select(part =>
    {
        var slice = 
            bits
            .Take(part.BitCount)
            .Batch(8)
            .ToBytes()
            .Pad(sizeof(int), (byte)0)
            .Take(sizeof(int))
            .ToArray()
            .ToInt();

        bits = bits.Skip(part.BitCount);

        int value = BitConverter.ToInt32(slice, 0);

        return new MessagePart(part.Name, value);
    }).ToArray();
}

Looks better, Eh?

Testing the implementation

It isn’t always easy to debug LINQ operations, thus in order to debug, test and visualize my examples, I used OzCode which is an awesome debugging extension for Visual Studio. I installed the EAP (Early Access Program) edition, which includes amazing LINQ debugging capabilities, and put it to work.

This is the code according to the example above

var layout = new[]
{
    new LayoutPart("Part 0", 0, 3),
    new LayoutPart("Part 1", 1, 4),
    new LayoutPart("Part 2", 2, 11),
};

var message = new[]
{
    new MessagePart("Part 0", 0, 3, 1),
    new MessagePart("Part 1", 1, 4, 0xA),
    new MessagePart("Part 2", 2, 11, 0x1F0),
};

var serializer = new MessageSerializer();

var bytes = serializer.Serialize(message);
var messageParts = serializer.Deserialize(layout, bytes);

I used OzCode’s LINQ analyser for visualizing and tracking the LINQ operations.

Here we can see what the actual bits are, and their order in the concatenated message-parts.

snis9uf
But according to the BitCount property of Part 0, its value should be projected only to 3 bits.

Fixing the bug:

byte[] Serialize(IEnumerable<MessagePart> messageParts)
{
    return messageParts.OrderBy(part => part.Index)
        .SelectMany(part => 
            BitConverter.GetBytes(part.Value)
            .ToBits()
            .Pad(part.BitCount, Bit.Off) //Added
            .Take(part.BitCount))  //Added
        .Batch(8)
        .Select(slice => slice.ToByte())
        .ToArray();
}

I added Pad and Take to be sure I project exactly the needed count of bits.

4fbbiu8

This looks much better.

Here we can see the value of each byte:
img-1
Testing the Deserialize method, we can see the value of each message part:
gd6uv4a

Summary

Even though actions that require bitwise operations are usually considered “low-level”, I was able to use LINQ, which is considered a “high-level” API, to perform the same thing. This kind of abstraction saved me time and effort, and made my code more maintainable, readable and clean. I suggest to use abstractions whenever you can, and whenever it make sense.

What I really learnt though, is that we can use skills we acquire in one domain to solve challenges in another.
There is a saying that polyglot programmers write better code than those who specializes only in one language. I think it is yet another example of the same idea – each skill you acquire, will make you better at what you do, regardless how unrelated that skill might look like at that time.

Note: this post is published also at OzCode’s blog.

Hello, Blog!

Hello 🙂

My name is Moaid Hathot. I am a software developer, consultant and OzCode Evangelist at CodeValue.

I am an enthusiastic developer, and writing code is one of my greatest passions. I usually code in C#, but you can catch me once in while using F#, C/C++ or even Java (Heaven forbid). Recently I started to be interested in Clean Code and architecture.

I LOVE to read, usually in English. Mainly about Fiction, Technology, Computer Science and the art of writing good Software.

Most of my blog posts will be about Software, Computer Science and Techonology, though occasionally I might write a little bit about my self.

Now that my blog is ready, and I’ve been introduced, the only part left is actually blogging. So stay tuned for future posts 😉