As I continue the development of my DMX control system, there have been features that I’ve avoided working on because of their apparent complexity. Scripting support is one such area; however, as you’ll see, my concerns were unfounded as Roslyn scripting support made it possible to implement in just a few hours.

AlterNET Code Editor UI Component as found in my app.

Most lighting control systems offer some form of macro support, which allow users to chain together commands from the domain specific language used in their command lines. This is a feature I used heavily when operating a Strand 500 (see below). I’d use macros to remap the ‘bump’ buttons found under the faders, making them fire effects and other tasks that usually require lengthy CLI input. You can see three lines of labels (one above, two below) the rows of faders, which indicates that I was using the shift functionality to double up my macros!

I’m operating a Strand 530i.

As I’ve been considering the how best to add this kind of macro support to my lighting console, I kept coming back to the idea of how to provide more flexibility than the DSL. After much deliberation, I’ve opted to allow users to write ‘macro’ scripts using C# and a small library that exposes some of the operating software’s key features.


What is Roslyn?

The Roslyn logo

Roslyn provides open-source C# and Visual Basic compilers with rich code analysis APIs. It enables building code analysis tools with the same APIs that are used by Visual Studio.

https://github.com/dotnet/roslyn

Roslyn is Microsoft’s .NET compiler platform, which includes a set of compilers and codes analysis APIs for .NET languages (including C#, F# and VB.NET). In simpler terms, Roslyn is the technology responsible for taking .NET source code and turning it into an executable binary. Its intended use is mostly concerning IDEs (Integrated Development Environments) and other development of tools. Though it additionally features a scripting API, making it possible to add execution of user-created code at runtime. 

Whilst there are obvious security concerns with allowing users to execute their own code at runtime, I’ll show a quite simple (and by no means fool proof) approach to mitigating some of the risk.


Setup & the basics

To make use of Roslyn’s scripting functionality, add the Microsoft.CodeAnalysis.CSharp.Scripting Nuget packages to your project.

You’ll then want to add the following using statement.

using Microsoft.CodeAnalysis.CSharp.Scripting;

That’ll give you everything you need to take some C# code and evaluate it as a script. Below is a simple console app example to demonstrate the equivalent of a Roslyn scripting hello world example.

public static async Task Main(string[] args)
{
    Console.WriteLine("Write some C#");
    var evaluate = true;

    while(evaluate)
    {
        var input = Console.ReadLine();

        if(input == "exit")
        {
            evaluate = false;
            break;
        }

        if(input == "clear")
        {
            Console.Clear();
            Console.WriteLine("Write some C#");
            continue;
        }

        object result = await CSharpScript.EvaluateAsync(input);
        Console.WriteLine(result.ToString());
    }
}

When you run the example above, you’ll likely notice the delay between the first expression being evaluated and the result being returned. This is just because of the JIT compiler doing its thing. You can work around this on multi-core devices using the ProfileOptimization class. You can find a branch of the demo that shows how you might enable this.

Console app running the sample app.
Sample app running.

Reusable Script objects & variable inspection

Unless your requirements are simply creating a C# REPL, you’ll want to use some more of the Roslyn APIs to add more functionality. One example might be to load in a txt file containing user generated C#, execute it and then inspect the variable values. Below is an example user provided C# snippet to run.

//User provided script stored in a .txt file 
System.TimeSpan sinceMidnight = System.DateTime.Now - System.DateTime.Today;
var seconds = sinceMidnight.TotalSeconds;

It can be loaded and executed using the following:

public static async Task Main(string[] args)
{
    var userScriptPath = Path.Combine(Environment.CurrentDirectory, "scriptInput.txt");
    var fileContents = File.ReadAllText(userScriptPath);

    Script script = CSharpScript.Create(fileContents, ScriptOptions.Default);
    var result = await script.RunAsync();

    Console.WriteLine($"Seconds in a week: {result.GetVariable("seconds").Value}");
}

In this example, the script object isn’t being persisted, which isn’t a problem as all the example code lives within the Main method. However, if there a separate method which was invoked twice, the script object would be created twice, and that would result Roslyn having to recompile both times.

If you know in advance that the script is going to be executed multiple times, then you will want to keep an instance around. Below you can see that I’m reusing the script instance within the for-loop. In this example, the script is only ever compiled once, on the first execution of RunAsync();

static Script script;

public static async Task Main(string[] args)
{
  var userScriptPath = Path.Combine(Environment.CurrentDirectory, 	"scriptInput.txt");
  var fileContents = File.ReadAllText(userScriptPath);

  script = CSharpScript.Create(fileContents, ScriptOptions.Default);

  for (int i = 0; i < 15; i++)
  {
    var result = await script.RunAsync();

    Console.WriteLine($"{i} - Seconds in a week: {result.GetVariable("seconds").Value}");
    await Task.Delay(500);
  }            
}

Scripting in the Light Console

The implementation of C# script in my lighting control project uses another element of Roslyn to help limit the scope of what can be executed. This extra element is Roslyn’s code analysis functionality.

This feature provides the ability to inspect the C# code to find illegal (this is defined within the app) syntax and ensures the script cannot be compiled if any illegal APIs are found.

The reasoning behind this requirement is based on the need to block users ability to serialise fixture objects and persist them to disk. This is due to supporting the Carallon Fixture Database, which has the following license requirement:

– Subscribers must make best efforts to ensure that they do not distribute libraries for their products in a format that releases Carallon’s proprietary data into the public domain. It is imperative that the data be distributed in a format that is not human-readable and cannot be trivially reverse-engineered to gain access to Carallon’s data.

Carallon Fixture Database License

Whilst the data provided is processed through a transpiler, making it impossible to return to it’s original form, the transpiled data still retains much of the valuable original information. This means it’s essential to block access to APIs that might allow users to write the contents of an in-memory fixture object to disk. The obvious APIs to block include System.Runtime.Serialization, System.Xml.Serialization and System.Text.Json. Focusing on Serialization APIs severely limits the ability to create a human-readable copy of any in-memory object.

Following on from blocking the serialisation APIs, I’ve blocking all of System.IO, it becomes impossible to read or write anything to disk. This means that even if a user did somehow manager to circumnavigate their way around the blocked serialisation APIs, they wouldn’t be able to persist the data as it’d only live in memory whilst the app is running.

You can see the entire list of illegal assemblies below but I thought it’d also be worth pointing out that all of the above is pointless if System.Reflection isn’t also blocked.


Now you understand the rationale behind why I need to block API usage, lets look at how. Usually Roslyn analysers are used to provide hints, tips and code fixes to developers using Visual Studio, but at the core of every analyser is Roslyn’s ability to provide detailed information about .NET source code.

Roslyn Analysis in its traditional setting (Visual Studio)

I discussed earlier about the script compilation when calling RunAsync(). This is the basis of how I control what can be executed within the application, through creating a two step process to run a script. First is compilation, which also validates the script. Secondly I handle the running of the script, if it has passed validation. If the script fails the validation checks, then the local variable to the Script object is set to null, ensuring that it cannot be executed.

Script Compilation & Validation

Below you can see the source for the compilation method, which is tried to a button presented in the editor UI view.

private Script userScript;

private void CompileScript()
{
    UpdateSource(); //Handles creating a copy of the script which includes variables the user never sees.
    string text = csSource.Text; //csSource is an object from the UI provider
    
  	userScript = CSharpScript.Create(text, options: userScriptOptions);
    userScript.Compile(); 

  	//Validation class
    ScriptValidator scriptValidator = new ScriptValidator(userScript);

    if(!scriptValidator.IsValid)
    {
        MessageBox.Show($"Found {scriptValidator.Errors.Count()} illegal APIs in script.", "Build Error");
    	userScript = null;
    }
}

The ScriptValidator class is responsible for creating a Semantic Model and Syntax Tree from the compiled user Script. It’s important to note that I compile the script before any validation occurs but if validation fails, I update the script instance to be null.

private Compilation compilation;
private SemanticModel semanticModel;
private CompilationUnitSyntax syntaxRootNode;

// Validation Class Constructor 
public ScriptValidator(Script script)
{
  compilation = script.GetCompilation();

  SyntaxTree syntaxTree = compilation.SyntaxTrees.First();
  syntaxRootNode = syntaxTree.GetRoot() as CompilationUnitSyntax;
  semanticModel = compilation.GetSemanticModel(compilation.SyntaxTrees.First());
  
  ...
}

Once I have both the Syntax tree and Semantic Model, it becomes a trivial exercise to validate the script against a set of rules of my choosing. Let’s look at how you might validate that System.IO is included in the using statements.

foreach(UsingDirectiveSyntax usingDirective in syntaxRootNode.Usings)
{
    ISymbol symbol = semanticModel.GetSymbolInfo(usingDirective.Name).Symbol;
    string name = symbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
    
  	if(name == "System.IO")
      throw new Exception("whoa!!, you can't be using that here!");
  	
}

Whilst this check is perfectly valid, it won’t detect any System.IO usage elsewhere in the script. To achieve this, I created a method to validate assembly usage by their names. This method is then called from various sub-validators (such as method and property declarations).

private bool IsIllegalAssemblyName(string assemblyName)
{
   return assemblyName switch
   {
       "System.Console" => true,
       "System.Diagnostics.Debug" => true,
       "System.IO" => true,
       "System.IO.FileSystem" => true,
       "System.IO.FileSystem.Primitives" => true,
       "System.Reflection" => true,
       "System.Reflection.Extensions" => true,
       "System.Runtime" => true,
       "System.Runtime.Extensions" => true,
       "System.Runtime.InteropServices" => true,
       "System.Diagnostics.Tracing" => true,
       "System.Runtime.CompilerServices.Unsafe" => true,
       "System.Memory" => true,
       "System.Reflection.Emit.ILGeneration" => true,
       "System.Diagnostics.Tools" => true,
       "System.Reflection.Metadata" => true,
       "System.IO.Compression" => true,
       "System.IO.MemoryMappedFiles" => true,
       "System.Diagnostics.FileVersionInfo" => true,
       "Microsoft.Win32.Registry" => true,
       "System.Security.AccessControl" => true,
       _ => false,
   };
}

One example of its usage can be found in the ValidatePropertyDeclarationSyntax method as seen below (the last validation).

private void ValidatePropertyDeclarationSyntax(PropertyDeclarationSyntax declarationSyntax)
{
  //Get the symbol for the property.
  IPropertySymbol symbol = semanticModel.GetDeclaredSymbol(declarationSyntax) as IPropertySymbol;

  if (symbol.IsStatic)
  {
      errors.Add((ValidationErrorType.Illegal_Feature, "Static properties are not allowed"));
      return;
  }

  if (symbol.IsOverride)
  {
      errors.Add((ValidationErrorType.Illegal_Feature, "Overriding properties is not allowed"));
      return;
  }

  if (symbol.IsAbstract)
  {
      errors.Add((ValidationErrorType.Illegal_Feature, "Abstract properties are not allowed"));
      return;
  }

  if (symbol.IsVirtual)
  {
      errors.Add((ValidationErrorType.Illegal_Feature, "Virtual properties are not allowed"));
      return;
  }

  string assemblyName = symbol.Type.ContainingAssembly.Name;
  if (IsIllegalAssemblyName(assemblyName))
  {
      errors.Add((ValidationErrorType.Illegal_Assembly, $"Types from assembly {assemblyName} are not allowed"));
      return;
  }
}

If the script passes validation then it can be executed using the RunScript method (again, tied to a UI button).

private async void RunMacro()
{
    if(userScript == null)
        MessageBox.Show("You must first compile the script.");

    try
    {
        ScriptState result = await userScript.RunAsync();
    }
    catch(Exception e)
    {
        MessageBox.Show(e.Message.ToString(), "Error");
    }
}

Whilst this is a rudimentary validation technique, I’ve found it works well enough that I’m happy to roll with it for the moment for internal testing. The key to this approach is that every syntax node is analysed, regardless of where in the script it belongs. This means that regardless of how users write their code, if types belong to illegal assemblies, the validation will always fail!

To develop this further, I have considered potentially using the new Roslyn Source Generator APIs to rewrite illegal APIs, replacing their usage a snippet that will present an in-app notification explaining that an illegal API was called, but this experiment will be for another day.


Providing an in-app API

I decided to create a library focused on abstracting away the complexities of the internal Light Console APIs. A library focused on providing an easy to use for set of APIs for script users.

Whilst most of Light Console’s functionality live in just a single library, LightConsole.Core, this library is extremely complex and requires knowledge of the general architecture. There also exist another 27 libraries which includes 20 UI libraries (Palettes, Colour, Configuration etc.. views all get separate libraries for a clear separation of concerns) and auxiliary libraries for things like command line parsing and fixture data processing.

The UI Views – Separated into descrete projects

The internal API amounts to 1,173 domain specific types, 180 namespaces and 6,932 methods, across 786 source files. It’s not a small API for anyone to learn, let alone a non C# developer. Providing unfettered access to all this would not only cause end-user confusion, but importantly require that I reduce the number of breaking changes I introduce to the internal API.

For this reason, I created a separate library to abstract away the key features of the console’s API that I want to offer script developers. This library has just a single entry point, through the ScriptApi object. This object provides access to the networking, shell, metrics and show APIs, providing the ability for the user to build complex functionality, in a control environment. Lets take a look at one of the Scripting APIs I provide.

Shell API

The Shell API is responsible for the user interface on the console that is executing the script. It lets users change create / delete views, present notifications and generally manage the aspects of the UI. Most of the interesting API belongs within the IWorkspaceScriptApi implementation, as this API is responsible for abstracting workspace and view interaction APIs.

public interface IShellScriptApi 
{
    // Provides workspace APIs including change workspace and add / remove views 
  	IWorkspaceScriptApi Workspaces { get; }
  
  	// Present local notifications. Supports message box & growl 
    INotifcationsScriptApi Notifications {get;}
  
    // Change the UI theme of the console (currently supports Light & Dark)
    void SetTheme(AppTheme theme);
}

Below you can see a small sample of the WorkspaceScriptApi definition and how it provides a simple, but useful set of features for operators to use.

public interface IWorkspaceScriptApi
{
    // Allows for querying the current workspace
  	IWorkspace ActiveWorkspace { get; }
  
    // Provides the ability to change the workspace. 
    void SetActiveWorkspace(IWorkspace workspace);
    // Overloads to allow for changing using only a workspace number 
    void SetActiveWorkspace(int workspaceNumber);
  
  
    /* Add a view to a workspace. Workspace name is from the KnownViews class,
    *  which contains nested classes for view areas, which themselves contain 
    *  enums for the known views. 
    *  If the workspaceNumber is null then the view is added to the active 
    *  workspace.  
    *
    *  Example usage: 
    *  Shell.Workspaces.AddView(KnowViews.Programming.Sheets.Fixtures);
    */
    void AddView(Enum windowNameEnum, int? workspaceNumber = null);
  
    // Overload to make it possible to execute with workspace object
  	void AddView(Enum windowNameEnum, IWorkspace workspace);
    
    // Overload but using string for view name
    void AddView(string viewIdentifier, int? workspaceNumber = null);
   
    // Overload to make it possible to execute with workspace object
  	void AddView(string viewIdentifier, IWorkspace workspace);
      
    //Close the current selected view. 
    void CloseActiveView();
  
    // Get a list of all the known view identifiers
    IReadOnlyList<string> KnownViewIdentifiers { get; }
}


// KnownView example 
 public static class KnownViews
 {
   //Defines the Programming View category
   public static class Programming
   {
     //Provides strongly type options for the available sheets
     public enum Sheets
     {
       FixturePlan,
       Fixtures,
       Programmer,
       Visualiser,
     }
     
     public enum Palettes
     {
       Fixtures,
       Intensity,
       Position,
       Colour,
       Beam,
       Effect
     }

     public enum Editors
     {
       Cue,
       Blade,
       Script,
       Position,
     }

     public enum Colours
     {
       ColourWheel,
       HueColourPicker,
     }

     public enum System
     {
       CommandLine,
     }
   }
}

Example user script

public void Execute()
{
	// Lets find all the parked fixtures in the show	
	var parkedFixtures = Show.Data.Fixtures.Get().Where(x => x.IsParked == true);

	foreach(var parkedFixture in parkedFixtures)
	{
		// Lets print out any a warning for any movers that happen to be parked.
		if(parkedFixture.Attributes.Any(x => x.FeatureGroup == AttributeGroup.PanTilt))
		{
			Shell.Notifications.Local.ShowWarning($"{parkedFixture.Name} {parkedFixture.UserNumber} mover is parked.");
			continue;
		}
		
		// Let's unpark any 8bit generic dimmers
		if(parkedFixture.Attributes.FirstOrDefault().Feature == Attribute.Dimmer &&
		   parkedFixture.DmxChannelFootPrint == 1)
		{
		    parkedFixture.IsParked = false;
			Show.Data.Fixtures.Update(parkedFixture);
			Shell.Notifications.Local.ShowSuccess($"Unparked Fixture {parkedFixture.UserNumber}.");
		}
	}
}

The above shows how users can query the state of fixtures, in this case notifying the user of parked moving lights and unparking generic dimmers.

WPF User Interface

To tie it all of the scripting functionality together, I wanted to provide users with a full-featured code editing experience. As a .NET developer, if intellisense stops working, so do I.

I wanted to provide something familiar for existing C# developers, whilst guiding non-programmers to discover the available console APIs. This meant finding a solution to provide auto-complete (intellisense) was an absolute must, as developing this from scratch is far outside the scope of the project.

For this reason, I looked at the existing options available, both commercial and open source. Below are the best I could find commercially and from the OSS world.

OSS – RoslynPad Editor

RoslynPad

You may have already heard of RoslynPad, an open source C# editor based on both Roslyn and AvalonEdit. The maintainer provides a Nuget package which offers the editor control a standalone WPF component. Whilst he project appears to still be maintained (looking though the commit history), the Nuget package is now incredibly out of date. I’d highly recommend building from source if you intend to use this component in your own apps.

Commercial – AlterNET Code Editor

AlterNet Code Editor – Sample App

The solution in the current branch of the project is built using AlterNet’s code editor. This was by far the best value for money editor component I could find which offered everything I was looking for. One of the things that impressed me most about AlterNET was the speed at which Andrew responds to questions. If you’re looking for an editor with good support, then look no further. It’s available for $499 for the WPF component or $699, which includes a WPF style form designer and Scripter component (which I evaluated before opting to rely on Roslyn).

Learn more

I have included a few links below if you’re interested in learning more about Roslyn & C# scripts in general.

Roslyn Documentation
Roslyn Script samples
Example project from article
RoslynPad

Learn about the DMX project

If you’re interested in learning more about the DMX control system project then check out the dedicated blog I put together.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.