DFrame — Distributed load testing framework for writing test scenarios in C#

6 min readMar 16, 2022

I’ve released a new library called DFrame.

DFrame.Controller, which is the real-time web UI, and DFrame.Worker, which writes the load test scenario in C#. Deploying DFrame.Worker in a cluster on the web, with one to thousands of workers working to generate a large number of requests.

The ability to write test scenarios in plain C# also means that you can cover all kinds of communication, not just HTTP/1. WebSocket, HTTP/2, gRPC, MagicOnion, or even Photon or your own TCP transport, even Redis or databases, etc.

DFrame.Worker supports Unity as well. This means that by deploying it on a large number of Headless Unity or device farms, even proprietary communication frameworks that only run on Unity can be load tested.

Significance of writing load test scenarios in C#

There are many load testing frameworks out there. Some of the most popular are ab, jMeter, k6, Artillery, Gatling, wrk, bombardier, Locust, etc. k6, Artillery, and Gatling are available as SaaS, and cloud services also provide load testing frameworks such as Azure Load Testing(Managed jMeter), .NET has officially dotnet/crank framework for load testing.

DFrame is similar to Locust in this context, including its architecture (Controller-Worker configuration, WebUI, etc.). One of the most important of its features is the ability to write scenarios in code. Instead of having the user configure the scenario in a weird UI or write complicated XML, YAML, or JSON, the scenario can be written in plain code. Locust is in Python, but k6 can be written in JavaScript.

The server or client language and the load test scenario writing language should be the same. For example, if you are developing a Unity game (any server-side language is fine), and the Unity game is written in C#, then if you can write the test scenario in C#

There is a client SDK (endpoints and typed Request/Response) equivalent
Fully equivalent to the client implementation so there is game domain logic

This greatly reduces the time and effort required to write test scenarios. If it is Python, JavaScript, Lua, or anyway a different language, the amount of work is incomparably greater.

If the server-side language is C# (ASP.NET), of course, it would be very easy to write the load testing scenario in C#.

DFrameApp.Run

Reference package from and write only one line. There are also lines of description of the test scenario (workload), but still, that’s all.

Now if you open http://localhost:7312 in your browser, you will find SampleWorkload.

Workloads are generated for the number of Concurrency, and ExecuteAsync is executed for the number of Total Request / Workers / Concurrency. As you can see, the code are simple, so you can use gRPC, MagicOnion, or anything else.

There is also SetupAsync for preparation before ExecuteAsync and TeardownAsync for cleanup. A simple gRPC test can be written as follows

It can also accept arguments, so you can create something that passes an arbitrary URL. The constructor can accept parameters or instances injected by DI.

A String url entry point appears on the WebUI screen, and the URL can now be entered.

In addition, if you want to test simple HTTP GET/POST/PUT/DELETE, you can have the IncludesDefaultHttpWorkload enabled, which will add a workload that accepts built-in parameters.

Performance

There are numerous load testing frameworks, but their performance varies considerably. Very detailed on the k6 blog Open source load testing tool review 2020. For example, wrk is tens to hundreds of times faster than others. Performance is very important, and sometimes workers are so inefficient that they can’t handle the load on the target. To cope with this, you will need a large number of machines and higher specs to build clusters.

DFrame is provided as a library and everything is pre-compiled as a C# executable at runtime. There is no such thing as dynamically reading and executing scripts, which slows things down. Performance is quite good because we used C# as it is, which is a fast language, and also because we tuned it quite a bit.

As a comparison, let’s take a look at the following results for ab and k6.

It targets an HTTP/1 server on localhost, running 32 parallel (-c 32, -32VUs, Concurrency=32). Note that for more accurate measurements, the target and worker must be on separate machines. This is because the server side is affected by the CPU due to worker load.

ab is 6287 req/sec, k6 is 125619 req/sec, and DFrame is 207634 req/sec. ab did not perform well in my environment (Windows).

Also, I didn’t provide an image, but unfortunately Locust is quite slow and has very high CPU usage (it’s Python. ……).

REST API for Automation

DFrame has REST Api for automation, For example, /api/connections gives the number of worker connections currently connected. Execution parameters, etc., are in the form of JSON thrown by Post.

Since JSON is only exchanged via the REST API, it can be used from any language, but in the case of C#, a typed client is provided in the DFrame.RestSdk package, so you can get started right away.

Together with “dotnet run”, it may be useful for periodic CI runs.

Runs on Unity

Unity is also supported. Possible uses include controlling a large number of headless Unitys. Even for normal load testing, there is a strategy to build with headless Unity without cutting out the C# for the communication part into a .NET Console App.

Library or Tool

Controller must be built by the user, not by a pre-built exe or Docket Container.

The advantage of having DFrame.Controller provided as a library is that configuration is by far simpler to get on top of the normal code and ASP.NET mechanisms. You can set up any logger you want in DI, specify URLs, SSL, etc. in appsettings. json would be much better than a large number of convoluted command line options.

The log persistence process should also be easier to use if you have it injected by DI normally (there is a thing called IExecutionResultHistoryProvider available, and if you have DI register an implementation of it, you can put the results into a database or time series DB for statistical reference), rather than providing it as a plug-in.

To begin with, since Worker is written in C#, you need to incorporate and build it yourself, so we made it possible to start Controller+Worker cohabitation with only one line of DFrameApp.Run, and made it easy to use by devising the library design.

Controller itself can be started from “Microsoft.NET.Sdk” of the console application template, instead of “Microsoft.NET.Sdk.Web”. It’s simpler that way, right?

Blazor Server + MagicOnion(grpc-dotnet)

DFrame.Controller has a co-located Blazor Server and MagicOnion (grpc-dotnet) architecture.

This is an interesting architecture. Since the Web UI and MagicOnion (Server side) share the same memory, changes in the MagicOnion (Client side) at the edge are directly delivered to the browser through C# alone.

From the server, everything is propagated in real time, including access from the API, and the screens are synchronized, but thanks to this architecture, the implementation is quite simple.

Load testing is important, and even if you don’t need it now, there will come a time when you have to choose something. If you are writing something in C# (server or client), DFrame would be a great help.

I have been launching various applications in C# for over 10 years, both server and client, and I have finally implemented something I have always wanted.

Please try it.