Skip to content

Observe Performance Tests From Within the Application

Bombardier shows us the view from the outside and answers the question on how fast our application responded. But what happens inside the application? Does everything work as expected? Or did the application only return success as a status code but failed internally? Let us explore our options to find out what is going on.

Use your existing instrumentation

The simplest way to find out what is going on is to use the instrumentation that is already in place. Do you write logfiles? If so, check them to see what goes on. Do you write to the Windows event log? Check in the Event Viewer if something shows up.

In many applications the logs help to explain what went wrong but do often not show what went right. It may catch unhandled exceptions and inform you about a problem after the fact. But can it also show the happy path in enough details to figure out what is going on? Often this concern is overlooked, and we only can follow parts of the user journey. Let us fix that.

Can we change the application?

The main deciding factor on how to get more information about what goes on in our application is whether we can change it.

If we can change the application, we can add additional log messages, switch to Serilog and Seq, or introduce OpenTelemetry. That will give us a ton of additional data and from that we can better follow along what happens inside our application.

Sometimes we are unable to make changes. Either because we got the application as a black box with no way to change it or we may be at a step in the development lifecycle where we cannot quickly make the necessary changes. In this case we need to fall back on instruments like the .Net diagnostic tools.

For the remainder of this post we assume that we can change the application. Next week we dive into the diagnostic tools for the applications we cannot change in a timely manner.

Use Serilog and Seq

I use Serilog and Seq now for more than 10 years. I still find the semantic log messages a great improvement over traditional plain text messages and use Seq excessively to track our apps in production.

Serilog comes with the package Serilog.AspNetCore that allows us to log ASP.NET's internal operations to the same Serilog sinks as our application events. In addition to the explicit log messages we write, we get entries like this one on the duration of a request:

[12:01:47 INF] HTTP GET / responded 200 in 95.0581 ms

To activate this feature, we need to install the package and add a few extra lines to Program.cs:

1
2
3
4
5
6
7
8
9
var name = Assembly.GetExecutingAssembly().GetName().Name ?? "API"
builder.Host.UseSerilog((context, loggerConfig) => {
    loggerConfig
        .ReadFrom.Configuration(context.Configuration)
        .Enrich.WithProperty("Application", name)
        .Enrich.FromLogContext()
        .WriteTo.Seq("http://localhost:5341")
        .WriteTo.Debug();
});

If we log to Seq, we can filter for the new message type and create dashboards to track the request duration.

Use OpenTelemetry

A different approach that we could combine with Serilog & Seq is OpenTelemetry. But be aware that OpenTelemetry writes a ton of messages, and it may be advisable to log them into another store.

We can install these 3 packages to our web application: • OpenTelemetry.Exporter.OpenTelemetryProtocol • OpenTelemetry.Extensions.Hosting • OpenTelemetry.Instrumentation.AspNetCore

To activate OpenTelemetry, we need this configuration in Program.cs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(builder.Environment.ApplicationName))
    .WithLogging(logger => logger.AddOtlpExporter()
    );

const string serviceName = "WebAppLoadDemo";

builder.Logging.AddOpenTelemetry(options =>
{
    options
        .SetResourceBuilder(
            ResourceBuilder.CreateDefault()
                .AddService(serviceName))
        .AddOtlpExporter();
});
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(serviceName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter());

If we do not specify an endpoint for OpenTelemetry, it will log to the default port localhost:18889. We can run a tool like the Aspire dashboard to catch those messages and visualise them. The simplest way to start is to run this command to create a Docker container:

docker run --rm -it -p 18888:18888 -p 4317:18889 -d --name aspire-dashboard \
    mcr.microsoft.com/dotnet/aspire-dashboard:9.0

The more parts of our system use OpenTelemetry, the more details we get into each request. If our web application uses a web API that itself uses another web API before it talks to SQL Server, we can track the request over all systems – nicely combined without any additional work at our end:

We can follow the user flow through multiple systems, like a web application, an API and another API that talks to SQL Server

Next

If we can change our application, we can use Serilog with Seq or OpenTelemetry to see in detail what happens inside our application. Those tools produce a lot of data that we need to turn into information. Do not underestimate the effort it takes at the beginning, especially if you go from nothing to OpenTelemetry. If you keep up with the challenge, you get to know what happens exactly in your application and where it breaks if the load gets too big.

Next week we explore the .Net diagnostic tools to get some insights into all the .Net applications that we cannot change.