dateo. Coding Blog

Health checks are a good thing - we all should do them at regular intervals. But do you utilize health checks in your ASP.NET applications to their full potential?
Let's take a quick look at how you can gain more insights into your app by implementing custom health checks.

As usual, you can find the related code in a GitHub repository to play around with.

Introduction

Health checks in ASP.NET applications are a very common tool to get a feeling on the status of our apps, especially in containerized environments. Many container platforms like EKS, and therefore also cloud environments like Amazon's EC2, will regularly check specific endpoints to monitor that the instance is performing fine.

But how can we make sure to provide the necessary information from within our application? Can we even customize those and build additional tests suited to our needs? And is that a good idea? Let's find out in this article.

Standard implementation in ASP.NET

Right from the beginning, if you create a new ASP.NET WebApi project, run it and navigate to, say localhost:5041/health or localhost:5041/hc nothing will happen.
First, we'll have to tell the application that we want to use health checks:

builder.Services.AddHealthChecks();
...
app.MapHealthChecks("health");

The first call on the service collection will perform necessary registrations on the DI of the applications.
The second call will map a route of our choice to query the health checks in our applications.

If we now navigate to localhost:5041/health we will be greeted with Healthy and nothing more.
So that is fine, but what is actually checked by default?

Well, nothing. If we were to take a look at the DefaultHealthCheckService, we would see that it simply resolves all registrations of IHealthCheck and queries those. By default, not a single health check is registered; so we will have to do that ourselves.

Implementing custom health checks

Let's start with something silly:

builder.Services
    .AddHealthChecks()
    .AddCheck("TimeCheck", x =>
    {
        return DateTime.Now.Second > 30 
            ? HealthCheckResult.Degraded() 
            : HealthCheckResult.Healthy();
    });

Health checks can be registered both as classes deriving from IHealthCheck as well as lambdas doing very simple checks. This one here will show us degraded for about 50% of the day.
If you take a close look at the console while the application is Degraded, the health check service will also measure the time it took to query all of the registered health checks in parallel:

Health check TimeCheck with status Degraded completed after 0.1255ms with message '(null)'

That might be a good indicator later on that your health checks are doing a bit too much lifting, but more on that later. We can also make sure that we see a proper message when returning a degraded result:

HealthCheckResult.Degraded("I cannot operate in the second half of the minute") 

Also, we can have the same result using a full implementation of this check:

public class TimeHealthCheck : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = new())
    {
        return DateTime.Now.Second > 30 
            ? HealthCheckResult.Degraded("I cannot operate in the second half of the minute") 
            : HealthCheckResult.Healthy();
    }
}

Note: You can also return HealthCheckResult.Unhealthy, indicating something fatal has happened to your application.

This gives us the ability to inject other services via DI into this class, making it possible to create more complex checks on our infrastructure. The registration will be performed as follows:

builder.Services
    .AddHealthChecks()
    .AddCheck<TimeHealthCheck>("TimeCheck");

More useful examples

Let's say we want to implement something a bit more useful on our health check; maybe we'd like to keep an eye on our database connection. Fortunately, the world already provides us with plenty of available health checks, many of which you can find online, for example here. For now, we add a simple SQlite database to our application and check the validity of the connection using the health check AspNetCore.HealthChecks.SqLite
By adding the connection string to our appsettings.json (and of course having a database in place), we can add the health check simply by calling:

builder.Services
    .AddHealthChecks()
    .AddSqlite(connectionString);

Custom responses

Currently, in case of failures, our health check would not give us much information on what is actually wrong. For this, we'd still have to take a look at the logs for example. Fortunately, the customization options give us the chance to adjust the content of the response, turning it from Healthy or Degraded into full JSON-formatted information.
Microsoft provides this snippet for a custom response writer:

### Tag-based mapping 

### Custom responses and response writer
```csharp
public static class ResponseWriter
{
    public static Task WriteResponse(HttpContext context, HealthReport healthReport)
    {
        context.Response.ContentType = "application/json; charset=utf-8";

        var options = new JsonWriterOptions { Indented = true };

        using var memoryStream = new MemoryStream();
        using (var jsonWriter = new Utf8JsonWriter(memoryStream, options))
        {
            jsonWriter.WriteStartObject();
            jsonWriter.WriteString("status", healthReport.Status.ToString());
            jsonWriter.WriteStartObject("results");

            foreach (var healthReportEntry in healthReport.Entries)
            {
                jsonWriter.WriteStartObject(healthReportEntry.Key);
                jsonWriter.WriteString("status",
                    healthReportEntry.Value.Status.ToString());
                jsonWriter.WriteString("description",
                    healthReportEntry.Value.Description);
                jsonWriter.WriteStartObject("data");

                foreach (var item in healthReportEntry.Value.Data)
                {
                    jsonWriter.WritePropertyName(item.Key);

                    JsonSerializer.Serialize(jsonWriter, item.Value,
                        item.Value?.GetType() ?? typeof(object));
                }

                jsonWriter.WriteEndObject();
                jsonWriter.WriteEndObject();
            }

            jsonWriter.WriteEndObject();
            jsonWriter.WriteEndObject();
        }

        return context.Response.WriteAsync(
            Encoding.UTF8.GetString(memoryStream.ToArray()));
    }
}

which we can make use of in the call to MapHealthChecks:

app.MapHealthChecks("/health", new HealthCheckOptions {ResponseWriter = ResponseWriter.WriteResponse});

Now, if we hit our endpoint, we will be greeted by a proper JSON object. I am assuming your browser will automatically render it, but when we switch to the underlying raw data it will look something like this:

{
  "status": "Healthy",
  "results": {
    "sqlite": {
      "status": "Healthy",
      "description": null,
      "data": {}
    }
  }
}

Now that we are this far, let's see how we can feed our own information into this response. For this, we'll reinstate our (stupid) time health check:

public class TimeHealthCheck : IHealthCheck
{
    private readonly TimeSpan Threshold = TimeSpan.FromSeconds(30);

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = new())
    {
        var currentTime = DateTime.Now;
        var data = new Dictionary<string, object>
        {
            {"CurrentTime", currentTime},
            {"Threshold", this.Threshold}
        };

        return DateTime.Now.Second > this.Threshold.Seconds
            ? HealthCheckResult.Degraded("I cannot operate in the second half of the minute", data: data)
            : HealthCheckResult.Healthy(data: data);
    }
}

This time, we are creating a dictionary that we can fill with basically any information we consider useful and pass it on via the HealthCheckResult. Now hitting the endpoint will return

{
  "status": "Degraded",
  "results": {
    "TimeCheck": {
      "status": "Degraded",
      "description": "I cannot operate in the second half of the minute",
      "data": {
        "CurrentTime": "2023-08-18T13:35:47.5327298+02:00",
        "Threshold": "00:00:30"
      }
    },
    "sqlite": {
      "status": "Healthy",
      "description": null,
      "data": {}
    }
  }
}

This is already much more descriptive and can help operators and developers very quickly in dire situations.

Splitting health checks

As soon as developers start seeing the potential of additional health checks, they are tempted to add checks for every infrastructural component they see. Health checks are certainly useful, but we need to be careful about the things we really need to know on a continuous basis, and things that we can take a look at only once in a while.

To achieve this split, ASP.NET provides us with the ability to tag health checks and create multiple endpoints associated with only a subset of checks:

builder.Services
    .AddHealthChecks()
    .AddCheck<TimeHealthCheck>("TimeCheck", tags: new[] {"core"})
    .AddSqlite(connectionString, tags: new[] {"infrastructure"});
...
app.MapHealthChecks("/health", new HealthCheckOptions {Predicate = x => x.Tags.Contains("core")});
app.MapHealthChecks("/health-details", new HealthCheckOptions {Predicate = x => x.Tags.Contains("infrastructure"), ResponseWriter = ResponseWriter.WriteResponse});

In the above snippet, we split our two available health checks into two separate categories. One is for simple and essential checks, so that for example our container engine can check in on our application in regular intervals. The other category is for checks that might require a bit more effort; calling out to databases, checking AWS resources or similar operations would fit into this category. We also map the two categories to two different endpoints, so that we can query the infrastructural checks on demand.

Please be aware that these two groups of health checks are fully independent of each other. If the SQLite check fails, it will not cause /health to be unhealthy. This is something to keep in mind. The proper use of multiple tags here can ensure that checks will show up in more than one category, but there is always a price to pay.

In general, the following things should be kept in mind when adding new health checks:

• Performance impact: Each new health check, especially if queried often, will require resources in your application.
• False positives: Having many health checks can increase the risk of false positives, mainly due to transient issues or inter-component dependencies.
• Signal-to-noise ratio: If multiple health checks are failing, finding the root cause might become difficult. This is also true if too much information is written into the response. Please KISS.

Conclusion

We covered the basics of one of ASP.NET's powerful tools. I hope you can take something from it.
As usual, the code for the examples can be found on GitHub.

Please share on social media, stay in touch via the contact form, and subscribe to our post newsletter!