Setting Up OpenTelemetry my .NET API

In this post, I’ll walk you through setting up OpenTelemetry in a .NET API for monitoring and tracing, based on my experience with the HelloOpenTelemetry example. OpenTelemetry is a powerful observability framework for cloud-native software, providing standardized methods for collecting, processing, and exporting telemetry data such as traces, metrics, and logs. Additionally, we’ll cover how to configure Prometheus and Grafana to visualize your telemetry data, like this:

Prerequisites

Before starting, ensure you have the following installed:

• .NET SDK
• Docker
• Docker Compose

1. Install OpenTelemetry NuGet Packages

OpenTelemetry provides various NuGet packages for tracing and monitoring. Install them by running the following commands in your terminal:

dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

2. Configure OpenTelemetry

In this step, we will define a reusable extension method for setting up OpenTelemetry instrumentation and exporters. Create a new file called OpenTelemetryExtensions.cs and add the following code:

using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

namespace HelloOpenTelemetry.Extensions;

public static class OpenTelemetryExtensions
{
    public static void AddCustomOpenTelemetry(this WebApplicationBuilder builder)
    {
        var resourceBuilder = ResourceBuilder.CreateDefault()
            .AddService(serviceName: "helloopentelemetry", serviceVersion: "1.0.0");

        builder.Services.AddOpenTelemetry()
            .WithTracing(tracing => tracing
                .SetResourceBuilder(resourceBuilder)
                .AddSource("HelloOpenTelemetry")
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddConsoleExporter()
            )
            .WithMetrics(metrics => metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddConsoleExporter()
                .AddPrometheusExporter()
            );

        builder.Logging.AddOpenTelemetry(logging => { logging.AddConsoleExporter(); });
    }
}

This code configures OpenTelemetry to handle traces, metrics, and logs, and exports the data to Prometheus.

3. Reference OpenTelemetry in Program.cs

Next, reference the extension method in your Program.cs to and app.UseOpenTelemetryPrometheusScrapingEndpoint() set up OpenTelemetry when the application starts. Update your Program.cs with the following:

using System.Diagnostics;
using HelloOpenTelemetry.Extensions;
using Microsoft.OpenApi.Models;

var builder = WebApplication.CreateBuilder(args);

...

builder.AddCustomOpenTelemetry();

var app = builder.Build();

...

app.UseOpenTelemetryPrometheusScrapingEndpoint();

app.MapGet("/", () =>
{
    return "Hello World!";
})
.WithName("GetHelloWorld")
.WithOpenApi();

app.Run();

Note: app.UseOpenTelemetryPrometheusScrapingEndpoint() sets up an endpoint in the app that exposes the metrics collected by OpenTelemetry in a format that Prometheus can understand. Simply put, it adds a URL (usually /metrics) to your app that Prometheus can regularly visit to grab performance data, like API response times, resource usage, and more. It makes it easy for Prometheus to monitor the app’s health and performance.

4. Prometheus and Grafana Configuration

Now we’re going to configure Prometheus to receive metrics from your .NET API and Grafana to display these metrics using the preconfigured dashboard (aspnetcore.json) that you’ve downloaded from here.

Folder Structure

Your updated project folder structure for Prometheus and Grafana configuration looks like this:Copy codedeploy/
  grafana/
    config/
      provisioning/
        dashboards/
          default.yml
        datasources/
          default.yml
      grafana.ini
    dashboards/
      aspnetcore_endpoint.json
      aspnetcore.json
  prometheus/
    prometheus.yml

1. Configuring Prometheus

The prometheus.yml file inside the prometheus folder defines the scraping configuration for Prometheus. This configuration ensures that Prometheus can collect metrics from your .NET API.

global:
  scrape_interval: 15s  # Scrape every 15 seconds

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "helloopentelemetry"  # Scraping our .NET API
    static_configs:
      - targets: ["helloopentelemetry:8080"]

• scrape_interval: Defines how frequently Prometheus will scrape metrics. Here it’s set to 15 seconds.
• targets: Specifies where Prometheus will fetch the metrics. For the API, it’s helloopentelemetry:8080, which is the internal Docker service.

2. Configuring Grafana

Grafana is configured to load metrics from Prometheus and display them on the dashboard aspnetcore.json. The configuration for Grafana is found in the provisioning folder, which includes two key files: default.yml for data sources and default.yml for dashboards.

a) provisioning/datasources/default.yml

This file configures Prometheus as the data source for Grafana.

apiVersion: 1

datasources:
  - name: Prometheus  # Data source name
    type: prometheus  # Data source type: Prometheus
    access: proxy     # Grafana accesses Prometheus via proxy
    url: http://prometheus:9090  # Prometheus URL inside Docker network
    isDefault: true   # Sets Prometheus as the default data source

• url: This is the endpoint for Prometheus inside the Docker network (prometheus:9090), where Grafana will retrieve the metrics.
• isDefault: Marks Prometheus as the default data source for Grafana.

b) provisioning/dashboards/default.yml

This file tells Grafana where to find the preconfigured dashboards. Here, we reference the aspnetcore.json dashboard inside the dashboards folder.

apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: '.Net'
    type: file
    options:
      path: /var/lib/grafana/dashboards

• path: Specifies the directory inside the Grafana container where dashboards are stored. This will point to /var/lib/grafana/dashboards, which is mapped to your local folder deploy/grafana/dashboards/.

c) grafana.ini

In addition to the provisioning configuration, grafana.ini allows you to customize how Grafana operates. For example, you can configure default users, themes, and other settings. Ensure that this file is correctly configured for your needs.

Steps to Use the Dashboard

1. Place the Dashboard Files: Ensure that the downloaded dashboard files aspnetcore.json and aspnetcore_endpoint.json are placed in the deploy/grafana/dashboards/ folder.
2. Grafana Dashboard Auto-Loading: Grafana will automatically load these dashboards from the dashboards directory thanks to the configuration in provisioning/dashboards/default.yml.

Important update

After downloading the aspnetcore.json dashboard file, I updated it to avoid the Grafana error: «Templating Failed to upgrade legacy queries». Specifically, I replaced "uid": "${DS_PROMETHEUS}" with a sample uid, so, like this: "uid": "4E7B5C8D9FA341A8B". This change addresses an issue where Grafana fails to interpret legacy data source queries properly, ensuring the dashboard loads without errors. By setting a specific UID, the problem with upgrading legacy queries is resolved, allowing the Prometheus data source to function correctly in Grafana.

5. Set Up Docker Compose

To enable telemetry monitoring, you need services like Prometheus and Grafana. Here is an example docker-compose.yml file to configure the required services:

version: "3.4"

services:
  helloopentelemetry:
    image: ${DOCKER_REGISTRY-}helloopentelemetry
    build:
      context: .
      dockerfile: src/HelloOpenTelemetry/Dockerfile
    ports:
      - "5000:8080"
    depends_on:
      - prometheus

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./deploy/prometheus/:/etc/prometheus/

  grafana:
    image: grafana/grafana-oss:latest
    ports:
      - "3000:3000"
    volumes:
      - ./deploy/grafana/config/:/etc/grafana/
      - ./deploy/grafana/dashboards/:/var/lib/grafana/dashboards/
    depends_on:
      - prometheus
networks:
  monitoring:
    driver: bridge

Conclusion

You’ve now configured OpenTelemetry in your .NET API with Prometheus and Grafana for monitoring and tracing. This setup provides a robust way to visualize and trace your application’s performance in real-time.
More details and code, here in my github.

Happy coding!