Sindre Smistad

Tag: Testing

  • Turning Hurl API Flows Into Repeatable Integration Tests With Captured Responses

    We have a .NET API that sits on top of Microsoft Dataverse. Writing integration tests for it has always been a bit of a pain. The API does a lot of orchestration behind the scenes, calling into Dataverse multiple times for a single user request, but we obviously don't want our test suite to need a live Dataverse instance. We want tests that are fast, deterministic, and work offline.
    Here's what we came up with: a three-part approach.
    1. Hurl scripts define end-to-end API test flows that we run against a live instance.
    2. A CapturingHttpMessageHandler sits in the HTTP pipeline and records every backend request/response as numbered JSON files while a hurl flow runs.
    3. A MultiRequestIntegrationTestFactory loads those captured responses and replays them during xUnit integration tests, so we get full coverage of complex multi-step logic without ever touching a real backend.

    The Problem

    Our API has some pretty involved workflows. A couple of examples:
    • A review-and-approval flow: Create a project, retrieve its auto-generated checklists, add items to those checklists (dependencies, focus points, risks), mark them “ready for QA”, then verify that items can’t be modified after approval. This one flow alone triggers 30+ backend requests.
    • A signing workflow with ordering constraints: Create an entity, assign team roles, create child items, sign things in a specific order (ready-for-QA then verified), check that doing it in the wrong order gets rejected, then archive and dearchive while verifying that signatures reset properly.
    We already had both of these flows written as hurl test files that we'd run against a real environment. The question was simple: how do we get these same flows running as automated tests in CI, without needing a live backend?

    Hurl as the Source of Truth

    Hurl is a command-line tool for running HTTP requests defined in plain text files. A .hurl file looks something like this:

    # Create a project
POST https://{{base_url}}/projects
Authorization: Bearer {{access_token}}
Content-Type: application/json
{
  "name": "Test Project",
  "department": "engineering"
}
HTTP 200
[Captures]
project_id: jsonpath "$.projectId"

# Get auto-generated items for the project
GET https://{{base_url}}/projects/{{project_id}}/items
Authorization: Bearer {{access_token}}
HTTP 200
[Captures]
first_item_id: jsonpath "$.items[0].itemId"

# Add a dependency to the first item
POST https://{{base_url}}/item-dependencies
Authorization: Bearer {{access_token}}
{
  "itemId": "{{first_item_id}}"
}
HTTP 200
[Captures]
dependency_id: jsonpath "$['dependencyId']"

# Delete the dependency
DELETE https://{{base_url}}/item-dependencies/{{dependency_id}}
Authorization: Bearer {{access_token}}
HTTP 204
    Hurl files are nice because they're self-contained and readable. Each request can capture values from the response and feed them into the next request. We use these to validate full user journeys against a live environment.
    The realization that kicked this whole thing off: the sequence of backend requests our API makes while processing these hurl flows is exactly the test data we need.

    Capturing Backend Responses

    The CapturingHttpMessageHandler

    We wrote a DelegatingHandler that sits in the HTTP pipeline and intercepts every request the API makes to Dataverse. It saves each request's metadata and response body to a numbered JSON file:
    public sealed class CapturingHttpMessageHandler : DelegatingHandler
{
    private readonly string _outputDirectory;
    private static int _requestCounter;

    public CapturingHttpMessageHandler(string? outputDirectory = null)
    {
        _outputDirectory = outputDirectory
            ?? Path.Combine(Path.GetTempPath(), "CapturedResponses",
                DateTime.Now.ToString("yyyyMMdd_HHmmss"));

        Directory.CreateDirectory(_outputDirectory);
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var response = await base.SendAsync(request, cancellationToken);
        await CaptureResponseAsync(request, response, cancellationToken);
        return response;
    }

    private async Task CaptureResponseAsync(
        HttpRequestMessage request, HttpResponseMessage response,
        CancellationToken cancellationToken)
    {
        var requestNumber = Interlocked.Increment(ref _requestCounter);

        // Read the response body without consuming it
        var contentBytes = await response.Content.ReadAsByteArrayAsync(cancellationToken);
        var contentString = System.Text.Encoding.UTF8.GetString(contentBytes);

        // Restore the response content so downstream code still works
        var contentType = response.Content.Headers.ContentType;
        response.Content = new ByteArrayContent(contentBytes);
        if (contentType != null)
            response.Content.Headers.ContentType = contentType;

        var pathAndQuery = RemoveSelectQueryParam(
            request.RequestUri?.PathAndQuery ?? string.Empty);

        var captured = new CapturedResponse
        {
            RequestNumber = requestNumber,
            Method = request.Method.Method,
            PathAndQuery = pathAndQuery,
            StatusCode = (int)response.StatusCode,
            ResponseBody = contentString,
        };

        var safeFileName = CreateSafeFileName(
            request.RequestUri?.PathAndQuery ?? "unknown");
        var filePath = Path.Combine(
            _outputDirectory, $"{requestNumber:D4}_{safeFileName}_metadata.json");

        var json = JsonSerializer.Serialize(captured,
            new JsonSerializerOptions { WriteIndented = true });
        await File.WriteAllTextAsync(filePath, json, cancellationToken);
    }
}

    How to use it

    You plug this handler into your HTTP client pipeline temporarily. Wherever your HttpClient is configured (usually Program.cs), you add it as a delegating handler:
    services.AddHttpClient("BackendClient", client =>
    {
        client.BaseAddress = new Uri(backendUrl);
        // ... headers ...
    })
    .AddHttpMessageHandler(ConfigureAuthentication(scopes))
    .AddHttpMessageHandler(() =>
    {
        var outputDir = Path.Combine(
            Directory.GetCurrentDirectory(), "CapturedResponses");
        return new CapturingHttpMessageHandler(outputDir);
    });
    Then run your hurl flow against localhost:
    hurl --variable base_url=localhost:5001 \
     --variable access_token=<your-token> \
     my-flow.hurl
    Every backend call the API makes gets saved as a numbered JSON file:
    CapturedResponses/
  0001__api_data_v9.2_projects_metadata.json
  0002__api_data_v9.2_items_metadata.json
  0003__api_data_v9.2_dependencies_metadata.json
  ...
    Each file looks like this:
    {
  "RequestNumber": 1,
  "Method": "POST",
  "PathAndQuery": "/api/data/v9.2/projects",
  "StatusCode": 200,
  "ResponseBody": "{ ... }"
}
    Once you're done capturing, you remove the handler from Program.cs. It's only there for generating the test fixtures.

    Replaying Responses in Integration Tests

    RequestResponseLoader

    The RequestResponseLoader reads all the numbered JSON files from a directory and serves them back in order, matching on Method + PathAndQuery:
    internal class RequestResponseLoader
{
    private readonly List<RequestResponseFileWithIndex> _requestResponseFiles;
    private readonly Dictionary<string, int> _usedIndexes;
    private readonly object _lock = new();

    public RequestResponseLoader(string directoryPath)
    {
        _requestResponseFiles = new List<RequestResponseFileWithIndex>();
        _usedIndexes = new Dictionary<string, int>();

        var jsonFiles = Directory.GetFiles(directoryPath, "*.json");
        Array.Sort(jsonFiles, StringComparer.OrdinalIgnoreCase);

        var index = 0;
        foreach (var filePath in jsonFiles)
        {
            var json = File.ReadAllText(filePath);
            var file = JsonSerializer.Deserialize<RequestResponseFile>(json);
            _requestResponseFiles.Add(new RequestResponseFileWithIndex
            {
                Index = ++index,
                Method = file.Method,
                PathAndQuery = file.PathAndQuery,
                StatusCode = file.StatusCode,
                ResponseBody = file.ResponseBody
            });
        }
    }

    public RequestResponse GetResponse(string pathAndQuery, string method)
    {
        pathAndQuery = RemoveSelectQueryParam(pathAndQuery);
        var key = $"{method}:{pathAndQuery}";

        lock (_lock)
        {
            var matchingFiles = _requestResponseFiles
                .Where(f => f.PathAndQuery == pathAndQuery
                    && f.Method.Equals(method, StringComparison.OrdinalIgnoreCase))
                .ToList();

            if (matchingFiles.Count == 0)
                throw new Exception(
                    $"No captured response found for request: {key}");

            if (!_usedIndexes.TryGetValue(key, out var highestUsed))
                highestUsed = 0;

            var nextFile = matchingFiles
                .FirstOrDefault(f => f.Index > highestUsed);

            if (nextFile.Index == 0)
                throw new Exception(
                    $"No more captured responses for request: {key}");

            _usedIndexes[key] = nextFile.Index;

            return new RequestResponse
            {
                StatusCode = nextFile.StatusCode,
                ResponseBody = nextFile.ResponseBody
            };
        }
    }
}
    The important bit here: when the same endpoint gets called multiple times (say POST /item-dependencies is called three times during a flow), the loader hands back the next unused captured response each time, in the original recorded order.

    MultiRequestIntegrationTestFactory

    The MultiRequestIntegrationTestFactory extends WebApplicationFactory<Program> and swaps out the real IHttpClientFactory with a fake one that routes all requests through the RequestResponseLoader:
    public sealed class MultiRequestIntegrationTestFactory(
    string storageConnectionString, string jsonPath, Guid? testUserId = null)
    : WebApplicationFactory<Program>
{
    private readonly RequestResponseLoader _loader = new(jsonPath);
    private readonly Guid _userId = testUserId
        ?? new Guid("12345678-1234-1234-1234-123456789012");

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureServices(services =>
        {
            services.RemoveAll<IHttpClientFactory>();
            services.AddSingleton<IHttpClientFactory>(_ =>
            {
                var handler = new FakeHttpMessageHandler(req =>
                {
                    // User-lookup requests get a hardcoded response
                    if (req.RequestUri?.PathAndQuery.Contains("systemusers") == true)
                        return CreateUserResponse();

                    // Everything else is served from captured files
                    var response = _loader.GetResponse(
                        req.RequestUri?.PathAndQuery ?? "", req.Method.Method);

                    return new HttpResponseMessage((HttpStatusCode)response.StatusCode)
                    {
                        Content = new StringContent(
                            response.ResponseBody, Encoding.UTF8, "application/json")
                    };
                });

                var client = new HttpClient(handler)
                {
                    BaseAddress = new Uri("https://localhost/api/data/v9.2/")
                };

                var factory = Substitute.For<IHttpClientFactory>();
                factory.CreateClient(Arg.Any<string>()).Returns(client);
                return factory;
            });

            // Set up test authentication
            services.AddAuthentication("Test")
                .AddScheme<AuthenticationSchemeOptions, TestAuthHandler>(
                    "Test", _ => { });
        });
    }
}

    The Test Itself

    With all this wired up, an integration test just mirrors the hurl flow step by step, but everything runs in-memory:
    [Fact]
public async Task FullApprovalWorkflow()
{
    var factory = new MultiRequestIntegrationTestFactory(
        azuriteFixture.GetConnectionString(),
        Path.Combine("Endpoints", "ApprovalWorkflow", "JSON"));

    var client = factory.CreateClient();

    // Step 1: Create project
    var projectId = await CreateProject(client);

    // Step 2: Get auto-generated checklist
    var checklistId = await GetChecklist(client, projectId);

    // Step 3: Add and remove items
    var depId = await CreateDependency(checklistId, client, HttpStatusCode.OK);
    await DeleteDependency(depId, client, HttpStatusCode.NoContent);
    depId = await CreateDependency(checklistId, client, HttpStatusCode.OK);

    // Step 4: Mark ready for QA
    await MarkReadyForQa(checklistId, client, projectId);

    // Step 5: Verify items cannot be added after approval
    await CreateDependency(checklistId, client, HttpStatusCode.OK);
    await DeleteDependency(depId, client, HttpStatusCode.NoContent);

    // ... more steps covering the full workflow ...
}
    Each await sends a request to our in-memory test server, which then makes backend calls. Those backend calls get answered by the captured JSON files. So the test ends up covering the full orchestration logic, validators, state machines, and error handling, all without any external dependencies.

    Trade-offs and Lessons Learned

    Parallel Requests (Task.WhenAll) and Non-Deterministic Ordering

    This was the biggest pain point. Our API sometimes fires off multiple backend requests in parallel using Task.WhenAll. For example, when archiving an entity we might PATCH five related records at the same time. The problem is that the order these parallel requests hit the HTTP handler is non-deterministic.
    The captured files are numbered in whatever order they happened during the original capture. But during replay, those same parallel requests might arrive in a completely different order. Since the RequestResponseLoader serves responses sequentially by index, a different arrival order means the wrong response can get returned.
    Our workaround: when a PATCH request can't find a matching captured response (which usually means it's a fire-and-forget side-effect from a parallel batch), we just return a `200 OK` with an empty JSON body instead of throwing:
    catch (Exception)
{
    // For PATCH calls that are side-effects in parallel batches,
    // return a success response instead of crashing
    if (req.Method == HttpMethod.Patch)
    {
        return new HttpResponseMessage(HttpStatusCode.OK)
        {
            Content = new StringContent("{}", Encoding.UTF8, "application/json")
        };
    }
    throw;
}
    This isn't ideal because it can hide real failures for PATCH requests. A better solution would be matching responses by request body content or using some more flexible matching strategy. But for now it works because the parallel PATCHes in our case are side-effect operations and nothing in our code actually reads their responses.

    Stripping $select Query Parameters

    We kept running into a frustrating issue: tests would break whenever someone added a new column to a $select clause in a backend query. This happened all the time. A frontend feature needs a new field displayed, so a developer adds it to the $select. But the captured response was stored with the old PathAndQuery, and now the paths don't match anymore.
    The fix was simple: both the CapturingHttpMessageHandler and the RequestResponseLoader strip out $select parameters before storing or matching:
    private static string RemoveSelectQueryParam(string pathAndQuery)
{
    if (string.IsNullOrEmpty(pathAndQuery) || !pathAndQuery.Contains('?'))
        return pathAndQuery;

    var parts = pathAndQuery.Split('?', 2);
    var path = parts[0];
    var query = parts[1];

    var queryParams = query.Split('&')
        .Where(p => !p.StartsWith("$select=", StringComparison.OrdinalIgnoreCase))
        .ToArray();

    return queryParams.Length == 0
        ? path
        : $"{path}?{string.Join("&", queryParams)}";
}
    This makes the tests immune to those kinds of changes. The selected columns don't affect any business logic; they only control which fields the backend includes in the response. And the captured response has the full object in it regardless. Once we added this, a whole category of annoying false failures just went away.

    Numbered Files Preserve Request Ordering

    The captured JSON files are numbered (0001_, 0002_, etc.), and the loader sorts them by filename before building the lookup index. So:
    • When the same endpoint is hit multiple times, each call gets the next captured response in sequence.
    • The ordering is stable and reproducible across runs, as long as the API’s own call ordering is deterministic (see point 1 for when it isn’t).

    This Is a One-Time Capture, Not a Record-Replay Framework

    One thing worth calling out: we do **not** capture responses on every test run. The capture is a one-time, manual step:
    1. Start the API locally with CapturingHttpMessageHandler enabled.
    2. Run the hurl flow against it.
    3. Copy the captured JSON files into the test project.
    4. Remove the handler from Program.cs.
    5. Commit the JSON files as test fixtures.
    So the test data is a snapshot. If the backend schema changes in a big way, you re-capture. In practice this rarely happens because the $select stripping handles most of the day-to-day query changes, and the response structure tends to stay stable.