Consumer-Driven Contract Testing

Context

Microservices are being adopted in the project. The internals of the microservices are hidden behind their APIs. The APIs of microservices need occasional or frequent changes to accommodate the implementation of new features and refactorings.

Further, all of the following criteria apply:

1. The consumers of the microservice APIs are known for testing the APIs; it is not a public API.
2. The API consumers and their needs should drive the APIs of the providing microservices instead of API providers specifying their APIs by themselves or using API negotiations.
3. Feedback on integration issues should be given earlier, favoring isolated tests over integration tests for testing interoperability; functional tests are viewed as a separate topic.
4. Resources are given to consistently learn, set up, and apply the technique throughout the project.

Problem

• There is uncertainty about whether an API change could break consumers. This either leads to a reduced innovation speed or, e.g., if API versioning is preemptively applied, to API evolution complexity and effort.

• Traditional integration testing to ensure compatibility requires coordination between teams and is expensive. Thus, tests do not ensure syntactic interoperability among microservices in every service interaction. The result is occasional or frequent deployments of API changes that break consuming microservices in the production environment, leading to service outages.

Solution

Use consumer-driven contract tests instead of traditional integration tests between microservices to test for syntactic interoperability.

Instead of testing both sides against the same interface specification, CDCT divides the testing into two isolated steps. The first step is running a consumer test, resulting in a contract file that encodes the consumer's expectations towards the API. The second step is the provider test, which replays the interactions encoded in the contract file against the provider microservice API. In this two-step process, the compatibility of the API-providing microservice is tested with the expectation of a consumer.

The technical decoupling of creating consumer expectations and testing the API of the provider does not require the consumer and provider to run simultaneously, leading to several advantages:

• The test isolation leads to a more stable and deterministic test environment, speeds up test execution and enables faster feedback cycles.
• CDCT can partially replace integration tests (which typically take longer to run). CDCT focuses on syntactic interoperability and does not test functional aspects, so they are only complementary to integration tests.
• If applied consistently, it becomes easily trackable which consumers use which APIs of a microservice.
• Contracts can serve as communication tools among teams as they make consumer expectations explicit. Like this, API consumers are empowered to drive APIs according to their actual needs.
• Introducing CDCT leads to revisiting existing consumer and API design, potentially improving their quality.
• Consumer tests validate the logic of adapter logic on the consumer side in isolation.

Example

For an illustrative example, we use Pact for a fictive microservice interaction where one microservice gets patient data like the name via a REST API from another microservice. The code snippets show an example of a consumer test written in C#, using the Pact.Net library.

API Consumer Side

public class PatientApiConsumerTests
{

    public PatientApiConsumerTests(ITestOutputHelper output)
    {
        // Setup pact library
        // ...
    }

    [Fact]
    public async Task GetPatient_WhenTheTesterPatientExists_ReturnsThePatient()
    {
        // Arrange
        _pactBuilder
            .UponReceiving("A GET request to retrieve the patient")
                .Given("There is a patient with id 'tester'")
                .WithRequest(HttpMethod.Get, "/patients/tester")
                .WithHeader("Accept", "application/json")
            .WillRespond()
                .WithStatus(HttpStatusCode.OK)
                .WithHeader("Content-Type", "application/json; charset=utf-8")
                .WithJsonBody(new
                {
                    id = "tester",
                    firstName = "John",
                    lastName = "Doe"
                });

        await _pactBuilder.VerifyAsync(async ctx =>
        {
            // Act
            var client = new PatientApiClient(ctx.MockServerUri);
            var patient = await client.GetPatient("tester");

            // Assert
            Assert.Equal("tester", patient.Id);
        });
    }
}

The following diagram contains the corresponding schematic sequence diagram to illustrate how the interaction with the Pact library works along general lines. First, the consumer test sets up a mock server and instructs how it should react to a given request. Then, the consumer test uses the adapter implementation to trigger the actual request, but it is directed to the mock server. The mock server responds according to the prior configuration. Finally, the consumer test validates the correct parsing and handling of the adapter functionality. Besides the consumer test results, the contract file is emitted as a secondary output based on the mock server interactions.

Output: Pact File containing consumer interactions and the expected responses - a single consumer contract. This Pact File is passed to the provider tests to verify if the consumer expectations are in line with the implementation of the provider.

API Provider Side

public class PatientApiTests : IClassFixture<PatientApiFixture>
{
    private readonly PatientApiFixture fixture;
    private readonly ITestOutputHelper output;

    public PatientApiTests(PatientApiFixture fixture, ITestOutputHelper output)
    {
        this.fixture = fixture;
        this.output = output;
    }

    [Fact]
    public void EnsurePatientApiHonoursPactWithConsumer()
    {
        //Arrange
        var config = new PactVerifierConfig
        {
            // Configure pact
            // ...
        };

        string pactPath = Path.Combine("..", "..","path", "to", "pacts", "Patient API Consumer-Patient API.json");

        // Act / Assert
        IPactVerifier pactVerifier = new PactVerifier(config);
        pactVerifier
            .ServiceProvider("Patient API", fixture.ServerUri)
            .WithFileSource(new FileInfo(pactPath))
            .WithProviderStateUrl(new Uri(fixture.ServerUri, "/provider-states"))
            .Verify();
    }
}

The following figure illustrates how the schematic interaction with the Pact library works on the API provider side. After reading the contract file, Pact is configured with functionality to initialize the different provider states. Under the hood, the pact library will emulate API requests based on the contract file generated by the consumer tests and match the responses to the expected ones to verify whether the API provider conforms to the consumer's expectations.

Maturity

Evaluated.

Sources of Evidence

L3:

• replaced integration tests with cdc tests
  • => independent testing of each service
  • uses consumer's expectations
  • (+) minimized inter-team coordination
    • => enabled forming smaller teams
    • => independent testing enabled to deploy services independently
  • (-) more complex testing strategy
• consumer-driven contracts could increase confidence in the team's responsible for service
  • most of its customers are satisfied if contracts olds

L12:

• same as gs3

L14:

• Context: additional concerns at otto.de not covered in paper
  • customer-driven contracts (among others)

L22:

• Consumer-driven contracts as complementary practice to microservices

L32:

• devs of consuming service write contract tests
  • to ensure producing services meet their expectations
• contract tests also run in pipeline of producing services
  • => devs of producing services know whether changes will break expected contracts of consumers
• => safety net for contract changes

L54:

• Cites consumer-driven contracts as alternative to service versioning to solve cross-configuration
  • cross-configuration = config change on service (e.g. IP) impacts other services

Interview A:

• Contracts when a new service consumption is implemented
  • contract over their REST interface
  • Tools like Swagger
  • Contract testing as one of the upcoming topics

Interview B:

• Where happens testing?
• Theoretically, if I test APIs diligently
  • e.g. automated CDC tests
  • and communicate every contract change
  • and inspect if contracts are still fulfilled or not
  • => detached testing possible, heavily automated via CI/CD
  • (still need for end-to-end tests)
• CDC add to preventing failures to make their way into production

Interview D:

• Advises to use CDC testing (over traditional integration tests)
  • if you want to be truly independent
  • need to think about mocks and stubs

Exit Interview (summary excerpt) CDCT-focused Action Research Study:

• Question 1.5: Does the following statement reflect your experience? If not, how would you rephrase it? Use consumer-driven contract testing when all of these apply: a) the pool of API consumers is limited, b) the API consumers and their needs should drive the APIs of the providing microservices (instead of API providers specifying the API by themselves), c) isolated tests are favored over integration tests, and d) resources are given to learn, set up, and apply the testing technique consistently. How does your new approach differ from CDCT? How does your new solution address your challenges compared to CDCT?
  • a)
    • Suggests to rephrase to “the pool of event consumers is limited” to resemble their case
    • Was not taken into account
    • Varies greatly for their events whether applies (also broadcast events)
      • Wouldn’t have left out those messages
    • Interviewer suggests “known” instead of “limited”
      • Fair
      • If consumer not known would run into problems
      • Then it is a valid statement!
  • b)
    • Not really true, it does not capture commands
    • Also for messages: more coordinated way to define contracts
    • Often more than one consumer
    • Interviewer probes whether CDCT is still a good solution even though not following consumer-driven API design?
      • CDCT is good if organizational setup maps well to it
      • Not the case for them
      • Imagines fair if setup is more decoupled, more decoupled events
      • Did not map well to their processes
  • c)
    • In general true
    • However, it is not substitution
      • Only takes away small part of integration testing; effort remains from functional perspective
      • Only figure out if they have a chance to work together because interfaces are matching
    • Would rephrase it is more about receiving feedback that can be provided earlier
      • Should be earlier provided on a lower test level
      • But no substitution
  • d)
    • Pact has good documentation
      • But focusing on HTTP interactions
      • Messaging a little bit left out, documentation scarce, examples scattered
      • Changes between versions of library
      • Effort to get started
      • Hard to define contracts with the given syntax
        • Would have created internal framework on top of Pact to make it more usable
        • Generator to generate consumer tests given a class or an object
        • This framework size same as complete solution as for new approach
        • Easier when not everybody needs to know how to write a test
  • Interviewer probes via observations: three things that made CDCT unappealing; (1) criterion b, did not align with org setup, no consumer-driven API definition; (2) criterion d, had resources to set everything up but not to apply it consistently or different prioritization; (3) the commands that did not fit scheme
    • Adds bad timing as another point
    • Agrees to this statement