Need some help with your project? Contact me

Simple Guzzle API mocking for functional testing in Drupal 8

In this article I am going to show you a technique I used recently to mock a relatively simple external service for functional tests in Drupal 8.

Imagine the following scenario: you have an API with one or a few endpoints and you write a service that handles the interaction with it. For example, one of the methods of this service takes an ID and calls the API in order to return the resource for that ID (using the Guzzle service available in Drupal 8). You then cast the Guzzle response stream to a string and return whatever from there to use in your application. How can you test your application with this kind of requirements?

The first thing you can do is unit test your service. In doing so, you can pass to it a mock client that can return whatever you set to it. Guzzle even provides a MockHandler that you can use with the client and specify what you want returned. Fair enough. But what about things like Kernel or Functional tests that need to use your client and make requests to this API? How can you handle this?

It’s not a good idea to use the live API endpoint in your tests for a number of reasons. For example, your testing pipeline would depend on an external, unpredictable service which can go down at any moment. Sure, it’s good to catch when this happens but clearly this is not the way to do it. Or you may have a limited amount of requests you can make to the endpoint. All these test runs will burn through your budget. And let’s not forget you need a network connection to run the tests.

So let’s see an interesting way of doing this using the Guzzle middleware architecture. Before diving into that, however, let’s cover a few theoretical aspects of this process.

Guzzle middlewares

A middleware is a piece of functionality that can be added to the pipeline of a process. For example, the process of turning a request into a response. Check out the StackPHP middlewares for a nice intro to this concept.

In Guzzle, middlewares are used inside the Guzzle handler stack that is responsible for turning a Guzzle request into a response. In this pipeline, middlewares are organised as part of the HandlerStack object which wraps the base handler that does the job, and are used to augment this pipeline. For example, let’s say a Guzzle client uses the base Curl handler to make the request. We can add a middleware to the handler stack to make changes to the outgoing request or to the incoming response. I highly recommend you read the Guzzle documentation on handlers and middlewares for more information.

Guzzle in Drupal 8

Guzzle is the default HTTP client used in Drupal 8 and is exposed as a service (http_client). So whenever we need to make external requests, we just inject that service and we are good to go. This service is instantiated by a ClientFactory that uses the default Guzzle handler stack (with some specific options for Drupal). The handler stack that gets injected into the client is configured by Drupal’s own HandlerStackConfigurator which also registers all the middlewares it finds.

Middlewares can be defined in Drupal as tagged services, with the tag http_client_middleware. There is currently only one available to look at as an example, used for the testing framework: TestHttpClientMiddleware.

Our OMDb (Open Movie Database) Mock

Now that we have an idea about how Guzzle processes a request, let’s see how we can use this to mock requests made to an example API: OMDb.

The client

Let’s assume a module called omdb which has this simple service that interacts with the OMDb API:

<?php

namespace Drupal\omdb;

use Drupal\Core\Site\Settings;
use Drupal\Core\Url;
use GuzzleHttp\ClientInterface;

/**
 * Client to interact with the OMDb API.
 */
class OmdbClient {

  /**
   * @var \GuzzleHttp\ClientInterface
   */
  protected $client;

  /**
   * Constructor.
   *
   * @param \GuzzleHttp\ClientInterface $client
   */
  public function __construct(ClientInterface $client) {
    $this->client = $client;
  }

  /**
   * Get a movie by ID.
   *
   * @param \Drupal\omdb\string $id
   *
   * @return \stdClass
   */
  public function getMovie(string $id) {
    $settings = $this->getSettings();
    $url = Url::fromUri($settings['url'], ['query' => ['apiKey' => $settings['key'], 'i' => $id]]);
    $response = $this->client->get($url->toString());
    return json_decode($response->getBody()->__toString());
  }

  /**
   * Returns the OMDb settings.
   *
   * @return array
   */
  protected function getSettings() {
    return Settings::get('omdb');
  }

}

We inject the http_client (Guzzle) and have a single method that retrieves a single movie from the API by its ID. Please disregard the complete lack of validation and error handling, I tried to keep things simple and to the point. To note, however, is that the API endpoint and key is stored in the settings.php file under the omdb key of $settings. That is if you want to play around with this example.

So assuming that we have defined this service inside omdb.services.yml as omdb.client and cleared the cache, we can now use this like so:

$client = \Drupal::service('omdb.client');
$movie = $client->getMovie('tt0068646');

Where $movie would become a stdClass representation of the movie The Godfather from the OMDb.

The mock

Now, let’s assume that we use this client to request movies all over the place in our application and we need to write some Kernel tests that verify that functionality, including the use of this movie data. One option we have is to switch out our OmdbClient client service completely as part of the test, with another one that has the same interface but returns whatever we want. This is ok, but it’s tightly connected to that test. Meaning that we cannot use it elsewhere, such as in Behat tests for example.

So let’s explore an alternative way by which we use middlewares to take over any requests made towards the API endpoint and return our own custom responses.

The first thing we need to do is create a test module where our middleware will live. This module will, of course, only be enabled during test runs or any time we want to play around with the mocked data. So the module can be called omdb_tests and we can place it inside the tests/module directory of the omdb module.

Next, inside the namespace of the test module we can create our middleware which looks like this:

<?php

namespace Drupal\omdb_tests;

use Drupal\Core\Site\Settings;
use GuzzleHttp\Promise\FulfilledPromise;
use GuzzleHttp\Psr7\Response;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

/**
 * Guzzle middleware for the OMDb API.
 */
class OmdbMiddleware {

  /**
   * Invoked method that returns a promise.
   */
  public function __invoke() {
    return function ($handler) {
      return function (RequestInterface $request, array $options) use ($handler) {
        $uri = $request->getUri();
        $settings = Settings::get('omdb');

        // API requests to OMDb.
        if ($uri->getScheme() . '://' . $uri->getHost() . $uri->getPath() === $settings['url']) {
          return $this->createPromise($request);
        }

        // Otherwise, no intervention. We defer to the handler stack.
        return $handler($request, $options);
      };
    };
  }


  /**
   * Creates a promise for the OMDb request.
   *
   * @param RequestInterface $request
   *
   * @return \GuzzleHttp\Promise\PromiseInterface
   */
  protected function createPromise(RequestInterface $request) {
    $uri = $request->getUri();
    $params = \GuzzleHttp\Psr7\parse_query($uri->getQuery());
    $id = $params['i'];
    $path = drupal_get_path('module', 'omdb_tests') . '/responses/movies';

    $json = FALSE;
    if (file_exists("$path/$id.json")) {
      $json = file_get_contents("$path/$id.json");
    }

    if ($json === FALSE) {
      $json = file_get_contents("$path/404.json");
    }

    $response = new Response(200, [], $json);
    return new FulfilledPromise($response);
  }

}

Before explaining what all this code does, we need to make sure we register this as a tagged service inside our test module:

services:
  omdb_tests.client_middleware:
    class: Drupal\omdb_tests\OmdbMiddleware
    tags:
      - { name: http_client_middleware }

Guzzle middleware services in Drupal have one single (magic) method called __invoke. This is because the service is treated as a callable. What the middleware needs to do is return a (callable) function which gets as a parameter the next handler from the stack that needs to be called. The returned function then has to return another function that takes the RequestInterface and some options as parameters. At this point, we can modify the request. Lastly, this function needs to make a call to that next handler by passing the RequestInterface and options, which in turn will return a PromiseInterface. Take a look at TestHttpClientMiddleware for an example in which Drupal core tampers with the request headers when Guzzle makes requests during test runs.

So what are we doing here?

We start by defining the first two (callable) functions I mentioned above. In the one which receives the current RequestInterface, we check for the URI of the request to see if it matches the one of our OMDb endpoint. If it doesn’t we simply call the next handler in the stack (which should return a PromiseInterface). If we wanted to alter the response that came back from the next handler(s) in the stack, we could call then() on the PromiseInterface returned by the stack, and pass to it a callback function which receives the ResponseInterface as a parameter. In there we could make the alterations. But alas, we don’t need to do that in our case.

A promise represents the eventual result of an asynchronous operation. The primary way of interacting with a promise is through its then method, which registers callbacks to receive either a promise's eventual value or the reason why the promise cannot be fulfilled.

Read this for more information on what promises are and how they work.

Now for the good stuff. If the request is made to the OMDb endpoint, we create our own PromiseInterface. And very importantly, we do not call the next handler. Meaning that we break out of the handler stack and skip the other middlewares and the base handler. This way we prevent Guzzle from going to the endpoint and instead have it return our own PromiseInterface.

In this example I decided to store a couple of JSON responses for OMDb movies in files located in the responses/movies folder of the test module. In these JSON files, I store actual JSON responses made by the endpoint for given IDs, as well as a catch-all for whenever a missing ID is being requested. And the createPromise() method is responsible for determining which file to load. Depending on your application, you can choose exactly based on what you would like to build the mocked responses.

The loaded JSON is then added to a new Response object that can be directly added to the FulfilledPromise object we return. This tells Guzzle that the process is done, the promise has been fulfilled, and there is a response to return. And that is pretty much it.

Considerations

This is a very simple implementation. The API has many other ways of querying for data and you could extend this to store also lists of movies based on a title keyword search, for example. Anything that serves the needs of the application. Moreover, you can dispatch an event and allow other modules to provide their own resources in JSON format for various types of requests. There are quite a lot of possibilities.

Finally, this approach is useful for “simple” APIs such as this one. Once you need to implement things like Oauth or need the service to call back to your application, some more complex mocking will be needed such as a dedicated library and/or containerised application that mocks the production one. But for many such cases in which we read data from an endpoint, we can go far with this approach.

Further information on the topic?

Comments

This is an interesting approach I hadn't thought of, thanks for sharing. I recently struggled with finding a way to mock API interactions on a Drupal 8 site with Behat tests as well. I spent a lot of time trying to make PHP-VCR (https://github.com/php-vcr/php-vcr) work, which I think is a really nice solution. I just couldn't get it working right with Behat though =/.

Thanks. The good thing about this is that you can abstract all this logic into a test module and whenever that test module is enabled, no requests actually exist the Drupal site to the API. Be that Kernel, Functional or Behat tests. One solution for all testing use cases.

And it's just as flexible as creating a mock client that returns it's own responses.

Where it does not really work is when you have complex APIs where you need 2 way communication (such as for authentication). But hey.. :)

I've done something similar, but with using a service provider to change out the client. I'm now thinking about rewriting these as middlewares.

Either of these approaches should work fine with Behat tests - you just have to make sure to enable the test module in a @BeforeFeature step. I've done it before but don't have any public code I can share.

The trickiest bit I've had is mocking inside of Functional or FunctionalJavascript tests, where you need to mock requests that are triggered by UI actions. Since they are in a separate thread, any alterations you do to the client in your test (such as with a MockHandler middleware) won't apply. The above works around this by having the test store it's queue of mock responses in the state table, which the client uses to read it's queue of responses from.

Where it does not really work is when you have complex APIs where you need 2 way communication (such as for authentication). But hey.. :)

You can mock responses with a callable (or with a middleware like you have) and hack into the authentication calls there. For example, in one project we just returned a 200 response with a response token in the body, unless the Request object had INVALID-CREDENTIAL as the API token. If the response requires something like a current date or expiration time, the callable can do that too, instead of just returning a static fixture from a file.

One more bit I've found has helped, especially when writing true functional tests that do hit a live API, is to inject a middleware for request and response logging. In particular, this helps with capturing test fixtures, because Guzzle can log complete requests as a string that can be parsed back in by your test with \GuzzleHttp\Psr7\parse_request. If you want to do this from Drupal, the cuzzle module can help.

Thanks for the article!

Add new comment

You must have Javascript enabled to use this form.