Asynchronous browser tests with Phoenix (making our tests faster 🚀)

6 March 2021

Recently I have been working on a project using Elixir and its web framework, Phoenix. I have enjoyed learning many new things but have thus far been too lazy to write about them. However there’s a time for everything, so let me tell you about something interesting I did this week!

In our project we are using Wallaby for full-stack browser testing, but the technique discussed here is not specific to Wallaby.

We are also using Mox to mock and stub various calls to external APIs during our tests. Unfortunately that meant we had many Wallaby tests that could not be run asynchronously.

When Wallaby fires up a browser, the browser makes a request to the application’s Plug endpoint. The request is handled by a different process than the one that is running the test. If two tests are running at the same time, and they both mock out the same function, how is Mox to know which mock to use? The HTTP request could have originated from either test.

The easy and slow solution is to put Mox into global mode, but that means we cannot run those tests asynchronously. We did this until now, but I wanted to add a “clock” mock to allow us to stub out the current time for time-dependent tests. This would force many more tests to become synchronous, so I wanted to find another way.

The better solution is to use Mox.allow/3 to share expectations and stubs between processes. The process that defines an expectation or stub is the ‘owner’ of that definition. We can then explicitly share the definition with another process:

Mox.allow(MyMock, owner_pid, pid_to_share_with)

This can be done in any process, because we’re passing in the relevant PIDs explicitly. This fact will prove useful later on!

So my problem was this: how can a test know the PID of the process that will accept the HTTP request that it is triggering? It can’t. That process doesn’t even exist at the moment that we’d want to know its PID.

I was vaguely aware that Ecto has a sandbox for database connections, but had never really dug into the details to properly understand how it works. Somehow, Wallaby enables the database connection used in a test to also be used by the process receiving an HTTP request from the browser. The Phoenix generators and Wallaby.Feature set this up for us so it’s not something I’d given a lot of thought to previously.

The Wallaby docs talk about adding the Phoenix.Ecto.SQL.Sandbox plug to your application’s endpoint in order to enable asynchronous tests. It took me a minute to realise that this module is in the phoenix_ecto package, and is not the same thing as Ecto.Adapters.SQL.Sandbox, which is in ecto_sql package.

Reading the Wallaby.Feature source code, I could see that for an asynchronous test it was basically doing two things when a test runs.

Firstly, its checks out a database connection, making the test process the owner of that connection:

:ok = Ecto.Adapters.SQL.Sandbox.checkout(repo)

Then, it calls Phoenix.Ecto.SQL.Sandbox.metadata_for/2 to get some “metadata”:

Phoenix.Ecto.SQL.Sandbox.metadata_for(repos, self())

self() gives the PID of the current process, which is the one running the test. Hmmm…

That metadata is passed through several more layers, but eventually winds up being passed to the module which orchestrates whichever browser we’re running our test with. In our case, that was Wallaby.Chrome, and the metadata (which is a map at this point) is encoded to a string and added to the browser’s User-Agent header that it sends when making HTTP requests:

user_agent =
  Metadata.append(
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36",
    opts[:metadata]
  )

The metadata generated by Phoenix.Ecto.SQL.Sandbox.metadata_for/2 contains two pieces of information: the Ecto repo module (which is an atom) and the pid that is passed in:

def metadata_for(repo_or_repos, pid) when is_pid(pid) do
  %{repo: repo_or_repos, owner: pid}
end

So, Wallaby is generating the metadata with the pid of the process running our test, and that putting that metadata into the User-Agent header that will be sent to the application’s endpoint.

What happens when the request hits the endpoint? Well, the Phoenix.Ecto.SQL.Sandbox plug picks it up here:

def call(conn, %{header: header, sandbox: sandbox}) do
  _result =
    conn
    |> extract_metadata(header)
    |> allow_sandbox_access(sandbox)

  conn
end

It parses out the metadata from the User-Agent header, and then allow_sandbox_access() effectively does this:

Ecto.Adapters.SQL.Sandbox.allow(repo, owner_pid_from_metadata, self())

The process receiving the HTTP request is able to obtain the pid of the process running the test that triggered it (the “owner”), and then use Ecto.Adapters.SQL.Sandbox.allow/4 to share the owner’s database connection with itself.

This is very similar to the Mox.allow/3 call we want to make! ✨

I initially thought I would need to use Phoenix.Ecto.SQL.Sandbox as inspiration for a new, custom plug that would call Mox.allow/3. I could re-use the public functions of Phoenix.Ecto.SQL.Sandbox for some of the heavy lifting. But then I realised that Phoenix.Ecto.SQL.Sandbox doesn’t have to call Ecto.Adapter.SQL.Sandbox.allow() – it has a :sandbox option that I could use to make it call an allow() function on any given module!

So I defined my own sandbox module, in test/support/sandbox.ex, like this:

defmodule MyApp.Sandbox do
  def allow(repo, owner_pid, child_pid) do
    Ecto.Adapters.SQL.Sandbox.allow(repo, owner_pid, child_pid)
    Mox.allow(MyMock, owner_pid, child_pid)
  end
end

I then altered config/test.exs to change this:

config :my_app, :sql_sandbox, true

To this:

config :my_app, :sandbox, MyApp.Sandbox

And updated lib/endpoint.ex:

if sandbox = Application.get_env(:my_app, :sandbox) do
  plug Phoenix.Ecto.SQL.Sandbox, sandbox: sandbox
end

(I removed the sql_ prefix because it’s no longer purely about the database.)

And voila, I was able to make all our Mox-using browser tests asychronous. This almost halved the time it took to run mix test on our CI! 😀