Automated testing

Our automated testing landscape

There are two main classes of automated tests:

• Tests for monitoring

  • they monitor an environment (production being the most important) to make sure it keeps on functioning as expected

  • we have smoke tests and what we call smoulder tests (the latter ones run less frequently and they monitor mainly the health of the Antivirus API)

• Regression tests

  • they try and ensure that changes to the code that we plan to release to production don’t introduce bugs

(In this section we are not going to consider performance tests, which are also automated tests).

Broadly speaking, there are two types of regression tests: we might call them environment-wide and in-app tests:

• environment-wide tests

  • they stand up the whole environment (that is, all the apps)

  • they do not mock dependencies (e.g. Notify, Dun & Bradstreet)

  • they live in a dedicated code repository (as opposed to the repositories of the apps)

  • there are two sets:

    • what we may call end-to-end tests (often referred to as functional tests in the team)

    • visual regression tests

• in-app tests

  • they live in the same repository as the app they test (e.g. tests for the user frontend app)

  • they mock dependencies

  • they are of two types:

    • tests that do not need to stand up the app under test

      • they test individual software components

      • we may call these unit tests

    • tests that stand up the app under test

      • they test how the app responds to requests

      • we may call these integration tests

(diagram source)

Guidelines for automated testing

Expensive tests

Keep the use of expensive tests to the minimum - use them only when strictly necessary.

A test is expensive when:

• it takes long to complete

• it’s not reliable (it is flaky or brittle) - it routinely fails w/o genuine issues in the test subject

In practical terms, a test is expensive if:

• it’s not automated :-0

• it tests behaviours via the user interface (normally with a browser)

• it needs to stand up some of the apps that make up the service (in general, the more the apps the more expensive)

This is well explained by Martin Fowler’s test pyramid.

Practical considerations for our regression testing

Applied to our landscape, this means that environment-wide tests should be used as the last resort - they are expensive and difficult to run on a developer laptop.

If you need to write a environment-wide test, follow these principles:

• keep it minimal

• use it only to test the “happy path”; other conditions should be tested with in-app tests as much as possible

Finally, keep the testing frameworks up-to-date (as you do with other types of dependencies).