Email integration

We integrate with two different email providers. These integrations are used directly in our applications but also in our scripts running on Jenkins. Integration is facilitated by our own email clients.

Details for how to create an account and login to an email provider can be found in Adding and removing access for new starters / leavers.

Note, email addresses that exist on local, preview and staging databases should come from a sanitised source (generally the cleaned dumps) so if an email is sent to it, then it will not get delivered to a real user (e.g. functional-test@user.marketplace.team, 10405@user.marketplace.team, example@gov.uk). Note, an exception to this rule is supplier contact information which does contain production emails as these details are publicly available - we shouldn’t be sending emails to those email addresses though.

GOV.UK Notify

Transactional (not marketing) emails should be sent by Notify by default.

Local development

Local environments may break during processes which should send a Notify email, this is likely because you do not have a valid Notify API key set in your application. You can set up a test (not live) key for yourself in the Notify interface to use for local development:

  • on the Notify dashboard, go to ‘API Integration’

  • go to ‘API keys’

  • select the ‘Create an API Key’ button at the bottom of the dashboard

  • give the new key a memorable name that will help other developers know who set it up, the type of key, and what it’s going to be used for. For example, ‘firstname-test-localdev’.

  • make sure to use ‘Test’ for the key type

  • copy the key as displayed, and save it somewhere safe

To use locally in an app, add the key to the Development class in the app’s config.py. Make sure not to commit it!

You can also add DM_NOTIFY_KEY as an environment variable across all your local apps, using Flask’s dotenv support.

“Live” stages

On all “live” stages (preview, staging & production), the DM_NOTIFY_REDIRECT_DOMAINS_TO_ADDRESS config setting is set on the frontends to “redirect” emails to a number of domains to a specific “safe” dummy email address. This is to prevent Notify getting annoyed by being made to send out many messages to nonexistent addresses because of our functional/integration tests (it reportedly gives them bad karma with their downstream providers, making them look like potential spammers). Functional tests should attempt to use dummy email addresses from only these specifically safe domains, update this setting (across all frontends) with any new domains employed anywhere in the functional tests. At time of writing these domains are:

example.com
example.gov.uk
user.marketplace.team

This gives a selection of both approved and invalid domains for for buyers, and approved admin domains, to use for the relevant tests.

Preview

Emails are sent to Notify’s sandbox, not real users.

If you want to view an email sent via Notify on preview, for example to see how the template has been populated with variables from a script, there is a sneaky workaround to do so:

  • head to the Digital Marketplace account on Notify.

  • click on ‘API integration’ in the menu.

  • find the email you want

  • pop it open from the arrow

  • click the ‘View email’ at the bottom

Staging

Emails will be sent only to specific email addresses of members of the Notify organisation (i.e. +-based aliases are not considered to be the same) or those addresses we’ve added to our Notify organisation’s “whitelist”. In particular, we have added the specific “safe” dummy email address to which emails have been configured to be redirected via DM_NOTIFY_REDIRECT_DOMAINS_TO_ADDRESS so that functional tests are able to check the status of sent messages. Beware that emails sent to non-whitelisted addresses or domains using this type of Notify key will generate errors instead of being silently swallowed.

The main reason for using this type of Notify key here instead of a “sandbox” key is to keep Staging’s configuration closer to that of Production, but it is also useful for QA purposes to be able to send real emails to developers’ addresses.

Production

Emails will be sent to real users. You can view a sent email in the Notify interface. Sensitive content such as links generated to reset passwords are generally redacted so you should not be able to use them (this behaviour is set up in the Notify templates themselves).

Note however that this sensitive content is still present, and would be retrievable via any “live” Notify API key, which is why we take particular care with live keys and try to limit the number we have.

Logging

Notify can be configured to send callbacks to provide delivery status updates about notifications. We have configured Notify to send these callbacks to our production API app.

As of 2021-03-19, the API emits a log message for every callback we receive. This means if you need to check the status of an email or sms sent from the Digital Marketplace by Notify (regardless of how the notification was requested), you can use one of our log aggregators to get information about it.

If the status of an email notification is permanent-failure, then the account associated with that email address will be made inactive, as a security feature.

Mailchimp

We use Mailchimp for two processes:

  • sending suppliers alerts about new Digital Outcomes and Specialists opportunities, and managing their subscriptions

  • allowing interested suppliers to subscribe to new framework updates (however we now export this contact list from Mailchimp, and send the actual ‘New Framework is coming’ emails through Notify).

Mailchimp does not have a sandbox. Instead for testing purposes we use a Mailchimp list for each environment. Then for local testing, preview and staging we instead sign up our own emails to test things. All emails that are sent can be seen through the Mailchimp interface.

Creating new lists in Mailchimp

A new set of Mailchimp lists (known as ‘audiences’ in the Mailchimp UI) must be created for each iteration of DOS, to ensure that the correct suppliers for each framework and lot will receive daily notifications about new opportunities.

To create a new list:

  • Log into the Mailchimp UI and select the ‘GDS’ organisation.

  • Go to the ‘Audience’ tab and select ‘View Audiences’ at the bottom of the drop down list.

  • Click ‘Create Audience’ and enter the details (copy the campaign/contact settings from a previous DOS mailing list, and update with the new framework iteration number)

  • Once created, go to the Settings page for the new list and copy the Audience ID (e.g. d35601203b).

In the scripts repository , add these IDs to the send-dos-opportunities-email.py and upload-dos-opportunities-email-list.py scripts.

Mandrill

We no longer use Mandrill in our service. It was retired in Jan 2018 after it was discovered that we longer required the additional features we had been using it for (Setting of reply-to header and complex templating)