Development and deployment process

Tip

If in doubt discuss with other developers in the #dm-development Slack channel

This manual page describes the day to day work of a developer on the Digital Marketplace, much of which will be writing and reviewing pull requests. It supplements (and in some cases supersedes) the GDS Way, particular the pages on how to store source code, how to review code, and the page on pull requests in general.

Story Kickoff

So, you’re looking at the ticket at the top of your team’s backlog and are about to start work building out the feature…

Before even thinking about beginning to touch a line of code, do a story kickoff. Talk to anyone on the team who might have an opinion about the technical implementation or is likely to be affected by changes to the interface and hash out what the solution to your particular problem might look like. A list of interested people may be in the story. By the time you claim the story on Jira it should be abundantly clear what your goals are.

Develop locally on feature branches

We develop all our code on Git feature branches and review these via GitHub pull requests.

We typically develop locally using Digital Marketplace Runner.

Open a pull request

When development of a feature is complete and the application level tests pass locally open a pull request. This will cause Github Actions to run the application tests and update the pull request. From this point on the developer who opened the pull request is responsible for ensuring the feature moves through the deployment process.

If there is a Jira card for the work you are doing paste the link to the card into the pull request message.

Request review

As soon as you’re happy with the pull request description it’s time to request review. Post a link to the pull request in the #dm-review channel. All developers on the Digital Marketplace should be a member of this channel and watch it regularly for new pull requests. If your pull request needs to be urgently reviewed make sure to mention that when posting the link to the pull request.

Note

All developers should be involved in reviewing pull requests (although only one approval is needed before merging).

It isn’t expected that everyone reads every pull request, but everyone should be spending a small amount of time each day reviewing. Don’t be shy about reviewing code you’re unfamiliar with - it’s a good time to ask questions and learn about the broader codebase. However make sure you understand the changes before approving.

We review pull requests with the review tool on GitHub. Discussion, questions, and constructive feedback are all encouraged and expected; also don’t forget to leave friendly emojis!

Reviewers may approve your pull request without comment, approve it with a comment and expect you to action the comment before merging (trusting that you will), or leave comments without approving if they have unresolved questions or concerns.

You can update your pull request as many times as you want before merging, although you might want to avoid making big changes after it has been approved. You should use git rebase liberally to keep the commits based on the current master branch and the history clean (the exception to this is when pairing on a pull request, when it’s better to wait until it’s ready for merging before rebasing).

Merge

Once your reviewers have reviewed the pull request and are happy with the proposed changes they will approve it using the GitHub review tool. This indicates that you’re clear to merge it when you’re ready. It is not easy to change a pull request once merged, so at the point of merging a pull request everyone implicated should agree on the implemented solution.

Merging a pull request on one of the main Flask applications will automatically kick off a series of steps involved in deploying to preview:

1. The application is tagged with the git hash that caused the build release-<git short hash>.

2. An artifact is built for this tag and deployed to preview.

3. Once the deployment is complete a run of the functional and visual regression tests will be started.

Working on shared repositories

Several of our repositories contain shared code used by our different applications:

• digitalmarketplace-apiclient

• digitalmarketplace-content-loader

• digitalmarketplace-frameworks

• digitalmarketplace-utils

• digitalmarketplace-test-utils

• digitalmarketplace-govuk-frontend

• digitalmarketplace-frontend-jinja

When updating any of these we need to publish a new version (tagged with a new version number) when we want to share changes. So before making a pull-request with changes in one of these repositories, you need to update the version number on your working branch – the location of this varies depending on the type of repository you’re working with.

• Node repositories such as the frameworks and frontend repos both define the version as part of their package.json manifest

• Python repositories such as the apiclient, content-loader and utils repos have an __init__.py file inside of their Python module folder which contains __version__

Once your pull request bumping the version is merged:

• A GitHub Action will run, creating a GitHub release and then publishing the package to the appropriate package repository (NPM or PyPI). For more information about these actions read GitHub Actions.

Test on development

Once the application has been deployed to development, the smoke tests, functional tests and visual regression tests will be run automatically. After they have both passed it is ready to be viewed on development.

You should consider doing some manual testing of your own when a change has been made on development. For larger changes and new features you may also want to take some screenshots to share with your team’s product manager, or have them go through the journey themselves.

Note

Larger changes should ideally have feature tests and visual regression tests added to the respective suites.

Development can be found at: https://www.development.marketplace.team/

Deploy to pre-production

Deploying to pre-production is the first step of deploying a feature to production. Our pre-production environment acts as a backstop to catch any breaking changes before deploying to production; usually only one change from production should be present in pre-production at a time, and not for very long.

Before deploying to pre-production or production the feature must have been OK’d for release by doing some manual testing and discussing with the product manager if necessary.

Note

Because of limitations of our current release process, the longer your change sits in development without being deployed, the more likely it will be that you will be unable to release it without also releasing another developer’s changes.

When you’re ready to go take the Gorilla of Deploy - YOLO - and keep it close!

Check the channel topic in the #dm-release Slack channel indicates it is currently free.

Update the channel topic in the #dm-release Slack channel to indicate you are deploying.

Use the build and deploy pipeline job and put in the app and Git reference for the code you want to deploy to pre-production.

Once deployment to pre-production is complete a run of the functional and visual regression tests will be started - you will be notified of the results in the #dm-release Slack channel once they complete. If tests fail either fix the tests or fix your code.

Deploy to production

Once the pre-production tests have passed, the pipeline will deploy the artefact currently in pre-production into production.

You should now check the feature works as expected on the live site. Once we’re happy the feature is live it is DONE!

Release YOLO back into the wild.

Update the channel topic in the #dm-release Slack channel to indicate it is free.