Development and deployment process ¶

Tip

If in doubt discuss with other developers in the #dm-development Slack channel

This manual page describes the day to day work of a developer on the Digital Marketplace, much of which will be writing and reviewing pull requests. It supplements (and in some cases supersedes) the GDS Way, particular the pages on how to store source code, how to review code, and the page on pull requests in general.

Story Kickoff ¶

So, you’re looking at the ticket at the top of your team’s backlog and are about to start work building out the feature…

Before even thinking about beginning to touch a line of code, do a story kickoff. Talk to anyone on the team who might have an opinion about the technical implementation or is likely to be affected by changes to the interface and hash out what the solution to your particular problem might look like. A list of interested people may be in the story. By the time you claim the story on Trello it should be abundantly clear what your goals are.

Develop locally on feature branches ¶

We develop all our code on Git feature branches and review these via GitHub pull requests.

We typically develop locally using Digital Marketplace Runner.

Open a pull request ¶

When development of a feature is complete and the application level tests pass locally open a pull request. This will cause Github Actions to run the application tests and update the pull request. From this point on the developer who opened the pull request is responsible for ensuring the feature moves through the deployment process.

If there is a Trello card for the work you are doing paste the link to the card into the pull request message. The GitHub Trello poster will then add a link to your pull request to the Trello card for you [1].

Request review ¶

As soon as you’re happy with the pull request description it’s time to request review. Post a link to the pull request in the #dm-review channel. All developers on the Digital Marketplace should be a member of this channel and watch it regularly for new pull requests. If your pull request needs to be urgently reviewed make sure to mention that when posting the link to the pull request.

Note

All developers should be involved in reviewing pull requests (although only one approval is needed before merging).

It isn’t expected that everyone reads every pull request, but everyone should be spending a small amount of time each day reviewing. Don’t be shy about reviewing code you’re unfamiliar with - it’s a good time to ask questions and learn about the broader codebase. However make sure you understand the changes before approving.

We review pull requests with the review tool on GitHub. Discussion, questions, and constructive feedback are all encouraged and expected; also don’t forget to leave friendly emojis!

Reviewers may approve your pull request without comment, approve it with a comment and expect you to action the comment before merging (trusting that you will), or leave comments without approving if they have unresolved questions or concerns.

You can update your pull request as many times as you want before merging, although you might want to avoid making big changes after it has been approved. You should use git rebase liberally to keep the commits based on the current master branch and the history clean (the exception to this is when pairing on a pull request, when it’s better to wait until it’s ready for merging before rebasing).

Merge ¶

Once your reviewers have reviewed the pull request and are happy with the proposed changes they will approve it using the GitHub review tool. This indicates that you’re clear to merge it when you’re ready. It is not easy to change a pull request once merged, so at the point of merging a pull request everyone implicated should agree on the implemented solution.

Merging a pull request on one of the main Flask applications will automatically kick off a series of steps involved in deploying to preview:

The application is tagged with the git hash that caused the build release-<git short hash>.
An artifact is built for this tag and deployed to preview.
Once the deployment is complete a run of the functional and visual regression tests will be started.

Working on shared repositories ¶

Several of our repositories contain shared code used by our different applications:

When updating any of these we need to publish a new version (tagged with a new version number) when we want to share changes. So before making a pull-request with changes in one of these repositories, you need to update the version number on your working branch – the location of this varies depending on the type of repository you’re working with.

Node repositories such as the frameworks and frontend-toolkit repos both define the version as part of their package.json manifest

Python repositories such as the apiclient, content-loader and utils repos have an __init__.py file inside of their Python module folder which contains __version__

Once your pull request bumping the version is merged, one of two things will happen: either

Jenkins will automatically run the push-tag.sh script. This will create a git tag with the version number you supplied, thus making it possible to clone the exact version of the code

or

A GitHub Action will run, creating a GitHub release and then publishing the package to the appropriate package repository (NPM or PyPI). For more information about these actions read GitHub Actions.

Test on preview ¶

Once the application has been deployed to preview, the smoke tests, functional tests and visual regression tests will be run automatically. After they have both passed it is ready to be viewed on preview.

You should consider doing some manual testing of your own when a change has been made on preview. For larger changes and new features you may also want to take some screenshots to share with your team’s product manager, or have them go through the journey themselves.

Note

Larger changes should ideally have feature tests and visual regression tests added to the respective suites.

Preview can be found at: https://www.preview.marketplace.team/

Deploy to staging ¶

Deploying to staging is the first step of deploying a feature to production. Our staging environment acts as a backstop to catch any breaking changes before deploying to production; usually only one change from production should be present in staging at a time, and not for very long.

Before deploying to staging or production the feature must have been OK’d for release by doing some manual testing and discussing with the product manager if necessary.

Note

Because of limitations of our current release process, the longer your change sits in preview without being deployed, the more likely it will be that you will be unable to release it without also releasing another developer’s changes.

Be sure to check that all changes going out in your release have been OK’d. We have a lag radiator that can help with this, it shows how many merges are undeployed for each app. Clicking on the name for each app links to a GitHub page with all undeployed commits. To access the lag radiator decrypt digitalmarketplace-credentials/links.yaml and bookmark the deploy_lag_radiator URL. Please don’t share it in plaintext, as it contains an API key.

When you’re ready to go take the Gorilla of Deploy - YOLO - and keep it close!

Check the channel topic in the #dm-release Slack channel indicates it is currently free.

If the Deploy Lag Radiator shows there are other team members’ commits included in the deploy; tag them in a message in the #dm-release Slack channel; so they are aware their changes are about to be deployed.

Update the channel topic in the #dm-release Slack channel to indicate you are deploying.

Find the relevant release pipeline job for the app you’re releasing in Jenkins, hover your mouse pointer over the “release to staging” box and click “Proceed”. This will take the artefact for the version of the application currently deployed into preview and deploy it to staging.

Once deployment to staging is complete a run of the functional and visual regression tests will be started - you will be notified of the results in the #dm-release Slack channel once they complete. If tests fail either fix the tests or fix your code.

Deploy to production ¶

Once the staging tests have passed, the pipeline will deploy the artefact currently in staging into production.

You should now check the feature works as expected on the live site. Once we’re happy the feature is live it is DONE!

Release YOLO back into the wild.

Update the channel topic in the #dm-release Slack channel to indicate it is free.