Rotating API keys

Rotating Data API and Search API keys

API keys for the Data API, Search API and Antivirus API are stored in the digitalmarketplace-credentials repository. Our applications (both API and frontend), other Digital Marketplace components (eg scripts), and developers use these keys to talk to the APIs.

digitalmarketplace-credentials/vars/preview.yaml:

api:
    auth_tokens:
    - A-first-token-is-used-by-other-applications
    - J-second-token-is-used-by-jenkins
    - D-third-token-is-be-used-by-developers

In the event that a key is leaked then it should be rotated. If a developer leaves the Digital Marketplace, then production tokens (at least) should be rotated. A Jenkins job called Rotate API tokens exists to generate new tokens and push them out to our applications, following the correct process for a seamless transition. We have three tokens for each API:

  • An application token that starts with ‘A’

  • A Jenkins token that starts with ‘J’

  • A developer token that starts with ‘D’

The Jenkins job is a Pipeline that has five main stages:

  1. The pipeline will generate a new Pull Request against the credentials repository to add three new tokens each for the Data API and Search API. A developer will need to manually merge this.

  2. Go back to the pipeline and proceed through the next two dialogue boxes to deploy the Search API, then deploy the Data API, and then deploy all other apps simultaneously.

  3. After all of the apps have released and the pipeline is paused at the ‘Update Jenkins config’ stage, a developer needs to update Jenkins config. To do this, prepare Jenkins for shutdown (Manage Jenkins -> Prepare for Shutdown) to stop any new jobs from being started. Wait for existing jobs to finish running and notify developers on #dm-release that Jenkins will be restarted shortly, then run make reconfigure from the digitalmarketplace-jenkins repository to push the updated tokens for Jenkins to inject into the environment for scripts/etc. You must have DM_CREDENTIALS_REPO defined and it must point at a version of digitalmarketplace-credentials that contains the newly-added tokens. After Jenkins has restarted, continue the Pipeline by clicking ‘Proceed’ twice.

  4. Pipeline stage 4 will generate another Pull Request against the digitalmarketplace-credentials repository to remove the old tokens from the Data API and the Search API. A developer will need to manually merge this.

  5. Finally, continue the pipeline again by clicking ‘proceed’ twice to deploy the Search API, then the Data API, then all other apps (apps don’t access the Antivirus API directly, so it’s unimportant when the Antivirus API is restarted).

Rotating Jenkins Docker credentials

Jenkins has its own Docker Hub account which it uses to publish Docker images that it builds. This account is protected by two-factor authentication, and is accessed by Jenkins using a Docker Hub personal access token.

The account token is kept in the digitalmarketplace-credentials repo, in the file jenkins-vars/docker_credentials_env.enc, as an environment variable called DOCKER_PASSWORD.

This is a manual process during which releases/merges should be paused.

  1. Post a message in the #dm-release Slack channel warning the team not to merge or release.

  2. Using the details for the Jenkins user in the credentials file pass/docker.com/jenkins-ci, log in to the Docker Hub console. To do this you will need to reset the password (generate a new one with your choice of reputable token generator).

  3. Create a new access token following the instructions in the Docker manual. Make sure to give the token a memorable name, like “jenkins (new)”. Copy the token into the credentials file jenkins-vars/docker_credentials_env.enc.

  4. Delete the old access token and rename the new access token (which should now be the only access token) to “jenkins”.

  5. Log out of Docker Hub, and throw the account password away.

  6. Commit the new token to digitalmarketplace-credentials and open a pull request. Get the PR reviewed and merged.

  7. Run the update-credentials job in the Jenkins console.

  8. SSH into Jenkins and delete the /var/lib/jenkins/.docker folder to remove the auth token.

  9. Still in the Jenkins box, log in with the new token: sudo -u jenkins docker login

  10. Test with a Docker image job (e.g. run the Refresh Docker base images job).

  11. Give the all clear in the #dm-release Slack channel.

Rotating Notify API keys

We should normally only have two “Live” Notify API keys at a time: one for the (production) applications and one for Jenkins. These are the only ones it’s particularly useful to rotate: “Test” and “Team and Whitelist” tokens are relatively powerless.

  1. Create two new “Live” Notify API keys using the Notify dashboard.

    • One named in the format APPLICATION-PRODUCTION-<date>. Use this key to replace the notify_api_key variable in the digitalmarketplace-credentials repo’s vars/production.yaml file.

    • One named in the format JENKINS-LIVE-<date>. Use this key to replace the jenkins_env_variables.NOTIFY_API_TOKEN and jenkins_env_variables.NOTIFY_API_FUNCTIONAL_TESTS_KEY_PRODUCTION variables in the digitalmarketplace-credentials repo’s jenkins-vars/jenkins.yaml file.

  2. Commit these changes and get the resulting PR approved and merged.

  3. Ensuring you have this latest credentials revision checked out in your local credentials repository, prepare Jenkins for shutdown (Manage Jenkins > Prepare for Shutdown), Wait for existing jobs to finish running and notify developers on #dm-release that Jenkins will be restarted shortly, then run make reconfigure from the digitalmarketplace-jenkins repository.

  4. Run the “Re-release all apps” jenkins job against production to ensure all the apps have picked up the new notify key.

  5. It’s probably best to take a look at the “Message log” in the Notify dashboard (under “API integration”) and check you can see the new keys in use (and that you can’t see the old keys still being in use).

  6. Delete the old keys in the Notify dashboard.

Rotating AWS Access keys

The apps hosted on PaaS use an AWS Access Key to connect with S3 buckets and other AWS resources. There is an access key for each AWS account (development and production).

To rotate an access key, use the Jenkins job. The job implements the old manual process:

  1. In the AWS console, switch role (to either infrastructure or admin role) for the relevant account.

  2. Create new key for the paas-app user for that account in AWS console: IAM > Users > paas-app > Security Credentials > Access Keys Each user can have a max of 2 keys.

  3. Add the new aws_access_key_id and aws_secret_access_key to the credentials repo in vars/<stage>.yml. Commit, review and merge the PR.

  4. Re-release all apps via Jenkins to inject the new AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment vars into the PaaS apps.

  5. Try accessing a restricted document on Digital Marketplace for the relevant stage(s). For example, log in as a supplier and download/upload a service document.

  6. In the AWS console, check that the new key has been used (and that the old key has not been used since the release). This information may take up to 5 minutes to appear in the AWS console. If it doesn’t appear, check if logs for the stage are appearing in CloudWatch (the PaaS apps use the AWS access key to send logs to CloudWatch). If there are no logs in CloudWatch, the key is probably invalid/misconfigured and you will need to start again.

  7. Once you’re confident the new key is being used, make the old key inactive in the AWS console (this is reversible and it doesn’t need to be deleted straight away).