What is PaaS?

Our web applications and services are hosted on the GOV.UK PaaS. The GOV.UK PaaS is a customised CloudFoundry deployment hosted on AWS open for use by government development teams.

You can read more about it on their service pages, technical documentation pages or get support/ ask questions on the #PaaS slack channel

Deploying Digital Marketplace apps to GOV.UK PaaS

Warning

Developers should generally NEVER have to do the things detailed here on a day-to-day basis. Our deploy pipeline is handled by our pipeline release jobs in Jenkins and you should use those. The details here are for information and extreme emergencies only.

The GOV.UK PaaS documentation explains the steps required to set up and login to the Cloud Foundry CLI.

Organisation name	Space	Description
digitalmarketplace	sandbox	Environment for experiments (remember to clean up after yourself!)
digitalmarketplace	monitoring	Environment for apps which monitor other apps
digitalmarketplace	labs	Environment for user research
digitalmarketplace	preview/staging/production	Environments for different stages

Setting up the target organisation and space:

cf target -o digitalmarketplace -s <space>

Deploying a Dockerised app

All Digital Marketplace apps run in Docker containers on the GOV.UK PaaS. Each application has an image tagged with each release. Images can be found on Docker Hub.

The digitalmarketplace-aws repo has a Makefile with rules for generating manifests for apps, and then pushing them to the GOV.UK PaaS. The most important being:

make deploy-app APPLICATION_NAME=<docker repository> RELEASE_NAME=<docker tag>

This is the easiest way to deploy an application. It pulls in variables from the credentials repo to populate the manifest template, and takes APPLICATION_NAME and RELEASE_NAME variables to make sure the correct Docker image is used.

Application manifests describe the application instances, hosts, launch commands and environment variables used. Secret env variables are pulled in from the credentials repo ‘vars’ directory when the manifest is generated.

The manifest also configures the routing for the applications within GOV.UK PaaS (e.g. requests with the /suppliers prefix are routed to the supplier-frontend app). The AWS repo has the full list of routes.

A healthcheck url for each app is defined in the base manifest, by appending _status?ignore-dependencies to the first item in the app’s routes list.

Debugging instances

Head to the ‘Debugging PaaS App Instances’ section of the manual for tips on debugging app instances on GOV.UK PaaS.

Creating services

We currently have services for our database, OpenSearch and cdn_router. Creating new services is only required when setting up a new space or creating a separate environment/instance of the API. For example:

cf create-service postgres "M-dedicated-9.5" <database_name>

The created database name can then be used in the application manifest services list.

Running cf services will display a list of all services in the current space, and which apps have a binding to them. For example:

+---------------------------+------------------+--------------+------------+
| name                      | service          | plan         | bound apps |
+---------------------------+------------------+--------------+------------+
| search_api_elasticsearch  | OpenSearch       | small-ha-1   | search-api |
+---------------------------+------------------+--------------+------------+

The credentials for connecting to GOV.UK PaaS services are automatically provided in the app environment cf env <application_name> (i.e. these are not in our credentials repo). The credentials are regenerated when a new instance is created. If you need to recycle the credentials following a leak, you must destroy and rebuild the service. First you must unbind any apps that use the service, and delete any service keys. Then create the new service instance, rebind the apps and re-add the service keys.

For example, to destroy/create a new OpenSearch instance:

cf unbind-service <application_name> <service_name>
cf delete-service-keys <service_name> <key_name>
cf delete-service <service_name>
cf create-service OpenSearch small-ha-1 <service_name>
cf add-service-key <service_name> <key_name>
cf bind-service <application_name> <service_name>

You may need to redeploy the application via Jenkins, in order to use the service’s new environment variables.

The cf delete-service and cf create-service commands will take several minutes each. To avoid downtime, you may need to create the new service with a different name beforehand, bind the app to it, then delete the old service.

If there were previously any custom settings for the service (e.g. our OpenSearch instances have a larger memory allocation than the default) these will have to be re-applied via a request to GOV.UK PaaS support.

Warning

If destroying/rebuilding OpenSearch, remember to recreate the indexes, aliases and populate the data for both services and briefs!

Deploying an app using a buildpack

None of the main Digital Marketplace apps use buildpacks. As of December 2020, only the PaaS Prometheus exporter uses buildpacks.

To deploy an application using an existing manifest, cd into the application directory and run:

cf push -f <path_to_manifest> <application_name>

Note

If your directory names match the repository names then you can use $(basename $(pwd)) as the application name to always deploy current directory to the corresponding application.

Warning

When started without <application_name> CloudFoundry will deploy the current directory to all applications in the manifest. If this happens, you need to redeploy each application separately from the correct application directory.

Cloud Foundry will push the contents of the current directory as the new application version. This isn’t tied to files tracked by git, so any uncommitted changes or files ignored by git will be pushed (this usually includes node_modules and app/content).

Warning

Make sure there are no secrets or confidential information (eg production logs, API tokens or database dumps) in the current working directory before pushing

Scaling apps

Our PaaS apps can be scaled horizontally (increasing the number of instances) or vertically (increasing the memory or disk space).

Warning

During a deployment, we will briefly have twice as many apps as normal running. When scaling ensure that 50% of our PaaS quota is still free. Otherwise you will break deployments.

To scale up or down:

1. Update the vars/<environment>.yml file in the digitalmarketplace-aws repo with the number of instances, memory and disk space for each environment.

2. Create a pull request and explain the change in the description. Get it approved and merged (see this example PR for scaling G-Cloud 12).

3. Alert the team in the #dm-release channel.

4. Run the Jenkins Re-release all apps job for the right environment. This uses the currently deployed images so you don’t have to go through the full pipeline.

Scaling manually

If you urgently need to increase the number of instances, this can be done with a simple CloudFoundry command:

cf scale <application_name> -i 10

You will still need to commit and merge the changes in the AWS repo afterwards.

Warning

Only use the CloudFoundry command for increasing the number of instances. If scaling down apps, or increasing/decreasing memory or disk space, this must be done via a re-release.