Developer setup

Contents

• Developer setup

  • Shortcut - Digital Marketplace Runner

  • Bootstrapping a Mac

  • Bootstrapping on Linux

  • Running the environment

  • Testing the applications

  • Ports

  • Making requests to your local API instance

  • Accessing frontend applications as a single website

  • Importing data into your local database

  • Elasticsearch

  • Using remote versions of services locally

  • Configuring Pycharm or IntelliJ IDEA Ultimate

    • Prerequisites

    • Python interpreter

    • Pytest configuration

  • Libraries and other repositories you will (eventually) need

    • Digitalmarketplace utils

    • Digitalmarketplace test utils

    • API client

    • Frontend toolkit

    • Frameworks

    • Content loader

    • Functional tests

    • Scripts

    • Manual

See also Adding and removing access for new starters / leavers

Shortcut - Digital Marketplace Runner

The Digital Marketplace Runner (‘dmrunner’) is an attempt to simplify setup for new developers, as we have a number of backing services, apps, and dependencies that don’t easily come together in a single place with existing tools like vagrant, docker, or nix. You might want to consider trying this first if you are setting up the Digital Marketplace from nothing, although the rest of the information on this page is still very useful reading to understand more about how the services and apps and repositories all come together to make the Digital Marketplace.

dmrunner:

git clone git@github.com:crown-commercial-service/digitalmarketplace-runner.git

Bootstrapping a Mac

Note

If you’re using Big Sur, you may encounter permissions problems. If you do, ask the other developers for assistance, and then update this page.

Do this first, if you’re new (or your Mac is new).

Ensure xcode is installed by running xcode-select --install.

Using Self Service, install Docker and Homebrew.

It is recommended to manage Python versions using pyenv.

Install pyenv

pyenv install the version specified in: https://github.com/crown-commercial-service/digitalmarketplace-runner/blob/master/.python-version

Using Homebrew, run brew install <package name> for the following:

• node@14

• postgres

• sops and jq (for credentials)

• cloudfoundry/tap/cf-cli

• rbenv and chromedriver (for functional tests)

Using Pip3, run pip3 install <package name> for the following:

• virtualenv

• awscli (for credentials)

Using NPM run the following command to install dependencies we require globally:

npm install -g gulp-cli standard

You will also need to generate an SSH key for your github account (see the Github instructions).

Bootstrapping on Linux

If you’re new and have a Linux laptop, you can mostly follow the instructions for a Mac above.

Don’t install packages using Homebrew. Instead, on Ubuntu run sudo apt-get install postgresql-client jq chromium-chromedriver. Then manually install terraform and cf. Install nvm, then switch to a digital marketplace repo, and run nvm use.

Running the environment

Tip

If you are using Digital Marketplace Runner (‘dmrunner’), you can simply use its invoke commands without needing to follow the instructions in this section.

There are eight Flask applications that make up the Digital Marketplace service. To run the service locally you need to clone each of the repositories and follow the detailed instructions in the README for each application, including initial bootstrapping of the database and elasticsearch.

You don’t have to do it in the exact order listed here, but starting with the APIs makes sense.

API:

git clone git@github.com:crown-commercial-service/digitalmarketplace-api.git

Search API:

git clone git@github.com:crown-commercial-service/digitalmarketplace-search-api.git

User frontend:

git clone git@github.com:crown-commercial-service/digitalmarketplace-user-frontend.git

Buyer frontend:

git clone git@github.com:crown-commercial-service/digitalmarketplace-buyer-frontend.git

Supplier frontend:

git clone git@github.com:crown-commercial-service/digitalmarketplace-supplier-frontend.git

Admin frontend:

git clone git@github.com:crown-commercial-service/digitalmarketplace-admin-frontend.git

Briefs frontend:

git clone git@github.com:crown-commercial-service/digitalmarketplace-briefs-frontend.git

Brief responses frontend:

git clone git@github.com:crown-commercial-service/digitalmarketplace-brief-responses-frontend.git

Each application has a Makefile with a run-app command that sets environment variables that enable the apps to communicate with each other for local development setup, and a run-all command that additionally installs all of the frontend and backend dependencies.

Testing the applications

Each application has a Makefile with a test command that first runs PEP 8 Python style checks and then (if they pass) runs the tests for the application.

Ports

The port number that the applications run on are set in the application.py startup script of each application. The default port numbers used are as follows:

App	Port
digitalmarketplace-api	5000
(Skipped as it is used by Cisco Anyconnect VPN)	5001
digitalmarketplace-buyer-frontend	5002
digitalmarketplace-supplier-frontend	5003
digitalmarketplace-admin-frontend	5004
digitalmarketplace-briefs-frontend	5005
digitalmarketplace-brief-responses-frontend	5006
digitalmarketplace-user-frontend	5007
digitalmarketplace-antivirus-api	5008
digitalmarketplace-search-api	5009

Making requests to your local API instance

When making requests to your local API instance you should use the bearer token myToken to authenticate, for example:

curl -H "Authorization: Bearer myToken" localhost:5000

Accessing frontend applications as a single website

The frontend applications are hyperlinked together but are running on different ports. This can cause links to error when they link between different applications. The way around this is to set up nginx as a reverse proxy so all front end applications can be accessed through port 80.

If you’re not using dmrunner, the easiest way to do this is to clone the digitalmarketplace-functional-tests repository and then run the script:

digitalmarketplace-functional-tests/nginx/bootstrap.sh

This will do all the setup of nginx that you need, and you should only ever need to run it once. You can subsequently just run:

sudo nginx

And that will start up the necessary reverse proxy on port 80, allowing you to access all of your locally running applications seamlessly linked together at http://localhost/

Importing data into your local database

It isn’t necessary (but is probably useful) to populate your database with some “realistic” data. To import a recent sanitised dump of our production database (i.e. without any real email addresses, etc.) you can download the exportdata.tar.gz file from the digitalmarketplace-cleaned-db-dumps S3 bucket, and psql-ing it into your local database.

Slightly more detailed instructions can be found in Importing data for developers.

Once you’ve imported data into your local database, you will need to index any briefs and services so that they will appear in search results (see next section).

Elasticsearch

Note

We switched to OpenSearch in March 2022. Update this section if you work out how to do the same for OpenSearch.

Elasticsearch is generally installed directly on the host/development machine at the moment. If you are a new joiner, you can use the Quickstart instructions in the README.md for digitalmarketplace-search-api.

If you are an existing developer with an older installation of Elasticsearch, you will need to stop the Elasticsearch service, uninstall the old version and purge its data directory (/usr/local/var/elasticsearch/ elasticsearch_<username>/, on a Mac) before installing via the Quickstart. You may need to remove other directories and files; if you have a problem, speak to other developers for help.

After installation, you will need to index (G-Cloud) services into Elasticsearch. You can use a script in digitalmarketplace-scripts to do this. Example:

./scripts/index-to-search-service.py services dev --index=g-cloud-9 --frameworks=g-cloud-9 --create-with-mapping=services

You will also need to index DOS opportunities (briefs), for example:

./scripts/index-to-search-service.py briefs dev --index=briefs-digital-outcomes-and-specialists --frameworks=digital-outcomes-and-specialists-2 --create-with-mapping=briefs-digital-outcomes-and-specialists-2

Note that this is slightly different to how we index data on preview, staging and production - see Search API.

See also: the Elasticsearch Head GUI, useful for viewing/removing Elasticsearch indices and aliases locally.

Using remote versions of services locally

When developing locally, apps can be configured to talk to deployed versions of the services. This is configured using environment variables. The default configuration points at local versions of the API and Search API (set in config.py), but these can be overridden.

For example, see the buyer app.

Configuring Pycharm or IntelliJ IDEA Ultimate

You are free to use the editor/IDE of your choice! If you use IntelliJ IDEA Ultimate or Pycharm the following instructions can be used to setup your environment.

Prerequisites

Ensure you have run invoke requirements-dev in the repository you wish to setup.

For IntelliJ only: Ensure you have installed the Python plugin

Python interpreter

Pycharm

Pycharm should be able to detect the interpreter automatically. You can check that in Pycharm > Preferences > Project > Project interpreter

IntelliJ

File > Project Structure… > Project Settings > Project

On the drop down for Project SDK select:

Add SDK > Python SDK

Virtualenv Environment > Existing environment > Interpreter > …

Select: <CHECKOUT DIRECTORY>/venv/bin/python

Click OK

Pytest configuration

Run > Edit Configurations > Add configuration > pytest

Target: Custom

Working directory: <CHECKOUT DIRECTORY>

Click OK

Follow the steps at Choosing Your Testing Framework, from the default test runner dropdown select: pytest.

For more configuration options see also Jetbrains documentation: Run/Debug Configuration: pytest

Libraries and other repositories you will (eventually) need

As well as the main Flask applications there are several other repositories that you are likely to need sooner or later as a developer on Digital Marketplace:

Digitalmarketplace utils

A home for utility functions, Jinja2 filters, constants and anything else that needs to be used by more than one app. It is pulled into all frontend apps by pip as a Python package, and flask_init.py gives a consistent application startup for all frontend apps.

https://github.com/crown-commercial-service/digitalmarketplace-utils

Digitalmarketplace test utils

A home for utility functions and scripts specific to unittests.

https://github.com/crown-commercial-service/digitalmarketplace-test-utils

API client

This is imported by pip as a Python package by all frontend applications and contains functions that wrap calls to both the data and search APIs:

https://github.com/crown-commercial-service/digitalmarketplace-apiclient

Frontend toolkit

Reusable design patterns and Jinja2 macros for frontend components on the Marketplace (tables, forms, buttons, fonts, etc.)

https://github.com/crown-commercial-service/digitalmarketplace-frontend-toolkit

The frontend toolkit is pulled in to all frontend apps by npm and gulp during the frontend build stage.

Frameworks

This is also known as the “content repository”. It contains all content for individual frameworks - the questions asked during the application process for supplier declarations and service descriptions, framework-specific messages, URLs and dates. All content files are in YAML format. The files here are also used to generate the JSON schemas used by the API to validate certain data fields - see the repository README for more detail.

https://github.com/crown-commercial-service/digitalmarketplace-frameworks

The frameworks repository is pulled in to all frontend apps (with the exception of user frontend) by npm and gulp (i.e. as part of the frontend build), not as a Python package.

Content loader

This is imported by pip as a Python package by all frontend applications. It loads the YAML files of content from the “frameworks” repository and processes them into objects that can be used by the frontend apps to generate the framework-specific pages they need to:

https://github.com/crown-commercial-service/digitalmarketplace-content-loader

Functional tests

This is our acceptance test suite that runs against our live environments. It’s a Ruby project using Cucumber and Capybara. The full functional test suite runs against preview whenever new code is merged to one of the five main Flask apps, and against staging after a release is promoted from preview to staging. The purpose is to check that everything still works as expected. A smaller set of smoketests runs periodically against all live environments to check that all apps are up and running.

https://github.com/crown-commercial-service/digitalmarketplace-functional-tests

Scripts

There are several processes, especially as part of framework submission and award, that are not (yet) built in to our web applications. We have a collection of scripts that interact directly with our APIs and S3 buckets to do some of these routine tasks.

https://github.com/crown-commercial-service/digitalmarketplace-scripts

Manual

The code for this manual! The files used to generate this site can be written either in reStructuredText or Markdown. Create a pull request and ask for a review by dropping a comment into #dm-review on the GDS Slack instance.

https://github.com/crown-commercial-service/digitalmarketplace-manual