Digital Marketplace team manual

Useful information for the Digital Marketplace team, including:

• History of the service and context for why the service is built as it is.

• How we operate our team and our service.

• How to run and work on our applications.

• What our technical architecture looks like.

• Our 2nd Line and Support processes.

Digital Marketplace background

• Finding Things
  • Developer Manual
  • Shared Drive
  • Jira
• GDPR
  • What is the GDPR?
  • The Digital Marketplace as a Data Processor
  • Sharing Data with CCS
  • Data Minimisation
  • Documentation
• Digital Marketplace 1.5

Developing the Digital Marketplace

• Accessing the API
  • Use the API client shell
  • Access the API directly
• Adding a new repository
  • Prerequisites
  • Setting up a repo on GitHub
  • Adding the repo to the world
• Application architecture
  • Overall architecture
  • Applications
  • Asynchronous processes
• Apply to Supply
  • Support
  • Development
  • Deployment
  • Environments
• Browser and accessibility support
  • Browser support
• Environments
  • Develpment (Preview)
  • Pre-Production
  • Production
• Developer setup
  • Shortcut - Digital Marketplace Runner
  • Bootstrapping a Mac
  • Bootstrapping on Linux
  • Running the environment
  • Testing the applications
  • Ports
  • Making requests to your local API instance
  • Accessing frontend applications as a single website
  • Importing data into your local database
  • OpenSearch
  • Using remote versions of services locally
  • Configuring Pycharm or IntelliJ IDEA Ultimate
  • Libraries and other repositories you will (eventually) need
• Repositories
  • Digital Marketplace 1.5 (Apply to Supply)
• Digital Marketplace Runner (‘dmrunner’)
  • Repository
  • Debugging
  • Possible improvements
• Data model
  • Overview of the API data model
  • Entities and what they are
• Development and deployment process
  • Story Kickoff
  • Develop locally on feature branches
  • Open a pull request
  • Request review
  • Merge
  • Working on shared repositories
  • Test on development
  • Deploy to pre-production
  • Deploy to production
• Moving data between environments
  • Overview
  • Moving data from production to preview or staging
  • Importing data for developers
• URLs
  • Buyer Frontend
  • User Frontend
  • Supplier Frontend
  • Admin Frontend
  • API
  • Search API
  • Special urls
• Users
  • Buyers
  • Suppliers
  • Admins
• Automated testing
  • Our automated testing landscape
  • Guidelines for automated testing
• Database migrations
  • Create a new migration
  • Browse migrations
  • New column defaults
• Docker strategies
  • Base images
  • Build and deploy
• Frontend assets and tools
  • Shared assets
  • Testing
• Python typing strategy
• Guidelines for accessing production data and systems
  • General Principles
  • On the Digital Marketplace

Content and frameworks

• Preparing to add a new framework
  • 1 - Make the content
  • 2 - Generate service validation schemas and add them to the API
  • 3 - Add a supplier declaration validator (optional)
  • 4 - Check application email templates
  • 5 - Create any new lots
  • 6 - Upload legal documents
  • 7 - Generate the assessment schemas
  • 8 - Create a new search mapping (G-Cloud only)
  • 9 - Load test
  • 10 - Let’s go!
• Framework lifecycle for developers
  • 0 - overview
  • 1 - coming
  • 2(a) - open (with clarification questions open)
  • 2(b) - open (with clarification questions closed)
  • 3 - pending
  • 4 - standstill
  • 5 - live
  • 6 - expired
• How suppliers apply to a framework
  • 0 - at any time
  • 1 - coming
  • 2 - open
  • 3 - pending
  • 4 - standstill
  • 5 - live
  • 6 - expired
• Framework agreements
  • 0 - What is a framework agreement?
  • 1 - Signing the agreement
  • 2 - Countersigning an agreement
  • 3 - Suspending suppliers without an agreement
• Finding content
  • Content finder
  • Where content is saved
• Changing content
  • GitHub content editor
  • How to edit content
• Testing supplier applications
  • Changing framework state
  • Getting a supplier account

Infrastructure

• The Antivirus API
  • How scans are initiated
  • Actions on virus detection
  • Functional/Smoulder tests
• AWS accounts and access
  • Accounts
  • AWS Console Access
  • Programmatic access
  • Billing and billing alerts
• AWS Resources
  • S3
  • IAM
• Code backups
  • Why code backups
  • What the backup jobs do
  • To add a repo to the backups
  • Further reading
• Database backups
  • How our database backups are created
  • What our backup script does
  • The S3 buckets
  • Signed S3 URL’s
  • GPG
  • Restoring from a backup
• Domains
  • DNS
  • SSL/TLS cerfiticates
• Email integration
  • GOV.UK Notify
  • Mailchimp
  • Mandrill
• OpenSearch
  • Where OpenSearch data is used
  • Data that Digital Marketplace gets from OpenSearch
  • Where Digital Marketplace finds that data
  • Benefits to multi-index single-type model
  • Where indexes are described
  • What our indexes contain
  • How our indexes are named
  • How indexes are created
  • How indexes are populated
  • Aliases
  • Jobs
  • Catch-up jobs
  • Manual (cleanup) jobs
  • Checking the OpenSearch version
• Jenkins
  • What we use Jenkins for
  • Accessing and deploying jobs to Jenkins
  • Jenkins Logging
  • Jenkins infrastructure
  • How Jenkins uses credentials
  • Jenkins chores
• GitHub Actions
  • What we use GitHub Actions for
  • Pre-merge checks
  • Publishing libraries
• Logging, monitoring and alerting
  • Logging
  • Monitoring
  • Alerting
• Maintenance Mode
  • Planned downtime
  • Accessing applications during maintenance
  • Before putting the application into maintenance mode
  • How to activate/deactivate
  • What does it look like?
  • What’s going on under the hood?
  • Lack of sticky sessions
• Python environments
  • System Python 3.10
  • Local Python 3.10
  • Python 3.10 virtual environment
  • Future Python compatibility
• What is PaaS?
• Deploying Digital Marketplace apps to GOV.UK PaaS
  • Deploying a Dockerised app
  • Debugging instances
  • Creating services
  • Deploying an app using a buildpack
  • Scaling apps
• Rate Limiting
  • How it works
  • Burst
  • Deactivating rate limiting
• Upgrading backend services
  • Nginx
  • PostgreSQL
  • Elasticsearch
  • Redis
• Session Management
  • Accessing redis sessions

Managing People and Secret Things

• Credentials and secrets
  • Credentials store
• Adding and removing access for new starters / leavers
  • Github
  • AWS
  • Credentials
  • Jenkins
  • GOV.UK PaaS
  • GOV.UK Notify
  • Mailchimp
  • Logit
  • API tokens and other shared credentials
  • Snyk
  • Digital Marketplace administrators
  • Leavers
• Requirements for access to production environments
  • Pairing or shadowing topics

2nd Line Runbook

• Incidents
  • Grading an incident
  • P1 and P2 process
  • P3 and P4 process
• Logging in to databases
• Resetting brief data with conduit
• IP Restrictions
  • Blocking individual IPs
• Upgrading Jenkins
• Jenkins spring clean
  • Updating and cleaning the Jenkins box
  • Update Chrom and Chromedriver
• Rebuilding Jenkins from scratch
  • Restoring an existing Jenkins instance
  • Creating a new Jenkins instance
• Rotating API keys
  • Rotating Data API and Search API keys
  • Rotating Jenkins Docker credentials
  • Rotating Notify API keys
  • Rotating AWS Access keys
• Debugging PaaS App Instances
  • Logging in
  • Showing the environment
  • Using SSH
  • Useful commands
  • Disabling health checks
  • Restarting apps or rebuilding app artifacts (causes downtime)
• Vulnerability scanning
  • Snyk
  • Jenkins
• Novations
  • What is a novation?
  • Steps for transferring services
• Support Tasks
  • Support process
  • Users
  • Updating DOS Briefs
  • Supplier queries
  • General investigation tips
• Responding to Cloudwatch alerts
  • Investigating alerts
  • Next steps
• Buyer Domain Allow List

Digital Marketplace archive

• History: From G-Cloud 5 to G-Cloud 8
• User Journeys on the Digital Marketplace
  • Visual Overview User Journeys
  • Walk-through User Journeys
  • User Task Maps
  • Preview environment
• Archived applications
  • Briefs Frontend
  • Brief Responses Frontend
  • Router app
• Browser and accessibility support
  • Browser review
  • Browser support considerations
  • Assistive technology support
• Digitla Marketplace 1.0 Environments
  • Preview
  • Staging
  • Production
• Repositories
  • Digital Marketplace 1.0
• Recording decisions on the Digital Marketplace
• Nix Environments
  • Who’s nick?
  • default.nix files in Digital Marketplace repositories
  • default.nix in the functional tests repo
• Migrating the Digital Marketplace frontend
  • Background
  • Pieces involved
• URLs
  • Buyer Frontend
  • User Frontend
  • Briefs Frontend
  • Supplier Frontend
  • Brief Responses Frontend
  • Admin Frontend
  • API
  • Search API
  • Special urls
• Google Analytics
  • Google Analytics dashboard
  • Analytics opt-in
  • Cross domain tracking
• Load testing
  • What to test
  • How much load
  • Writing the tests
  • Running the tests
  • Overload testing
• Splitting or adding a frontend application
  • Prerequisites
  • Setting up infrastructure
• Managing database backups for the NFT environment
  • Prerequisites
  • Create backup of the database
  • Telling the NFT environement to use the backup
  • Clean up
• Contract variations
  • What is a contract variation?
  • The workflow for a contract variation
  • How to create a new contract variation
  • How to countersign a contract variation
  • Contract variations in pictures
• Adding a new form field type
  • Purpose
  • Intro
  • How this all works
  • How to
  • Examples
  • References
• Logging in to databases
• Resetting brief data with conduit
• Useful API endpoints for frameworks
  • Users
  • Draft services
  • Declarations
  • Supplier accounts

What is this?

• README