Support Tasks ¶

There are a number of common support requests we receive that admins cannot carry out themselves. See the Trello board for more information about how we do second line.

Support process ¶

Support requests should be sent to the developer team via ccsrequests@digitalmarketplace.service.gov.uk. This gets forwarded to the Digital Marketplace 2nd line google group and creates a ticket in Zendesk which is our primary support tool. When you receive a support request, check the following before responding:

Did it come in to the ccsrequests/DM 2nd line email group? If the request was sent to you individually, push back asking to resend to correct address. This is important for team visibility and tracking how many requests we receive.
Is the request asking for help with something admins can already do? Check the permissions for different admin roles.
Check whether the request is something we should be handling (see https://trello.com/c/Uji4WJ1A/1001-handling-ccs-support-requests). If you’re not sure or need to escalate a serious issue, ask a senior developer or product/delivery manager.
Check that someone has not already responded in Zendesk.

If the request meets the criteria above, respond in Zendesk, setting an appropriate ticket status e.g. ‘Pending’ or ‘Solved’.

If you use a script or API query in preparing your response, it is useful to include this as an Internal note on the ticket for others to refer back to.

If you’re not sure about any of the above, ask! Talk to a Product or Delivery Manager (or your friendly senior dev).

Responding to requests via the Salesforce integration ¶

The Customer Service Centre (CSC) are the 1st line responding to user support requests for the Digital Marketplace. They use Salesforce, but can escalate tickets to our Zendesk group via an integration. There is some subtlety to responding:

to reply to the user, make a public reply
to reply to the CSC person, apply the “Send Internal Comment back to Salesforce” macro and then make an internal note. You need to apply the macro every time you do this - the tag it adds gets removed after you submit the update
to make an internal note, make a internal note

Users ¶

Inviting a new admin user ¶

Requests to setup an admin account must be approved and come from a user with an existing admin account. The request should also specify the role for the account. Note that you can only select one role per account. For more information on admin roles and permissions, check the section on admin accounts under users.

To invite a new admin user or to deactivate or change permissions for an existing user, follow the same steps used to add or remove Digital Marketplace administrator accounts for developers.

Changing a user’s email ¶

Buyers cannot change the email address on their account. We recommend two options for buyers:

Create a new account with their new email address. They can still log in to their old account (e.g. to view past opportunities) as long as they know the password for it.
Ask their own IT department to set up email forwarding to their new email address, so that they still receive anything sent to the old address.

Suppliers can add a new email address to their account by inviting a contributor. They can remove any inactive email addresses without needing to raise a support request, however admins can also invite/remove contributors if needed.

Admin accounts must be invited/removed by a developer, using the admin-manager account.

User account keeps becoming inactive ¶

Sometimes you’ll be made aware of an account that is repeatedly becoming inactive. This is likely because when we send a Notify email to a user and the mailserver responds with an undeliverable error (also commonly referred to as a hard bounce) we deactivate the account.

This is the correct behaviour because emails, particularly those sent by Notify, are an integral part of our service.

The actual mechanism by which this is achieved is a Notify callback which POSTs to a view on the api which deactivates the user.

The method to check whether or not this is the cause of the user becoming inactive is helpfully detailed above in the Checking audit events section.

Something like the below will find this type of audit event:

  digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.find_audit_events_iter(object_id=<user_id>, object_type='users', audit_type=AuditTypes.update_user)

Removing personal data ¶

As per the Digital Marketplace’s commitment to upholding GDPR regulations our users have the right to request that their personal data be removed from our datasets. This is detailed in our GDPR documentation.

Note

Note that their data will still exist in our backups but the removal will be propagated over 2 years. This is inline with the policy outlined in our GDPR documentation.

For all user types run the script to remove their data from our database and mailchimp:

digitalmarketplace-scripts$ ./scripts/remove-user-personal-information.py production <user email address>

Removing personal data from a supplier ¶

If the user is a ‘supplier’ user then they may have been entered as a contact for the relevant supplier. This can be deleted by issuing the following:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.get_supplier(<SUPPLIER_ID>)
data.remove_contact_information_personal_data(<SUPPLIER_ID>, <CONTACT_ID>)

If there is a request to remove all personal information from a supplier, and the supplier is not on any framework, then the steps above should be followed for all users and contact information on the supplier object.

You should also check whether the supplier has an application in progress for any frameworks that might happen to be open at the time of the request. If they have started an application you should remove the supplier declaration:

data.remove_supplier_declaration(<SUPPLIER_ID>, <FRAMEWORK_SLUG>)

Getting email lists for user research ¶

Users of the Digital Marketplace can opt in to being invited to participate in user research. To get the list of email addresses of opted-in users, log in to the Digital Marketplace as a Framework Manager admin. The links to download the lists are on the admin dashboard.

Updating DOS Briefs ¶

Updating incorrect Brief award details ¶

Buyers must supply details of their awarded contract value and start date. If they enter the wrong award details (e.g. ‘£10.000’ instead of ‘£10,000’) then they must raise a support request. A developer then needs to update the details via an API request.

Award details may also need to be manually updated if the buyer account is inactive (e.g. the user has left the buying organisation).

To update the award details:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

brief_id = 12345
brief_response_id = data.get_brief(brief_id)['briefs']['awardedBriefResponseId']
award_details = data.get_brief_response(brief_response_id)['briefResponses']['awardDetails']

# Make your changes to award_details...
award_details['awardedContractStartDate'] = '2525-01-01'
award_details['awardedContractValue'] = "20000.01"  # £20,000.01

data.update_brief_award_details(
    brief_id,
    brief_response_id,
    award_details,
)

or:

POST `/briefs/<int:brief_id>/award/<int:brief_response_id>/contract-details`
{
    "awardDetails": {
        "awardedContractStartDate": "2525-01-01",
        "awardedContractValue": "20000.01"
    },
    "updated_by": "developers.email@digital.cabinet-office.gov.uk"
}

If the buyer has not awarded the brief at all, the status on the winning BriefResponse must be changed first:

data.update_brief_award_brief_response(
    brief_id,
    brief_response_id,
  )

or:

POST `/briefs/<int:brief_id>/award`
{
    "briefResponseId": 12345,
    "updated_by": "your.email@digital.cabinet-office.gov.uk"
}

To mark a brief as unsuccessful or cancelled:

POST `/briefs/<int:brief_id>/<cancel|unsuccessful>`
{"updated_by": "developers.email@digital.cabinet-office.gov.uk"}

Make sure to check the public brief page to confirm the correct outcome is displayed.

Making changes to a Brief ¶

Warning

You should seek explicit DOS Category approval to perform any of the following actions

Briefs are designed not to be changed once published as that may constitute a breach of procurement regulations. Changing a brief part-way through a procurement could unfairly benefit one supplier over another.

However a brief is not a binding contract to actually exchange money or sign a contract (commonly referred to as a call-off contract).

A buyer may decide they no longer want whatever it was they were procuring so we allow buyers to:

withdraw a brief while it is live and suppliers are making bids
cancel a brief once its tender period has ended

These states signify that the buyer will not be signing a contract based on the outcome of the brief.

Sometimes a supplier needs more information than the buyer provided on the original brief. Because we can’t allow the buyer to change the brief, and because we don’t want to unfairly benefit one supplier over another by giving them more information than everyone else, this information is given by the clarification question process.

The supplier can ask a question about the information they think should have been included in the brief, and the buyer publishes a response to that question openly on the brief. This supplementary information is provided transparently and all suppliers are notified that a question has been answered and supplementary information is available.

This all means that it is very rare that a developer should have to withdraw or remove a brief but is has been discussed as part of threat modelling scenarios.

A buyer wants to legitimately withdraw an opportunity but has lost access to their email address
A malicious buyer account posts defamatory information
A buyer accidentally posts a brief containing security-breaching data or personally identifiable information

Withdrawing ¶

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.withdraw_brief(<BRIEF_ID>)

Removing ¶

Briefs cannot be deleted, or unpublished, because brief responses may be orphaned as a result.

Removing data from public view would have to be done directly in the production database. Publicly visible brief details are stored in the briefs table data JSON column. The current accepted wisdom would be to overwrite the offending field of this data blob with a string such as '<REMOVED>'

You should then create a new briefs index and redirect the briefs-digital-outcomes-and-specialist alias to your new index.

Finally, extract a list of suppliers who have responded to the brief and ask the category team to message them concerning the removal.

Unawarding ¶

Buyers may want to change the brief response that was awarded the opportunity. For example, to correct a mistake.

First, unaward the brief:

data.unaward_brief_response(BRIEF_ID, AWARDED_BRIEF_RESPONSE_ID)

This will set the brief back to closed. The buyer can then award the brief to the correct brief response in the normal way.

Changing the owner ¶

We sometimes get asked to change the owner of a brief. We can’t do this safely. The API gives us no way to do this, and it’s hard to know whether the requester is authorised.

Instead, suggest that the requester accesses the owner’s email to get access to the owning buyer account.

Supplier queries ¶

Updating Supplier DUNS numbers ¶

Warning

A supplier’s DUNS number should only be changed in response to a specific request, giving the supplier ID(s).

Make sure the CCS data controller is aware of any changes requested if they aren’t the one making the request.

To update a DUNS number for one supplier, first check that the new number isn’t already in use:

GET `/suppliers?duns_number=123456789`

or:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.find_suppliers(duns_number=<DUNS_NUMBER>)

Update the DUNS number:

POST `/suppliers/<int:supplier_id>`
{
    "updated_by": "developers.email@digital.cabinet-office.gov.uk",
    "suppliers": {
        "dunsNumber": "123456789"   # Must be a string, not an integer
    }
}

or:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

data.update_supplier(<SUPPLIER_ID>, {'dunsNumber': '<DUNS_NUMBER>'})

Any declarations for open applications will be updated, but any other declarations won’t.

To swap DUNS numbers for two suppliers, run oneoff/swap-supplier-duns.py from the root of the scripts repo:

scripts/oneoff/swap-supplier-duns.py <SUPPLIER_ID_1> <SUPPLIER_ID_2> --updated-by=<YOUR_WORK_EMAIL> --stage=production

The two keyword options are not optional and must to be included.

Novations ¶

This is when one supplier takes over another supplier, and wants to sell their services.

See Novations for more details.

Supplier has made a mistake when signing framework agreement ¶

Note

If you are unfamiliar with framework agreements and how they work, see the Framework Agreements manual page for detailed info.

A supplier may have made a mistake when signing a framework agreement, or wish for a different user associated with the supplier account to sign the agreement.

Tell CSC to ask the user to find the link in the email they received inviting them to sign their framework agreement. From this page, they can re-sign the agreement and a new countersigned document will be generated with the updated details.

In the case where the agreement is not standard, the supplier should collaborate with CSC to produce the countersigned agreement. When ready, a developer can upload it following these steps:

Confirm that the FrameworkAgreement object exists. You can find this using data.get_supplier_framework_info in the API client.
Confirm that the countersigned field is True, countersignedAt has a timestamp and countersignedDetails is not empty.
Confirm that the agreement document is signed and countersigned.
If all points above match, use https://github.com/Crown-Commercial-Service/ccs-digitalmarketplace-scripts/blob/master/scripts/upload-counterpart-agreements.py to upload the new updated countersigned agreement (check script for instructions)
Confirm on the API that the FrameworkAgreement object points to the new document

Supplier wishes to update Supplier Authorised Representative ¶

Note

If you are unfamiliar with framework agreements and how they work, see the Framework Agreements manual page for detailed info.

The DOS5 contract produced by the e-signature flow includes a ‘Supplier Authorised Representative’ section which is populated based on the declaration questions answered by the supplier. A supplier may wish to change these so that different details appear on the contract.

These fields are based on the contactNameContractNotice and contactEmailContractNotice fields in the supplier’s declaration, to update them run the following in the API client shell:

data.update_supplier_declaration(<SUPPLIER_ID>, '<FRAMEWORK_SLUG>',{'contactNameContractNotice': '<NAME>',  'contactEmailContractNotice': '<EMAIL_ADDRESS>'})

Adding or updating a Modern Slavery statement ¶

Once framework applications have closed, suppliers can’t edit their Modern Slavery statement themselves, so they will have to raise a support ticket.

For DOS, modern slavery statements are not publicly available. However the support team may request them, so we should update them anyway.

Make sure CSC provided the supplier ID and the new document.
Check that the document is under 5MB and is either Open Document Format (ODF) or PDF/A (eg .pdf, .odt).

Run the script from digitalmarketplace-scripts to upload the file and set it as the supplier’s modern slavery statement for the framework:

AWS_PROFILE=production-infrastructure scripts/upload-modern-slavery-statement.py <file_path> <supplier_id> <framework> <s3_file_name>

For example:

AWS_PROFILE=production-infrastructure scripts/upload-modern-slavery-statement.py /tmp/file.pdf 123456 g-cloud-12 modern-slavery-declaration-2020-11-19.pdf

For G-Cloud, check one of the supplier’s services to make sure the new document is visible.

Adding or updating a service document ¶

Suppliers can replace their own Service Definition Document once the framework is live, but they cannot remove it completely. Occasionally it’s necessary to remove the Service Definition Document (e.g. following a novation), however from G-Cloud 12 onwards the document is mandatory, so a replacement must be provided.

To remove a Service Definition Document:

data.update_service(<service_id>, {'serviceDefinitionDocumentURL': None})

To update a service document, ensure the new file is present in the production S3 documents bucket. The filename should be in the format <service_id>-<document-type>-<datestamp>.pdf, for example 12345678901-pricing-document-2020-01-01.pdf.

Then update the service with the full https://assets.digitalmarketplace.service.gov.uk/… path:

data.update_service(<service_id>, {'serviceDefinitionDocumentURL': 'https://assets...'})
data.update_service(<service_id>, {'sfiaRateDocumentURL': 'https://assets...'})
data.update_service(<service_id>, {'termsAndConditionsDocumentURL': 'https://assets...'})
data.update_service(<service_id>, {'pricingDocumentURL': 'https://assets...'})

Change supplier details on DOS service CSVs ¶

When buyers start a DOS opportunity on a particular lot, they have access to a CSV download of supplier services for that lot. These CSVs are generated by a developer when the framework opens and uploaded to a public S3 bucket (see more info on this process at making DOS services live).

Sometimes suppliers need to change the details of their services during a framework, for example adding a new specialist role or lowering day rates. These service changes can be made by a CCS Category admin, however they are not automatically reflected in the public CSVs available to buyers.

The DOS service export scripts take a long time to run, and need ‘cleaning’ before they’re suitable for publishing to a public bucket to remove any sensitive information. For ad hoc change requests it’s quicker and easier to manually edit the CSV instead. Make sure to follow the same upload process on S3, remembering to:

update the ‘Last updated’ line at the top of the CSV
follow the existing file name pattern
set the Content-Disposition header to attachment
set the Cache-Control header to max-age=3600
make the file readable to the public

Supplier querying their DOS brief submission ¶

It is important to suppliers that their brief responses are received by the buyer but there is some evidence that suppliers can neglect to either complete their response or to hit submit before the deadline. In these cases we often recieve support requests from suppliers, via CSC, who want to find out if there was any platform instability that could have impacted the submission of their brief.

Digital Marketplace has robust Monitoring, Alerting and Logging and we use this to investigate every error returned from the platform. The case of user not being able to submit a brief would constitute a P1-P3 incident in line with our incident grading scale detailed in the Incidents section of this manual. This is significant because the steps taken in managing such incidents include proactively contacting CSC to notify them of impact to users and degradation of service.

What this all means is that if a user is served an error from a Digital Marketplace server whilst submitting a brief CSC would be aware of it.

If you have received a support request asking whether Digital Marketplace has experienced any instability that could affect the submission of brief responses or if you can investigate whether any users were served errors whilst submitting a brief response please reply with the standardised response on the template card on the Digital Marketplace 2ndLine Trello Board.

In the case of such instability it is possible to check whether:

the user made a request to submit their brief response
whether that request actioned the submission of their brief response

You will need the supplier id, brief id, access to credentials and the scripts repo and access to Kibana:

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

supplier_id=<SUPPLIER_ID>
brief_id=<BRIEF_ID>

from dmapiclient.audit import AuditTypes

brief_response_id = data.find_brief_responses(
    supplier_id= supplier_id,
    brief_id=brief_id,
    status='draft,awarded,pending-awarded,submitted'
)['briefResponses'][0]['id']

audits = data.find_audit_events(
    object_type='brief-responses',
    object_id=brief_response_id,
    data_supplier_id= supplier_id,
    audit_type=AuditTypes.submit_brief_response
)['auditEvents']

print('"/brief-responses/' + str(brief_response_id) + '/submit"')
print(audits)

The above will print a string containing the brief_response_id of the brief response (like "/brief-responses/12345/submit"). You can then search for this in Kibana which will tell you if we received a request from the user to submit their brief response.

It will also print whether we have an audit event record of the request having actioned the submission of their brief response.

Note

If you want to look up the suppliers updates to the brief response you can change the audit_type passed to find_audit_events to AuditTypes.update_brief_response.

General investigation tips ¶

Checking logs ¶

Our platform logs are available in Kibana for 30 days. For logs going further back, use AWS Cloudwatch. See the Logging and Monitoring manual page for more information.

Notify email logs are stored for 7 days. See the Email integration manual page for more information. We also log messages about Notify delivery receipts using our API app, so if an email was sent more than 7 days ago there will still be a record of it in our own logs. See our Notify callbacks manual page.

Checking audit events ¶

Occasionally we’ll need to get a list of supplier actions from the audit events, (for example, to check whether a supplier completed part of their framework application).

One way to do this is via the API client. The easiest way to initialise it is via scripts/api-clients-shell.py in the scripts repo. This will initialise the DataAPIClient in a Python shell, fetching the credentials from your local environment rather than having to copy and paste the key into the shell.

For example, to find all the update draft service events for supplier ID 12345 (using the API client AuditTypes):

digitalmarketplace-scripts$ ./scripts/api-clients-shell.py preview

> audits = data.find_audit_events_iter(data_supplier_id=12345, audit_type=AuditTypes.update_draft_service)
> for audit in audits:
>    print(audit)

{'acknowledged': False, 'createdAt': '2019-01-23T00:12:23.456789Z', 'data': {'draftId': 654321, 'serviceId': None, 'supplierId': 12345, 'updateJson': {'serviceManagerPriceMax': '890'}}, 'id': 5123456, 'links': {'self': 'https://api.digitalmarketplace.service.gov.uk/audit-events'}, 'objectId': 123456, 'objectType': 'DraftService', 'type': 'update_draft_service', 'user': '12345@example.com'}
{'acknowledged': False, 'createdAt': '2019-02-23T00:12:23.456789Z', 'data': {'draftId': 654321, 'serviceId': None, 'supplierId': 12345, 'updateJson': {'technicalArchitectLocations': ['Offsite']}}, 'id': 5123457, 'links': {'self': 'https://api.digitalmarketplace.service.gov.uk/audit-events'}, 'objectId': 123456, 'objectType': 'DraftService', 'type': 'update_draft_service', 'user': '12345@example.com'}

Make sure to use find_audit_events_iter(), otherwise only the first page of audit events will be returned.

Other useful API client filters:

> client.find_audit_events_iter(object_type="suppliers", object_id=12345)
> client.find_audit_events_iter(user="12345@example.com")

Note that the list of queryable object types is found in the API repo’s audits.py view.

Checking user login events ¶

A user’s ‘last logged in date’ is recorded in the database, along with the number of subsequent failed logins.

Successful and failed logins are also recorded in the User Frontend application logs. These are available for 30 days in Kibana, or in Cloudwatch.

Get the hashed version of the user’s email address with the utils helper function:

> from dmutils.email.helpers import hash_string
> hash_string('info@example.com')
> '-xpHV_g7dOWofBVUyGibqxLQlnT54V2zZsNjarRSAEw='

Kibana search query:

message: "-xpHV_g7dOWofBVUyGibqxLQlnT54V2zZsNjarRSAEw="

To search in Cloudwatch:

Log in to the AWS console
Switch to the correct region (e.g. eu-west-1)
Switch to a production role
Go to CloudWatch > CloudWatch Logs > Log groups > production-user-frontend-application
Select ‘Search All’
Set a date range
Search for the hashed string

Finding historic support requests ¶

Tickets in the CCS Zendesk date back to October 2021, when the Digital Marketplace moved to CCS. If you’re on the Digital Marketplace development team, you should have a login for the GDS Zendesk to see historic support requests. Other people with Google access can see the Zendesk export.