Skip to main content
Bulk Data Extracts

Bulk Data Extracts

Bulk data extracts (IM1 Bulk) provide scheduled, bulk exports of practice data as CSV files, in contrast to the Transactional API, which handles individual real-time operations. Medicus queries the reporting database, writes each query to a CSV, bundles them into a ZIP, and delivers it over SFTP or MESH. All extracts are processed in the United Kingdom.

See the extract schema for the available tables and columns, and onboarding and assurance for how to get set up.

Definitions​

Direct care​

The term "direct care" is defined as a clinical, social or public health activity concerned with the prevention, investigation and treatment of illness and the alleviation of suffering of individuals (all activities that directly contribute to the diagnosis, care and treatment of an individual).

It includes:

  • supporting individuals' ability to function and improve their participation in life and society; the local audit/assurance of the quality of care provided;
  • the management of untoward or adverse incidents;
  • the measurement of outcomes undertaken by one or more registered and regulated health or social care professionals and their team with whom the individual has a legitimate relationship for their care.

It does not include research, teaching, financial audit, service management activities or risk stratification (see note below on borderline cases).

Source: NHS Digital, A guide to confidentiality in health and social care.

Indirect care​

The term "indirect care" is defined as activities that contribute to the overall provision of services to a population as a whole or a group of patients with a particular condition, but which fall outside the scope of direct care. It covers health services management, preventative medicine, and medical research. Examples of indirect care activities include risk prediction and stratification (see note below on borderline cases), service evaluation, needs assessment, and financial audit.

The key reason for distinguishing between purposes in this way is that it is generally possible to imply consent for the use of confidential information for direct care purposes but not for other purposes.

There are some exceptions and some tricky borderline cases on which specific guidance is provided.

Source: NHS Digital, A guide to confidentiality in health and social care.

Supported use cases​

IM1 Bulk may be used for:

  • Targeted extracts for specific reporting use cases (for example CH-IS extracts).
  • Broad extracts transferred to a data warehouse (for example PHM extracts).
  • Direct-care planning where the organisation has a legitimate relationship (for example a practice running its own call/recall).

It must not be used for:

  • Direct care for third-party organisations delivering care (record viewers, clinical decision support): use the Transactional API.
  • Patient-facing services where patients view their own records: use IM1 PFS (the NHS GP Connect Patient Facing Services specification).
  • Data migration: use the NHS Data Migration standard.

Extract definition​

An extract is defined as SQL queries against the reporting tables. In line with minimising data egress, queries must select the minimum data and fields needed for the use case. Multiple CSV files are produced to handle multiple data sources, and all CSVs for a run are delivered together in one ZIP (Medicus sends all files at once).

The definition states:

  • Which reporting data and fields are required.
  • Full extract or deltas. A full extract regenerates the full CSV each run; a delta extract runs an initial full extract, then subsequent runs contain only the changes since the last run. Changing an extract definition restarts the process from a fresh full extract.
  • Frequency: daily.
  • Timing: overnight is preferred, to minimise impact during core hours.

Data protection obligations​

All consumers must:

  • Provide a definition of the intended use of the data.
  • Outline the legal basis for extraction: details of any data provision notice or explanation of the purpose.
  • Provide a DPIA and other documentation explaining the data flows. This documentation must detail how data will be access controlled, particularly when the feed is multi-use (for example direct and indirect care) and controlled via RBAC within the consumer system.
  • Maintain ISO 27001 certification from a UKAS-approved accreditation body, with an SoA covering all processing activities.
  • Maintain Cyber Essentials Plus certification, with scope covering all processing activities.
  • Maintain DSP Toolkit certification to "Standards Exceeded" level.
  • Legally guarantee that all processing and sub-processing only happens within the United Kingdom, unless an exception has been agreed with the data controllers. Medicus shares no liability for any breaches of this agreement.
  • Legally guarantee that there will be no secondary transfers without explicit data controller approval. Medicus shares no liability for any breaches of this agreement.
  • Have an agreed data retention policy which includes a mechanism for cascading GDPR obligations from data controllers.
  • Enforce at least NIST AAL2 authentication levels for access to the data.
  • Detail the procedures they have in place to protect sensitive data from illegal breach.
  • Provide a statement outlining how data remains encrypted once it has been downloaded from MESH or SFTP.
  • Provide a statement that confirms all data will remain in the United Kingdom during the end-to-end data flow, including technical details.
  • Patients with a Type 1 opt-out must be excluded from onward processing unless the use case is direct care, or a statutory data provision notice overrides it.
  • For patients with a Type 1 opt-out preference, if the patient explicitly requests to be removed from historical data extracts, the data controller and consumer are jointly responsible for removing historical data from their systems.
  • Data controllers are responsible for ensuring removal of historical data by the consumer where it is no longer appropriate or legal (for example historical data extracted over many years and then retained beyond what is reasonable or lawful).

More information about patient consent for information sharing can be found on the NHS opt-out page.

Anonymisation and data minimisation​

For all data extracts, the consumer must give special consideration to, and provide a statement on the risk of:

  • "Singling out".
  • "Linkability" (otherwise known as the mosaic effect).

See the ICO guidance on singling out and linkability for more information.

Other information-governance considerations:

  • GDPR, minimisation of data egress: the consumer will only be granted access to data fields that have a clear legal basis for disclosure. As part of the extract definition, consumers must provide clear justification for the data extracted.
  • NHS Act 2006 s251, sensitive medical records: without an explicit justification, no sensitive medical records will be included (for example codes on RCGP exclusion lists).

Data controller obligations and sensitive-data exclusions​

Bulk data extracts are always approved by the GP practice, acting in their capacity as data controller; no bulk data is extracted without prior approval. Where the practice and ICB act as joint data controllers, the practice must approve the extract from their Medicus database.

Unless there is a clear legal basis, the following information should not be used in onward processing flows. This list is not exhaustive, and consumers should take all reasonable efforts to protect patient privacy.

Gender reassignment

  • Any concept in 999004351000000109 | General practice summary data sharing exclusion for gender related issues simple reference set (foundation metadata concept).
  • Any concept in 1955851000000101 | National Health Service secondary use data exclusions due to Gender Recognition Act 2004 simple reference set (foundation metadata concept).

Sexuality

  • Descendants of 118199002 | Finding relating to sexuality and sexual activity (finding).
  • Descendants of 365957000 | Finding related to sexual relationship (finding).
  • Descendants of 118200004 | Finding related to sexual state (finding).
  • Descendants of 171023003 | Psychosexual counseling (procedure).

HIV status

  • Any concept in 999004381000000103 | General practice summary data sharing exclusion for sexually transmitted disease simple reference set (the most robust way to exclude HIV status).

Legal allegations against others

Partly covered by not extracting free text or documents.

Other legally restricted information, to protect against breaching data controller legal obligations, including any concept in the RCGP sensitive dataset:

  • 999004361000000107 | General practice summary data sharing exclusion for termination of pregnancy simple reference set.
  • 999004351000000109 | General practice summary data sharing exclusion for gender related issues simple reference set.
  • 999004371000000100 | General practice summary data sharing exclusion for assisted fertilisation simple reference set.
  • 999004381000000103 | General practice summary data sharing exclusion for sexually transmitted disease simple reference set.

Free text and documents cannot be included, as there is no mechanism to guarantee this type of legally restricted information is not exported, which would lead to a legal breach. Whilst Medicus endeavours to protect sensitive data, the data controller holds the ultimate responsibility for ensuring that data is captured in a way that allows for automatic redaction from bulk data extracts.

Transmission​

Extracts are delivered over SFTP or MESH.

SFTP (hosted by the consumer)​

Pros:

  • Consumers do not need to be assured to use MESH.
  • Works in all contexts (NHS, Channel Islands).

Cons:

  • Additional cost for consumers to host.

What we require:

  • SFTP server host.
  • SFTP server port.
  • SFTP server username.
  • SFTP directory (where the files will be uploaded).

Medicus authenticates using public/private SSH keys (algorithm: ed25519). Add the Medicus public key for each environment you receive extracts from to your SFTP server:

UAT

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHmuZOGYiUQVyOsso8SDhx8GmRhsno2oNmuBM0tE5WPg support@medicus.health

Staging

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJDCqiL07SBXzIFwxgloWrhkLH5mXLHfr9dIc8/EOF7q support@medicus.health

Production

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICgV1uyg1WqW6W/e4j9b5KL3C/2iTCbOhfbFeCn+SZ9t support@medicus.health

The SFTP server must be on the public internet, not HSCN. If you need to add our source IP addresses to your firewall to allow access:

  • UAT: 3.11.49.207
  • Staging: 3.8.218.189
  • Production: 18.135.65.70

MESH​

Pros:

  • Very high availability (NHS Platinum service).
  • Retry mechanism built into the outbox sending pattern.
  • Secure by design: an established pattern for sharing PII.
  • Low cost for consumers.

Cons:

  • Only suitable when transferring data from NHS England organisations; not eligible outside England or for private organisations.

What we require:

  • MESH mailbox ID.

Medicus sends the files from the local organisation's MESH mailbox, using MESH workflow ID GFTD_INIT. Data is sent via MESH over HTTPS and downloaded by the consumer from MESH over HTTPS.

File encryption (optional)​

ZIP files can optionally be encrypted with GPG (OpenPGP): Medicus encrypts the ZIP directly to your public key using AES-256, and only your private key can decrypt it. This means there are no passwords to transmit or guess: security depends on possession of your private key.

You provide an ASCII-armored OpenPGP public key during onboarding (through the practice's extract-enablement screen); it must start with -----BEGIN PGP PUBLIC KEY BLOCK----- and contain exactly one encryption-capable key. The encrypted file is named with a .gpg suffix.

# 1. Generate an encryption-capable key pair (RSA 4096, valid 2 years)
gpg --quick-generate-key "Your Organisation <integrations@example.com>" rsa4096 encrypt 2y

# 2. Export the public key and provide it to the practice when they enable the extract
gpg --armor --export "Your Organisation <integrations@example.com>" > public.asc

# 3. Decrypt the .gpg file Medicus sends (after importing your private key)
gpg --import private.asc
gpg --output medicus-data-extract.zip --decrypt medicus-data-extract.zip.gpg