diff --git a/OLIDS/Documentation/Schema/Allergy_Intolerance.md b/OLIDS/Documentation/Schema/Allergy_Intolerance.md index 798592e..6a7f932 100644 --- a/OLIDS/Documentation/Schema/Allergy_Intolerance.md +++ b/OLIDS/Documentation/Schema/Allergy_Intolerance.md @@ -49,6 +49,7 @@ Substances include, but are not limited to: a therapeutic substance administered | `LDS_IS_DELETED` | `BOOLEAN` | standardised representation of soft-deletes. | - | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | - | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP_NTZ` | The timestamp when the record was supplied to, or acquired by, LDS. | - | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_NTZ` | The timestamp when the record was transformed by LDS into OLIDS. | - | -- | ## Entity relationships diff --git a/OLIDS/Documentation/Schema/Appointment.md b/OLIDS/Documentation/Schema/Appointment.md index 61261c6..57f76b3 100644 --- a/OLIDS/Documentation/Schema/Appointment.md +++ b/OLIDS/Documentation/Schema/Appointment.md @@ -62,6 +62,7 @@ This definition takes the concepts of appointments in a clinical setting and als | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes th data. | | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | | -- | ## Entity relationships diff --git a/OLIDS/Documentation/Schema/Appointment_Practitioner.md b/OLIDS/Documentation/Schema/Appointment_Practitioner.md index a554a9c..ed8fe79 100644 --- a/OLIDS/Documentation/Schema/Appointment_Practitioner.md +++ b/OLIDS/Documentation/Schema/Appointment_Practitioner.md @@ -36,6 +36,7 @@ List of practitioner participants involved in an appointment. | `LDS_IS_DELETED` | `BOOLEAN` | standardised representation of soft-deletes. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP_NTZ` | The timestamp when the record was supplied to, or acquired by, LDS. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | LDS transform date time. | | -- | ## Entity relationship diff --git a/OLIDS/Documentation/Schema/Diagnostic_Order.md b/OLIDS/Documentation/Schema/Diagnostic_Order.md index d244960..8fe4b93 100644 --- a/OLIDS/Documentation/Schema/Diagnostic_Order.md +++ b/OLIDS/Documentation/Schema/Diagnostic_Order.md @@ -53,6 +53,7 @@ The resource allows requesting only a single procedure. If a workflow requires r | `LDS_IS_DELETED` | `BOOLEAN` | standardised representation of soft-deletes. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | The timestamp when the record was supplied to, or acquired by, LDS. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | ## Entity relations diff --git a/OLIDS/Documentation/Schema/Encounter.md b/OLIDS/Documentation/Schema/Encounter.md index 2b925db..4237c14 100644 --- a/OLIDS/Documentation/Schema/Encounter.md +++ b/OLIDS/Documentation/Schema/Encounter.md @@ -47,6 +47,7 @@ A patient encounter is further characterized by the setting in which it takes pl | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | ## Entity relationship diff --git a/OLIDS/Documentation/Schema/Episode_Of_Care.md b/OLIDS/Documentation/Schema/Episode_Of_Care.md index 3f1342f..d53574c 100644 --- a/OLIDS/Documentation/Schema/Episode_Of_Care.md +++ b/OLIDS/Documentation/Schema/Episode_Of_Care.md @@ -37,6 +37,7 @@ An association between a patient and an organisation / healthcare provider(s) du | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | ## Entity relationships diff --git a/OLIDS/Documentation/Schema/Location.md b/OLIDS/Documentation/Schema/Location.md index 8f2b147..5ed1be3 100644 --- a/OLIDS/Documentation/Schema/Location.md +++ b/OLIDS/Documentation/Schema/Location.md @@ -39,6 +39,7 @@ A Location includes both incidental locations (a place which is used for healthc | `IS_OBSOLETE` | `BOOLEAN` | is obsolete. | | -- | | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | -- | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP_NTZ` | source extraction date. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | ## Entity Relationships diff --git a/OLIDS/Documentation/Schema/Medication_Order.md b/OLIDS/Documentation/Schema/Medication_Order.md index 3db2518..a2dfacc 100644 --- a/OLIDS/Documentation/Schema/Medication_Order.md +++ b/OLIDS/Documentation/Schema/Medication_Order.md @@ -62,6 +62,7 @@ The MedicationRequest resource allows requesting only a single medication. If a | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | 1. See the [schema notes section on publisher, provider, author organisation definitions](_schema_notes.md#provider-author-publisher-organisation-id) diff --git a/OLIDS/Documentation/Schema/Medication_Statement.md b/OLIDS/Documentation/Schema/Medication_Statement.md index ec6366c..3a95074 100644 --- a/OLIDS/Documentation/Schema/Medication_Statement.md +++ b/OLIDS/Documentation/Schema/Medication_Statement.md @@ -3,9 +3,11 @@ - [Medication\_Statement](#medication_statement) - [Overview](#overview) - [Columns](#columns) + - [Cancellation date derived](#cancellation-date-derived) - [Entity Relationships](#entity-relationships) - [Notes](#notes) + ## Overview Linked FHIR resource: [🔥 Medication Statement](https://hl7.org/fhir/medicationstatement.html) @@ -36,7 +38,8 @@ The primary difference between a medicationstatement and a medicationadministrat | `REFERRAL_REQUEST_ID` | `UUID` | referral request id. | FK -> [Referral_Request](Referral_Request.md).ID | -- | | `CLINICAL_EFFECTIVE_DATE` | `DATE` | clinical effective date. | | `clinical_effective_date` | | `CLINICAL_EFFECTIVE_DATE_PRECISION_SOURCE_CONCEPT_ID` | `UUID` | date precision concept id. | FK -> [Concept](Concept.md).ID | `date_precision_concept_id` | -| `CANCELLATION_DATE` | `DATE` | cancellation date. | | `cancellation_date` | +| `CANCELLATION_DATE` | `DATE` | supplied cancellation date. | | - | +| `CANCELLATION_DATE_DERIVED` | `DATE` | cancellation date derived using logic (See below) | | `cancellation_date` | | `DOSE` | `VARCHAR` | dose. | | `dose` | | `QUANTITY_VALUE_DESCRIPTION` | `VARCHAR` | quantity value description. | | -- | | `QUANTITY_VALUE` | `DOUBLE` | quantity value. | | `quantity_value` | @@ -56,10 +59,40 @@ The primary difference between a medicationstatement and a medicationadministrat | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | 1. See the [schema notes section on publisher, provider, author organisation definitions](_schema_notes.md#provider-author-publisher-organisation-id) +### Cancellation date derived + +The `CANCELLATION_DATE_DERIVED` column has been added to this model to allow for parity with prior systems that used logic to determine the cancellation date of a medication statement, rather than the supplied value alone. As we do not wish to hide or obfuscate supplied values, we have chosen to supplement the table with an additional derived value for parity. + +The logic is as below: + +- For EMIS: + - If the drug record is marked as active then it must not be cancelled -> set cancellation_date_derived to `null` + - If the drug record cancellation date is populated then use this date -> set cancellation_date_derived to the supplied `cancellation_date` + - If the drug record cancellation date is null then: + - if the latest issue of the corresponding medication is not null, use the latest issue date plus the expected course duration according to the latest issue + - if there is no latest issue of the corresponding medication, then simply use the 'effective date' of the statement + +```sql +, CASE + WHEN PDR1.IS_ACTIVE = TRUE + THEN NULL -- Java explicitly nulls active med cancellation dates + WHEN PDR1.CANCELLATION_DATE IS NOT NULL + THEN PDR1.CANCELLATION_DATE -- use source value if present + WHEN LI.EFFECTIVE_DATE IS NOT NULL + THEN DATEADD( + DAY, + COALESCE(LI.COURSE_DURATION_IN_DAYS, 0), + LI.EFFECTIVE_DATE -- infer: latest issue date + course duration + ) + ELSE PDR1.EFFECTIVE_DATE -- fallback: DrugRecord EffectiveDate +END AS CANCELLATION_DATE_DERIVED +``` + ## Entity Relationships > [!NOTE] diff --git a/OLIDS/Documentation/Schema/Observation.md b/OLIDS/Documentation/Schema/Observation.md index 18b8a36..0965131 100644 --- a/OLIDS/Documentation/Schema/Observation.md +++ b/OLIDS/Documentation/Schema/Observation.md @@ -62,6 +62,7 @@ Uses for the Observation resource include: | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | 1. See the [schema notes section on publisher, provider, author organisation definitions](_schema_notes.md#provider-author-publisher-organisation-id) diff --git a/OLIDS/Documentation/Schema/Patient.md b/OLIDS/Documentation/Schema/Patient.md index 2b4b24f..521679a 100644 --- a/OLIDS/Documentation/Schema/Patient.md +++ b/OLIDS/Documentation/Schema/Patient.md @@ -64,6 +64,7 @@ The data in the Resource covers the "who" information about the patient: its att | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the |data. | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | - | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | | -- | 1. See the [schema notes section on publisher, provider, author organisation definitions](_schema_notes.md#provider-author-publisher-organisation-id) diff --git a/OLIDS/Documentation/Schema/Patient_Address.md b/OLIDS/Documentation/Schema/Patient_Address.md index fd28283..9b65a4d 100644 --- a/OLIDS/Documentation/Schema/Patient_Address.md +++ b/OLIDS/Documentation/Schema/Patient_Address.md @@ -3,7 +3,6 @@ - [Patient\_Address](#patient_address) - [Overview](#overview) - [Columns](#columns) - - [pseudo view](#pseudo-view) - [Entity Relationships](#entity-relationships) - [Notes](#notes) @@ -19,8 +18,6 @@ Patient addresses with different uses or applicable periods. ## Columns -### pseudo view - | Column Name | Data Type (Size) | Description | PK/FK | Masking Policy | Compass Equivalent | | --- | --- | --- | --- | --- | --- | | `ID` | `UUID` | id. | | | `id` | @@ -43,6 +40,7 @@ Patient addresses with different uses or applicable periods. | `LDS_IS_DELETED` | `BOOLEAN` | True if the record has been marked as deleted. | | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | ODS code of the organisation who, acting as the data controller, permitted the release of data. | | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | Timestamp extracted from source file name indicating extraction time. | | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | - | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP` WITH TIME ZONE | lds transform date time. | | | -- | ## Entity Relationships diff --git a/OLIDS/Documentation/Schema/Patient_Age_Flags.md b/OLIDS/Documentation/Schema/Patient_Age_Flags.md new file mode 100644 index 0000000..8e3057f --- /dev/null +++ b/OLIDS/Documentation/Schema/Patient_Age_Flags.md @@ -0,0 +1,50 @@ +# Patient_Age_Flags + +- [Patient\_Age\_Flags](#patient_age_flags) + - [Overview](#overview) + - [Columns](#columns) + - [Entity Relationships](#entity-relationships) + +## Overview + +Related FHIR resource: _not applicable_ + +This table allows for the service to provide specific age related flags. These flags can be used by pseudonymised access users to determine if the patient is within age eligibility criteria for specific treatments or pathways. + +## Columns + +| Column Name | Data Type (Size) | Description | PK/FK | Masking Policy | Compass Equivalent | +| --- | --- | --- | --- | --- | --- | +| `ID` | `UUID` | The Patient ID | FK ->[PATIENT](Patient.md).ID | - | - | +| `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | record owner organisation code. | - | - | - | +| `IS_LESS_THAN_8_WEEKS_OLD` | `BOOLEAN` | Is the patient under 8 weeks of age today | - | - | - | +| `IS_LESS_THAN_12_WEEKS_OLD` | `BOOLEAN` | Is the patient under 12 weeks of age today | - | - | - | +| `IS_LESS_THAN_15_WEEKS_OLD` | `BOOLEAN` | Is the patient under 15 weeks of age today | - | - | - | +| `IS_LESS_THAN_16_WEEKS_OLD` | `BOOLEAN` | Is the patient under 16 weeks of age today | - | - | - | +| `IS_LESS_THAN_24_WEEKS_OLD` | `BOOLEAN` | Is the patient under 24 weeks of age today | - | - | - | +| `IS_LESS_THAN_6_MONTHS_OLD` | `BOOLEAN` | Is the patient under 6 months of age today | - | - | - | +| `IS_LESS_THAN_8_MONTHS_OLD` | `BOOLEAN` | Is the patient under 8 months of age today | - | - | - | +| `IS_LESS_THAN_12_MONTHS_OLD` | `BOOLEAN` | Is the patient under 12 months of age today | - | - | - | +| `IS_LESS_THAN_18_MONTHS_OLD` | `BOOLEAN` | Is the patient under 18 months of age today | - | - | - | +| `IS_LESS_THAN_1000_DAYS_OLD` | `BOOLEAN` | Is the patient under 1000 days of age today | - | - | - | +| `IS_BETWEEN_1_TO_18_MONTHS_OLD` | `BOOLEAN` | Is the patient between 1 to 18 months of age today | - | - | - | +| `IS_LESS_THAN_2_YEARS_OLD` | `BOOLEAN` | Is the patient under 2 years of age today | - | - | - | +| `IS_BETWEEN_3_TO_4_YEARS_OLD` | `BOOLEAN` | Is the patient between 3 to 4 years of age today | - | - | - | +| `IS_LESS_THAN_5_YEARS_OLD` | `BOOLEAN` | Is the patient under 5 years of age today | - | - | - | +| `IS_BETWEEN_11_TO_18_YEARS_OLD` | `BOOLEAN` | Is the patient between 11 to 18 years of age today | - | - | - | + + +## Entity Relationships + +> [!NOTE] +> Diagrams below are currently indicative. The precise optional/mandatory nature of certain relationships remains to be clarified. + +```mermaid +erDiagram + + PATIENT ||--|| PATIENT_AGE_FLAGS: ID +``` + +| Related Table | Relationship Type | Local Key | Related Key | Notes | +| --- | --- | --- | --- | --- | +| [Patient](Patient.md) | FK | ID | ID | one-to-one relation | diff --git a/OLIDS/Documentation/Schema/Patient_Contact.md b/OLIDS/Documentation/Schema/Patient_Contact.md index 8dc50bb..54ccd49 100644 --- a/OLIDS/Documentation/Schema/Patient_Contact.md +++ b/OLIDS/Documentation/Schema/Patient_Contact.md @@ -30,6 +30,7 @@ A Patient may have multiple ways to be contacted with different uses or applicab | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | | | `organization_id` | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | - | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | | -- | ## Entity Relationships diff --git a/OLIDS/Documentation/Schema/Patient_Person.md b/OLIDS/Documentation/Schema/Patient_Person.md index 8130e72..5332da2 100644 --- a/OLIDS/Documentation/Schema/Patient_Person.md +++ b/OLIDS/Documentation/Schema/Patient_Person.md @@ -24,6 +24,7 @@ | `LDS_SOURCE_RECORD_ID_PERSON` | `UUID` | Record-level identifier of the linked person record, sourced from Person. Retained separately from LDS_SOURCE_RECORD_ID to distinguish the person record identifier from the patient record identifier in this link table. | | | -- | | `GP_PRACTICE_CODE` | `VARCHAR` | ODS code of the patient's registered GP practice, sourced from Person. 'Unknown' when the organisation could not be resolved. | | | -- | | `LDS_IS_DELETED` | `BOOLEAN` | Indicates whether the source patient record has been logically deleted. Coalesced to FALSE when the source value is NULL. | | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | - | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | | -- | ## Entity Relationships diff --git a/OLIDS/Documentation/Schema/Practitioner.md b/OLIDS/Documentation/Schema/Practitioner.md index 15a75f0..2554544 100644 --- a/OLIDS/Documentation/Schema/Practitioner.md +++ b/OLIDS/Documentation/Schema/Practitioner.md @@ -54,6 +54,7 @@ The Practitioner resource is used for anyone involved in the provision of care o | `IS_OBSOLETE` | `BOOLEAN` | is practitioner record obsolete. | - | -- | | `LDS_IS_DELETED` | `BOOLEAN` | is the practitioner record deleted in the source system. | - | -- | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | - | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | the timestamp when the transform process that generated this record. | - | -- | ## Entity Relationships diff --git a/OLIDS/Documentation/Schema/Practitioner_In_Role.md b/OLIDS/Documentation/Schema/Practitioner_In_Role.md index e93cb0b..de1e81d 100644 --- a/OLIDS/Documentation/Schema/Practitioner_In_Role.md +++ b/OLIDS/Documentation/Schema/Practitioner_In_Role.md @@ -50,6 +50,7 @@ For use cases where an organisation has activities where a practitioner is not d | `DATE_EMPLOYMENT_END` | `DATE` | date employment end. | | -- | | `LDS_IS_DELETED` | `BOOLEAN` | True if the record has been marked as deleted. | | -- | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | Timestamp extracted from source file name indicating extraction time. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | ## Entity Relationships diff --git a/OLIDS/Documentation/Schema/Procedure_Request.md b/OLIDS/Documentation/Schema/Procedure_Request.md index 7c02ad2..695da32 100644 --- a/OLIDS/Documentation/Schema/Procedure_Request.md +++ b/OLIDS/Documentation/Schema/Procedure_Request.md @@ -52,6 +52,7 @@ The ProcedureRequest resource allows requesting only a single procedure. If a wo | `LDS_IS_DELETED` | `BOOLEAN` | standardised representation of soft-deletes. | - | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | The Organisation Data Service (ODS) code of the organisation who, acting as the data controller, publishes the data. | - | -- | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP_NTZ` | The timestamp when the record was supplied to, or acquired by, LDS. | - | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_NTZ` | The timestamp when the record was transformed by LDS into OLIDS. | - | -- | ## Entity relationships diff --git a/OLIDS/Documentation/Schema/Referral_Request.md b/OLIDS/Documentation/Schema/Referral_Request.md index 236ee0a..13710ab 100644 --- a/OLIDS/Documentation/Schema/Referral_Request.md +++ b/OLIDS/Documentation/Schema/Referral_Request.md @@ -62,6 +62,7 @@ ReferralRequest is also intended for use when there is a complete and more perma | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | record owner organisation code. | | -- | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_NTZ` | The timestamp when the record was transformed by LDS into OLIDS. | - | -- | ## Entity Relationships diff --git a/OLIDS/Documentation/Schema/Schedule.md b/OLIDS/Documentation/Schema/Schedule.md index 15fee6b..163c0d9 100644 --- a/OLIDS/Documentation/Schema/Schedule.md +++ b/OLIDS/Documentation/Schema/Schedule.md @@ -36,6 +36,7 @@ The schedule does not provide any information about actual appointments. This se | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | record owner organisation code. | | | -- | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | - | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_NTZ` | The timestamp when the record was transformed by LDS into OLIDS. | - | | -- | ## Entity Relationships diff --git a/OLIDS/Documentation/Schema/Schedule_Practitioner.md b/OLIDS/Documentation/Schema/Schedule_Practitioner.md index ea4ba70..6dd83e5 100644 --- a/OLIDS/Documentation/Schema/Schedule_Practitioner.md +++ b/OLIDS/Documentation/Schema/Schedule_Practitioner.md @@ -31,6 +31,7 @@ If the schedule needed to be consulted, then there would be one created covering | `LDS_IS_DELETED` | `BOOLEAN` | lds is deleted. | | -- | | `PUBLISHER_ORGANISATION_CODE` | `VARCHAR` | record owner organisation code. | | -- | | `SOURCE_EXTRACTION_DATE` | `TIMESTAMP` | source extraction date. | | -- | +| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- | | `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- | ## Entity Relationships diff --git a/OLIDS/Release-notes/1_OLIDS_Share/v1.4.0.md b/OLIDS/Release-notes/1_OLIDS_Share/v1.4.0.md new file mode 100644 index 0000000..9fc95f7 --- /dev/null +++ b/OLIDS/Release-notes/1_OLIDS_Share/v1.4.0.md @@ -0,0 +1,32 @@ +# Release v1.4.0 - 2026-08-05 + +> [!NOTE] +> The below is a summary of the changes since the previous release [v1.3.1](v1.3.1.md). + +## Summary + +This release introduces a new shared macro for postcode formatting, enhancing consistency and reusability across the project. The update streamlines postcode handling and removes obsolete log files, resulting in a cleaner repository. + +### New Features + +- Added a `format_postcode` shared macro for standardized postcode formatting. [PR: #17](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/17) + > ✨ ***Feature:*** Enables consistent and reusable postcode formatting across different parts of the project, reducing duplication and potential errors. + >
Supports ongoing development for hashed postcode migration to UKHFD. + +### Inputs + +This changelog was automatically produced from: + +- 1 Pull Requests +- 0 Issues +- 0 WorkItems +- 5 Commits + +## Full Details + +Pull Requests included in this release: + +- [PR #17](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/17): Added format_postcode shared macro + - No linked issues. + +[View the diff between v1.3.1 and v1.4.0](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/compare/releases%2Fv1.3.1...releases%2Fv1.4.0) \ No newline at end of file diff --git a/OLIDS/Release-notes/2_EMIS_OLIDS/v2.3.0.md b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.3.0.md new file mode 100644 index 0000000..f489a0e --- /dev/null +++ b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.3.0.md @@ -0,0 +1,40 @@ +# Release v2.3.0 - 2026-08-05 + +> [!NOTE] +> The below is a summary of the changes since the previous release [v2.2.0](v2.2.0.md). + +## Summary + +This release addresses a critical issue in patient deduplication logic, ensuring that individuals with placeholder NHS numbers are no longer incorrectly merged into a single record. Comprehensive unit tests have been added to validate this fix and prevent future regressions. Additionally, the OLIDS_Share package has been updated to the latest version, bringing in recent features and improvements from upstream dependencies. + +### Improvements + +- Update OLIDS_Share revision to v1.4.0 *[PR: [#115](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/115)]* + > 🎯 ***Impact:*** Ensures the project benefits from the latest features and bug fixes in the OLIDS_Share package, improving overall compatibility and stability. + +### Bug Fixes + +- Fix deduplication logic and add tests for NHS_NUMBER placeholders *[PR: [#114](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/114), WorkItems: [AB#30111](https://dev.azure.com/NELAnalytics/aef6c175-110f-4ba4-82fc-70761f459236/_workitems/edit/30111), [AB#29871](https://dev.azure.com/NELAnalytics/aef6c175-110f-4ba4-82fc-70761f459236/_workitems/edit/29871)]* + > 🐞 ***Fix:*** Resolves an issue where patients with placeholder NHS numbers (e.g., '0000000000', '9999999999', or empty) were incorrectly collapsed into a single row. The deduplication logic now correctly distinguishes patients using the generated person ID, and new unit tests ensure ongoing accuracy and data integrity. + >
**This directly addresses bug [#AB29871 - (SEL) - Person_ID is null for small number of records](https://dev.azure.com/NELAnalytics/LondonDataService/_workitems/edit/29871)** + +### Inputs + +This changelog was automatically produced from: + +- 2 Pull Requests +- 0 Issues +- 2 WorkItems +- 5 Commits + +## Full Details + +Pull Requests included in this release: + +- [PR #114](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/114): Fix deduplication logic and add tests for NHS_NUMBER placeholders + - [WorkItem AB#30111](https://dev.azure.com/NELAnalytics/aef6c175-110f-4ba4-82fc-70761f459236/_workitems/edit/30111) + - [WorkItem AB#29871](https://dev.azure.com/NELAnalytics/aef6c175-110f-4ba4-82fc-70761f459236/_workitems/edit/29871) +- [PR #115](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/115): Update OLIDS_Share revision to v1.4.0 + - No linked issues. + +[View the diff between v2.2.0 and v2.3.0](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/compare/releases%2Fv2.2.0...releases%2Fv2.3.0) \ No newline at end of file diff --git a/OLIDS/Release-notes/3_OLIDS_Enrichment/v2.3.0.md b/OLIDS/Release-notes/3_OLIDS_Enrichment/v2.3.0.md new file mode 100644 index 0000000..1b8802b --- /dev/null +++ b/OLIDS/Release-notes/3_OLIDS_Enrichment/v2.3.0.md @@ -0,0 +1,61 @@ +# Release v2.3.0 - 2026-08-05 + +> [!NOTE] +> The below is a summary of the changes since the previous release [v2.2.1](v2.2.1.md). + +## Summary + +This release introduces several enhancements and improvements across the OLIDS Enrichment project. Key highlights include a major overhaul of the NDOO request handling pipeline for better tracking and retry management, consistent inclusion of source dataset fields across PID and Pseudo models, and the addition of clustering to the Person_PDS_Versioned table for improved query performance. The release also standardizes age flag logic in patient models, adds derived cancellation dates to medication statements, and updates package dependencies to the latest versions. These changes collectively improve data quality, maintainability, and platform robustness. + +### New Features + +- Enhance NDOO request handling with new view and refactoring [PR: #58](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/58) + > ✨ ***Feature:*** Introduces a new, robust NDOO request tracking model with improved retry logic, outbound request selection, and legacy data seeding, providing a more auditable and maintainable export pipeline. + +- Add derived cancellation date field to Medication Statement models [PR: #59](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/59) + > ✨ ***Feature:*** Adds a derived cancellation date field to medication statement outputs, improving data completeness for downstream consumers. + >
**This directly addresses the bug raised regarding differences with legacy system derivation of cancellation dates [AB#29871 (SEL) - Cancellation Date differences](https://dev.azure.com/NELAnalytics/LondonDataService/_workitems/edit/29871).** + +### Improvements + +- Add LDS_SOURCE_DATASET to various PID and Pseudo SQL models [PR: #61](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/61) + > 🎯 ***Impact:*** Ensures consistent capture of source dataset information across all PID and Pseudo models, enhancing data traceability and readiness for multi-source processing. + +- Add clustering to Person_PDS_Versioned (on Publisher Code) [PR: #62](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/62) + > 🎯 ***Impact:*** Improves query performance and scalability by clustering the Person_PDS_Versioned table, and aligns warehouse variable usage and masking controls for consistency. + +- Update package revisions to latest releases [PR: #63](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/63) + > 🎯 ***Impact:*** Keeps the project up to date and compatible by upgrading to the latest versions of the EMIS_OLIDS and OLIDS_Share packages. + +### Refactoring + +- Rename Patient_Date_Flags_PID to Patient_Age_Flags_PID [PR: #60](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/60) + > ♻️ ***Refactor:*** Standardizes flag naming and logic in patient models, switching from ambiguous string flags to explicit boolean age flags, improving clarity and maintainability of model outputs and tests. + +### Inputs + +This changelog was automatically produced from: + +- 6 Pull Requests +- 0 Issues +- 0 WorkItems +- 17 Commits + +## Full Details + +Pull Requests included in this release: + +- [PR #58](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/58): Enhance NDOO request handling with new view and refactoring + - No linked issues. +- [PR #59](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/59): Add derived cancellation date field to Medication Statement models + - No linked issues. +- [PR #60](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/60): Rename Patient_Date_Flags_PID to Patient_Age_Flags_PID + - No linked issues. +- [PR #61](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/61): Add LDS_SOURCE_DATASET to various PID and Pseudo SQL models + - No linked issues. +- [PR #62](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/62): Add clustering to Person_PDS_Versioned (on Publisher Code) + - No linked issues. +- [PR #63](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/63): Update package revisions to latest releases + - No linked issues. + +[View the diff between v2.2.1 and v2.3.0](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/compare/releases%2Fv2.2.1...releases%2Fv2.3.0) \ No newline at end of file diff --git a/OLIDS/Release-notes/README.md b/OLIDS/Release-notes/README.md index 6f74a86..8579500 100644 --- a/OLIDS/Release-notes/README.md +++ b/OLIDS/Release-notes/README.md @@ -4,9 +4,9 @@ This location provides a single place to review changes across the OLIDS deliver ## Latest releases -- OLIDS_Share: [v1.3.1](./1_OLIDS_Share/v1.3.1.md) -- EMIS_OLIDS: [v2.2.0](./2_EMIS_OLIDS/v2.2.0.md) -- OLIDS_Enrichment: [v2.2.1](./3_OLIDS_Enrichment/v2.2.1.md) +- OLIDS_Share: [v1.4.0](./1_OLIDS_Share/v1.4.0.md) +- EMIS_OLIDS: [v2.3.0](./2_EMIS_OLIDS/v2.3.0.md) +- OLIDS_Enrichment: [v2.3.0](./3_OLIDS_Enrichment/v2.3.0.md) ## Structure