Information for Data Providers

To maximise the value derived from data, Data Linkage Services at the Department of Health is currently collaborating with public sector agencies and other stakeholders to establish, and sustainably maintain, a greater number of enduring infrastructure linkages within and between commonly used health and non-health datasets.

Effective data linkage services involve staff at all levels (governance, ethics, administration and coordination, data linkage, data extraction and systems support) across the entire data life cycle.

The information on this page is designed to support understanding of effective data linkage practices, with the goal of providing more timely access to linked data within Western Australia. 

Data providers seeking guidance to enable the application of high quality and efficient data linkage should refer to our Minimum Requirements for Data Linkage (PDF, 780 KB).

Overview of Infrastructure Linkages

Infrastructure linkages are linkages of datasets that are regularly updated and are generally available for a range of approved projects (as opposed to linkages of datasets for a single approved project only, often referred to as ‘ad-hoc linkages’).

The most accessed infrastructure linkages are the Department of Health’s ‘core’ linkages, such as the Hospital Morbidity Data Collection, Emergency Department Data Collection and Death Registrations, however the Department of Health also maintains a wide range of other non-core infrastructure linkages. For more information on the infrastructure datasets available please see our Available Datasets.

Infrastructure linkages are established according to ten principles:

1. The data provided for linkage meets the minimum data requirements and is technically feasible.
2. Data updates for linkage are provided on a regular schedule (i.e., weekly, monthly, or annually).
3. Data can be accessed by multiple projects and/or groups according to the relevant application and approval process.
4. There is ongoing value for the linkage to be performed as infrastructure work rather than project-specific work.
5. The data flow, privacy considerations and associated data governance arrangements are fully documented in a Data Agreement between the Data Provider and the Department of Health.
6. The new linkage is approved by the Department of Health WA Human Research Ethics Committee.
7. The Data Provider is ‘linkage ready’ insofar as having the staff, skills and resources needed to supply the data for linkage, answer data queries, and fulfil project-specific data extraction requirements once all of the necessary approvals are in place.
8. There is sufficient funding available for initial development and ongoing operational costs in line with the Department of Health’s charging model.
9. The Data Provider and/or a Research Group is responsible for meeting the aforementioned costs.
10. The Department of Health schedules work (including linkages) to optimise the balance between capacity and demand.

All linkage activities involve logistical risks, and this increases with complexity. Generally, infrastructure linkages are associated with the greatest complexity (and risk) due to a variety of inter-related factors, including legal, ethical and legislative considerations, the stakeholders involved, the underlying data quality, and the effort, duration, skills and budget required.

Often, it is difficult to estimate complexity with absolute certainty prior to commencement of an infrastructure linkage. For this reason, it can take the Department of Health several months to link data before it is available for approved requests. Sometimes, this duration can take longer due to unforeseen circumstances (e.g., changes in staff, data collection systems, data set specifications, available resources, etc.).

 
Standard linkage processes
 

Data can be transferred to the Department of Health via secure online file transfer systems such as MyFT. New datasets provided for linkage go through multiple stages before they can be linked, including:

  • liaison with the Data Provider;
  • evaluation;
  • process preparation;
  • importing;
  • cleaning;
  • assignment of linkage specific record IDs; and
  • linkage strategy development.

Theoretically, data updates (‘refreshes’) are more straightforward to complete, because processes and linkage strategies can be reused. Data Providers should be aware that changes to the data, or poor consolidation against what has been received before, can lead to problems (especially impacting time and cost).

General advice for making data as ‘linkage ready’ as possible
Data linkage uses information about ‘who’ a person is more than data about ‘what has happened’ to them.

The most useful fields for linkage are:

  • Given names;
  • Surname;
  • Date of birth;
  • Street address;
  • Suburb; and
  • Postcode.

Information that ‘deterministically connects’ (matches exactly) the new data to another dataset is very valuable. Examples include Unit Medical Record Number (UMRN) and Elector Number.

The data formats that are accepted are character-delimited (e.g., by commas or tabs), fixed width text files and Excel spreadsheets. If a Data Provider cannot provide data in this format, then they should contact ISPD Client Services to discuss.

Every record must have a unique ID that maps back to the original system/data collection. If the data does not have a unique record ID, Data Providers should consider creating one, or providing the Linkage Team with a list of fields which, when taken together, are unique (e.g., person ID and event date).

If the new data is ‘person based’ rather than ‘event based’ (i.e., everyone has one record with all of their information contained within), this could be problematic, particularly if the Data Provider overwrites old values with new ones (e.g., updating addresses). This should be discussed with ISPD Client Services.

Non-human records need to be removed before linkage, noting some datasets may include, e.g., animals or vehicles. ISPD Client Services request that Data Providers endeavour to remove these records before they are supplied.

Text equivalents of NULL, such as ‘no fixed permanent address’ and ‘N/A’ can be problematic, as the Linkage Team wants to avoid mistaking them for a match to one another. Data will be easier to link if these values are left empty instead.

Metadata is very useful when determining how to use fields for linkage, particularly when they are understood by only a select group of people with specialised knowledge. Data Providers must ensure they provide ISPD Client Services with data dictionaries and code lists, if they exist.

Data should be checked closely for errors after exporting it from the original system.

Some of the most common errors that are encountered:

  • single records wrapping onto two lines;
  • inconsistent number of delimiters or fields;
  • values being stored in the wrong field (e.g., dates of birth in the given name field);
  • inclusion of the separating character in the field value (e.g., addresses with commas in them, within a comma-delimited file);
  • the same record ID being assigned to more than one record; and
  • inadvertently reformatting fields (e.g., expressing a number in truncated scientific notation in Excel).

A Data Provider should cross-check the field list in the relevant Data Agreement to ensure they have supplied all approved fields.

If ongoing data updates will be provided to the Department of Health, these will be easiest to process if to the Data Provider can provide the Department of Health with records that are new or have been modified since their previous provision. If this is impossible, they can theoretically provide the Department of Health with a full refresh of all records, however, this could be problematic if the data has undergone any changes since the last update.

Data changes such as the addition of new fields, moving the data to a new storage system, or reallocation of record IDs, will impact processing requirements for updates. It is preferable that the Department of Health is provided with as much information as possible to bridge the gap between the old version of the data and the new one (e.g., new-to-old ID mapping files, change of field name information, etc.). It is helpful if a data provider can format the data to look as similar as possible to the previous version (e.g., same format, same field order, same naming conventions, with newly added fields appended to the end of each record).

Data Providers should ensure old records do not disappear from their system. It is also vital that Data Providers thoroughly document any related data processes to prevent knowledge loss at the source. This is especially important during periods of staff changeover or system redevelopment/replacement.