Raw Data Description

Raw Data Description#

Data Streams from Daiser#

This data will be ingressed via a company Daiser. Daiser are building an app that will collate data from participants’ wearable devices on their smart phones. This data will be provided to Data Connect by ingress to our RONIN Isolate Secure Data Environment (SDE).

The Daiser app will receive data from the following streams:

  • Fitbit API

  • Health app data, e.g. Apple Health, Google Fit

  • GPS data from wearable devices

  • Follow-up health questionnaires

../_images/daiser_streams.png

Daiser data flow#

The inital MVP phase of the pipeline build includes processing Fitbit and GPS data recieved from Daiser.

Incoming Data Structure#

Daily data will be received in a 7z17z compression is a high-efficiency compression method primarily associated with open-source software 7-Zip. archive. Within this archive there will be 8 directories: One for each of the previous 7 days (with the date as the directory name, in the format YYYMMDD), and one for the follow-up questionnaire data.

The internal structure of each of the daily data directories will be identical. Each will contain a “GPS” and “Device” subdirectory, and within those there will be a directory for each individual participant.

Within each participant in the “Device” directory structure, there will always be a Health Application directory (this is the data collected through the Daiser Smart Health mobile app) containing the daily summary data for their wearable devices. If the participant has a Fitbit device and has consented to sharing that data then there will also be a Fitbit directory containing the daily summary data and the intraday data directory from their Fitbit device.

The diagram below shows the internal directory structure for data processed by the pipeline developed within the MVP.

../_images/incoming_data.jpg

Diagram of the directory structure within the compressed daily archive recieved from Daiser. Note that this diagram represents the data processed by the pipeline developed in the MVP.#

Example folder structure#

Below is an example of the file structure and naming within the compressed daily archive.

├── DaiserArchive_20251030​​/
|    ├── 20251023/
|    |   ├── Device/
|    |	   |     ├── UOS001/Fitbit/
|    |	   |     |	├── UOS001_20251023_heart.json
|    |	   |     |	├── UOS001_20251023_fitbit.json
|    |	   |     ├── UOS001/healthapp/
|    |	   |     |	├── UOS001_20251023_healthdata.json
|    |   ├── GPS
|    |	   |     ├── UOS001/UOS001_20241023_gps.json
|    ├── 20251024/
|    |    ├── Device/
|    |	   |     ├── UOS001/
|    |	   |     |	├── UOS001_20251024_heart.json
|    |	   |     |	├── UOS001_20251024_fitbit.json
|    |   ├── GPS
|    |	   |     ├── UOS001/UOS001_20241024_gps.json
|    ├── 20251025/
|    |    ├── Device/
|    |	   |     ├── UOS001/
|    |	   |     |	├── UOS001_20251025_heart.json
|    |	   |     |	├── UOS001_20251025_fitbit.json
|    |   ├── GPS
|    |	   |     ├── UOS001/UOS001_20251025_gps.json
|    ├── 20251026/
|    |    ├── Device/
|    |	   |     ├── UOS001/
|    |	   |     |	├── UOS001_20251026_heart.json
|    |	   |     |	├── UOS001_20251026_fitbit.json
|    |   ├── GPS
|    |	   |     ├── UOS001/UOS001_20251026_gps.json
....same for 20251027,20251028 and 20251029

Example JSON structures#

This section contains more information about the structure of the .json files expected for each data stream.

Each data stream has a number of required data fields. The Fitbit daily summary file may also contain a number of optional fields. More information is provided below.

Where a participants’ file contains data fields that were not expected, a warning is raised, the file is not processed, but the pipeline continues processing other participants’ data.

Fitbit Daily Summary#

The daily summary data file contains a number of daily activity summary metrics. The full list of fields can be found in the data dictionary.

Each summary metric object will contain an array of a single object containing the necessary dateTime and value fields of that particular metric, as in the example below:

{
  "calories": [
    {
      "dateTime": "2018-12-26",
      "value": "11"
    }
  ],
  ...,
}

An example Fitbit Daily Summary .json file can be found on the source repository.

Fitbit Intraday Activities#

Each activity stream, e.g. heart or steps, will be recieved in a separate file.

This file will have a nested structure. An activities-<activity>-intraday object which contains a dataset array of the individual date-value objects. These individual objects must contain a d (datetime) and v (value). See the example below.

{
    "activities-steps-intraday": {
        "dataset": [
            {
                "d": "2025-07-14T00:00:05Z",
                "v": 0
            },
            {
                "d": "2025-07-14T00:01:11Z",
                "v": 0
            },
        ... ]
    }
    ...,
}

An example Fitbit intraday .json file containing steps data can be found on the source repository.

GPS data#

The GPS data file has a nested object structure. A gps-coordinates object contains a dataset array containing the individual value objects.

Each individual value contains the required fields d (datetime), lat (latitude) and long (longitude), as in the example below:

"gps-coordinates": {
        "dataset": [
            {
                "d": "2025-10-09T00:00:00",
                "lat": 53.3806626,
                "long": -1.4702278
            },
]
}

An example GPS data .json file can be found on the source repository.

Health App and Baseline/ Follow-up Questionnaire data#

These datasets are outside of the scope of the initial MVP phase.

On this page