Apple Health Data : How to extract and analyse sleep data in Python (Part 1)

Over the past decade, since Apple launched their Apple Health platform at WWDC in 2016, I’ve collected huge amounts of data on my body. Specifically – with a focus on getting up earlier and earlier again, I thought I would start by extracting the data I do have, and exploring what secrets lie within.

In this post – i’m going to walk you through how to extract your data from Apple Health, parse it into a format that can be read – then analyse this data to find any trends or correlations. This will just be an introduction to

Step 1 : Extracting Apple Health Sleep Data

Although there is no automated method to get your Apple Health data off your iPhone, its not difficult either, and Apple have made it exceptionally easy to package and extract all your data in one go.

You just need to open the “Health” app on your phone, tap the profile icon in the top right hand corner,

And select “Export All Health Data”

Your phone will then package all your health data as an zipped XML file and allow you to drop this into another app. In my case my file was over 1.6GB, and it was easier to AirDrop this to my Mac to analyse than via email or another method.

Once you have dropped this across, extract the files. You may find you have a bunch of other data (for instance – I had workout routes as well). The main file we’re looking to analyse for today is called “export.xml”.

Step 2 : Converting Apple Health XML for use with Pandas

As mentioned above, my file was quite large. I could – if I had the time, import this file every time I needed to analyse it, however due to the size it takes around 10-15 minutes to process and load the dataframe in memory. Instead, we’re going to create a short ETL process (that stands for “Extract – Transform – Load”) to extract the information we need out of the XML file, change the structure to match our requirements, then load this into a more efficient binary file format for processing.

To do this you’re going to need the pandas and the pyarrow packages installed. Credit goes to Alejandro Rodríguez over at towardsdatascience.com for providing most of these steps (you can read their more in-depth overview of Apple Health data analysis here). If you want more detail on the below I suggest you head over to their post, but essentially this extracts the data, normalises the dates, and removes some of the “Apple-y” string naming.

import xml.etree.ElementTree as ET
import pandas as pd

# create element tree object 
tree = ET.parse('apple_health_export/export.xml') 

# for every health record, extract the attributes into a dictionary (columns). Then create a list (rows).
root = tree.getroot()
record_list = [x.attrib for x in root.iter('Record')]

# create DataFrame from a list (rows) of dictionaries (columns)
data = pd.DataFrame(record_list)

# proper type to dates
for col in ['creationDate', 'startDate', 'endDate']:
    data[col] = pd.to_datetime(data[col])

# value is numeric, NaN if fails
data['value'] = pd.to_numeric(data['value'], errors='coerce')

# some records do not measure anything, just count occurences
# filling with 1.0 (= one time) makes it easier to aggregate
data['value'] = data['value'].fillna(1.0)

# shorter observation names: use vectorized replace function
data['type'] = data['type'].str.replace('HKQuantityTypeIdentifier', '')
data['type'] = data['type'].str.replace('HKCategoryTypeIdentifier', '')

# save into feather as this is a more efficient data format
data.to_feather("converted_data/apple_export.ftr")

Step 3 : Group and Aggregate Apple Health Sleep data

For this tutorial – and for my own goals – I’m focusing currently on my sleep data, so in the next few steps we’re going to extract the Apple sleep data, specifically the Apple Watch focused sleep data (as this contains information on how many times I’ve woken up in the night), and produce some derivative variables to analyse.

The first step is to re-load the data from the last script into a new dataframe

import pandas as pd
import datetime

data = pd.read_feather('converted_data/apple_export.ftr')

From here we want to just filter on the sleep data and specifically the Apple Watch sleep data. Thankfully, Apple provides two columns we can filter on here “type” – which we want to set to “SleepAnalysis” and “sourceName” which we want to set to “Jon’s Apple Watch”.

# filter on sleep data and apple watch info ONLY
sleep_data = data[data['type'] == "SleepAnalysis"]
sleep_data = sleep_data[sleep_data['sourceName'] == "Jon’s Apple Watch"]

This gives us a dataframe that looks something like the following;

	type	sourceName	sourceVersion	unit	creationDate	startDate	endDate	value	device
4014637	SleepAnalysis	Jon’s Apple Watch	7		2020-09-18 05:48:57+01:00	2020-09-17 22:33:31+01:00	2020-09-17 23:57:01+01:00	1
4014638	SleepAnalysis	Jon’s Apple Watch	7		2020-09-18 05:48:57+01:00	2020-09-18 00:00:31+01:00	2020-09-18 00:45:01+01:00	1
4014639	SleepAnalysis	Jon’s Apple Watch	7		2020-09-18 05:48:57+01:00	2020-09-18 00:47:01+01:00	2020-09-18 01:59:31+01:00	1
4014640	SleepAnalysis	Jon’s Apple Watch	7		2020-09-18 05:48:57+01:00	2020-09-18 02:01:31+01:00	2020-09-18 02:29:01+01:00	1
4014641	SleepAnalysis	Jon’s Apple Watch	7		2020-09-18 05:48:57+01:00	2020-09-18 02:32:01+01:00	2020-09-18 05:47:31+01:00	1

Nothing too useful yet, but a few items to note;

• startDate – provides the start of a sleep cycle
• endData – provides, you’ve guessed it, the end of a sleep cycle
• creationDate – provides when this data was pushed from the Apple Watch to the Apple Health app. Usefully – this is provided in a batch in the morning – making this a great field to group data by.

Although this may only be 2 fields, we can still derive a lot of data from this. I looking to try and analyse any sleep patters such as average wake time, restlessness and so on. To do this, we’re going to create the following fields;

• Time Asleep – How long am I asleep for in each record (endDate minus startDate)
• Total Time Asleep – How long am I asleep for in total each night (sum of time asleep for the night)
• Bed Time – When do I actually hit the hay and get to bed? (minimum startDate for the night)
• Awake Time – When does my watch detect when I am awake (maximum endDate for the night)
• Sleep Counts – How many times am I restless during the night (count of records for the night)
• REM cycles – How many REM (deep sleep) cycles have I got (number of records over 90 minutes, divided by 90 minutes – this is the most complex of the aggregations)
• Total Time in Bed – How long did I spend in bed (awake time minus bed time)
• Restless time – How restless was I during the night (total time in bed minus total time asleep )

8 fields from 2 data points – thats pretty cool! Lets get into the code and how I’ve calculated each one.

# calulate time between date(s)
sleep_data['time_asleep'] = sleep_data['endDate'] - sleep_data['startDate']

# records are grouped by creation date, so lets used that to sum up the values we need here
# total time asleep as a sum of the asleep time
# awake and bed times are max's and min's
# sleep count is the number of times the Apple Watch detected movement
# rem is the number of sleep cycles over 90 minutes (divded by 90 if they were longer than 1 cycle)
sleep_data = sleep_data.groupby('creationDate').agg(total_time_asleep=('time_asleep', 'sum'),
    bed_time=('startDate', 'min'), 
    awake_time=('endDate', 'max'), 
    sleep_counts=('creationDate','count'), 
    rem_cycles=pd.NamedAgg(column='time_asleep', aggfunc=lambda x: (x // datetime.timedelta(minutes=90)).sum()))

# Time in Bed will be different to Apple's reported figure - 
# as Apple uses the time you place your iPhone down as an additional 
# datapoint, which of course, is incorrect if you try to maintain 
# some device separation in the evenings.
# For now - we will just use Apple Watch data here
sleep_data['time_in_bed'] = sleep_data['awake_time'] - sleep_data['bed_time']
sleep_data['restless_time'] = sleep_data['time_in_bed'] - sleep_data['total_time_asleep']

Step 4 : Analyse the data

Now we’ve pivoted and aggregated our data, the next step is to identify any trends from our data. I’m going to be doing some very basic initial analysis here – and I’ll leave the deeper dive to a later post, but for now, lets look for any overall trends in the data using Matplotlib.

# convert time duration to minutes for easier plotting and comparison
sleep_data['time_in_bed'] = (sleep_data['time_in_bed'].dt.total_seconds()/60)
sleep_data['total_time_asleep'] = (sleep_data['total_time_asleep'].dt.total_seconds()/60)

import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt

from matplotlib.dates import DateFormatter, MonthLocator

chart1 = sleep_data[['time_in_bed','total_time_asleep']].plot(use_index=True)
plt.show()

As we can see from this initial chart, my sleep time is very cyclical based on the week, and I had a major crash around last September (this was when I had some big issues with my teeth – requiring a root canal). In later posts I’ll be looking to diving into these trends a little deeper – using AI to identify any trends throughout the months, weeks and years.

If you’ve founds todays analysis useful, feel free to drop me a comment on twitter @busbyjon to follow future posts on this subject and others in this series.