Switching between space and time: Spatio-temporal analysis with
cubble

H. Sherry Zhang

The Australian spatial econometrics and statistics workshop
Monash University, Australia
2023 Feb 17

Hi!

• A final year PhD student in the Department of Econometrics and Business Statistics

• My research centers on exploring multivariate spatio-temporal data with data wrangling and visualisation tool.

• Find me on

  • Twitter: huizezhangsh,
  • GitHub: huizezhang-sherry, and
  • https://huizezhangsh.netlify.app/

• Thanks for the invitation to speak

• Today I will be talking about …

• First, a little bit about myself

• I’m Sherry Zhang, …

• …

• Here are the details to find me on Twitter, GitHub, and my website

Spatio-temporal data

People can talk about a whole range of different things when they refer to their data as spatio-temporal!

The focus of today will be on vector data.

Today, we have listened to many talks on spatio-temporal data and you may have noticed that […]

Here we have three different types: vector, raster, trajectory data (explain each)

vector data have time series measured at a collection of locations

raster data use gridded cells to represent a continuous space and each cell, or pixel, has variables measured at different time points. An example of raster data could be satellite images.

There is also trajectory data where points are moving in the space and time in the same time. This type of data can be found in ecology to track animal movements.

In my talk today, we will focus on vector data. But I want to start by giving you the big picture.

Example of vector data

Physical sensors that measure the temperature, rainfall, and wind speed & direction

Physical sensors are a common source of vector spatio-temporal data. Here is the picture of a fancy weather sensor.

On the right, the wind speed cup and the wind vane measure the wind speed and direction.

The rain gauge on the left has a standard diameter, based on which rainfall is measured in mm and the thermo sensor below it measures the temperature.

Instruments like this are placed across Australia to record climate data on a daily or sometimes half hourly basis.

But vector data are more than just points

This simplified Victoria polygon contains 360 points!

# A tibble: 1 × 2
  NAME                                                             geometry
  <chr>                                                       <POLYGON [°]>
1 Victoria ((140.9657 -38.05599, 140.9711 -37.79145, 140.9739 -37.46209, 1…

While stations data may be the most common type of vector data, vector data also includes linestrings, polygons, and others, which is defined by a standard called simple features.

In R, simple feature is implemented by the package sf. The reason we need these complex features is because it allows us to do operations based on shapes. For example, we may have weather stations with coordinates and want to find out those located in Victoria. This is to filter points that are within a polygon. The sf package allows us to do it through interfacing with external geo software.

Here on the left is a polygon of Victoria and when printed in R, you may have noticed that it somehow wraps up points into a polygon class, as you can read from the print.

Actually, this is already a simplified polygon and it contains 360 points on the border. So sometimes, these geometrical objects can be quite big.

Temporal data

A wide table 😢

year-by-month table

Year	Jan	Feb	Mar	…
1946	26.663	23.598	26.931	…
1947	21.439	21.089	23.709	…
1948	21.937	20.035	23.590	…

A long table 😄

time stamp forms rows, variable forms columns.

Year	month	value
1946	Jan	26.663
1946	Feb	23.598
1946	Mar	26.931
…	…	…
1947	Jan	21.439
1947	Feb	21.089
1947	Mar	23.709
…	…	…

On the other hand, these weather stations collect time series data.

Here I show you two different ways to arrange your data.

Sometimes, you may receive your time series in a wide table. One year per row and each column represents a month.

A long table is your preferred structure if you do some panel data modelling, say with the package plm, or you would like to include month as a regressor in your model.

The package tsibble supports the long table approach. It and its associated packages in tidyverts, not tidyverse, provide functionalities to work with time series models.

When the data has both spatial and temporal dimensions

• In a long table with duplicated spatial variables? That would give a lot of duplication if daily data & large spatial objects.

• Sometimes, we would like to make per station summary, ideally, each station forms a row

• Other time, we would like to work on temporal variables in the long form.

• A lot of padding work to arrange the spatio-temporal data in the format convenient for spatial & temporal operations!

What raises an issue here is how you would organise your data if they have both spatial and temporal dimension.

This creates

Cubble: a spatio-temporal vector data structure

• Today I will introduce a new data structure, called cubble, for vector spatio-temporal data

• in short, cubble is a nested object built on tibble that allow easy pivoting between spatial and temporal form.

• we will first talk about how the two forms look like and then how you cna pivot beween them for different tasks.

• In the nested form, spatial variables are in columns out and temporal variables are nested into a list column called ts

• In the long form, the time series data are shown in columns and each row is cross identified by the site and date in a long table

Cubble: a spatio-temporal vector data structure

Cubble is a nested object built on tibble that allow easy pivoting between spatial and temporal form.

• The pair face_temporal() and face_spatial() to switch the cubble between the two forms.

• With face_temporal(), the focus of the data is now on the temporal face of the spatio-temporal cube and this corresponds to switch the data to the long form.

• With face_spatial(), the long cubble is switched back to the nested form, the spatial face of the datacube.

Australian weather station data:

stations

# A tibble: 30 × 6
  id            lat  long  elev name                       wmo_id
  <chr>       <dbl> <dbl> <dbl> <chr>                       <dbl>
1 ASN00060139 -31.4  153.   4.2 port macquarie airport aws  94786
2 ASN00068228 -34.4  151.  10   bellambi aws                94749
3 ASN00017123 -28.1  140.  37.8 moomba airport              95481
4 ASN00081049 -36.4  145. 114   tatura inst sustainable ag  95836
5 ASN00018201 -32.5  138.  14   port augusta aero           95666
# … with 25 more rows

ts

# A tibble: 10,632 × 5
  id          date        prcp  tmax  tmin
  <chr>       <date>     <dbl> <dbl> <dbl>
1 ASN00003057 2020-01-01     0  36.7  26.9
2 ASN00003057 2020-01-02    41  34.2  24  
3 ASN00003057 2020-01-03     0  35    25.4
4 ASN00003057 2020-01-04    40  29.1  25.4
5 ASN00003057 2020-01-05  1640  27.3  24.3
# … with 10,627 more rows

• Let’s put these in a data context.

• The stations data records 30 Australia weather stations, along with their longitude, latitude, elevation, and name

[breath]

• On the temporal side, we have precipitation, maximum and minimum temperature collected daily for each station in 2020.

Cast your data into a cubble

(weather <- as_cubble(
  list(spatial = stations, temporal = ts),
  key = id, index = date, coords = c(long, lat)
))

# cubble:   id [30]: nested form
# bbox:     [114.09, -41.88, 152.87, -11.65]
# temporal: date [date], prcp [dbl], tmax [dbl], tmin [dbl]
  id            lat  long  elev name              wmo_id ts                
  <chr>       <dbl> <dbl> <dbl> <chr>              <dbl> <list>            
1 ASN00003057 -16.5  123.     7 cygnet bay         94201 <tibble [316 × 4]>
2 ASN00005007 -22.2  114.     5 learmonth airport  94302 <tibble [363 × 4]>
3 ASN00005084 -21.5  115.     5 thevenard island   94303 <tibble [366 × 4]>
4 ASN00010515 -32.1  117.   199 beverley           95615 <tibble [354 × 4]>
5 ASN00012314 -27.8  121.   497 leinster aero      95448 <tibble [366 × 4]>
# … with 25 more rows

• the spatial data (stations) can be an sf object and temporal data (ts) can be a tsibble object.

• To cast the two separate tables into a cubble, you can supply them in a named list.

• You also need to tell cubble some identifiers it looks for

• The key argument is the spatial identifier that connects the two tables.

• The index argument is the temporal identifier that prescribes the timestamp.

• The coords argument is to used to specify the coordinate

[breath]

• From the cubble header, you can read that the key is id, there are 30 stations and it is in the nested form.

• The third line here shows you the available temporal variables and their types.

• Also, if the spatial and temporal data is an sf or tsibble object, they will be indicated in the header as well.

Switch between the two forms

long form

(weather_long <- weather %>% 
  face_temporal())

# cubble:  date, id [30]: long form
# bbox:    [114.09, -41.88, 152.87, -11.65]
# spatial: lat [dbl], long [dbl], elev [dbl],
#   name [chr], wmo_id [dbl]
  id          date        prcp  tmax  tmin
  <chr>       <date>     <dbl> <dbl> <dbl>
1 ASN00003057 2020-01-01     0  36.7  26.9
2 ASN00003057 2020-01-02    41  34.2  24  
3 ASN00003057 2020-01-03     0  35    25.4
4 ASN00003057 2020-01-04    40  29.1  25.4
5 ASN00003057 2020-01-05  1640  27.3  24.3
# … with 10,627 more rows

back to the nested form:

(weather_back <- weather_long %>% 
   face_spatial())

# cubble:   id [30]: nested form
# bbox:     [114.09, -41.88, 152.87, -11.65]
# temporal: date [date], prcp [dbl], tmax [dbl],
#   tmin [dbl]
  id         lat  long  elev name  wmo_id ts      
  <chr>    <dbl> <dbl> <dbl> <chr>  <dbl> <list>  
1 ASN0000… -16.5  123.     7 cygn…  94201 <tibble>
2 ASN0000… -22.2  114.     5 lear…  94302 <tibble>
3 ASN0000… -21.5  115.     5 thev…  94303 <tibble>
4 ASN0001… -32.1  117.   199 beve…  95615 <tibble>
5 ASN0001… -27.8  121.   497 lein…  95448 <tibble>
# … with 25 more rows

identical(weather_back, weather)

[1] TRUE

• Here is what a cubble look like when being switched between the long and the nested form.

  • With the weather object we just created, we turn it into the long form with the function face_temporal()

• Notice that the third line in the header now changes to see the available spatial variables

[breath]

• On the right, weather_long is switched back the nested form with the function face_spatial()

• As you can see from the last line of code, face_temporal() and face_spatial() are the exact inverse.

• Hence weather_back and weather are identical

Access variables in the other form

Reference temporal variables with $

weather %>% 
  mutate(avg_tmax = mean(ts$tmax, na.rm = TRUE))

# cubble:   id [30]: nested form
# bbox:     [114.09, -41.88, 152.87, -11.65]
# temporal: date [date], prcp [dbl], tmax [dbl], tmin [dbl]
  id            lat  long  elev name              wmo_id ts                 avg_tmax
  <chr>       <dbl> <dbl> <dbl> <chr>              <dbl> <list>                <dbl>
1 ASN00003057 -16.5  123.     7 cygnet bay         94201 <tibble [316 × 4]>     32.4
2 ASN00005007 -22.2  114.     5 learmonth airport  94302 <tibble [363 × 4]>     33.2
3 ASN00005084 -21.5  115.     5 thevenard island   94303 <tibble [366 × 4]>     30.7
4 ASN00010515 -32.1  117.   199 beverley           95615 <tibble [354 × 4]>     26.4
5 ASN00012314 -27.8  121.   497 leinster aero      95448 <tibble [366 × 4]>     29.6
# … with 25 more rows

Move spatial variables into the long form

weather_long %>% unfold(long, lat)

# cubble:  date, id [30]: long form
# bbox:    [114.09, -41.88, 152.87, -11.65]
# spatial: lat [dbl], long [dbl], elev [dbl], name [chr], wmo_id [dbl]
  id          date        prcp  tmax  tmin  long   lat
  <chr>       <date>     <dbl> <dbl> <dbl> <dbl> <dbl>
1 ASN00003057 2020-01-01     0  36.7  26.9  123. -16.5
2 ASN00003057 2020-01-02    41  34.2  24    123. -16.5
3 ASN00003057 2020-01-03     0  35    25.4  123. -16.5
4 ASN00003057 2020-01-04    40  29.1  25.4  123. -16.5
5 ASN00003057 2020-01-05  1640  27.3  24.3  123. -16.5
# … with 10,627 more rows

• Sometimes, you may need to access variables from the other form for your analysis.

• For example, we may want to calculate some per station summary of the time series data.

• We can refer to the temporal variables from the nested form with the $ sign.

• Here I’m calculating the average maximum temperature across the whole year for each station and I need to get access to tmax from the list-column ts.

• In the long form, you need the cubble verb unfold() to move the spatial variables into the long form.

• Here I move the two coordinate columns into the long form and later we will see how it can help us to create a glyph map.

Explore temporal pattern across space

Sometimes, although we technically have spatio-temporal data, we may choose to fix a few stations to explore their temporal patterns, or select a few timestamps to explore their spatial distributions.

A more holistic approach is to explore the space and time simultaneously and now we will see an example of how to use cubble to explore temporal pattern across space in a glyph map.

Why do you need a glyph map?

Here is a typical plot you may have seen when someone tries to visualise their spatio-temporal data. The x and y axes are the coordinates, here I simplify it with only two points, but in reality you may see a collection of points in space or a raster image. Each facet here shows the space in different timestamp and the values are mapped into color.

The problem of this type of visualisation is that it becomes difficult to comprehend the temporal structure of the data since you have to compare points at the same location across panels to digest the pattern.

Why do you need a glyph map?

Instead the temporal pattern is much easier to observe if shown in a time series plot.

What a glyph map do is to put the time series glyph in the place of the location, so you can see the temporal trend in the space.

Glyph map transformation

DATA %>%
  ggplot() +
  geom_glyph(
    aes(x_major = X_MAJOR, x_minor = X_MINOR,
        y_major = Y_MAJOR, y_minor = Y_MINOR)) +
  ...

• I have a short illustration to show you how the transformation works

• Here (1) shows a single station on the map with its long and lat coordinate and (2) is its associated time series.

• Here you know the range of your x and y axis and you can use linear algebra to transform them into a different scale.

• In step (3), the time series in still the same but its scale has been transformed to a width of 1 and heights of 0.3 and the center in this scale is where the original point lays.

• Once we have the time series in the transformed axes, they can be placed onto the map as in (4)

• To make a glyph map, you can use the geom_glyph function from the cubble package.

• It requires a pair of major and a pair of minor variable as required aesthetics

• The major variable are the spatial coordinates, long and lat here and the minor variable are the temporal coordinates, date and tmax here.

Avg. max. temperature on the map

cb <- as_cubble(
  list(spatial = stations, temporal = ts),
  key = id, index = date, coords = c(long, lat)
)

set.seed(0927)
cb_glyph <- cb %>%
  slice_sample(n = 20) %>%
  face_temporal() %>%
  mutate(month = lubridate::month(date)) %>%
  group_by(month) %>% 
  summarise(tmax = mean(tmax, na.rm = TRUE)) %>%
  unfold(long, lat)

ggplot() +
  geom_sf(data = oz_simp, 
          fill = "grey95", 
          color = "white") +
  geom_glyph(
    data = cb_glyph,
    aes(x_major = long, x_minor = month,
        y_major = lat, y_minor = tmax),
    width = 2, height = 0.7) + 
  ggthemes::theme_map()

• This is a full example of using glyph map to explore Australian weather pattern.

• The code has three blocks.

• In the first block we first create a cubble object form the stations and ts data.

• The second block involves wrangling the data using the nested and long form.

• Sampling 20 stations is a spatial operations, so it is performed in the nested form.

• Then we need to do a summary of average maximum temperature by month. It is a temporal operation, so the cubble is then switched to the long form with face_temporal().

• The glyph map requires both the spatial and temporal axes variables as aesthetics, so we move the column long and lat with unfold() into the long form.

• The last chunk shows the ggplot2 code to make the glyph map with geom_glyph()

[breath]

• On the map, you can see that the temperature curve in the north and south (the Tasmania Island) are relative constant throughout the year.

• Those inland stations, for example in the eastern Australia, have a much visible variation in the year, as compared to the coastline ones.

• And remember Australia is in the southern hemisphere, so winter is in the June, July, and August and the temperature is in the U-shape.

Remark

• Nowadays, data collection can take many forms and the research process begins long before a cleaned dataset is available for modeling.

• I hope you view data wrangling as an equally important part as your model.

• With research on creating data tools, you can more easily reproduce results with more recent data in the future, without having to hire a new RA to redo the data preparation work your previous RA has already done (if you ever hire one).

Data preparation, including arranging data into different forms, may not sound like the most exciting part of research for some people.

There should be tools available out there to make it simple to do these tasks, same as someone who has already coded up an estimation methods or model that you have searched online for a while.

Research starts before data served to according to the recipe….

just as

Acknowledgements

• The slides are made with Quarto, available at

sherryzhang-monashspatial2023.netlify.app

• All the materials used to prepare the slides are available at

https://github.com/huizezhang-sherry/MONASHspatial2023

Reference

• cubble: https://huizezhang-sherry.github.io/cubble/

• Wickham, H., Hofmann, H., Wickham, C., & Cook, D. (2012). Glyph‐maps for visually exploring temporal patterns in climate data and models. Environmetrics, 23(5), 382-393: https://vita.had.co.nz/papers/glyph-maps.pdf

• This wraps up my talk today.

• Cubble has already made its way to CRAN

• There will be a version update on cubble in the next two weeks, so stay tuned!

• Thanks for listening