Statistics students (and instructors) need experience wrestling with large, messy, complex, challenging data sets, for which there is no obvious goal or specially-curated statistical method. In this example, we consider a case study from a subset of the 180 million record Airline Delays dataset (see http://stat-computing.org/dataexpo/2009) that includes n=336,776 domestic commercial flights originating in New York City area airports (Newark, JFK, and LaGuardia) in 2013. These data are made available as a series of comma separated variable (CSV) files or through Hadley Wickham’s nycflights13 package on CRAN and allow students to explore a variety of statistical questions related to flights from NYC airports.

These five separate datasets can easily be merged (see the appendix for a list of the first few observations in each of these tables.) More details and extended examples can be found at http://www.amherst.edu/~nhorton/precursors.

# derive variables of interest...
len <- nchar(flights$dep_time)
hour <- as.numeric(substring(flights$dep_time, 1, len-2))
min <- as.numeric(substring(flights$dep_time, len-1, len))
flights <- mutate(flights, deptime = hour+min/60)
flights <- mutate(flights, realdelay = ifelse(is.na(arr_delay), 240, arr_delay))

Students can use this dataset to address questions that they find real and relevant. (It is not hard to find motivation for investigating patterns of flight delays. Ask students: have you ever been stuck in an airport because your flight was delayed or cancelled and wondered if you could have predicted the delay if you’d had more data?) This dataset is attractive because it is more similar to what analysts actually see in the wild than what is typically taught in the introductory statistics classroom.

Flights to San Francisco Bay

We start with an analysis focused on three airports in the San Francisco Bay area (OAK, SFO, and SJC) for flights that depart from New York City airports.

How many flights are there to each airport in January, 2013?

Almost all are to San Francisco International (SFO). Let’s take a closer look at what carriers service this route.

United is the largest carrier (it accounts for more than half of the flights).

Are there different delays by carrier?

Each of the carriers has at least a thousand flights, so it’s likely that estimates of arrival delays may be reasonable to estimate. Let’s calculate summary statistics of the arrival delay for the flights to SFO by carrier.

The “average” results (as provided by the median) is that flights arrive a few minutes early for each of these carriers. And even the 3rd quartile or the mean are relatively modest delays (all less than 20 minutes after the scheduled arrival time). But the maximum delays can be large (e.g., more than 10 hours for Virgin America and American Airlines).

We also observe that a number of flights are missing their arrival delay. Those missing values are likely cancelled flights. We might be interested in which month they occurred?

Cancelled flights seem to be most common in July, February, and December.

How should the cancelled flights be handled? (Note that there were excluded from the calculation of the summary statistics displayed earlier.)

One option might be to recode these as 4 hour (240 minute) delays, since it’s likely that if a flight is cancelled the expected delay might be of that duration on average. (This is an arbitrary choice: students might be asked what other options are reasonable. More sophisticated approaches could implement a “hurdle” method with a model for the probability of not being cancelled along with a model for the “average” delay for those flights that were not cancelled.)

Let’s revisit the distribution of real delays (accounting for cancellations) by carrier.

A parallel graphical description of the flights delays to San Francisco airport can be used to judge the airlines.

Note that the distributions have been rescaled so that only those flights between 2 hours early and 5 hours late are displayed (this excludes some of the extreme outliers).

The distributions appear to be somewhat symmetrically distributed around zero delays but with extremely long right tails. Different information is conveyed in the two representations: the overlapping density plots provide a clear sense of the shape of the distributions but are somewhat crowded. The boxplots make it easy to compare airline reliability, and to see the quantiles.

Is there seasonality to the number of flights?

We can consider whether the number of flights changes month by month.

We observe that there are some interesting patterns over the course of the year for SFO: the number of flights in January, February, and March is considerably less than the other nine months.

Predictors of delays

How is month of flight associated with delays?

We see that the largest average delays occur in the summer (with both June and July having an average above 20 minutes).

Is there an association between the number of flights in a month and the average delay?

There is not much of a pattern, but the delays seem to be more variable on months with more flights.

Another question that travelers might consider is whether the departure time matter as a predictor of flight delays?

A number of observations can be made from this graphical display. Very few flights depart between midnight and 5:30am. Most flights are on time, but there does appear to be a pattern that more delays occur for flights that are scheduled to depart later in the day.

We can improve the display by zooming in and adding a scatterplot smooth.

While there is some indication that delays tend to be more common (and slightly longer) as the day proceeds, the effect is modest for flights to San Francisco.

Weather

Other factors affect airline delays. This might include the weather.
The nycflights13 package in R includes other data scraped from the Internet (in this case detailed weather information). We can display the temperature (in degrees Fahrenheit) on New Year’s Day, 2013.

Let’s take a look at daily averages for delays as well as total precipation and maximum wind speed. First we undertake the merge and display a set of values.

A dramatic outlier is immediately spotted: windspeeds of 1000 mph are not common! This must be an error.

Let’s remove this outlier and consider the association between any precipiation and average delays.

Precipitation seems to be associated with delays:

Max windspeed also seems to be associated with delays.

After stratifying by precipitation status, we see that windspeed does not appear to be a major determinant of delays. Precipitation seems to be the issue.

Closing thoughts and further resources

The dataset (as a series of comma separated variable files), copies of the R Markdown and formatted files for these analyses (to allow replication of the analyses) along with further background on the Airline Delays dataset can be found at http://www.amherst.edu/~nhorton/precursors.

Horton, N.J., Baumer, B.S., and Wichham H. (2015) Setting the stage for data science: integration of data management skills in introductory and second courses in statistics“, Nicholas J. Horton, Benjamin S. Baumer, and Hadley Wickham, CHANCE, 28(2):40-50, http://chance.amstat.org/2015/04/setting-the-stage.

Wickham, H. (2011). ASA 2009 Data Expo, Journal of Computational and Graphical Statistics, 20(2):281-283.

Appendix

In this appendix, the first few rows of each of the datasets is displayed.

airlines
## Source: local data frame [16 x 2]
## 
##    carrier                        name
## 1       9E           Endeavor Air Inc.
## 2       AA      American Airlines Inc.
## 3       AS        Alaska Airlines Inc.
## 4       B6             JetBlue Airways
## 5       DL        Delta Air Lines Inc.
## 6       EV    ExpressJet Airlines Inc.
## 7       F9      Frontier Airlines Inc.
## 8       FL AirTran Airways Corporation
## 9       HA      Hawaiian Airlines Inc.
## 10      MQ                   Envoy Air
## 11      OO       SkyWest Airlines Inc.
## 12      UA       United Air Lines Inc.
## 13      US             US Airways Inc.
## 14      VX              Virgin America
## 15      WN      Southwest Airlines Co.
## 16      YV          Mesa Airlines Inc.
airports
## Source: local data frame [1,397 x 7]
## 
##    faa                           name  lat    lon  alt tz dst
## 1  04G              Lansdowne Airport 41.1  -80.6 1044 -5   A
## 2  06A  Moton Field Municipal Airport 32.5  -85.7  264 -5   A
## 3  06C            Schaumburg Regional 42.0  -88.1  801 -6   A
## 4  06N                Randall Airport 41.4  -74.4  523 -5   A
## 5  09J          Jekyll Island Airport 31.1  -81.4   11 -4   A
## 6  0A9 Elizabethton Municipal Airport 36.4  -82.2 1593 -4   A
## 7  0G6        Williams County Airport 41.5  -84.5  730 -5   A
## 8  0G7  Finger Lakes Regional Airport 42.9  -76.8  492 -5   A
## 9  0P2   Shoestring Aviation Airfield 39.8  -76.6 1000 -5   U
## 10 0S9          Jefferson County Intl 48.1 -122.8  108 -8   A
## .. ...                            ...  ...    ...  ... .. ...
planes
## Source: local data frame [3,322 x 9]
## 
##    tailnum year                    type     manufacturer     model engines
## 1   N10156 2004 Fixed wing multi engine          EMBRAER EMB-145XR       2
## 2   N102UW 1998 Fixed wing multi engine AIRBUS INDUSTRIE  A320-214       2
## 3   N103US 1999 Fixed wing multi engine AIRBUS INDUSTRIE  A320-214       2
## 4   N104UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE  A320-214       2
## 5   N10575 2002 Fixed wing multi engine          EMBRAER EMB-145LR       2
## 6   N105UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE  A320-214       2
## 7   N107US 1999 Fixed wing multi engine AIRBUS INDUSTRIE  A320-214       2
## 8   N108UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE  A320-214       2
## 9   N109UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE  A320-214       2
## 10  N110UW 1999 Fixed wing multi engine AIRBUS INDUSTRIE  A320-214       2
## ..     ...  ...                     ...              ...       ...     ...
## Variables not shown: seats (int), speed (int), engine (chr)
flights
## Source: local data frame [336,776 x 18]
## 
##    year month day dep_time dep_delay arr_time arr_delay carrier tailnum
## 1  2013     1   1      517         2      830        11      UA  N14228
## 2  2013     1   1      533         4      850        20      UA  N24211
## 3  2013     1   1      542         2      923        33      AA  N619AA
## 4  2013     1   1      544        -1     1004       -18      B6  N804JB
## 5  2013     1   1      554        -6      812       -25      DL  N668DN
## 6  2013     1   1      554        -4      740        12      UA  N39463
## 7  2013     1   1      555        -5      913        19      B6  N516JB
## 8  2013     1   1      557        -3      709       -14      EV  N829AS
## 9  2013     1   1      557        -3      838        -8      B6  N593JB
## 10 2013     1   1      558        -2      753         8      AA  N3ALAA
## ..  ...   ... ...      ...       ...      ...       ...     ...     ...
## Variables not shown: flight (int), origin (chr), dest (chr), air_time
##   (dbl), distance (dbl), hour (dbl), minute (dbl), deptime (dbl),
##   realdelay (dbl)
weather
## Source: local data frame [8,719 x 14]
## Groups: month, day, hour
## 
##    origin year month day hour temp dewp humid wind_dir wind_speed
## 1     EWR 2013     1   1    0 37.0 21.9  54.0      230      10.36
## 2     EWR 2013     1   1    1 37.0 21.9  54.0      230      13.81
## 3     EWR 2013     1   1    2 37.9 21.9  52.1      230      12.66
## 4     EWR 2013     1   1    3 37.9 23.0  54.5      230      13.81
## 5     EWR 2013     1   1    4 37.9 24.1  57.0      240      14.96
## 6     EWR 2013     1   1    6 39.0 26.1  59.4      270      10.36
## 7     EWR 2013     1   1    7 39.0 27.0  61.6      250       8.06
## 8     EWR 2013     1   1    8 39.0 28.0  64.4      240      11.51
## 9     EWR 2013     1   1    9 39.9 28.0  62.2      250      12.66
## 10    EWR 2013     1   1   10 39.0 28.0  64.4      260      12.66
## ..    ...  ...   ... ...  ...  ...  ...   ...      ...        ...
## Variables not shown: wind_gust (dbl), precip (dbl), pressure (dbl), visib
##   (dbl)