Some "state of the art of statistics" . I find weather forecasting (and its partner climate modelling) an incredibly interesting statistical activity in part because of the size (petabytes) of the data. The vaccine trials are as large as I've encountered in pharma clinical trials, with about 50,000 volunteers per trial (e.g. pfizer /biontech, moderna , J&J). The datasets can be processed on 2022 laptop.
Also interesting (to me) is the application of our "standard statistical tools" to petabytes of data for forecasting weather worldwide.
Berkeley earth has global datasets as far back as from 1870
By contrast Many years ago (more than I care to admit:)) , when i was in graduate school it was a "very big deal" when my advisor bought a then "cutting edge" apple lisa and bragged about being able to invert a large matrix for a standard regression problem on his desktop computer vs using the so-called "big iron" IBM 360 mainframes. I vaguely recall "large" on a PC was perhaps a dataset with perhaps 250 rows (patients) and perhaps 20 variables.
I supervised a data processsing for a small NIH funded project where the (about) $250,000 IBM Series 1 computer had about 250 K(kilo)bytes of virtual memory. ONe computer programmer and a friend liked to complain to me (because he knew there was nothing i could do about it) that If I thought the project was not running fast enough, it was because he could only fit one calculation into memory at a time due to the virtual memory limits. :)
The article and science paper (without paywall) below describe recent advances in weather forecasting using AI (artificial intelligence) and
some pragmatic methods of comparing outputs of several forecast models. there are not enough details on whether or not the bootstrapping is using petabytes of data to compute the confidence intervals
Below The "light" read is yahoo news. Then the editors overview . Folllowed by More technical details in the science paper
a key problem in prediction is "extreme" weather events - cyclones are mentioned by the phrase is not well defined in the articles below
And some meteorology technical jargon "resolution" refers to square miles
for example
...The frames of this animation illustrate spatial resolution of a model. The lowest resolution grid shown here (T42) covers most of the entire state of Florida with just two large grid cells, which was typical of model resolution in the mid-1990s. It has grid cells of about 200 by 300 km (120 by 180 miles) at mid-latitudes. The T85 resolution is typical for current models. Its grid cells are about 100 by 150 km (60 by 90 miles) across at mid-latitudes.
------------------------------
Chris Barker, Ph.D.
2023 Chair Statistical Consulting Section
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------