ASA Connect

Dear Colleagues: 

The increasing focus on data and benchmark tracks at conferences like NeurIPS and KDD highlights a critical area where statisticians can make significant contributions. These tracks prioritize the creation, curation, and evaluation of datasets and benchmarks that drive research progress, and statisticians are uniquely equipped to shape their development.

Opportunities:

• Data Design: Bringing rigor to ensure datasets reflect real-world complexity.
• Robust Benchmarks: Promoting metrics that emphasize generalizability, fairness, and robustness.
• Uncertainty Quantification: Enhancing interpretability and reliability in model evaluations.
• Ethics: Embedding fairness, inclusivity, and transparency in datasets and benchmarks.

Challenges:

• Scalability: Adapting statistical methods for large-scale data.
• Metrics: Advocating for nuanced, multidimensional benchmarks over simplistic metrics.
• Collaboration: Strengthening partnerships with ML researchers.
• Recognition: Elevating the value of data and benchmark contributions within our field.

Importantly, our community must recognize and reward the significant efforts involved in data and benchmark work. These efforts are foundational for advancing science, yet they often lack the visibility and prestige of methodological or theoretical contributions. As statisticians, we should advocate for greater appreciation, funding, and academic credit for those who dedicate themselves to this critical work.

By collectively addressing these challenges and championing those advancing data and benchmarks, we can ensure that our field remains integral to shaping the future of statistics and data science.

Best regards,

Hongtu Zhu