great question, I am not aware of a "validated" instrument. Possibly it may be useful for you to present a graph that leads astray and then show an improved graph. one potential example , I have colleagues who despise (the most polite word I can find after rejecting the word frothing) "dynamite graphs". Vertical bar graphs with say a "T" for the upper confidence limit. Looks sort of like a detonator for dynamite from a very old Hollywood movie

https://techcommunity.microsoft.com/t5/excel/95-confidence-interval-to-bar-graph/m-p/334804?lightbox-message-images-335108=72849i542DBE4AF424F24

https://techcommunity.microsoft.com/t5/image/serverpage/image-id/72849i542DBE4AF424F248/image-size/large?v=v2&px=999

Permit me some historical context. As recently as the 1990's decade, we were ->forbidden<- to use SAS proc plot or gplot (a luxury at the time) to prepare graphs for inclusion in a submission or any other materials sent to FDA. That happened because one of our team members took a SAS proc plot output and tried to match the letters (SAS printed using letters A B, ....etc.) with the actual (pk type ) data and couldn't. Of course a letter is not a point on graph. That was escalated up the corporate executive food chain and the instruction forbidding use of proc plot anywhere in our filing was sent back. The concern was that an FDA reviewer might also try to match points on a graph with actual data and <by a chain of logic I fortunately have long ago forgotten> decided there was a risk of FDA refusing the entire filing we were preparing for the new drug because of a problem with a graph :) I shelved my graph - is it turns out only for a decade or so. By coincidence I found that graph (in acetate) in my notes about a year ago. I scanned those This was the type of graph we were forbidden to send to FDA https://flic.kr/p/2oWjjkh I drew that graph , the color version was created with help from an expert SAS programmer. An exception of some kind was made for the presentation at the Advisory meeting . Several of the SAS programmers were tasked with "validating " each graph that might be presented at advisory.

we were also forbidden to use one other alternative. For a hefty per plot fee ($$$$) one could arrange for a flat bed plotter from HP (we were near silicon valley and HP was just a few doors down the street) . That required use of an arcane plotting language unique to the flatbed plotter. There was at most one (1) test plot- because those cost $$$$$ the same as the production plot. Long ago forgotten. What the flatbed plot could do was plot at a precisely defined x-y position on the flatbed. in the present era, one can simply plot in R or SAS in an exact x-y location in the plotting area. To the best of my knowledge FDA has never refused a filing because of a plot. FDA will include plots in a drug label. For example labels have appeared with 'forest plots" and "waterfall plots" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4093310/ FDA has organized a workshop about graphics. https://www.fda.gov/media/96653/download and I know of at least one pharma company that has routine meetings where data is presented in a graphical format. The scientists in the laboratories often prepared graphs and one package I recall hey used was sigmaplot. And some of those plots may have appeared in some reports sent to FDA.

The only actual examples I know of researchers trying to read actual data values from a graph is in some meta-analyses. There are software packages one can purchase which let one use a stylus, on say a Kaplan Meier in a published article to extract the values for the KM curve.

You mentioned Cleveland and on one occasion he and his collaborators used randomization in a design to evaluate a plot. (page 219 in the attached link)

https://cloud.r-project.org/web/packages/ggcleveland/ggcleveland.pdf

Please be sure to check Frank Harrell's notes, lectures and textbooks for advice on graph construction. for example the following is quite detailed.

https://hbiostat.org/doc/graphscourse.pdf

Hadley Wickham, developed ggplot ,and likely has written more extensively on proper graph design

from a google search, Hadley comments on 3d graphs in this interview

https://simplystatistics.org/posts/2012-05-11-ha/

- most important have fun!

------------------------------

Chris Barker, Ph.D.

Past Chair

Statistical Consulting Section

Consultant and

Adjunct Associate Professor of Biostatistics

www.barkerstats.com---

"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."

-Steve Lacy

------------------------------

Original Message:

Sent: 09-24-2024 23:08

From: Christopher Ryan

Subject: validated instrument for rating "quality" or "effectiveness" of graphical display of data

(Cross-posted to Consultants and Graphics forums)

I attended a medical conference recently, along with about 350 medical students, residents, and faculty. I gave a presentation on effective graphical display of data, with my take on the principles of Tufte, Cleveland, and Cairo (although clearly not up to their caliber). About 6 people attended. (But I'm over that . . . .)

There were over 100 poster presentations. It was great to see such enthusiasm for scholarly activity among physicians early in their careers. (Full disclosure, I'm a physician too, although far from "early.")

Among the 100+ posters, There were 2 with scatterplots (one from the team I was consulting with.) The most common type of graph seemed to be pie charts---some of them monocolor ("100%"). Bar charts of means and counts were running a close second. A lot of 3-D embellishments in use also.

This experience has gotten me thinking about ways to help this group improve their graphs. If I can devise and pitch an intervention, I'd like a way of assessing its effect (if any). Can anyone point me to validated instruments for measuring the "quality" or "effectiveness" of a graphical display? I don't want this to be about my personal, idiosyncratic, and perhaps misguided aesthetic opinions.

Thanks.

------------------------------

Christopher Ryan

Agency Statistical Consulting, LLC

------------------------------