Giordano, Snow, Yu, Stark, Recht

My Neyman Seminar in the Statistics Department at Berkeley was followed by a lively panel discussion including 4 Berkeley faculty, orchestrated by Ryan Giordano (Dept of Statistics):

• Xueyin Snow Zhang (Dept. of Philosophy)
• Bin Yu (Depts. of Statistics, Electrical Engineering and Computer Sciences)
• Philip Stark (Dept. of Statistics)
• Ben Recht (Dept. of Electrical Engineering and Computer Sciences)

I want to share the many fascinating (and difficult) questions put forward by panelists earlier on the day of my talk. Of course, once the live discussion began it took on a life of its own, and we rarely looked back at this list. I’m only now returning to many of those that we didn’t cover–and I’m interested in reader comments. I’m extremely grateful to the organizers, panelists, and audience for creating such a uniquely enriching and provocative exchange of ideas–one that is sure to be continued!

Philip Stark:

• When and how did Statistics lose its way and become (largely) a mechanical way to bless results rather than a serious attempt to avoid fooling ourselves and others?
• To what extent have statisticians been complicit in the corruption of Statistics?
• Are there any clear turning points where things got noticeably worse?
• Is this a problem of statistics instruction (teaching methodology rather than teaching how to answer scientific questions, deemphasizing assumptions, encouraging mechanical calculations and ignoring the interpretation of those calculations), of disciplinary myopia (to publish in the literature of particular disciplines, you are required to use inappropriate methods), of moral hazard (statisticians are often funded on scientific projects and have a strong incentive to do whatever it takes to bless “discoveries”), or something else?
• What can academic statisticians do to help get the train back on the tracks? Can you point to good examples?

Snow Zhang

• How does severity testing inform practical deliberation (e.g. policy-making, data-gathering)? More generally, what do you take to be the relationship between inference and decision-making?
• Do you think we can be pluralists about “evidence”/”confirmation”/”warrant”? (E.g. Can it depend on the stakeholder (experimenter vs. policymaker vs. informant) and practical context, e.g. courtroom vs. academia vs. industry?)
• In philosophy, Bayesianism is often taken to be a normative ideal (in two somewhat conflicting senses: 1. The closer we are to approximating Bayesian inference, the more rational we are; Idealized agents with no cognitive bounds should practice Bayesian inference, though the same is not necessarily true of bounded agents like us.) How do you think this version of Bayesianism relates to Bayesianism in statistical inference, and do you think it is subject to similar criticisms that you raised in your book?

Bin Yu

• What does probability mean in severity testing? Could it go beyond the stochastic generative modeling? What does severity testing say specifically about model checking?
• How does severity testing relate to the other sources of uncertainty in a data science life cycle such as those from data cleaning choices and modeling choices made by analysts?
• Is severity testing necessary in most physical science data problems, say climate modeling?
• What would be an example that you can share where severity testing are largely not needed?
• Statistics is How do you see severity testing evolve into the AI age?

Ben Recht

• What are your favorite examples of statistical methods being employed to definitively prove or disprove the effectiveness of an intervention? What was it about this application that elevates it above the common misapplications we all harp on?
• Even if the epistemological value of statistical tests is highly questionable, is it reasonable to use statistical tests as benchmarks for regulatory approval (say for drugs or policy)?

I’ve had a few ‘blogologues’ with Recht recently on his blog and mine, e.g., here, here, and here.

Slides from my presentation are in my previous post. Please share your thoughts, and proposed replies, in the comments.