This is one reasonable view, but there are several others:
1. Starry eyed Computer Scientists view the holy grail of AI to be the
construction of computers that "work like the human mind". While they
may not be particularly close yet, they have their share of rather
amazing successes (a poster child being self-driving cars). It seems
likely this will continue over the long term in the future, and there
are folks who definitely have their eye on that goal.
2. If you look at the actual contributions over the years, there is a
rather long history of big ballyhoos (often very successful at
attracting gobs of research funding):
a) AI itself was just such a buzzword at one point. Some of you may
remember the gross over-advertisement that was done (I think around the
1970s?) of "computers will put doctors out of business at the task of
medical diagnosis". After it didn't quite work out as advertised, the
community learned from the backlash by physicians that threatening
people is not a great way to get one's methods adopted.
b) Another major event in that spirit was "neural networks", which
really was about rather directly attempting to mimic how the brain
works. There is an important set of problems that have been solved by
this approach, including the voice recognition software we deal with
when we phone any major corporation today. Once again this came with
serious over-advertising (and a massive infusion of research funding).
Folks again claimed to have methods that could solve all problems, and
of course that later turned out to be too ambitious, with many attendant
failures.
c) Another round of this type of history is "machine learning". This is
mostly about trying to do statistical tasks, but with much less emphasis
on probability distributions, and much more on optimization. Again gobs
of research funding followed the hot new ideas, and again there is a set
of analytical problems where that set of ideas has proved to be very
effective.
d) One more round relevant to this AI discussion is "Deep Learning".
This is really just a repackaging of the Neural Nets in (b) above, but
there are two major differences: [1] typical data sets are now far
larger, and it is appearing that many of the earlier failures came from
data sets too small to effectively train the methods. [2] Vastly better
computational resources are now available, to the point that much better
training of these things is giving much better results. It is worth
noting that Deep Learning methods have almost completely taken over very
active research areas such as Computer Vision (an important aspect of
e.g. self-driving cars).
3. Now let's consider the issue of multiple false positives. First, from
a classical statistical viewpoint, I completely agree. However, it could
be useful to also consider a bigger picture. One relevant dichotomy of
data analytic tasks is "Causal" versus "Correlational". Here Causal
refers to the goal of really understanding the driving factors in an
analysis, and the desire to quantify the strength of the evidence for
that. The poster child for that is the "Scientific Method" (developed
over quite some period more than 100 years ago), and the basis of most
modern statistical inference. From the causal perspective, many of the
approaches mentioned above have precisely this weakness. Correlational
is just about getting answers to very specific problems (voice
recognition, or keeping a self-driving car on task). Note that these
explicitly do NOT attempt to say anything about underlying driving
factors, but in fact just give particular (but often very useful)
practical results.
4. Where should we go from here? For me both approaches have their
place. I agree with the concerns expressed, but I also believe we
should teach more correlation methodologies (e.g. machine learning or
deep learning related) in our courses, as they are important tools in
the future of data analysis.
Best,
Steve
Original Message------
AI opens the door wide to multiple testing false positives.