ASA Connect

 View Only
Back to eGroups
Expand all | Collapse all

Have you used Chat GPT to answer your stats exam questions?

  • 1.  Have you used Chat GPT to answer your stats exam questions?

    Posted 03-23-2023 13:03

    Hello Everyone,

    I'm wondering if anyone here has used Chat GPT to answer questions on their stats exams? 

    Over the last 5 years, I have taught as many in person stats classes as I have online. I give my online students a few days to get teh exams done. My in person students get the class period. I've noticed that their grades tend to be the same. 

    This semester, I have given my students exams and asked Chat GPT for answers. The problems my students did well on, Chat GPT did poorly on. The problems my students did poorly on, Chat GPT did well on. 

    So, I'm wondering:

    1) If anyone else has done this with their exams? If so, how did it do? 

    2) If my online students do just as well as my in class students, and clearly are not using ChatGPT or other sources for answers, should I be worried about them cheating or should I be more inclined to let my in person students do take home exams? 



    ------------------------------
    Andrew Ekstrom

    Statistician, Chemist, HPC Abuser;-)
    ------------------------------


  • 2.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-24-2023 07:29

    Good question!  I asked it some questions, starting at the most basic and worked forward in increasing difficulty.  Here are the results I attained:

    1) Control question.  I asked it what time it was.  Did not know, but told me how I could find out.

    2) Asked it to describe difference between mean and median.  Response was correct.

    3) Asked under what conditions should I use mean and median.  Response was correct.

    4) Asked if I should use the mean when my continuous data are positively skewed.  Response was correct.

    5) Asked if I could use Chi-squared statistic to do a between group comparison of continuous data.  Response was correct.

    6) Asked what the assumptions are for linear regression.  Response was incomplete and out of scope (so incorrect).  It provided an incomplete list of assumptions and how to interpret residuals.

    Happy chatting!



    ------------------------------
    Robert
    ------------------------------

    Original Message


  • 3.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-27-2023 10:36

    I give my students problems like those they might find in industry. So, not things you can easily look up in a textbook. I'm fairly unimpressed so far. 



    ------------------------------
    Andrew Ekstrom

    Statistician, Chemist, HPC Abuser;-)
    ------------------------------

    Original Message


  • 4.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-27-2023 11:14

    A colleague of mine and I have played around with ChatGPT and exams. Here are the results:

    So I ran (or attempted to run) all of the first introductory stats exam through ChatGPT.

    First note that this had to be done question by question, and this was pretty time consuming. Several times the AI timed out, and I sometimes had to log back in again. In practice this would not be a very efficient method of cheating.

    The exam has 30 questions. 25 are multiple choice, and 5 are true false.

    Multiple choice: Two of the problems had pictures in them, which I couldn't figure out how to easily load into the AI. This fact in of itself could help to deter AI use.

    One other problem was just too difficult to format for the AI, it was the 4th of a series of problems referring to some tables, and in general the AI does not handle sequential problems very well. It is almost certain that if I had successfully loaded the problem, the AI would get it wrong.

    Again, this provides another tip for discouraging AI use, make problems sequential rather than stand alone.

    Anyway, of the 25 multiple choice problems, I was able to get the AI to try 22 of them.

    In one problem, I tried 3 times and each time the AI timed out, so I marked that as wrong.

    In the remaining 21 problems, the AI got 9 correct. Clearly the AI gets a failing grade on my multiple choice problems.

    True/False was a different story. I got the AI to respond to all 5 problems, sometimes very quickly, and the AI got 4 of the 5 correct. This is actually better than the performance of many of the students on T/F problems.

    Overall, the AI got 13 correct of the 27 problems I gave it, which is an F.

    So to discourage AI use in exams, I recommend the following:

    1. Include figures and graphs.
    2. Make questions sequential rather than stand alone.
    3. Avoid true/false problems.

    Obviously these conclusions are based on only one exam.

    Cheers,

    Monnie



    ------------------------------
    Monnie McGee
    Associate Professor
    Southern Methodist University
    Dallas,TX United States
    ------------------------------

    Original Message


  • 5.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-30-2023 11:19

    Cool.

    My problems are sequential.

    My students always make the mistake of asking, "How many problems are on the exam?" I tell them, 10.... But there is part i to xiii on many.

    Perhaps one of my departments should look into making their dept final exams sequential instead of "Find the mean of this data. Find the median of that data. Find the variance of a 3rd set of data." 

    Thanks for the info. 



    ------------------------------
    Andrew Ekstrom

    Statistician, Chemist, HPC Abuser;-)
    ------------------------------

    Original Message


  • 6.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-24-2023 08:59

    I have tried this.  I teach a grad level public health/biostats course.  With ethics questions, I found ChatGPT did ok but not great (not very deep).  It did better with questions on concepts such as validity, generalizability, bias, etc.  For those questions, I think it could actually be a good learning tool - I probed a bit on issues of internal validity and bias, and it did a good job explaining distinctions.

    Nilupa Gunaratna

    Purdue University



    ------------------------------
    Nilupa Gunaratna
    Associate Professor
    Purdue University
    ------------------------------

    Original Message


  • 7.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-27-2023 10:34

    I give all my stats students problems with medical testing. I have them calculate the probabilities a medical test gives correct results, positive tests are correct, etc. ChatGPT uses Bayesian methods to calculate that. And gets it wrong every time. It WAY OFF TOO!! I have my students assume there are say 10,000 people taking the test, and have them fill out a 2x2 table and use that for calculations. Chat GPT uses Bayesian method because that is what is written about the most for those types of questions. 

    I even asked it what is the probability that a person that tested positive on a medical test will test positive a second time on that same test. It came back with another Bayesian calculation that was totally wrong. However, it gave answers I've read on ASA chat boards and in some articles. (But, those responses are wrong because they ignore realities about how a medical test works.) 



    ------------------------------
    Andrew Ekstrom

    Statistician, Chemist, HPC Abuser;-)
    ------------------------------

    Original Message


  • 8.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-24-2023 13:44

    Andrew, I'm afraid I don't have an answer for the general issue of potential cheating on take-home exams. 

    But so far, in my experience, ChatGPT is still error-prone when it comes to mathematics, and so probably is not a main reason for the concern.   Not unlike some students, perhaps, it's very poor at self-checking:   It comes up with an answer, somehow, but if you probe, you can find its answers are inconsistent, and it doesn't seem to notice or care about that.  

    I mentioned this to some colleagues with experience in AI, and they pointed out that ChatGPT is really focused on efficiently sampling mega-examples of text and predicting plausible next things to say.  But so far, it doesn't then take the time to double-check for inconsistencies, etc.

    One interesting weakness it has (which is more relevant for essay questions than stats questions) is when it comes to citations.   If it appears to cite something, it is probably echoing other people's citations of the thing---and it doesn't actually go off and find and read the source document for itself.   So, if I were assigning essay questions now, I'd spot-check some of the essays' citations to see if they accurately point to something. 



    ------------------------------
    William (Bill) Goodman
    Professor (Retired) and Adjunct Professor, Faculty of Business and Information Technology
    Ontario Tech University
    ------------------------------

    Original Message


  • 9.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-27-2023 10:41

    I'm working as a tutor at a couple difference colleges right now. I was talking to the English/writing tutors. They noticed that students have more convincing but not self consistent results. When they have ESL students that write the same way as native English speakers, they've found that those students are using ChatGPT to write the entire essay, instead of using it a starting point.  They've also had to go back and discuss essay writing with their students. 



    ------------------------------
    Andrew Ekstrom

    Statistician, Chemist, HPC Abuser;-)
    ------------------------------

    Original Message


  • 10.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-24-2023 14:18
    Edited by Trudy Beerman 03-24-2023 14:18

    Unless you are teaching how to use ChatGPT as a resource tool, and by your own word choice - you see it as "cheating," then look into the tools for checking if ChatGPT was used and warn against it. That may not stop your online students from using it, but then they are breaking an honor code at the very least.

    Using a calculator in math is not considered cheating by the time certain foundational learning is cleared, then students are taught to use the resource tool, the shortcut tool of the calculator. I believe Chat GPT will change how education happens, and when it does, teaching will focus more on accessing tools and how to effectively use them rather than needing to know the thing without the tool. Until then, ChatGPT to some and not to others, is an unfair advantage.



    ------------------------------
    Trudy Beerman
    Owner
    PSI TV Network (Profitable Stewardship Inc.)
    ------------------------------

    Original Message


  • 11.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-27-2023 10:44

    I'm looking forward to the time when I can use something like ChatGPT as an aid to teaching. So far, when it comes to the math I have my students doing, it's pretty horrible. 

    But, this weekend, I did spend about 20 minutes discussing with a student how to use thier calculator to get the right answers to their hw problems. Order of operations and the use of brackets is still an issue. 



    ------------------------------
    Andrew Ekstrom

    Statistician, Chemist, HPC Abuser;-)
    ------------------------------

    Original Message


  • 12.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-27-2023 12:27

    Just for fun I ran a few questions from Casella and Berger's probability and inference textbook through ChatGPT.  I found that it handled stuff like finding MLEs and the pdfs of transformed random variables fine.  A more conceptual question like showing that a power function for a exponential family distribution is differentiable required several tries and ended up with what looked like a superficially correct answer but I think may have contained "hallucinations" (at one point it said a pdf by definition was continuous and differentiable).  That said, in a year or two these kinds of things might be cleaned up.  Already intro MS stat/biostat courses will face challenges determining if homework was completed by the student or the AI I think.

    We didn't stop teaching multiplication when calculators became available, or calculus when Mathematica came on the market.  But there will have to be some way to a) convince students of the need to learn material and b) confirm that they are doing so.  I don't know what the answer to that is.



    ------------------------------
    Michael Elliott
    University of Michigan
    ------------------------------

    Original Message


  • 13.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-30-2023 11:32

    When I was at Eastern Michigan University, I took a class on numerical analysis. The prof would ask us for functions to deal with. Many of the students would shout out something like Y = exp(-7x). He'd get mad and ask for a real function. 

    At some point I asked for Y = 12*sin(cot(exp(-17.3*pi*x)))/(exp( x^2^ cos(89.32*x)). He smiled. The class looked at me funny. Until he did that problem. Then they realized what he was looking for. 

    So far, my MS Biostat students haven't been able to use GPT to answer their problems. But, I ask extensions of ideas from class and show how to get the right answers with say simulations. So, if all my students have the exact same answer, real easy to see who cheated;-) 



    ------------------------------
    Andrew Ekstrom

    Statistician, Chemist, HPC Abuser;-)
    ------------------------------

    Original Message


  • 14.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-27-2023 12:54

    I am teaching an intermediate level business statistics course for undergraduate. I fed one of the exams to GPT4. The results are mixed. GPT4 can easily points students to the right direction, i.e., what formula to use, what information is given. However, it makes obvious errors in the calculation. One simple example is calculating the average. It got the average wrong for as few as six numbers. GPT4 may not easily detect trick questions. One question asks the students to calculate the probability of a normal RV equals 0. It should be zero by definition. Yet, GPT4 calculate P(X=0) as P(X<=0). 



    ------------------------------
    Jiaxiu He
    Assistant Professor
    University of Texas Arlington
    ------------------------------

    Original Message


  • 15.  RE: Have you used Chat GPT to answer your stats exam questions?

    Posted 03-30-2023 12:48

    I have a table of problems with P(a<X<b), P(a<=X<b), P(X = int), P(X = float) for the normal, exponential, logistic, Poisson and Binomial distributions. For the continuous distributions, I allow students to answer the X=?? questions as 1/infinity or 0. I ask them to write what their calculators give them with say the binomial dist when X=float. It should give them an error message. 

    I also give them the trick question, "What is the probability of getting 12 or less comparisons correct if we have 10 comparisons to make?" Should the student blindly put those values into their calculator without thinking, they get the right answer. If they think a little bit about it, they get the wrong answer... but it's based upon proper reasoning. If they think about it a lot, they get the right answer with proper reasoning. I haven't tried entering this problem in ChatGPT yet. But, I will soon. 



    ------------------------------
    Andrew Ekstrom

    Statistician, Chemist, HPC Abuser;-)
    ------------------------------

    Original Message


Related Content