A business school professor in the United States wondered if ChatGPT could get a college degree. His essay shows yes… but OpenAI’s chatbot remains a math dunce.
Students who imagine themselves doing their homework with ChatGPT would do well to beware. Not only has the conversational agent developed by OpenAI been faulted several times on various issues… but in addition, the algorithm could well be an unexpected “rival”! This is shown by a somewhat astonishing story from the United States.
CNBC reports in its January 23 edition a successful test related to a university degree, the MBA (Master of Business Administration), which qualifies a person in fields ranging from management to marketing, including finance and human resources. And the professor who framed this test shared his observations in a 26-page note.
A prestigious diploma obtained by a mediocre student
However, ChatGPT did not turn out to be a brilliant student – a sign of an artificial intelligence that was ultimately quite limited. Admittedly, in theory, he would have obtained the MBA from the Wharton School – this is a very prestigious business school located in Pennsylvania, considered the best of its kind according to the Financial Times –, but without obtaining an excellent grade, observes the teacher.
” Given this performance, the Cat GPT3 would have received a grade of B to B- on the exam. writes Christian Terwiesch, who supervised this essay. A B- is equivalent to a mark of 10 or 11/20 in France. A B is a grade of 11 or 12. It’s very passable and far from great. According to Christian Terwiesch, it is above all his performance in maths that has planted him.
ChatGPT is not a good mathematician
” Chat GPT3 sometimes makes surprising errors in relatively simple calculations from the 6th grade level to middle school. These errors can be of considerable magnitude “, he notes in his synthesis. This is obviously critical for a degree where the manipulation of numbers, through marketing or finance, is central.
Another limit highlighted during this test: “ the current version of Chat GPT is not capable of handling more advanced process analysis questions, even when based on fairly standard templates. This includes process flows with multiple products and issues with stochastic effects such as demand variability. »
But the OpenAI chatbot has necessarily shone elsewhere, otherwise it could not have snatched the average in the test. The teacher, in his analysis, recognizes a ” outstanding work on basic operations management and process analysis questions », including the case studies. He picked up responses correct “, supported by explanations ” excellent “.
The ability to take into account the advice of a human expert was also noted: ChatGPT sometimes made mistakes, but corrected its answers after being given new indications. Thus, beyond its mistakes, which a student could also make, ChatGPT has improved its production. In a way, he has returned his knowledge better.
The story does not say what appreciation would have left Christian Terwiesch in ChatGPT if he had been a real student. But in an interview with the Financial Times, he summarized his opinion as follows: I was overwhelmed by the beauty of the wording – the conciseness, the choice of words, the structure. It was absolutely brilliant…but the math is horrible.” Holy emotional lift on the report card.
Subscribe to Numerama on Google News to not miss any news!