Much has been said lately about the enormous promise of OpenAI’s ChatGPT program for generating natural language statements in various styles and themes.
ChatGPT fascinates the crowds so much that many think this program is one of a kind. Still, AI specialists don’t quite agree.
“In terms of the underlying techniques, ChatGPT isn’t particularly innovative,” Yann LeCun, chief AI scientist at Meta, said during a small press and executive meeting on Zoom last week. “There’s nothing revolutionary about it, even if that’s how it’s perceived by the public,” said the deep learning pioneer.
“It’s just, you know, it’s well put together, it’s nicely done. »
OpenAI is not an isolated case
Such data-driven AI systems have been built in the past by many companies and research labs, Yann LeCun pointed out. The idea that OpenAI is alone in this type of work is therefore inaccurate, according to the researcher.
“OpenAI is not at all a step up from other labs,” he said. “It’s not just Google and Meta, but there are half a dozen startups that basically have technology very similar to this,” he added. “I don’t mean it’s not rocket science, but it’s rarely shared, there’s no secret behind it, if you will. »
According to Yann LeCun, ChatGPT and the program it is based on, GPT-3, is made up of multiple pieces of technology developed over many years by many parties. “You should know that ChatGPT uses Transformer architectures that are pre-trained in a self-supervised way,” observed Yann LeCun. “Self-supervised learning is something I have been advocating for a long time, even before OpenAI existed,” he added.
“Transformers is a Google invention,” noted Yann LeCun, referring to the linguistic neural network unveiled by Google in 2017, which has become the basis for a wide range of linguistic programs, including GPT-3.
decades of research
Furthermore, Yann LeCun points out that work on such language programs goes back decades.
“Large language models, the first neural network language model – at the time it was large, by today’s standards it is tiny – was made by Yoshua Bengio, about 20 years,” said the researcher, referring to the head of the MILA Institute for AI in Canada. Yoshua Bengio’s work on the concept of attention was then taken over by Google for the Transformer and became a central element of all language models.
OpenAI’s program also made extensive use of a technique called reinforcement learning through human feedback, which allows human agents to help rank machine results in order to improve them, much like the Google Page Rank for the Web. This approach was pioneered not by OpenAI, but by Google’s DeepMind unit, he said.
The ChatGPT program is less a scientific breakthrough than an example of decent engineering, according to Yann LeCun. To illustrate his point, he compared the program to the IBM Watson computer that participated in the 2011 game show Jeopardy! and entrepreneur Sebastian Thrun’s self-driving vehicle that won DARPA’s 2005 Grand Challenge, the defense research and development agency in the United States. Sebastian Thrun’s award-winning technology “wasn’t particularly innovative in terms of the science behind it”, said Yann LeCun, “it was just very well designed”.
“That’s kind of what OpenAI did,” he added. “I’m not going to criticize them for that. »
Source: ZDNet.com