AI21 Labs conducted a social experiment last spring, in which more than 2 million participants participated in more than 15 million online conversations. At the end of each conversation, each participant had to guess whether their interlocutor was a human or a robot. Nearly a third of them made a mistake (and the French did quite well in this exercise).
AI21 Labs was inspired for the “Human or Not” experiment by Alan Turing’s assessment of a machine’s ability to exhibit a level of intelligence indistinguishable from that of a human being.
This type of experiment will be known as the “Turing test” because of the observation made by the mathematician in 1950: “I think that in 50 years it will be possible to make computers play so well the imitation game that an average interrogator will not have more than a 70% chance of making the correct identification after 5 minutes of questioning”.
The results of the Human or Not experiment confirm Turing’s prediction
The results of the Human or Not experiment confirm Turing’s prediction: Overall, participants in the experiment guessed correctly 68% of the time. When paired with an AI chatbot, participants only guessed correctly 60% of the time. When the caller was another human, they guessed correctly 73% of the time.
While not a perfect Turing test, AI21 Labs’ Human or Not experiment showed that AI models can mimic human conversation convincingly enough to fool people. This challenges assumptions about the limits of AI and could have implications for the ethics of AI.
The experiment showed that human participants used different strategies to try to spot the AI bots, including asking personal questions, learning about current events, and rating the level of politeness of responses.
Bots trick players into adopting human-like behaviors
On the other hand, the study authors found that bots tricked gamers into human-like behaviors like using slang, typos, rudeness in responses.
“We created ‘Human or Not’ with the aim of giving the general public, researchers and policy makers a better understanding of the state of AI at the start of 2023,” said Amos Meron, Head of creative products at AI21 Labs at the time of the experiment. One of the goals, he added, was to “see AI not just as a productivity tool, but as future members of our online world, in a time when people are questioning how AI should be implemented in our future”.
Source: “ZDNet.com”