Deciding which programming language to learn is an important question for budding developers because of the huge investment of time it takes. However, this question could quickly become superfluous as artificial intelligence (AI) models continue to improve and could eventually do all the work, from describing the problem to executing code to solve it.
Researchers from DeepMind, Google’s AI unit, have just put another pebble in developers’ backyards by claiming that their AlphaCode system is now able to express solutions to problems in code up to achieve a median score in programming contests held for new programmers. So many competitions that ask humans to understand a problem described in natural language, then code an algorithm effectively.
In a new (non-peer-reviewed) article, DeepMind researchers explain how AlphaCode achieved an average ranking among the top 54.3% of entrants in 10 previously held programming contests with over 5,000 entrants. The competitions were hosted on the Codeforces code competition platform. DeepMind claims that AlphaCode is the first AI code generation system that achieves competitive results in code competitions for human developers. This research could improve the productivity of programmers and help non-programmers to express a solution without knowing how to write code.
A strong competitor
Human competitors, and therefore AlphaCode, had to analyze the description of a challenge or riddle and quickly write a program to solve them. This task is more difficult than training a model using GitHub data to solve a simple coding challenge. Like humans, AlphaCode needed to include a multi-paragraph natural language description of the problem, narrative details, and a description of the desired solution in terms of input and output.
To solve the problem, each participant had to create an algorithm and then implement it efficiently, which might require them to choose, for example, a faster programming language like C++ over Python to overcome these constraints. AlphaCode’s pre-training dataset included 715 GB of code from files from GitHub repositories written in C++, C#, Go, Java, JavaScript/TypeScript, Lua, Python, PHP, Ruby, Rust, and Scala . The team refined the model using datasets of competitive programming problems taken from Codeforces and other similar datasets.
The impetus given by DeepMind to AlphaCode was obtained by combining large-scale transformation models. Some of these models include OpenAI’s GPT-3 and Google’s BERT language model. DeepMind used transformer-based language models to generate code and then filter the output to a small set of “promising programs” that were submitted for review.
DeepMind wants to be reassuring
“At evaluation time, we create a massive amount of C++ and Python programs for each problem, orders of magnitude larger than previous work,” the AlphaCode team at DeepMind explains in a blog post. “Then we filter, aggregate and re-rank these solutions into a small set of 10 candidate programs which we submit for external review. This automated system replaces competitors’ trial and error process of debugging, compiling, testing, and finally submitting. »
This new breakthrough from DeepMind is not without its flaws. For example, models can generate code with exploitable weaknesses, including “unintentional vulnerabilities originating from outdated code or intentional vulnerabilities injected by malicious actors into the training set.” The model also has a certain environmental impact, its training having taken “hundreds of petaFLOPS days” in Google’s data centers. But AI code generation could in the long run “lead to systems that can write and improve recursively, which would quickly lead to increasingly advanced systems.”
The question obviously arises of the automation of tasks currently carried out by developers. DeepMind wants to be reassuring by pointing out the limitations of current code completion tools that greatly improve programming productivity, but which have so far been limited to one-line suggestions and restricted to certain languages or short snippets. of code. For DeepMind, its research is far from posing a threat to developers and should instead be seen as a tool to enhance the problem-solving capabilities available to humanity.
“Our exploration of code generation leaves a lot of room for improvement and hints at even more exciting ideas that could help developers improve their productivity and open up the field to people who don’t currently write code,” say assert the DeepMind researchers.
Source: ZDNet.com