In the past year, machine learning has demonstrated its new skills -- learning by absorbing billions of words collected from the Internet, generating text according to various prompts, and performing various tasks -- creating novels, generating images, chatting with historical people and objects according to prompts, and hopefully helping to realize automatic programming. Automatic programming refers to the automatic generation of program code by artificial intelligence based on a brief description of the functions that program code should perform. Overseas academic websites call it "Ai's latest trick".

Gpt-3 will be born in May 2020. Gpt-3 (general trained transformer-3) was created by openai, an artificial intelligence research laboratory in San Francisco (resources [1]). Gpt-3 uses deep learning. Its training object is a large number of texts collected from the Internet. It has trained about 200 billion words and can generate texts similar to those written by human beings.

Gpt-3 is a huge artificial neural network, full version, which can hold 175 billion machine learning parameters. It can well capture the language pattern, write an article on a given topic, concisely summarize an article, or answer questions about the content of the document. Recently, openai announced that by the end of March 2021, "tens of thousands" of developers have used gpt-3 in more than 300 different applications, generating 4.5 billion words of text every day. This may be a milestone for openai to celebrate, and also a sign of the growing scale, impact and business potential of machine learning artificial intelligence text generation.

Gpt-3 is a language model. It is a statistical program to predict possible word sequences. Gpt-3 is trained through a huge data set (from the public Internet, Wikipedia and other resources). It has learned millions of conversations and can calculate which word (or even character) should be behind the words around it. When you enter a set of initial words, such as "go to the store to buy...", gpt-3 will start to predict what will happen next based on the knowledge acquired through training, which may be like this: "eggs", "milk", "bread", "fruit", etc. The output is obtained by counting the context of thousands of different potential scenarios and tasks. Gpt-3 is special in that it can intelligently respond to the smallest input. It has been widely trained on billions of parameters. Now it only needs some hints or examples to perform the specific tasks you want, which is called "little shot learning". For example, after analyzing thousands of poems and poets, you just need to enter the name of a poet, and gpt-3 can create an original poem with similar style to the author. Gpt-3 reproduces the texture, rhythm, genre, rhythm, vocabulary and style of the poet's previous works, and produces a new poem.

When openai first submitted a paper, it provided only a few examples. When beta testing was conducted in July 2020, openai allowed selected users to access the beta version of gpt-3. It's amazing that in just a few months, developers quickly applied it to various fields: gpt3 shows excellent new skills of natural language processing. It can complete an article according to our tips, write poetry, blog, reporter report, fictitious interview, change the tone of sentences, pun and other creative works, crack jokes Write realistic business memos, generate cooking recipes, create website models, automate programming, interpret code, customer query services, medical diagnostics, create music, and even generate images from text descriptions.

For example, in a demonstration in January 2021, openai researchers released Dall · e, a trained neural network that can use gpt-3 to create images from text titles expressed in natural language. Figure 1 shows an example of an image generated by artificial intelligence according to the text prompt: "an armchair in the shape of a avocado" (reference [4]).

Gpt-3, this kind of artificial intelligence, has aroused the interest of some software giants. Microsoft invested $1 billion in openai as early as 2019, and announced a cooperation agreement with openai. Kevin Scott, CTO of Microsoft, released the good news on social network“ Today, we are excited to announce that Microsoft has reached a cooperation with openai and obtained the exclusive license of gpt-3, which allows us to develop and deliver more advanced artificial intelligence solutions for users. ".

It has been commented that the release of gpt-3 may be one of the most important news events in 2020. There's a lot of hype about gpt-3 (a lot of media contribution). In some people's eyes, it is omnipotent, exclaiming that it "will take over the world". However, others emphasize the limitations of gpt-3: Although gpt-3 seems to understand the context of text and produce output like human beings, it does not understand things. The output of gpt-3 is not the result of logical reasoning. All it does is look at the text entered into the model and give the results based on statistical possibilities. If you give it a normal sentence, it will perform well, but when you give something unusual to the model, it will give an absurd output based on statistical possibility. In the test, gpt-3 has produced some meaningless answers, or biased, wrong answers. Some experts warn that people should not have totally unrealistic expectations about what a large-scale language model like gpt-3 can do. As Turing prize winner Yann Lecun points out, "it's interesting, and it may be useful as a way to help creativity," but trying to build intelligent machines by expanding the language model is like "making a high-altitude plane to the moon.". It may break the altitude record, but landing on the moon will require a completely different approach. "

Automatic programming

Automatic programming was once the dream of early computer scientists. For example, when FORTRAN was introduced in the 1950s (invented by John Backus for IBM in 1954 and commercially released in 1957), its full name was FORTRAN automatic coding system. FORTRAN, as a general compiler language, is especially suitable for numerical calculation and scientific calculation. It replaces most of the tedious work of programming directly with machine code or combined language. However, FORTRAN (and later programming languages, such as C, C + +, Java, python, etc.) is not an automatic programming system. Programmers need to write the program architecture and correctly fill in every detail of the program code. No wonder the career of programmers is so frustrating (think, there are 10 million lines of code in the operating system of smart phones). Artificial intelligence researchers have been pursuing automation in many fields, but for a long time, programming automation seems to be a blind spot.

In the decade of the 21st century, the technological breakthrough of deep learning (see the last blog "machine learning ramble: the brilliance of deep learning") has re inspired the dream of pursuing automatic programming. For example, in 2019, researchers at MIT's CSAIL (Artificial Intelligence Laboratory for Computer Science) developed a programming artificial intelligence sketchadapt, which teaches computers to write short computer programs by combining deep learning and symbolic reasoning. The key idea is that the flexible combination of pattern recognition and explicit reasoning can be used to solve these complex programming problems (reference [5]). Sketchadapt has been trained with thousands of program examples. Instead of relying on experts to define the program structure, it uses deep learning and artificial intelligence to automatically build a high-level structure. When the neural network is not sure where to put the code, sketchadapt is programmed to leave a blank so that the search algorithm can find the appropriate subroutine to fill in the details. The system consists of two main components (Figure 2): Sketch generator and program synthesizer. The program specs are input into the sketch generator in the form of examples. The sketch generator is parameterized by the recurrent neural network (RNN) and outputs the program sketches the outline of the whole program. The program sketch is passed to the program synthesizer, which searches for the complete program that meets the specification. The intermediate representation and training algorithm of the system allow the program synthesis system to learn when to rely on pattern recognition and when to perform symbol search without direct supervision. At present, sketchadapt can only write very short programs. The aim, the researchers say, is to supplement rather than replace programmers.

In 2020, the breakthrough of gpt-3 machine learning text generation technology leads to a new hope that artificial intelligence can be used to generate computer program code more effectively and automatically. On April 26, 2021, the website of CACM (International Computer Society Communication) reprinted the article of wired website with the title of "now for AI's latest trick: writing computer code", reporting the latest skill of artificial intelligence: writing computer code (reference [6]). Sourceai, a start-up in Paris, is fine-tuning a tool that uses artificial intelligence to write code based on a brief description of what code should do, the report said. For example, tell the tool to "multiply two numbers given by the user," and it will write about 12 lines of code in Python to accomplish the task.

This tool refers to gpt-3 code generator sourceai. The report points out that sourceai's ambition marks a revolution in software development. Advances in machine learning make it possible to program automatically, from automatically completing code segments and fine-tuning algorithms to searching source code and finding errors in code.

According to the introduction of sourceai website, gpt-3 code generator sourceai is a powerful tool based on gpt-3, which can generate any programming language source code required by users. The tool is open to all (even non developers); Easy to use - clear and intuitive interface; Fast - save development time, generate code with one click, use your time more effectively; AI driven -- the next generation development technology driven by the most advanced AI technology gpt-3. For example, in Figure 3, the left side represents the user's input, telling the tool to use the python language, "calculate factor of number given by user", and the right side represents the Python code generated by the tool (resources [7]).

"When we tested the gpt-3 tool, we realized that it could generate code," said furkan bektes, founder and CEO of sourceai. At that time, we had the idea of developing sourceai. "

The goal of sourceai is to enable users to generate a wider range of programs in many different languages, thus helping to create more software automatically. " Developers will save coding time, and people without coding knowledge will be able to develop applications, "Becks said. He is not the first to notice the possibility. Shortly after the release of gpt-3, a programmer showed that it could create custom web applications by mixing input code fragments, including buttons, text input fields, and colors. Debild, another company, plans to commercialize the technology.

How sourceai tools actually work remains to be seen. At present, there is no public demonstration of this technology, but it is said that for simple commands, it can work 80% to 90% of the time. Becks seems confident that it can change some aspects of software development. "Students will use it to do their homework quickly," he joked.