AI writes its own code to let agents evolve! The big model of openai has the flavor of "human thought"

Jin Lei Dream morning From the Aofei temple
qubits | official account QbitAI

Do things ！

AI“ see ” I took a look GitHub How do humans submit updates on the Internet （commit） Of , Then imitate human beings Programmers modify code ……

Final , This AI Also successful “ Training ” There's a agent robot ：

No kidding , such Consider very fear Things about , stay OpenAI In a newly published study , It really happened ……

Originally , What the researchers want to solve is a genetic programming （GP） problem —— Let an intelligent robot learn to move .

（GP It is a special field in evolutionary computation , It is mainly aimed at automatically building programs to solve problems independently .）

but OpenAI The sword moves sideways , Put your own large-scale language model （LLM） Put it in , The result is a big “ Never in my wildest dreams ”.

Previously, in the process of agent evolution , Human researchers need to be involved in making some detailed adjustments 、 Determine the evolution direction , Let agents develop in a good direction .

Ok now , These jobs are all taken care of by the big model , Learn by yourself 、 Write your own code 、 Go by yourself “ Training ”：

As soon as this matter is discussed Joel Lehman Exposure on the Internet , It immediately attracted a lot of attention from netizens ：

A programmer netizen called directly after reading “ To keep pace with （ technology ） The pace of development ” 了 ：

even to the extent that OpenAI I have said in my research that ：

It has bridged the gap between evolutionary algorithm and human thought .

So this one “ Magic ” Things about ,AI How did you do it ？

Take a look at GitHub,AI Knock the code yourself

Design mobile robot in virtual environment , It is a very popular project in genetic algorithm research .

especially Sodarace The competition needs less calculation , Easy visualization of processes is very popular .

The rules are simple , from “ The joints ” and “ muscle ” The robot races on all kinds of terrain .

OpenAI It also specially rewrites the entire competition program from a special genetic code to Python edition , To demonstrate the versatility of the new approach to modern programming languages .

For example, such a paragraph Python Code , Can be used as the initial seed robot .

Define the four vertex joints of a square 、 End joint , Both of them use “ muscle ” After connecting , give the result as follows .

However, such a square structure cannot move , Next, we need to modify the code by genetic algorithm .

According to the research team , Modify the code with traditional genetic algorithm VS Human programmers do it themselves , There are still two gaps in efficiency ：

One is that the software is becoming more and more complex , Humans can do modular code reuse to deal with , At present, the most advanced genetic algorithm can not do this in the programming language used by human beings .

The other is that almost all genetic algorithms rely on random mutation （mutation）, Every time a human programmer changes his code, he has a purpose , Or add functions 、 Or improve efficiency 、 Or repair bug.

So is there any way to make AI Learn how humans modify code ？

There are , All the required training data exist GitHub On .

Good programmers write code every time they submit it commit describe , Make it clear what has been modified in this submission .

commit Describe the code comparison before and after the submission diff data , Namely AI Great learning materials .

The researchers sifted through some descriptions with clear intent 、 Submit data to train a small amount of modified code GPT-3 Architecturally AI Model .

It's equivalent to letting AI Learn from human programmers how to purposefully modify a piece of code .

The model used in this paper does not need a full version GPT-3 Of 1750 Billion parameters , The highest 7.5 Billion parameters are enough .

Thus the basic AI Model , It will play the role of mutation operator in genetic algorithm .

Next let's AI The process of designing a new robot is divided into three steps .

First step , Start with the classic MAP-Elites The algorithm generates a set of initial robots .

This is a kind of QD（ Quality diversity ） Algorithm , It can ensure that robots have different behaviors and high quality .

The second step , Use the initial data generated in the first step for pre training , Give Way AI First learn to design a robot within the training data distribution .

That is, the moving picture that amazed everyone on the Internet at the beginning , It shows AI How to move the immovable “ square ” Transformed into a mobile robot with alternating legs .

The third step , Combined with reinforcement learning algorithm to make fine adjustment , Give Way AI It can generate robots that can adapt to the environment according to different terrain conditions .

Final , The researchers selected the robot evolved from the first three seeds to show the effect .

It can be seen that their structure and movement are completely different .

The netizen exclaimed “ The train of thought is strange ”

Once the study was published , It can be said that one stone aroused thousands of waves .

Many netizens are amazed at this “ Big model + Evolutionary algorithm ” A novel way of combining ：

Researchers who have done related work also said , I never thought I could use a big model to diffs To learn about mutation ：

In addition to the discussion of the research form and itself , There are also netizens with this picture ：

Emmm…… It's a little like that .

team introduction

The team members of this study are all from OpenAI.

The first paper is Joel Lehman, Is a machine learning scientist . Its focus areas include artificial intelligence security 、 Reinforcement learning and open search algorithms .

meanwhile ,Joel Lehman Previously, he wrote a scientific book based on his thinking on the development of artificial intelligence 《 Why can't greatness be planned ： The secret of objectivity 》：

And the next step for this study ,Joel Lehman I said ：

There is another important issue , It is how much the model can be applied to other environments .

GP The mutation effect in can now be achieved by ELM Greatly improve , This will stimulate a wide range of new applications and research directions .

So does this study have any new implications for you ？

Welcome to comment and discuss ~

Reference link ：

[1]https://arxiv.org/abs/2206.08896
[2]https://twitter.com/joelbot3000/status/1538770905119150080?s=21&t=l8AASYjgC6RAEEimcQaFog