How to understand the mechanism of memory and attention in deep learning? This 212 page doctoral thesis of Deakin University gives you answers

Thesis link

Intelligence needs memory . There is no memory , Human beings cannot complete all kinds of important tasks , For example, reading novels 、 Play games or solve math problems . The ultimate goal of machine learning is to develop intelligent systems that automatically learn and act like humans , Therefore, the memory construction of the machine is inevitable . The artificial neural network connects the calculation units through weights , Modeling neurons and synapses in the brain , This is a typical machine learning algorithm similar to memory structure . Their descendants have more sophisticated modeling techniques ( Deep learning ), It has been successfully applied to many practical problems , And proved the importance of memory in the process of machine system learning . In recent years , The research of memory modeling in deep learning mainly focuses on the external memory structure , Inspired by computational Turing model and biological neuron system . Attention mechanism is produced to support the acquisition and retention of external memory . Despite the lack of theoretical basis , These methods have shown the hope of helping the machine system reach a higher level of intelligence . The purpose of this paper is to improve the understanding of memory and attention in deep learning . Its contributions include : (i) Present a collection of memory categories ,(ii) Construct a new memory enhancing neural network supporting multiple control and memory units (MANN), (iii) Introduce variability through memory in the sequence generation model ,(iv) Find the best write operation in the slot based memory network to maximize the memory ability ;(v) A new type of external memory through neural network —— Neural memory program memory to simulate general-purpose Turing machine .

The contents of the catalog ：

1 Introduction

2 Taxonomy for Memory in RNNs 9

3 Memory-augmented Neural Networks 40

4 Memory Models for Multiple Processes 57

5 Variational Memory in Generative Models 89

6 Optimal Writing Memory 103

7 Neural Stored-Program Memory 120

8 Conclusions 134

In this paper , We introduce several memory types of Neural Networks , Especially recurrent neural networks (RNNs). We emphasize memory as RNN The concept of external storage , among rnn You can learn to read and write external memory , To support their working memory ( The first 2 Chapter ). We reviewed the solution training RNN Difficult progress , Such as gating and attention mechanism , Especially slot based MANN, This is the main content of the new model proposed in Chapter 3 . Our main contributions are in four aspects . First , We MANN As a multi process multi viewpoint model to deal with complex problems , Such as sequence-to-sequence Mapping and multi view sequence learning ( Chapter four ). We further expand MANNs As a model of discrete sequence, the variability and consistency of session data are required ( The first 5 Chapter ). We also explain to memory last , We introduce a new class MANN, It follows the principle of storing programs , Different functions can be performed by switching the program of the controller .