A neural Turing machine (NTM) is a recurrent neural network model of a Turing machine. The approach was published by Alex Graves et al. in 2014. [1] NTMs combine the fuzzy pattern matching capabilities of neural networks with the algorithmic power of programmable computers.
An NTM has a neural network controller coupled to external memory resources, which it interacts with through attentional mechanisms. The memory interactions are differentiable end-to-end, making it possible to optimize them using gradient descent. [2] An NTM with a long short-term memory (LSTM) network controller can infer simple algorithms such as copying, sorting, and associative recall from examples alone. [1]
The authors of the original NTM paper did not publish their source code. [1] The first stable open-source implementation was published in 2018 at the 27th International Conference on Artificial Neural Networks, receiving a best-paper award. [3] [4] [5] Other open source implementations of NTMs exist but as of 2018 they are not sufficiently stable for production use. [6] [7] [8] [9] [10] [11] [12] The developers either report that the gradients of their implementation sometimes become NaN during training for unknown reasons and cause training to fail; [10] [11] [9] report slow convergence; [7] [6] or do not report the speed of learning of their implementation. [12] [8]
Differentiable neural computers are an outgrowth of Neural Turing machines, with attention mechanisms that control where the memory is active, and improve performance. [13]
A neural Turing machine (NTM) is a recurrent neural network model of a Turing machine. The approach was published by Alex Graves et al. in 2014. [1] NTMs combine the fuzzy pattern matching capabilities of neural networks with the algorithmic power of programmable computers.
An NTM has a neural network controller coupled to external memory resources, which it interacts with through attentional mechanisms. The memory interactions are differentiable end-to-end, making it possible to optimize them using gradient descent. [2] An NTM with a long short-term memory (LSTM) network controller can infer simple algorithms such as copying, sorting, and associative recall from examples alone. [1]
The authors of the original NTM paper did not publish their source code. [1] The first stable open-source implementation was published in 2018 at the 27th International Conference on Artificial Neural Networks, receiving a best-paper award. [3] [4] [5] Other open source implementations of NTMs exist but as of 2018 they are not sufficiently stable for production use. [6] [7] [8] [9] [10] [11] [12] The developers either report that the gradients of their implementation sometimes become NaN during training for unknown reasons and cause training to fail; [10] [11] [9] report slow convergence; [7] [6] or do not report the speed of learning of their implementation. [12] [8]
Differentiable neural computers are an outgrowth of Neural Turing machines, with attention mechanisms that control where the memory is active, and improve performance. [13]