This is the
talk page for discussing improvements to the
Recurrent neural network article. This is not a forum for general discussion of the article's subject. |
Article policies
|
Find sources: Google ( books · news · scholar · free images · WP refs) · FENS · JSTOR · TWL |
![]() | This article is rated C-class on Wikipedia's
content assessment scale. It is of interest to the following WikiProjects: | ||||||||||||||||||||||||||||||||||||||||||||||||||
|
The orphaned article "Bidirectional associative memory" (BAM) references this article, claiming that BAM is a kind of RNN. If this is correct, then this article should mention BAM and reference the BAM article. Likewise if it is appropriate to mention "Content-addressable memory" in this context, this should be done (however the article about that is biased towards computer hardware technology and not machine learning). 195.60.183.2 ( talk) 16:06, 17 July 2008 (UTC)
I think the article is in much better shape now than it was a couple of months ago, although it still needs to be polished. But I guess one could take out this template now:
![]() | This article needs attention from an expert in Computer science. Please add a reason or a talk parameter to this template to explain the issue with the article.(November 2008) |
Epsiloner ( talk) 15:41, 7 December 2010 (UTC)
This whole article needs a major rewrite. The word "recurrent" means that part or the whole output is used as input. The very first sentence "...where connections between nodes form a directed or undirected graph along a temporal sequence" is inaccurate (see quote below). A much better statement is "where the connection graph has cycles.". By the way, how can a neural network have "undirected" edges? Every node is a computation in one direction. And it just gets worse from that - there are *many* false statements in this article derived from this basic misunderstanding.
In the neural network literature, neural networks with one or more feedback loops are referred to as recurrent networks.
— Simon Haykin, "Neural networks: a comprehensive foundation", (p. 686)
Carlosayam ( talk) 00:45, 18 August 2022 (UTC)
"RNN can use their internal memory to process arbitrary sequences of inputs."
Some types can, but the typical RNN has nodes with binary threshold outputs, which makes it a finite state machine. This article needs clarification of what types are turing-complete. — Preceding unsigned comment added by Mister Mormon ( talk • contribs) 13:22, 18 December 2010 (UTC)
Yeah, thanks. It's super-Turing complete. Seriously, are all published RNNs either finite or uncomputable? Where's the middle ground with rational/integer weights and no thresholds in literature? I would be surprised if there were none to be found, since sub-symbolic AI has been in use for 30 years. Mister Mormon ( talk) 17:58, 28 December 2010 (UTC)
Hey, this paper on a Turing-complete net could be helpful: http://www.math.rutgers.edu/~sontag/FTP_DIR/aml-turing.pdf Mister Mormon ( talk) 02:29, 10 September 2011 (UTC)
about the picture, it seems to me that there supposed to be multi connections from the context layer forward to the hidden layer and not just one to one. although the save state connections from the hidden layer to the context are indeed one to one. [1] [2]
References
{{
cite journal}}
: Cite journal requires |journal=
(
help)
Indeed, the network in the image is not an Elman network, for the reason stated above. The image should be fixed, removed, or at least relabelled.
2604:3D08:2486:5D00:88FA:2817:2657:E641 ( talk) 21:14, 6 May 2022 (UTC)
QUOTE:In particular, RNNs cannot be easily trained for large numbers of neuron units nor for large numbers of inputs units. Successful training has been mostly in time series problems with few inputs.
Current(2013) state of art in speech recognition technique do use RNN. And speech require a lot of input. Check this: SPEECH RECOGNITION WITH DEEP RECURRENT NEURAL NETWORKS Alex Graves, Abdel-rahman Mohamed and Geoffrey Hinton. 50.100.193.20 ( talk) 11:42, 5 August 2013 (UTC)
Can somebody add this if it is of note?
Thanks! -- Potguru ( talk) 19:01, 4 March 2016 (UTC)
Hi, the Architectures sections currently contains 18 subsections. I think the readability of the article would be improved if there is some kind of structure or order in them. I do not (yet) have the knowledge to rearrange them. Maybe someone else is willing to do this. VeniVidiVicipedia ( talk) 11:06, 2 January 2017 (UTC)
This article is difficult for non-experts. It relies heavily on a lot of terminology and jargon without defining or explaining what much of it means. And so this article is only useful if you already know a lot about these concepts. 108.29.37.131 ( talk) 19:03, 25 July 2018 (UTC)
The second and third paragraphs of this article talk about "finite impulse response", and are introduced with
The term "recurrent neural network" is used indiscriminately to refer to two broad classes of networks with a similar general structure, where one is finite impulse and the other is infinite impulse.
As one of the very first researchers in recurrent neural networks I find these two paragraphs very misleading. The whole point of a recurrent neural network is that it's recurrent, that is some of the output is fed back to the input. It is possible to do this in a restricted way that results in finite impulse response (e.g. time delay neural networks) but it is very misleading to give prominence to this uninteresting restricted subset.
Supporting evidence for the above can be seen in the whole of the rest of the article, in every case there is infinite impulse response (even in the case of Hopfield networks, and it's arguable whether they should be included here).
Also, the History section is very sparse (one brief mention of Hopfield then lots on TDNN). I believe that I was the first to write a PhD thesis on recurrent networks ("Dynamic Error Propagation Networks". A. J. Robinson. PhD thesis, Cambridge University Engineering Department, February 1989.) so if there is interest I can help flesh this bit out (in order not to violate the conflict of interest policy someone else should do the edits).
DrTonyR ( talk) 04:59, 8 December 2018 (UTC)
Just fixed it. Atcold ( talk) — Preceding undated comment added 20:20, 15 November 2021 (UTC)
"A finite impulse recurrent network is a directed acyclic graph" In general, a finite impulse recurrent network may contain cycles, it's just that these cycles are not directed. Only feedforward networks are truly acyclic. — Preceding unsigned comment added by 142.184.185.164 ( talk) 03:15, 24 December 2019 (UTC)
There are two references to RTRL and both seem to have issues. The first is to one of my publications but it leads to an empty Google books page. https://www.academia.edu/30351853/The_utility_driven_dynamic_error_propagation_network would be better. The second has a 2013 date, but I think it should be R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270–280, June 1989. ISSN 0899-7667. doi: 10.1162/neco.1989.1.2.270.
Things seem to have changed since I last understood references here. I can't fix with my current knowledge, but I'll learn if needed. DrTonyR ( talk) 08:06, 21 June 2020 (UTC)
This is the
talk page for discussing improvements to the
Recurrent neural network article. This is not a forum for general discussion of the article's subject. |
Article policies
|
Find sources: Google ( books · news · scholar · free images · WP refs) · FENS · JSTOR · TWL |
![]() | This article is rated C-class on Wikipedia's
content assessment scale. It is of interest to the following WikiProjects: | ||||||||||||||||||||||||||||||||||||||||||||||||||
|
The orphaned article "Bidirectional associative memory" (BAM) references this article, claiming that BAM is a kind of RNN. If this is correct, then this article should mention BAM and reference the BAM article. Likewise if it is appropriate to mention "Content-addressable memory" in this context, this should be done (however the article about that is biased towards computer hardware technology and not machine learning). 195.60.183.2 ( talk) 16:06, 17 July 2008 (UTC)
I think the article is in much better shape now than it was a couple of months ago, although it still needs to be polished. But I guess one could take out this template now:
![]() | This article needs attention from an expert in Computer science. Please add a reason or a talk parameter to this template to explain the issue with the article.(November 2008) |
Epsiloner ( talk) 15:41, 7 December 2010 (UTC)
This whole article needs a major rewrite. The word "recurrent" means that part or the whole output is used as input. The very first sentence "...where connections between nodes form a directed or undirected graph along a temporal sequence" is inaccurate (see quote below). A much better statement is "where the connection graph has cycles.". By the way, how can a neural network have "undirected" edges? Every node is a computation in one direction. And it just gets worse from that - there are *many* false statements in this article derived from this basic misunderstanding.
In the neural network literature, neural networks with one or more feedback loops are referred to as recurrent networks.
— Simon Haykin, "Neural networks: a comprehensive foundation", (p. 686)
Carlosayam ( talk) 00:45, 18 August 2022 (UTC)
"RNN can use their internal memory to process arbitrary sequences of inputs."
Some types can, but the typical RNN has nodes with binary threshold outputs, which makes it a finite state machine. This article needs clarification of what types are turing-complete. — Preceding unsigned comment added by Mister Mormon ( talk • contribs) 13:22, 18 December 2010 (UTC)
Yeah, thanks. It's super-Turing complete. Seriously, are all published RNNs either finite or uncomputable? Where's the middle ground with rational/integer weights and no thresholds in literature? I would be surprised if there were none to be found, since sub-symbolic AI has been in use for 30 years. Mister Mormon ( talk) 17:58, 28 December 2010 (UTC)
Hey, this paper on a Turing-complete net could be helpful: http://www.math.rutgers.edu/~sontag/FTP_DIR/aml-turing.pdf Mister Mormon ( talk) 02:29, 10 September 2011 (UTC)
about the picture, it seems to me that there supposed to be multi connections from the context layer forward to the hidden layer and not just one to one. although the save state connections from the hidden layer to the context are indeed one to one. [1] [2]
References
{{
cite journal}}
: Cite journal requires |journal=
(
help)
Indeed, the network in the image is not an Elman network, for the reason stated above. The image should be fixed, removed, or at least relabelled.
2604:3D08:2486:5D00:88FA:2817:2657:E641 ( talk) 21:14, 6 May 2022 (UTC)
QUOTE:In particular, RNNs cannot be easily trained for large numbers of neuron units nor for large numbers of inputs units. Successful training has been mostly in time series problems with few inputs.
Current(2013) state of art in speech recognition technique do use RNN. And speech require a lot of input. Check this: SPEECH RECOGNITION WITH DEEP RECURRENT NEURAL NETWORKS Alex Graves, Abdel-rahman Mohamed and Geoffrey Hinton. 50.100.193.20 ( talk) 11:42, 5 August 2013 (UTC)
Can somebody add this if it is of note?
Thanks! -- Potguru ( talk) 19:01, 4 March 2016 (UTC)
Hi, the Architectures sections currently contains 18 subsections. I think the readability of the article would be improved if there is some kind of structure or order in them. I do not (yet) have the knowledge to rearrange them. Maybe someone else is willing to do this. VeniVidiVicipedia ( talk) 11:06, 2 January 2017 (UTC)
This article is difficult for non-experts. It relies heavily on a lot of terminology and jargon without defining or explaining what much of it means. And so this article is only useful if you already know a lot about these concepts. 108.29.37.131 ( talk) 19:03, 25 July 2018 (UTC)
The second and third paragraphs of this article talk about "finite impulse response", and are introduced with
The term "recurrent neural network" is used indiscriminately to refer to two broad classes of networks with a similar general structure, where one is finite impulse and the other is infinite impulse.
As one of the very first researchers in recurrent neural networks I find these two paragraphs very misleading. The whole point of a recurrent neural network is that it's recurrent, that is some of the output is fed back to the input. It is possible to do this in a restricted way that results in finite impulse response (e.g. time delay neural networks) but it is very misleading to give prominence to this uninteresting restricted subset.
Supporting evidence for the above can be seen in the whole of the rest of the article, in every case there is infinite impulse response (even in the case of Hopfield networks, and it's arguable whether they should be included here).
Also, the History section is very sparse (one brief mention of Hopfield then lots on TDNN). I believe that I was the first to write a PhD thesis on recurrent networks ("Dynamic Error Propagation Networks". A. J. Robinson. PhD thesis, Cambridge University Engineering Department, February 1989.) so if there is interest I can help flesh this bit out (in order not to violate the conflict of interest policy someone else should do the edits).
DrTonyR ( talk) 04:59, 8 December 2018 (UTC)
Just fixed it. Atcold ( talk) — Preceding undated comment added 20:20, 15 November 2021 (UTC)
"A finite impulse recurrent network is a directed acyclic graph" In general, a finite impulse recurrent network may contain cycles, it's just that these cycles are not directed. Only feedforward networks are truly acyclic. — Preceding unsigned comment added by 142.184.185.164 ( talk) 03:15, 24 December 2019 (UTC)
There are two references to RTRL and both seem to have issues. The first is to one of my publications but it leads to an empty Google books page. https://www.academia.edu/30351853/The_utility_driven_dynamic_error_propagation_network would be better. The second has a 2013 date, but I think it should be R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270–280, June 1989. ISSN 0899-7667. doi: 10.1162/neco.1989.1.2.270.
Things seem to have changed since I last understood references here. I can't fix with my current knowledge, but I'll learn if needed. DrTonyR ( talk) 08:06, 21 June 2020 (UTC)