Index

accepting state ,

activation function ,

activation function

differentiable

activation function

,

activation function

properties

activation function

,

radial basis functions

activation function

rational

threshold|seethreshold linear unit

activation

function

derivative

Adali@Adali, T.

Alon, N. , , , , ,

Alopex

alphabet ,

Alquezar@Alquézar, R. , ,

asynchronous DTRNN

asynchronous transduction

attractor

attractor

limit cycle

Aussem, A.

automata

deterministic finite-state , , , , , , , , , , , , ,

finite-state , , , ,

neural nets and finite-state automata

pushdown

backpropagation , ,

backpropagation

focused

through time , , , ,

Baltersee, J.

batch learning , , , ,

Bengio, Y. ,

Bengio. Y.

bias

as a learnable parameter ,

notation

Blair, A. , , , , ,

blank symbol

in Turing Machines ,

bounds to number of units ,

Box, G.E.

BPS

BPTT|seebackpropagation through time

Bradley, M.J.

Bridle, J.S.

Bulsari, A.B. ,

Carrasco, R.C. , , , , , , , , , ,

Casey, M. ,

Cauwenberghs, G. ,

Chalmers, D.J.

Chambers, J.

channel equalization

Chen, W.-Y. ,

Cheng, Y.

Chiu, C.-C.

Chomsky's hierarchy ,

Chomsky, N.

Chovan, T.

Chrisman, L.

Cid-Sueiro, J. ,

Cleeremans, A. , , , , ,

clock

external

Clouse, D.S. ,

clustering

hierarchical ,

of DTRNN state vectors , , , ,

compression of signals

compressor

in RAAM

computability of natural functions

Connor, J.T.

construction of FSM in DTRNN , , , , , ,

context-free

grammar , , , , ,

language

context-sensitive grammar , ,

continuous-time recurrent neural networks

control

countable set

cycles in neural networks , , ,

Das, R. ,

Das, S. , , , , ,

decision function

decoder

in RAAM ,

in RAAM networks|seerecursive auto-associative memory

definite-memory machines

denumerable set

derivatives

of error , , ,

deterministic finite-state automata , , , , , , , , , , , , , , ,

DFA|seedeterministic finite-state automata

discrete-time recurrent neural network , , , , , , , , , , , , , , , , , , , , , , , , , , ,

discrete-time recurrent neural network

first-order

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

, ,

second-order

discrete-time recurrent neural network

, ,

discrete-time recurrent neural network

, ,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

, ,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

, ,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

, ,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

,

discrete-time recurrent neural network

Turing computability

discrete-time recurrent neural network

,

discretization

of DTRNN state space

of neuron outputs

Draye, J.P.

Dreider, J.F. ,

DTRNN|seediscrete-time recurrent neural network

dynamical system

dynamics

atractor

continuous-state

next state

of a DTRNN , , , , , ,

Dzielinski, A.

EKF

Elman net , , , ,

Elman, J.L. , , , , , , , , , , , ,

empty string

encoder

in RAAM , ,

encoding

of FSM in DTRNN , , ,

of input symbols

of Turing machines in DTRNN

equivalence

of BPTT and RTRL

of DFA and Moore machines

of FSM and DTRNN , ,

of Moore and Mealy machines

of sequence classification and transduction

of TLU and McCulloch-Pitts units

error function ,

error function

differentiable ,

error function

gradient of

error function

, ,

error function

in batch learning

error function

, ,

in online learning

in pattern learning

error function

,

error function

local minima ,

error function

minima

minima

multiple

error function

quadratic

special

excitation threshold

excitatory connection ,

exclusive encoding

of symbols ,

exclusive-or function , , ,

expression

regular ,

regular

equivalence with DTRNN

equivalence with finite-state machines

temporal propositional

extended Kalman filter

extraction

of FSM from DTRNN , , ,

of FSM from DTRNN

through partition of state space , ,

of FSM from DTRNN

trivial

of FSM from DTRNN

using clustering

of FSM from DTRNN

,

of FSM from DTRNN

,

of FSM from DTRNN

using self-organizing maps

of FSM from DTRNN

,

Fahlman, S.E. , , , ,

Fallside, F. , , , , ,

fan-in ,

fan-out

Fanelli, R. , ,

feedback

in DTRNN ,

in DTRNN

local

in DTRNN

, ,

feedforward neural net

as next-state function

as output function

feedforward neural net

,

feedforward neural net

, ,

feedforward neural net

in BPTT training

feedforward neural net

in RAAM

layered

lower-triangular ,

training

feedforward neural net

two-layer

feedforward neural net

,

feedforward neural net

, ,

Feldkamp, L.A

FFNN|seefeedforward neural net

Figueiras-Vidal, A.R.

filtering of discrete-time signals

final state

of a Turing machine

final state

,

finite-memory machines

finite-state automata

deterministic

finite-state automata

,

finite-state machine

finite-state machines

finite-state machines

and neural nets

finite-state machines

approximate

finite-state machines

as transducers

classes

finite-state machines

compatible with learning set

finite-state machines

definite-memory machines

deterministic

DTRNN behaving as , , ,

finite-state machines

,

earliest DTRNN as

emulation by DTRNN

finite-state machines

, , , ,

finite-state machines

,

finite-state machines

encoding in DTRNN

finite-state machines

,

finite-state machines

,

encoding in threshold DTRNN , , ,

equivalence to DTRNN ,

extraction

extraction from DTRNN , ,

finite-state machines

finite-memory machines

finite-state machines

inference

finite-state machines

, , ,

finite-state machines

learning by DTRNN

finite-state machines

, ,

learning in sigmoid DTRNN

next-state function

probabilistic

pushdown automata as

states

as clusters in state space , , , , ,

transition

finite-state machines

,

with stack

finite-state

automata

automata

deterministic , , , , , , , , , , , , ,

behavior

behavior

of DTRNN

of DTRNN , , , , , , , , , , , , ,

computation , ,

fixed points ,

Forcada, M.L. , , , , , , ,

fractal dimension

Frasconi, P. ,

FSM|seefinite-state machines

functions

computable by TLU

recursively computable

generalization , ,

Gershenfeld, N.A.

Giles, C.L. , , , , , , , , , , , , , , , , , , ,

Gori, M. ,

Goudreau, M.

Goudreau, M.W. , , , ,

gradient

descent , , , , ,

learning algorithm , , , ,

Gradient

,

gradient

, , , , ,

pseudo-gradient learning

vanishing

grammar , , ,

grammar

as generator

Chomsky's hierarchy ,

context-free , , , ,

context-sensitive ,

inference , , , , , ,

language generated by

regular ,

rules in DTRNN ,

Tomita's

unrestricted ,

grammatical inference , , , , ,

grammatical inference

using DTRNN

grammatical inference

,

grammatical inference

, , , , , ,

Haykin, S. , , , , ,

hidden

layer , , , ,

state , ,

state

neural architectures without ,

unit , , , ,

unit

activation patterns

units

hierarchy

Chomsky's ,

Hopfield, J.J.

Horne, B.G. , , ,

Hush, D.R. , , ,

Ifeachor, E.C.

implementation

of FSM in DTRNN , , , , , ,

inductive bias ,

inhibitory connection ,

initial state

as a learnable parameter

as a learnable parameter

of a DTRNN ,

of a DTRNN

derivatives

of a DTRNN

learning

of a FSM

of a pushdown automaton

of a Turing machine

input

alphabet , , , ,

layer

neurons , ,

sequence , ,

sequence

length

string

symbol

symbol

representation ,

symbols

representation

to a DTRNN , , , ,

to a TLU , , ,

window

window

in a NARX ,

in a TDNN ,

instability

internal representation , ,

interpretation

of DTRNN output

as symbols

of DTRNN outputs

of DTRNN outputs

as probabilities , ,

of DTRNN outputs

as symbols

Janacek, G.

Jordan, M.I. ,

Kaiser, J.F.

Kalman filter

extended

Kalman, B.L.

Kechriotis, G.

Kleene's theorem

Kleene, S.C. , , , , , , , ,

Kohavi, Z.

Kohonen, T. , , ,

Kolen, J.F. , ,

Kremer, S.C , , , , , ,

Kremer, S.C. , , ,

Kuhn, G.M. , , ,

Kwasny, S.C.

language

recognizer

accepted by DTRNN

accepted by Turing machines

acceptor , ,

acceptor

neural ,

Turing machine as

concatenation

context-free

defined by grammar

finite-state

generated by a grammar

generator

generator

probabilistic

learning by DTRNN

natural

recognition by DTRNN

recognizer , ,

regular , , , ,

regular

acceptor

recognition by DTRNN

transducer

Lawrence, S.C.

layers

hidden , , , ,

in BPTT ,

in feedforward neural net ,

output , ,

LBA

learnable parameters , , , , , , , , , , ,

learnable parameters

updating

batch

learnable parameters

updating

gradient

learnable parameters

updating

in BPTT

in perturbative methods

in RTRL

learnable parameters

updating

online

learnable parameters

updating

,

learnable parameters

updating

,

pattern ,

learnable parameters

updating

random

learning algorithm , ,

learning algorithm

backpropagation

for DTRNN ,

generalization test

learning algorithm

gradient-based

learning algorithm

, , , , ,

learning algorithm

,

learning algorithm

,

gradient-descent

inductive bias

learning algorithm

long-term dependencies

learning algorithm

non-gradient-based

learning algorithm

,

learning algorithm

, ,

learning algorithm

pseudo-gradient-based

learning algorithm

recurrent cascade correlation

learning set , , , , , , , , , , ,

learning set

noisy

of trees

learning set

partition

Li, C.J.

Li, L.

Lin, T.

linearly-bounded automata

local minima , ,

local-feedback DTRNN , ,

logical functions , ,

logical functions

computability

logistic function , ,

long-term dependencies , , , , ,

lower-triangular feedforward neural net ,

Manolios, P. , ,

Mars, P.

Martin, R.D.

McClelland, J.L.

McCulloch, W.S. , , , , , ,

McCulloch-Pitts net

Mealy machines , , , , , , , ,

Mealy machines

as sequence procesors

binary

neural

Mealy machines

,

Mealy machines

,

Mealy machines

,

Mealy machines

,

minimization

of error function

of FSM ,

Minsky, M.L. , , , , , , , , ,

Mitra, S.K.

Moore machines , , , ,

Moore machines

neural

Moore machines

,

Moore machines

, , , , , ,

Moore machines

,

Mozer, M.C. , , , , , ,

multilayer perceptron ,

Narendra, K.S. ,

NARX (nonlinear auto-regressive with exogenous inputs)

natural language

natural numbers

natural numbers

functions of ,

Neco@Ñeco, R.P. ,

Nerrand, O.

neural

Mealy machines

Moore machine

state machine , , , , , , , , , , , ,

neurocontrol

neuron field ,

next move function

next-state function , , , , , , , , , , , , , , ,

next-state function

of a TDNN

node

of a tree , ,

non-gradient-based

learning algorithms , ,

nonterminal symbols ,

NSM|seeneural state machine

observability of state ,

Omlin, C.W. , ,

one-hot encoding

of symbols , ,

online learning , ,

online learning

using RTRL

online learning

,

Oppenheim, A.V.

Ortiz-Fuentes, J.D.

output

alphabet , ,

desired , ,

function , , , , , , , , , , , , , , , , , , ,

of a DTRNN

of a TLU , , ,

of DTRNN

as projection of state vector , , ,

of sigmoid units

sequence , , ,

sequence

length

space ,

space

of a DTRNN

symbols

representation

unit , ,

window

window

in a NARX ,

P/Poly

computational class

Parberry, I.

Parisi, R.

parsing

parsing

shift-reduce

Parthasarathy, K. ,

partition

of learning sets

of state space , , , , , , ,

pattern learning , ,

PDA ,

Pearlmutter, B.

perceptron

multilayer

two-layer

Perrin, D.

Pineda, F.J.

Pitts, W.H. , , , , , ,

Pollack, J.B. , , , , , , , , , , , , , , , , , , , , ,

prediction

by DTRNN

of a sequence ,

of next symbol

of next symbol

using DTRNN , ,

time-series ,

time-series

using DTRNN

predictive coding ,

probabilistic

finite-state machine

language generator

probabilities

in DTRNN outputs , , ,

probability distribution , , ,

processing

element

of natural language

of sequences , , ,

of sequences

adaptive

classification

of sequences

using DTRNN , , ,

of sequences

,

of sequences

, ,

of strings ,

sequential

synchronous

production

of a grammar ,

pseudo-gradient

learning algorithm

pushdown automaton , , , ,

pushdown automaton

as Turing machine simulator

Puskorius, G.V

Qian, N.

RAAM|seerecursive auto-associative memory

radial basis functions

rational activation function , ,

real-time recurrent learning , , , , , ,

real-time recurrent learning

relation to extended Kalman filter

recognition

of languages

by DTRNN

of sequences ,

of speech

recognizer

dynamical , ,

finite-state ,

for context-free languages

language , ,

neural

reconstructor

in RAAM

recurrent cascade correlation ,

recurrent neural network

discrete-time ,

recurrent neural network

, , , ,

recurrent neural network

,

recurrent neural network

,

recurrent neural network

,

recurrent neural network

,

recurrent neural network

, ,

recurrent neural network

,

recurrent neural network

,

recurrent neural network

,

recurrent neural network

, , ,

recurrent neural network

, , ,

recurrent neural network

,

recurrent neural network

, , , , ,

recurrent neural network

,

recurrent neural network

, ,

recurrent neural network

,

recurrent neural network

, , ,

recurrent neural network

,

recurrent neural network

, , ,

recursive auto-associative memory , , ,

recursive auto-associative memory

labeling

recursive hetero-associative memory

recursivity

in grammars

region

of output space

in DTRNN

of output space

,

in RAAM

of state space

in DTRNN

of state space

,

in RAAM

regular

events

expression

grammar ,

language , , ,

language

acceptor

recognition by DTRNN

representation

learned by DTRNN , , ,

of FSM in DTRNN

of FSM

in DTRNN

of inputs in DTRNN , , ,

of outputs in DTRNN

of parse trees in RAAM

of sequences in RAAM ,

of terminals in RAAM

rewrite rules

in a grammar

recursive

Robinson, A.J. , , , , ,

Robinson-Fallside network

augmented

Rosenberg, C.R.

RTRL|seereal-time recurrent learning

rule

extraction from DTRNN

representation in DTRNN

Rumelhart, D.E. , ,

Sajda, J. , ,

sampling

discrete-time ,

Sanfeliu, A. , ,

Saxen@Saxén, H. ,

Schafer, R.W.

second-order DTRNN , , , , , , , , , , , , , , , ,

Sejnowski, T.J. ,

self-organizing feature maps ,

sequence

classification

continuation

generation ,

input , ,

output , , ,

prediction , ,

prediction

for speech coding

processing , ,

processing

adaptive

as language recognition

long-term dependencies ,

sequential

synchronous

using DTRNN , , , , , , ,

processor

discrete-time

state-based

recognition ,

representation

in RAAM

transduction

transduction

synchronous ,

sequential

processing

processing

in Mealy and Moore machines

transduction

by DTRNN

Shanblatt, M.A.

Siegelmann, H.T. , ,

sigmoid

function

function

logistic , ,

units

DTRNN using

signal

compression

discrete-time ,

filtering

processing

Sima@Šíma, J. ,

single-layer

DTRNN ,

feedforward neural net

feedforward neural net

as output function

feedforward neural net

,

Siu, K.Y.

Sluijter, R.J. ,

SOFM ,

Sontag, E.D. ,

space complexity of learning

speech

coding

through sequence prediction

using DTRNN

modelling of coarticulatory features

recognition

Sperduti, A. ,

stability of DTRNN as language recognizers

stability

of DTRNN as language recognizers

stack

empty

external , , , , ,

in pushdown automata ,

simulating Turing machines

unary

Starita, A.

start symbol of a grammar

start symbol

of a grammar ,

state space

of DTRNN , ,

of DTRNN

clustering

state space

of DTRNN

, ,

state space

of DTRNN

,

state space

of DTRNN

,

of DTRNN

,

partition

state space

of DTRNN

, ,

of DTRNN

,

of DTRNN

,

state space

of DTRNN

,

state space

of DTRNN

regions

topological mapping

of RAAM

state space

of RAAM

regions

state space

partition

to extract FSM ,

state

accepting

accepting

of a pushdown automaton

final

of a Turing machine ,

initial

of a DTRNN ,

of a DTRNN

learning ,

of a DTRNN

,

of a pushdown automaton

of a Turing machine

initial|seealsoinitial state

of a FSM

observable ,

of a FSM

of a pushdown automaton

of a Turing machine

of DFA

representation in DTRNN

transition ,

transition

in a FSM , , , ,

in a Turing machine

unit

units , , , , , ,

units

hidden , ,

in DTRNN ,

vector , , , ,

vector

clustering , ,

in a NARX

state-based sequence processor

step function

string ,

string

acceptance

by a DFA

by a DTRNN

by a pushdown automaton

by a Turing machine

continuation

by DTRNN ,

empty

generation

by a grammar

by DTRNN

input ,

output

processing

by DTRNN ,

by FSM

recognition

by automata

rejection

by DFA

by DTRNN

transduction

translation

translation

by DTRNN

valid

as defined by grammar

super-Turing

computation

by DTRNN

subclass P/Poly

Swift, L.

symbol

blank

in Turing machine ,

encoding

one-hot, local, or exclusive

end-of-string ,

in stack of pushdown automaton

in tape of Turing machine

input symbol ,

input symbol

representation in DTRNN , ,

next symbol

prediction

next symbol

,

next symbol

,

nonterminal ,

one-hot encoding ,

output , , ,

output

by DTRNN ,

representation in DTRNN

stack ,

start symbol

of a grammar

start symbol

, ,

string ,

terminal , ,

useless

variable

synapse ,

synapse

excitatory ,

inhibitory ,

synchronous

processing

sequence transduction

transduction ,

transduction

by DTRNN

system identification

tape

alphabet, in a Turing machine

in linearly bounded automata

of a nonuniform Turing machine

of a Turing machine , , , ,

target , , ,

target`don't care'' targets

TDNN

teacher forcing

temporal propositional expression ,

terminal symbol , , ,

test set

threshold linear unit , , , , , , , , , , ,

threshold unit

time complexity

of learning ,

time-delay neural network

time-series prediction ,

time-series prediction

using DTRNN

Tino@Tino, P. , ,

TLU|seethreshold linear unit

TM|seeTuring machine

Tomita, M. ,

TPE|seetemporal propositional expression

training algorithm|seelearning algorithm

transducer

finite-state

of strings

transduction

of sequences

of sequences

asynchronous

of sequences

sequential

of sequences

synchronous ,

using DTRNN

of trees

transition

function|seenext-state function

in a FSM , , , , , ,

in a pushdown automaton ,

in a Turing machine

translation|seetransduction

tree

learning set

node , ,

storing in RAAM , ,

transduction

Turing machine , , , , , , , , , , , , ,

Turing machine

deterministic

Turing machine

nondeterministic

Turing machine

nonuniform ,

universal ,

Turing, A.M.

two-layer feedforward neural net , , ,

type 0 grammar

type 1 grammar

type 2 grammar

type 3 grammar

unfolding

in BPTT

unit

demon

threshold

threshold linear , ,

universal Turing machine ,

Unnikrishnan, K.P.

unrestricted grammar ,

updating of learnable parameters

batch

in BPTT

in perturbative methods

in RTRL

online ,

pattern ,

random

using gradient

valence

of a tree

vanishing gradient

variable

in a grammar , , ,

vector

input vector

input vector

to a DTRNN

to a TLU

of weights

space

of inputs

of outputs

state vector , ,

state vector

clustering

state vector

,

state vector

, ,

in a NARX

Venugopal, K.P.

Waibel, A.

Wang, J.

Watrous, R.L. , , ,

Weigend, A.S.

weights

as learnable parameters ,

derivatives of error with respect to

derivatives of state with respect to

equivalence in BPTT

feedback

in DTRNN

in TDNN

organized in blocks

notation

perturbation

perturbation

learning algorithm ,

space

vector

Werbos, P.J. , , ,

Williams, R.J. , , , , , , ,

window

of inputs

of inputs

in a NARX , ,

of inputs

in a TDNN

of outputs

of outputs

in a NARX ,

Wu, G.

Wu, L.

Zbikowski, R.

Zeng, Z. , , ,

Zipser, D. , , , , , ,

[author]

2