Kremer (1995):

This paper generalizes the result by Minsky (1967) by showing that TLU-based Elman (1990) nets may actually represent finite automata using $\Theta(\vert Q\vert\vert\Sigma\vert)$ states,^5.4 but the construction by Kremer (1995) allows for input and output symbols to be represented by arbitrary patterns of `0' and `1' activations (instead of being represented by exclusive or one-hot patterns having precisely one `1'). This has the effect of reducing the number of learnable parameters in the input-state (${\bf W}^{xu}$) and the state-output (${\bf W}^{yx})$ weight matrices (see section 3.2.2).