this concerns “my own” lstm implementation in c/c++, i think i’ve had three (3) separate periods of identifying i/j transposition errors since i posted here, me no get sleepy,
but not try to bug fix my code today, asking about activation options.
i’ve been told “for audio, you can use NO activation function, just pass the output layer prediction” and tried it with and without clipping in the training loop. very slow progress in output if any. let’s ask a people.
i do try to keep track of what others try to express but so far i can’t tell if humanity has determined that audio in an lstm requires an ‘‘0.5 * (y * y)’’ shaping for backprop without tanh or not. i’m getting faster learning but a ways to go before my little lappy churns out anything click/burp i’m guessing. the prediction in training sound pretty good .. immediately similar, highs improve.. but as said, a ways to go for generative output (error is nll on abs(predict - target).
i’m wondering if i still have errors. i take my fixes back to my text lstm, train there since text is easy to observe, and find gradient clipping produces recognisable words, even during initial training. gradient normalisation and ADAM can train down to 0.0n error but just be “english shaped nonsense” (hot one chars, not tokenised text). i haven’t added regularisation because i’m guessing that will mess it up also, being another ‘generaliser’.