Vanilla Neural Networks

A neural network is just a math function: y = FNN(x) .

FNN

has a nested form. Think of it as a stack of layers. A 3-layer neural network that returns a scalar value looks like this:

y = FNN(x) = f₃(f₂(f₁(x))) Each f

— f₁

, f₂

, … fₙ

— has the same form:

f(x) = g(Wx + b) W

(the weight matrix) and b (a bias vector) are the learned parameters, usually trained via gradient descent. g

is the activation function, and it can be chosen differently for each layer.