In order to map x to linear models, we need to perform non-linear mapping on the inputs.

1. Choose a very generic φ, like an infinite-dimensional that is implicitly used by kernel machines based on the RBF kernel. (Not good for advanced problems)

2. Manually engineer φ.

3. Use deep learning to learn φ. y=f(x;θ, w) =φ(x;θ)Tw

Goal Function

${\displaystyle \mathbb {X} =\{[0,0]^{\top },[0,1]^{\top },[1,0]^{\top },[1,1]^{\top }\}}$

${\displaystyle f(\mathbb {X} )=[0,1,1,0]^{\top }}$

If we choose a linear model with ${\displaystyle \theta }$ and ${\displaystyle b}$, then the model is defined as ${\displaystyle f(x;w,b)=x^{\top }w+b}$, which will output ${\displaystyle w=0}$ and ${\displaystyle b={\frac {1}{2}}}$ when solved. This isn't very useful to have a function that outputs a constant value over all of the inputs.

If we add a hidden layer, such that ${\displaystyle h=f^{(1)}(x;W,c)}$ and ${\displaystyle y=f^{(2)}(h;w,b)}$ with the complete model being ${\displaystyle f(x;W,c,w,b)=f^{(2)}(f^{(1)}(x))}$

There must be a non-linear function to be able to turn our linear model into a non-linear. This is usually done by affine transformation followed by a fixed nonlinear activation function.

${\displaystyle h=g(W^{\top }x+c)}$, where W provides the weight of a linear transformation and c the biases.

The default recommended activation function is rectified linear unit, or ReLU, defined as ${\displaystyle g(z)=max\{0,z\}}$

The complete network can now be specified as ${\displaystyle f(x;W,c,w,b)=w^{\top }max\{0,W^{\top }x+c\}+b}$