Some time ago during a training session a colleague asked me what a surface plot of a two input neural net would look like. That is, if you have two inputs x_1 and x_2 and plot the output y as a surface what do you get? We thought it might look like a set of waterfalls stacked on each other.
Since neural nets are often considered a black box solution I thought it would be interesting to check this. If you can picture the output for simple case it makes things less mysterious in the more general case.
Let’s look at a neural net with a single hidden layer of two nodes and a sigmoid activation function. In other words something that looks like this. (If you need to catch up on the theory please see this brilliant book by David MacKay that’s free to view online)
Drawn using the lovely draw.io
Output for a single node
We can break the task down a little by doing a surface plot of a single node, say node 1. We are using the sigmoid activation function so what we are plotting is the application of the sigmoid function to what is essentially a function describing a plane.
The activation (i.e. the plane) is
Applying the sigmoid function we get
Another good tool I use now and then when testing out some maths is to is wolframe alpha. I’m going to pretend that weights w4 and w3 have been fitted as 0.1 and 0.2 and so the equation I input is
What we see is sort of waterfall shape angled in the direction of the plane. This is what we’d expect. As the alpha values get higher and lower the output of the sigmoid function tends to one and zero respectively.
Output for multiple nodes
Lets now look at the output for the whole network. The outputs from nodes 1 and 2 feed into node 3 creating a new activation
This is just a weighted sum of the outputs of the nodes 1 and 2 and so should look as predicted like one waterfall stacked on another (sort of). We can check this out in wolfram alpha by submitting the following (picking say w1=0.8, w2=0.7, w5=0.3, w6=-0.2)
Adding the final sigmoid function to the activation of node 3 doesn’t change things much. It just puts things back on the scale from zero to one.
Exploring changes in the weights