Neural Representation of AND, OR, NOT, XOR and XNOR Logic Gates Perceptron Algorithm by Obumneme Stanley Dukor

Neural networks are now widespread and are used in practical tasks such as speech recognition, automatic text translation, image processing, analysis of complex processes and so on. Another very useful approach to solve the XOR problem would be engineering a third dimension. The first and second features will remain the same, we will just engineer the third feature. Just by looking at this graph, we can say that the data was almost perfectly classified. After printing our result we can see that we get a value that is close to zero and the original value is also zero.

Hidden layer gradient

There is no intuitive way to distinguish or separate the green and the blue points from each other such that they can be classified into respective classes. Naturally, due to rising limitations, the use of linear classifiers started to decline in the 1970s and more research time was being devoted to solving non-linear problems. It is evident that the XOR problem CAN NOT be solved linearly. This is the reason why this XOR problem has been a topic of interest among researchers for a long time. Using the output values between this range of 0 and 1, we can determine whether the input \(x\) belongs to Class 1 or Class 0.

Brief introduction of Deep Learning

The loss function we used in our MLP model is the Mean Squared loss function. Though this is a very popular loss function, it makes some assumptions on the data (like it being gaussian) and isn’t always convex when it comes to a classification problem. It was used here to make it easier to understand how a perceptron works, but for classification tasks, there are better alternatives, like binary cross-entropy loss. Adding more layers or nodes gives increasingly complex decision boundaries. But this could also lead to something called overfitting — where a model achieves very high accuracies on the training data, but fails to generalize. As discussed, it’s applied to the output of each hidden layer node and the output node.

Building and training XOR neural network

1. Then the softmax function is applied.1 The softmax function makes sure that the output of every single neuron is in \([0, 1]\) and the sum of all outputs is exactly \(1\).
2. It abruptely falls towards a small value and over epochs it slowly decreases.
3. The table on the right below displays the output of the 4 inputs taken as the input.
4. To bring everything together, we create a simple Perceptron class with the functions we just discussed.

If we represent the problem at hand in a more suitable way, many difficult scenarios become easy to solve as we saw in the case of the XOR problem. Observe how the green points are below the plane and the red points are above the plane. This plane is nothing but the XOR operator’s xor neural network decision boundary. This provides formal proof of the fact that the XOR operators cannot be solved linearly. To better visualize the above classification, let’s see the graph below. For better understanding, let us assume that all our input values are in the 2-dimensional domain.

We know that a datapoint’s evaluation is expressed by the relation wX + b . Similarly, for the (1,0) case, the value of W0 will be -3 and that of W1 can be +2. Remember you can take any values of the weights W0, W1, and W2 as long as the inequality is preserved. The XOR problem can be solved using just two neurons, according to a statement made on January 18th, 2017.

A total of 6 weights from the input layer to the 2nd layer and a total of 3 weights from the 2nd layer to the output layer. This process is said to be continued until the actual output is gained by the neural network. A neural network is a network structure, by the presence of computing units(neurons) the neural network has gained the ability to compute the function. The neurons are connected with the help of edges, and it is said to have an assigned activation function and also contains the adjustable parameters.

Very often when training neural networks, we can get to the local minimum of the function without finding an adjacent minimum with the best values. Also, gradient descent can be very slow and makes too many iterations if we are close to the local minimum. Multi-layer feedforward neural networks, also known as deep neural networks, are artificial neural networks that have more than one hidden layer. These networks can learn complex patterns in data by using multiple layers of neurons. If we change weights on the next step of gradient descent methods, we will minimize the difference between output on the neurons and training set of the vector. As a result, we will have the necessary values of weights and biases in the neural network and output values on the neurons will be the same as the training vector.

In the above figure, we can see that above the linear separable line the red triangle is overlapping with the pink dot and linear separability of data points is not possible using the XOR logic. So now let us understand how to solve the XOR problem with neural networks. To solve the XOR problem using neural networks, one can use either Multi-Layer Perceptrons or a https://forexhero.info/ neural network that consists of an input layer, a hidden layer, and an output layer. As the neural network processes data through forward propagation, the weights of each layer are adjusted accordingly and the XOR logic is executed. Traditional neural networks also use a single layer of neurons which makes it difficult for them to learn complex patterns in data.

We are running 1000 iterations to fit the model to given data. Batch size is 4 i.e. full data set as our data set is very small. In practice, we use very large data sets and then defining batch size becomes important to apply stochastic gradient descent[sgd].

Perceptrons got a lot of attention at that time and later on many variations and extensions of perceptrons appeared with time. But, not everyone believed in the potential of Perceptrons, there were people who believed that true AI is rule based and perceptron is not a rule based. Minsky and Papert did an analysis of Perceptron and conluded that perceptrons only separated linearly separable classes. Their paper gave birth to the Exclusive-OR(X-OR) problem. This completes a single forward pass, where our predicted_output needs to be compared with the expected_output. Based on this comparison, the weights for both the hidden layers and the output layers are changed using backpropagation.

This process is repeated until the predicted_output converges to the expected_output. It is easier to repeat this process a certain number of times (iterations/epochs) rather than setting a threshold for how much convergence should be expected. A good resource is the Tensorflow Neural Net playground, where you can try out different network architectures and view the results.

On the other hand, when we test the fifth number in the dataset we get the value that is close to 1 and the original value is also 1. We will just change the y array to be equal to (0,1,1,0). We can see that now only one point with coordinates (0,0) belongs to class 0, while the other points belong to class 1. As you can see, the classifier classified one set of points to belong to class 0 and another set of points to belong to class 1. Now, we can also plot the loss that we already saved in the variable all_loss.

The XOR problem with neural networks can be solved by using Multi-Layer Perceptrons or a neural network architecture with an input layer, hidden layer, and output layer. So during the forward propagation through the neural networks, the weights get updated to the corresponding layers and the XOR logic gets executed. The Neural network architecture to solve the XOR problem will be as shown below.