Machine Learning? I think neuron to something.

As somewhat of a part 2 to my Hobby Dev Post, and part 1 of its own little series, I’ve started dabbling with machine learning. This isn’t meant as a tutorial so much as a narrative of my journey into trying to create a neural network from the ground up, using just the python standard library!

Why this undertaking? As I discussed in my Hobby Dev post, jobifying my hobbies has been a surefire way to kill them. But in every field I have worked, the people who were best at the job were always the people who spent their free time continuing to pursue the hobby. So since I am not content with my mediocrity, I have decided to try and breathe new life into this hobby.

Foundation

I really started digging into programming to do data analysis of the logs I kept while working at the climbing gym. At the time, machine learning seemed like magic, a magic I very much wanted to possess. It was, however, way beyond my reach. But I’ve learned a thing or two (or four… or six) since then, and decided machine learning was a great place to jump back in and see what I can do.

I am intentionally not using the big wigs of machine learning because my goal is not to write the worlds most accurate neural network. My goal is simply to understand the mechanics, and using a fully fleshed out library like Sci-kit Learn doesn’t really afford me that opportunity. So here I go, reinventing the wheel. (My wheel is definitely still kinda square… but we are working on it!)

A kinda perceptive Perceptron

My first step is writing just a single neuron. The goal of my standalone neuron, referred to as a Perceptron, is simple classification. Take in any number of inputs (initially, we’ll just use 2 inputs) and output -1 for Class A and 1 for Class B. The basic structure of a neuron is, on one end, all the receptors. These receptors feed their data to the “body” of the neuron, via weighted channels. In the body of the neuron, we calculate the weighted sum of all the data from the receptors. This sum is passed into some kind of activation function that essentially handles translating that sum into a usable value, and then spits that value out of our neuron.

input 1
     \
   weight 1
       \
        +-----------+
        | weighted  |     +----------+
        |           | --> |activation| --> classification value
        |    sum    |     +----------+
        +-----------+
       /
   weight 2
     /
input 2

Step one: creating the neuron class

This is some pretty straightforward code just setting up our skeleton on which to build.

class Neuron:
    def __init__(self, number_of_inputs):
        self.number_of_inputs = number_of_inputs

n = Neuron(number_of_inputs=2)

In order to do some simple error checking down the road, I chose not to make input numbers dynamic. This will let me do things like:

if len(inputs) != self.number_of_inputs:
    raise NeuronException('Length of inputs must be the same as the declared number of inputs.')

It is also a handy counter for doing loops for weighted sums and training our model later. We know how many times we’ll have to do a loop without having to grab the len() of our inputs or weights or whatever. And it is unlikely we wouldn’t know the dimensionality of our data before sending it to a neuron, so the slight potential inconvenience of having to declare our size is offset by the gains in functionality.

Step two: weights

Now we need weights. Can’t do a weighted sum without weights!

I want to be able to supply the neuron with a list of specific weights (for debugging or reproducing results for plotting), or let it randomly generate weights on its own. We’ll want these random weights to be normally distributed around 0, random.normalvariate(0, .5) will be perfect for this.

import random


class Neuron:
    def __init__(self, number_of_inputs, weights=None):
        self.number_of_inputs = number_of_inputs
        self.weights = weights if weights is not None else self._random_weights()

    def _random_weights(self):
        return [random.normalvariate(0, .5) for _ in range(self.number_of_inputs)]

This is another helpful use of having number_of_inputs explicit to the class; we don’t have to wait until we see inputs for the first time to generate our random weights. We should also check that the supplied list of weights has the same number of entries as our expected number of inputs, so that every input has a weight. I’m omitting that code here for clarity.

Step three: inputs and sums

Now that we have our weights, we also need to be able to take in a list of inputs and calculate our weighted sum. Again, I’m omitting the error checking for clarity.

class Neuron:
    # ...
    def _weighted_sum(self, inputs):
        return sum([inputs[i] * self.weights[i] for i in range(self.number_of_inputs)])

Damn python is a pretty language!

So we can take inputs and calculate their weighted sum, but that probably isn’t how we want to actually interact with our neuron. We probably want to call a method that encompasses both the weighted sum and our activation function, something like guess() or predict(). To do this, we should write our activation function first.

Step four: activate

There are many types of activation functions, some of which sound like diseases. If you find yourself with a Leaky ReLU, consult a medical professional immediately! ;)

Anyway, I don’t fully understand the math of many of these activation functions, so I’m starting with something I do understand, identity. Since the goal here is to output a -1 for Class A and a 1 for Class B, we can simply output based on the sign of the sum. If the sum is negative, output a -1, positive, a +1. Simple. (We do need to handle the off chance that our weighted sum comes back 0. I’m choosing to lump it in to positive numbers)

class Neuron:
    # ...
    def _activate(self, w_sum):
        return -1 if w_sum < 0 else 1

Now we have all the parts in place to make a simple prediction.

Step five: predict

We haven’t done any training yet, so our prediction will be more like a blind guess, but it is a building block we need to be able to train. This prediction function will really just be hooking together the plumbing we have laid thus far. Here is our whole Neuron class and the prediction function:

class Neuron:
    def __init__(self, number_of_inputs, weights=None):
        self.number_of_inputs = number_of_inputs
        self.weights = weights if weights is not None else self._random_weights()

    def _random_weights(self):
        return [random.normalvariate(0, .5) for _ in range(self.number_of_inputs)]

    def _weighted_sum(self, inputs):
        return sum([inputs[i] * self.weights[i] for i in range(self.number_of_inputs)])

    def _activate(self, w_sum):
        return -1 if w_sum < 0 else 1

    def predict(self, inputs):
        w_sum = self._weighted_sum(inputs)
        return self._activate(w_sum)


n = Neuron(number_of_inputs=2)
n.predict([100, 50])

After running this a few times, the distribution of -1 and 1 was about equal. The randomly generated weights mean this prediction is pretty much coin toss, at least until we train our neuron.

To be continued…

In the next post, we’ll return with a tale of how to train your dra… neuron. How to train your neuron.

Jimmy Keith wrote this on 15 July 2019