<p>As somewhat of a part 2 to my Hobby Dev Post, and part 1 of its own little series, I've started dabbling
with machine learning. This isn't meant as a tutorial so much as a narrative of my journey into trying to create a
neural network from the ground up, using just the python standard library!</p>

<p>Why this undertaking? As I discussed in my [Hobby Dev]({% post_url 2019-05-30-hobby-dev %}) post, jobifying my hobbies
has been a surefire way to kill them. But in every field I have worked, the people who were best at the job were always
the people who spent their free time continuing to pursue the hobby. So since I am not content with my mediocrity,
I have decided to try and breathe new life into this hobby.</p>
<h1>Foundation</h1>
<p>I really started digging into programming to do data analysis of the logs I kept while working at the climbing gym.
At the time, machine learning seemed like magic, a magic I very much wanted to possess. It was, however, <em>way</em> beyond
my reach. But I've learned a thing or two (or four... or six) since then, and decided machine learning was a great
place to jump back in and see what I can do.</p>
<p>I am intentionally not using the big wigs of machine learning because my goal is not to write the worlds most accurate
neural network. My goal is simply to understand the mechanics, and using a fully fleshed out library like Sci-kit Learn
doesn't really afford me that opportunity. So here I go, reinventing the wheel. (My wheel is definitely still
kinda square... but we are working on it!)</p>
<h2>A kinda perceptive Perceptron</h2>
<p>My first step is writing just a single neuron. The goal of my standalone neuron, referred to as a Perceptron, is simple
classification. Take in any number of inputs (initially, we'll just use 2 inputs) and output -1 for Class A and 1
for Class B. The basic structure of a neuron is, on one end, all the receptors. These receptors feed their data to the
"body" of the neuron, via weighted channels. In the body of the neuron, we calculate the weighted sum of all the data
from the receptors. This sum is passed into some kind of activation function that essentially handles translating
that sum into a usable value, and then spits that value out of our neuron.</p>
<pre><code>input 1
     \
   weight 1
       \
        +-----------+
        | weighted  |     +----------+
        |           | --> |activation| --> classification value
        |    sum    |     +----------+
        +-----------+
       /
   weight 2
     /
input 2
</code></pre>
<h3>Step one: creating the neuron class</h3>
<p>This is some pretty straightforward code just setting up our skeleton on which to build.</p>
<pre><code class="language-python">class Neuron:
    def __init__(self, number_of_inputs):
        self.number_of_inputs = number_of_inputs

n = Neuron(number_of_inputs=2)
</code></pre>
<p>In order to do some simple error checking down the road, I chose not to make input numbers dynamic. This will let me do
things like:</p>
<pre><code class="language-python">if len(inputs) != self.number_of_inputs:
    raise NeuronException('Length of inputs must be the same as the declared number of inputs.')
</code></pre>
<p>It is also a handy counter for doing loops for weighted sums and training our model later. We know how many times we'll
have to do a loop without having to grab the <code>len()</code> of our inputs or weights or whatever. And it is unlikely we
wouldn't know the dimensionality of our data before sending it to a neuron, so the slight <em>potential</em> inconvenience of
having to declare our size is offset by the gains in functionality.</p>
<h3>Step two: weights</h3>
<p>Now we need weights. Can't do a weighted sum without weights!</p>
<p>I want to be able to supply the neuron with a list of specific weights (for debugging or reproducing results for
plotting), or let it randomly generate weights on its own. We'll want these random weights to be normally distributed
around 0, <code>random.normalvariate(0, .5)</code> will be perfect for this.</p>
<pre><code class="language-python">import random


class Neuron:
    def __init__(self, number_of_inputs, weights=None):
        self.number_of_inputs = number_of_inputs
        self.weights = weights if weights is not None else self._random_weights()

    def _random_weights(self):
        return [random.normalvariate(0, .5) for _ in range(self.number_of_inputs)]
</code></pre>
<p>This is another helpful use of having <code>number_of_inputs</code> explicit to the class; we don't have to wait until we see inputs
for the first time to generate our random weights. We should also check that the supplied list of weights has the same
number of entries as our expected number of inputs, so that every input has a weight. I'm omitting that code here for
clarity.</p>
<h3>Step three: inputs and sums</h3>
<p>Now that we have our weights, we also need to be able to take in a list of inputs and calculate our weighted sum.
Again, I'm omitting the error checking for clarity.</p>
<pre><code class="language-python">class Neuron:
    # ...
    def _weighted_sum(self, inputs):
        return sum([inputs[i] * self.weights[i] for i in range(self.number_of_inputs)])
</code></pre>
<p>Damn python is a pretty language!</p>
<p>So we can take inputs and calculate their weighted sum, but that probably isn't how we want to actually interact with
our neuron. We probably want to call a method that encompasses both the weighted sum <em>and</em> our activation function,
something like <code>guess()</code> or <code>predict()</code>. To do this, we should write our activation function first.</p>
<h3>Step four: activate</h3>
<p>There are <em>many</em> types of activation functions, some of which sound like diseases. If you find yourself with a Leaky ReLU,
consult a medical professional immediately! ;)</p>
<p>Anyway, I don't fully understand the math of many of these activation functions, so I'm starting with something I do
understand, identity. Since the goal here is to output a -1 for Class A and a 1 for Class B, we can simply output based
on the sign of the sum. If the sum is negative, output a -1, positive, a +1. Simple. (We do need to handle the off chance
that our weighted sum comes back 0. I'm choosing to lump it in to positive numbers)</p>
<pre><code class="language-python">class Neuron:
    # ...
    def _activate(self, w_sum):
        return -1 if w_sum &#x3C; 0 else 1
</code></pre>
<p>Now we have all the parts in place to make a simple prediction.</p>
<h3>Step five: predict</h3>
<p>We haven't done any training yet, so our prediction will be more like a blind guess, but it is a building block we need
to be able to train. This prediction function will really just be hooking together the plumbing we have laid thus far.
Here is our whole Neuron class and the prediction function:</p>
<pre><code class="language-python">class Neuron:
    def __init__(self, number_of_inputs, weights=None):
        self.number_of_inputs = number_of_inputs
        self.weights = weights if weights is not None else self._random_weights()

    def _random_weights(self):
        return [random.normalvariate(0, .5) for _ in range(self.number_of_inputs)]

    def _weighted_sum(self, inputs):
        return sum([inputs[i] * self.weights[i] for i in range(self.number_of_inputs)])

    def _activate(self, w_sum):
        return -1 if w_sum &#x3C; 0 else 1

    def predict(self, inputs):
        w_sum = self._weighted_sum(inputs)
        return self._activate(w_sum)


n = Neuron(number_of_inputs=2)
n.predict([100, 50])
</code></pre>
<p>After running this a few times, the distribution of -1 and 1 was about equal. The randomly generated weights
mean this prediction is pretty much coin toss, at least until we train our neuron.</p>
<h3>To be continued...</h3>
<p>In the [next post]({% post_url 2019-07-31-neuron2 %}), we'll return with a tale of how to train your dra... neuron.
How to train your neuron.</p>

Machine Learning? I think neuron to something.