Building Patterns from Randomized Inputs

In the previous post, we looked at information theory from a physical perspective.

We used the metaphor of a key, a keyhole, and a lock to show that, in a physical sense, a single line of code is being input from a key to the receptor of a keyhole, which in turn is being used to activate a binary response, a lock opening or closing:

Key = Input with Features called Characteristics
Keyhole = Receptor with Features called Topography
Lock = Output with Activation

We used this example to discuss the ways that physical characteristics like dimples or ridges can be used to differentiate objects, allowing for each individual object, in this case a key, to act as a store of information.

We also noted that a keyhole, the receptor, can be equally thought of as a store of information. Through its topography, it too has a certain set of physical features that differentiate it from other key holes and allow it to serve as a prerequisite for activation.

The lock will only open when the set of features on our key and our keyhole match precisely. In this way, the activation of our function, opening the lock, only happens when we want it too.

Instead of requiring us to actively approve or deny each attempted entry, we have preprogrammed our system to, in a sense, only approve entries that were already preapproved, by having the lock turn away any key combination that doesn’t match with our designed topography.

Working backwards from our output

But what would happen if the topography was not designed by us? What if we knew that certain outputs were possible, but we didn’t know what type of input we needed to use in order to achieve them?

For a physical example of this, you could look at the image above and imagine trying to fashion a key for a keyhole you didn’t design. How would you go about that process?

Well, the most accurate way would be to take the keyhole apart and map its topography, and then put it back together and build a key that matches those features.

But taking a keyhole apart, analyzing it and putting it back together, requires you to have a lot of control over your environment.

In a realistic setting, if we have an output that we know is possible, but we don’t know which input features achieve it, it’s usually because:

1) We don’t have access the topography of the receptor, and we have to find out what it looks like.
2) Even when we know what the topography looks like, finding out how an input’s features would match it appropriately is a nontrivial problem.

So how would we go about building a key that matched an unknown keyhole? Well, I looked it up and the first thing that locksmiths do is create a prototype (in this case a blank thin key, like a blade) that is able to reach the end of the keyhole without being obstructed.

Once you have that, you try to turn it unsuccessfully until markings show up on the face of the key, indicating where it needs to be cut in order for the cylinder inside the keyhole, and therefore the lock, to turn.

Finally, you make incisions on that key face according to the markings and rinse and repeat until it turns the lock.

Working towards complex real world environments

Let’s take a moment to note that, in this example (physical keys, keyholes and locks) our theory of information is simplified in several ways.

Firstly, we have a single activation function which is binary (ON/OFF or LOCK/UNLOCK).

Secondly, our set of inputs (keys) and receptors (keyholes) both have just two parameters, in this case: (Number of ridges, Size of ridges).

Thirdly, the set of possible features for our inputs and outputs are part of a finite, static, and transparent set.

In this post, i’d like to expand our discussion to cover complex realistic domains that have the following criteria:

  1. The number of activation functions is both greater than two and not at our discretion.
  2. One set of features (either the input or receptors) required to activate an output function come from a dynamic, infinite, and/or opaque set.
  3. We only have agency over either the input or the receptor’s features, but not both.

If we are able to satisfy these three criteria, then we can use our simplified example as a stepping stone towards discussing forms of information transfer that are able to effectively represent meaning in highly complex scenarios.

Using an Example (Wilderness Survival)

I will try to flesh this out by imagining that we have been placed in the wilderness where we need to figure out how to survive.

In this case, Survival is our ultimate goal, but there are many different underlying functions that need to be activated in order for Survival to be maintained.

So our first criterion for a complex realistic domain has been satisfied already:

  1. “The number of output functions that can be activated is both greater than two and not at our discretion”.

This criterion applies because there are many things we need in order to Survive and we don’t get to choose what they are.

Our prerequisites for survival were chosen by our environment before we were born. If we could choose, it would be easy! We could say “I choose starving and freezing as the only prerequisites for my survival!” and we would not have to do anything. Alas, these are out of our control and they happen to look more like this:

Prerequisites for Survival in the Wilderness:

1. Stay fed

2. Stay warm

3. Don’t get eaten

4. Don’t poison yourself

In order to reach our ultimate goal of Survival, it is not enough to achieve one of these goals, we need to achieve all of them simultaneously, and they do not necessarily collapse into a single function. Staying fed may require information processing that doesn't help in staying warm, or not getting eaten.

The intelligence required to achieve any of these tasks is likely generalizable, and the structure of our brain is flexible enough to use its processing power towards different tasks when needed.

So, if Survival is our main goal, and the four subgoals are prerequisites to our main goal, then we can consider each of these an output, or activation of a function based on parameters being met within the input and the receptor. Let’s take a look at the first subgoal as an example: Stay fed.

Inputs and Receptors in Complex Environments

So, if Staying fed is our activation function, or output in this example, what are the inputs and receptors that determine whether we have succeeded in this task?

In this case, the input is repeatedly chosen from the set of edible organic matter that exists in our environment, and the receptors for metabolizing this organic matter are a matter human anatomy.

This is where our second criterion for a complex realistic domain comes in:

2. One set of features (either the input or receptors) required to activate an output function come from a dynamic, infinite, and/or opaque set.

Why?

Because in this case, we get to choose which types of organic matter to put into our body, but we do not get to choose the ways that the human body reacts to different types of organic matter.
Note that the types of organic matter that exist in a given environment are also dynamic, changing with the seasons, and opaque, we don’t have perfect information over the quantities and breadth of all the organic matter in our environment. But we have much more agency over the features of our inputs than the features of our receptors.
Again, if we could choose the features of our receptors, this would be easy! We could find the most abundant form of matter, say leaves, and choose for that to be the only type of matter that we need to Stay fed.

Alas, our receptor’s features in this case are part of a dynamic and opaque set. The set is dynamic because our bodies need different types of energy at different times, sometimes we require more sugars, fats, protein etc. It is opaque because we don’t have perfect information about what our body needs at any given time. We get signals in the form of hunger and satiation that provide hints into what types of food we should seek out and store, but there is no handbook telling us exactly what we need at every given time.

In this case, we have a much more complex information processing task. One where we need to use fuzzy clues in order to guess at the topography of our bodies metabolic receptors, while simultaneously figuring out which types of organic matter have features that might satisfy our needs.

Complex information processing from a high level perspective

Now that we know we can successfully structure a real world example in this way, satisfying our criteria for a complex realistic environment, let’s take a high level overview of the topic in order to have a clearer understanding of the dynamics involved.

Inputs: In a complex realistic environment, like the wilderness survival example from above, our inputs are chosen from a large set of possible inputs, in this case edible organic matter in the environment, because of specific features that match with our receptors. And we do not have perfect information over the set of all possible inputs.

Receptors: In a complex realistic environment, like the wilderness survival example from above, our receptors are predetermined by human anatomy, which in turn was determined by environmental factors.
These biological features serve as the ground truth that determines what features we require from inputs.” For example our biological benchmark for protein intake in a healthy adult male will determine how strongly we prioritize protein content as an input feature. Again, we do not have perfect information over the topography (features) of our receptors.

Outputs: In a complex realistic environment, like the wilderness survival example from above, our outputs: Stay fed, Stay warm, Don’t get eaten, Don’t poison yourself have instrumental value towards our intrinsic goal of Survival. Additionally, because Survival is not a binary output, there are levels of gradation. The better able we are to process information in our environment, in order to find the best match between the features of our inputs to our receptors, the more likely we are to survive for one extra day.

Music, resonance, and synchronization as a visual example

Lets imagine that we are a musician playing in a band. We are a keyboard piano player and we are being accompanied by a drummer and a bassist.

In the case of making music, we have one simple goal. Make something that is enjoyable to (you) our listeners. So our output is Enjoyability, which is not a binary output (i.e. we can tweak our music to make it more or less enjoyable to listen to), and our receptor is each individual listener. Even though everybody has their own idiosyncratic musical tastes, we will be appealing here to a general audience, so we can think of our receptor generally as the human brain.

Our inputs are simply the notes that we play and how we play them. However, notes have many elements underlying them, things like pitch, timbre, texture, volume, duration and form. From a mathematical perspective, each one of these elements acts as a parameter that increases the dimensions of the gradient of Enjoyability that we can traverse by tweaking our inputs.

In this case, we have a simplified example where each note has three parameters (pitch, duration, form) [x,y,z]: pitch referring to which note we are playing, duration referring to how long we are playing, or not playing, a note, and form referring to which instrument we are using to play a note.

Eventually, we could represent our entire input mathematically as a function of all of the parameterized notes that make up our overall sound.

Our drummer starts off playing a very simple rhythm, switching off between a lower and higher tone every second. This is analogous to the sound of a clock ticking.

In order to sound in sync with the drummer, the bassist plays a repetition of four notes every two seconds (C-G-A-G), where the note pair C-G is played every time our drummer plays the lower tone, and A-G are played every time the drummer plays the higher tone.

Simple Drum and Bass