Building Patterns from Randomized Inputs

In the previous post, we looked at information theory from a physical perspective.

We used the metaphor of a key, a keyhole, and a lock to show that, in a physical sense, a single line of code is being input from a key to the receptor of a keyhole, which in turn is being used to activate a binary response, a lock opening or closing:

We used this example to discuss the ways that physical characteristics like dimples or ridges can be used to differentiate objects, allowing for each individual object, in this case a key, to act as a store of information.

We also noted that a keyhole, the receptor, can be equally thought of as a store of information. Through its topography, it too has a certain set of physical features that differentiate it from other key holes and allow it to serve as a prerequisite for activation.

The lock will only open when the set of features on our key and our keyhole match precisely. In this way, the activation of our function, opening the lock, only happens when we want it too.

Instead of requiring us to actively approve or deny each attempted entry, we have preprogrammed our system to, in a sense, only approve entries that were already preapproved, by having the lock turn away any key combination that doesn’t match with our designed topography.

Working backwards from our output

But what would happen if the topography was not designed by us? What if we knew that certain outputs were possible, but we didn’t know what type of input we needed to use in order to achieve them?

For a physical example of this, you could look at the image above and imagine trying to fashion a key for a keyhole you didn’t design. How would you go about that process?

Well, the most accurate way would be to take the keyhole apart and map its topography, and then put it back together and build a key that matches those features.

But taking a keyhole apart, analyzing it and putting it back together, requires you to have a lot of control over your environment.

In a realistic setting, if we have an output that we know is possible, but we don’t know which input features achieve it, it’s usually because:

1) We don’t have access the topography of the receptor, and we have to find out what it looks like.
2) Even when we know what the topography looks like, finding out how an input’s features would match it appropriately is a nontrivial problem.

So how would we go about building a key that matched an unknown keyhole? Well, I looked it up and the first thing that locksmiths do is create a prototype (in this case a blank thin key, like a blade) that is able to reach the end of the keyhole without being obstructed.

Once you have that, you try to turn it unsuccessfully until markings show up on the face of the key, indicating where it needs to be cut in order for the cylinder inside the keyhole, and therefore the lock, to turn.

Finally, you make incisions on that key face according to the markings and rinse and repeat until it turns the lock.

Working towards complex real world environments

Let’s take a moment to note that, in this example (physical keys, keyholes and locks) our theory of information is simplified in several ways.

Firstly, we have a single activation function which is binary (ON/OFF or LOCK/UNLOCK).

Secondly, our set of inputs (keys) and receptors (keyholes) both have just two parameters, in this case: (Number of ridges, Size of ridges).

Thirdly, the set of possible features for our inputs and outputs are part of a finite, static, and transparent set.

In this post, i’d like to expand our discussion to cover complex realistic domains that have the following criteria:

1. The number of activation functions is both greater than two and not at our discretion.
2. One set of features (either the input or receptors) required to activate an output function come from a dynamic, infinite, and/or opaque set.
3. We only have agency over either the input or the receptor’s features, but not both.

If we are able to satisfy these three criteria, then we can use our simplified example as a stepping stone towards discussing forms of information transfer that are able to effectively represent meaning in highly complex scenarios.

Using an Example (Wilderness Survival)

I will try to flesh this out by imagining that we have been placed in the wilderness where we need to figure out how to survive.

In this case, Survival is our ultimate goal, but there are many different underlying functions that need to be activated in order for Survival to be maintained.

So our first criterion for a complex realistic domain has been satisfied already:

1. “The number of output functions that can be activated is both greater than two and not at our discretion”.

This criterion applies because there are many things we need in order to Survive and we don’t get to choose what they are.

Our prerequisites for survival were chosen by our environment before we were born. If we could choose, it would be easy! We could say “I choose starving and freezing as the only prerequisites for my survival!” and we would not have to do anything. Alas, these are out of our control and they happen to look more like this:

Prerequisites for Survival in the Wilderness:

1. Stay fed

2. Stay warm

3. Don’t get eaten

4. Don’t poison yourself

In order to reach our ultimate goal of Survival, it is not enough to achieve one of these goals, we need to achieve all of them simultaneously, and they do not necessarily collapse into a single function. Staying fed may require information processing that doesn't help in staying warm, or not getting eaten.

So, if Survival is our main goal, and the four subgoals are prerequisites to our main goal, then we can consider each of these an output, or activation of a function based on parameters being met within the input and the receptor. Let’s take a look at the first subgoal as an example: Stay fed.

Inputs and Receptors in Complex Environments

So, if Staying fed is our activation function, or output in this example, what are the inputs and receptors that determine whether we have succeeded in this task?

In this case, the input is repeatedly chosen from the set of edible organic matter that exists in our environment, and the receptors for metabolizing this organic matter are a matter human anatomy.

This is where our second criterion for a complex realistic domain comes in:

2. One set of features (either the input or receptors) required to activate an output function come from a dynamic, infinite, and/or opaque set.

Why?

Because in this case, we get to choose which types of organic matter to put into our body, but we do not get to choose the ways that the human body reacts to different types of organic matter. Again, if we could choose the features of our receptors, this would be easy! We could find the most abundant form of matter, say leaves, and choose for that to be the only type of matter that we need to Stay fed.

Alas, our receptor’s features in this case are part of a dynamic and opaque set. The set is dynamic because our bodies need different types of energy at different times, sometimes we require more sugars, fats, protein etc. It is opaque because we don’t have perfect information about what our body needs at any given time. We get signals in the form of hunger and satiation that provide hints into what types of food we should seek out and store, but there is no handbook telling us exactly what we need at every given time.

In this case, we have a much more complex information processing task. One where we need to use fuzzy clues in order to guess at the topography of our bodies metabolic receptors, while simultaneously figuring out which types of organic matter have features that might satisfy our needs.

Complex information processing from a high level perspective

Now that we know we can successfully structure a real world example in this way, satisfying our criteria for a complex realistic environment, let’s take a high level overview of the topic in order to have a clearer understanding of the dynamics involved.

Inputs: In a complex realistic environment, like the wilderness survival example from above, our inputs are chosen from a large set of possible inputs, in this case edible organic matter in the environment, because of specific features that match with our receptors. And we do not have perfect information over the set of all possible inputs.

Receptors: In a complex realistic environment, like the wilderness survival example from above, our receptors are predetermined by human anatomy, which in turn was determined by environmental factors.
These biological features serve as the ground truth that determines what features we require from inputs.” For example our biological benchmark for protein intake in a healthy adult male will determine how strongly we prioritize protein content as an input feature. Again, we do not have perfect information over the topography (features) of our receptors.

Outputs: In a complex realistic environment, like the wilderness survival example from above, our outputs: Stay fed, Stay warm, Don’t get eaten, Don’t poison yourself have instrumental value towards our intrinsic goal of Survival. Additionally, because Survival is not a binary output, there are levels of gradation. The better able we are to process information in our environment, in order to find the best match between the features of our inputs to our receptors, the more likely we are to survive for one extra day.