Predictive Processing as a unifying framework for human cognition

Introduction

The purpose of this paper is to provide a simplified framework for talking about a paradigm that has been called “the grand unifying theory of neuroscience.” While there is an emerging consensus about its validity among neuroscientists, it is not yet communicated clearly outside of academia. This paradigm is called predictive processing (or predictive coding), and it details what might be described as the master algorithm of the brain. It provides a plausible, testable explanation of how the brain constructs its experience, relating to its environment as well as to the body. And it boils down to this: your brain is a simulator, which attempts at all times to optimize its predictive model of what is happening in this moment. You are not experiencing reality directly, but your brain’s best simulation of what’s going on. The brain compares its generative model (meaning, one that it generates) moment by moment to incoming sensory data in order to optimize its own operations and reduce its processing power. This is a lot to take in so let’s break it down a bit.

Don’t worry about sounding professional. Sound like you. There are over 1.5 billion websites out there, but your story is what’s going to separate this one from the rest. If you read the words back and don’t hear your own voice in your head, that’s a good sign you still have more work to do.

Be clear, be confident and don’t overthink it. The beauty of your story is that it’s going to continue to evolve and your site can evolve with it. Your goal should be to make it feel right for right now. Later will take care of itself. It always does.

Plato’s Cave?

What does it mean to say we’re living in a simulation? The simulation that’s meant refers not to the Elon Musk variety - that we all exist within a giant computerized simulation - but to the notion that our experience is like a virtual reality simulation inside our own brains. Anil Seth explains this well in his TED talk, titled “Your brain hallucinates your conscious reality.” Start by remembering that everything we experience is experienced by the brain. There are no holes in the skull where light, sound, or any other sensory information comes in directly to the brain. The only inputs into the brain are electrical signals transduced from our senses and relayed by neurons. If someone’s visual cortex is damaged, they can’t see even if their eyes are perfectly functional. 

Take a second to let this fully sink in. Realize that what this means is that you always experience only a representation constructed by the brain, and never reality itself, as convincing as your experience of “reality” is. We simply have no direct access. Our body’s sensory receptors do, but without the brain to experience the senses of the body, the body’s experience is not perceived - think of a paraplegic whose body is intact but whose nerves to the brain are severed and who cannot feel his limbs. In the Allegory of the Cave, Plato describes a society chained inside a cave who can see nothing from the outside except for shadows cast on a nearby wall. In a very literal sense, we are those people, experiencing only the electrical signals transduced by our brains. So what is it that we actually experience?

The brain predicts

In the classical view of the brain, neuroscientists believed that sensory data was relayed through a hierarchical processing pathway that combines inputs for higher and higher levels of complexity and integration. The visual pathway is one of the best understood, so let’s use it as an example. The lowest levels of the hierarchy were viewed as directly connected to our sensory receptors, such as ones that receive direct input from photoreceptor cells in the eye. It was found that vision is processed almost like pixels in the lowest levels, combining into edges, then shapes, and later combining in the cortex with other information such as colors, patterns/textures, movement. At the higher levels of the hierarchy, the classical view says, visual data integrates further with other senses, concepts, and memories in our association cortices. It turns out that this classical view is not exactly right. The hierarchical pathway piece is largely correct, but the directional data transmission piece is off. 

As we began saying, the brain is a simulation machine - it constantly simulates its best model of “what’s actually going on out there.” The directional flow of this information is from the top-down, meaning that it’s relayed in its entirety rather than constructed from its pieces (which would be referred to as bottom-up). These simulated predictions relate to everything you already know about the world and have learned over the course of your life - beliefs, factual knowledge, heuristics, past experiences. At the same time that it’s creating this simulation, the brain simultaneously compares the simulated top-down predictions with sensory inputs it’s receiving from the environment (bottom-up) in order to refine its model and optimize its predictions in real time. Every layer of the hierarchy receives the predictions from the layer above it and the sensory data (which we’ll later see is in the form of prediction error) from the layer below it, and compares. What it’s looking for is, ideally, an agreement handshake between the top down data representing the brain’s best hypothesis (simulated model) and the bottom-up information related to sensory data coming in through our external and internal senses. 

The Bayesian Brain and Free Energy Principle (getting a bit technical)

[This section may be an unsatisfying mix of too technical and not technical enough. There is a TL;DR at the bottom for those who wish to skip.]

To understand that handshake better, let’s take a step back and discuss Bayesian Inference. Bayes theorem describes how a system updates its knowledge about the world given new evidence. We start with a prior belief about a specific situation, which is probabilistic in nature, and a piece of evidence. Bayes Theorem answers the question, “what is the posterior (after the fact) belief about a specific situation given a new piece of evidence?” And it does so with this formula:

P(B|E)= P(E|B)*P(B) / P(E)

B = Belief; E = Evidence; P = Probability; | = Given that. 

Without digging into the math of this equation, what it means is that after a piece of evidence, your posterior belief is related to your prior belief and the likelihood of seeing that evidence in the first place. Let’s try to get an intuitive sense for this with an example. Imagine you’re lying in bed at night feeling pretty relaxed and you hear a loud crack. If you live in a pretty safe neighborhood and have a cat, then what happens to your belief in your own safety? In this case, it probably wouldn't change very much. Your posterior belief in your safety given that loud crack (P(B|E)) remains pretty high because 1. your belief in your safety was high to begin with P(B), and 2. the probability of that loud crack happening even though it’s safe is pretty high (remember, the cat), and the probability of hearing a loud crack at all isn’t out of ordinary - so the ratio if P(E|B)/P(E) is almost 1 in this case. However, what happens if you don’t have a cat and you heard there’s a serial killer on the loose? In this case, the probability of the loud crack overall, P(E) is slightly higher than P(E/B). This is because P(E) is equal to the probability of the loud crack if you’re safe P(E/B) plus the probability of the loud crack if you’re not safe, <let’s refer to this as P(E|murder)>. Because P(E|B)/P(E) <1, even slightly (no reason to think that axe murderer is after you, but you never know…) then your posterior belief in your safety is slightly lower than before. These formulas are not very clear and require reading several time 

Ok… And the brain?

Before seeing how this applies to the brain, let’s touch upon the free energy principle, theorized by Karl Friston, the genius neuroscientist credited with much of predictive processing on the whole. The free energy principle equations are far beyond the scope of this writing, but its bottom line is that any self-organized system maintains equilibrium by minimizing its chances of encountering things that it cannot anticipate. 

Any self-organized system, from the human brain to a single drop of water that’s maintaining its integrity, is composed of four functions: sensory input coming in from the environment, motor action acted out on the environment, things that are occurring outside of the system and to which the system does not have direct access (called hidden states) and things occurring within the system (internal states). (This four-function model is commonly referred to as a Markov Blanket.) At any given time, the organism is attempting to optimize its chances for survival through reducing high free energy states to the best of its ability, maintaining its steady states. For complicated reasons (that the smartest people I know still don’t always fully grasp), this amounts to attempting to reduce its surprise at its environment by predicting it as accurately as it can. The system’s ideal low free-energy state is unsurprised - it accurately predicts what’s going on and reacts appropriately at all times. In the human brain, this translates to reducing the discrepancy between the predicted simulation model it generated of its environment and the data that’s coming in about that same environment. 

Combining these principles, we get the Bayesian Brain, which builds its best guess to explain (or infer, put another way) the causes for the pattern of sensory data it’s receiving. It does so by constantly minimizing the difference between its model of the hidden states of the world - remember, no holes in the brain, only electrical signals coming in - and the incoming electrical signals about those hidden states. The brain maintains this inferred internal model of the world - a simulation - and updates it according to new information that comes in. In predictive processing language, we say that the brain is trying to minimize its surprise or surprisal, but it is important not to confuse this with the human emotional experience of surprise - these terms refer to data discrepancy, not lived experience. A better term for this surprise is prediction error.

TL;DR: Computational neuroscientists, theoretical physicists, and many other experts have deduced that the brain follows a Bayesian algorithm. This means it is constantly updating its beliefs about the world from new sensory evidence. In order to maintain a low free-energy state, it attempts to optimize its internal model of the outside world, the true state of which remains out of view. 

Prediction error 

Let’s return to that agreement handshake between the top-down modeling and the bottom-up sensory data. The brain reducing its free energy/surprise/prediction error in this way translates to it constantly optimizing its model so that surprising or discrepant information does not need to be processed. On all levels of the hierarchy, the top-down simulation is compared to the bottom-up sensory information flowing up, and the difference is referred to as the prediction error. When the prediction error is very low, meaning the incoming data and simulation are pretty well matched, we get the handshake. When the prediction error is slightly higher but still sufficiently small, the brain usually ignores it - the brain does not propagate the signal up the hierarchy, instead choosing to “explain it away.” This is because processing is energetically costly to the brain; the brain would be drained if it analyzed every single incoming input, of which there are many at any moment. Our senses are imperfect and the world is constantly shifting (e.g. slight light changes, dust, subtle movement in water, as well as spontaneous eye saccades - what is actually coming into your eyes is not nearly as stable as it seems). The signal to noise ratio is lower than we imagine. If your attention captured every movement in your visual system you’d never get anything done because you’d be so overwhelmed. In fact, some theories explain autism by suggesting a faulty mechanism for prediction error, allowing too much prediction error to flow upward and overwhelm the system with sensory data; for example, while most of our brains “explain away” a slightly scratchy tag on our shirt, the brain of a person with autism may attend to it. As a result, the autistic person may find the slight discrepancy unbearable, hence the sensory sensitivities we often see in autism. Very good explanation!

Active Inference

So when error is exceedingly small, there’s a handshake. When it’s minimal, it’s explained away. What happens when prediction error is substantial - when there is significant discrepancy between the brain’s simulation and the incoming sensory data? One of two things. The first is adoption of an alternative hypothesis. Take, for example, that loud crack in the night. Perhaps the first hypothesis is that it’s the cat, but upon realizing that the cat is actually laying beside you in bed, the brain adopts an alternative hypothesis: it was that broom that was leaning precariously in the hallway. Perhaps you’ll avert your gaze to the hallway to investigate. The second thing that can happen is a resampling of data in order to attempt to better support the original hypothesis. This is referred to as active inference. Often, this is external sensory data, for example eye movement (known as saccades) that seek new information that might better support the hypothesis. For example, take fluid levels. When you’re thirsty and dehydrated, the brain begins to issue predictions of greater levels of hydration, then begins to act upon the world (by drinking) to reduce the prediction error of its current thirsty state and its predicted satiated state. This also happens with our internal senses - interoception and thought - and this is where things get very interesting (at least to me, someone interested in psychological experience).

Embodied Active Inference

Much of the brain’s information about your current experience comes from its mapping out of your internal state. Some of this is a basic feeling of good, bad, or neutral, which is sometimes referred to as proto-emotion, and probably explains why newborn babies sometimes cry and sometimes sleep peacefully. It’s just a basic way of how our body feels. Much of it was bootstrapped over the course of your life, providing your brain much more intricate context about the feelings of good or bad. Where it gets a little wild is, because the brain loves optimizing the accuracy of its model, one thing that happens is the brain predicts, “I’m going to feel and think about this situation a certain way,” and then it appears that it recruits the relevant body sensations and thoughts to make its own prediction come true, thereby giving itself the handshake that it craves so much. For example, let’s say that over the years, you’ve found that you really don’t like spending time with an acquaintance who makes you feel angry. When you hear that this acquaintance will be present at a dinner party this evening, you begin to form a prediction that you will grow angry this evening. Your brain starts to recruit “annoyed” circuitry before the evening even starts, and you arrive on edge. Throughout the night, whenever an ambiguous statement is made that could be interpreted as innocuous or as offensive, your brain favors the offensive interpretation and adopts it as further evidence of its hypothesis, recruiting more resources to fulfill its prediction of irritation, including thoughts such as “this guy is such a jerk,” past memories, and body sensations that corroborate your annoyance. As this feedback loop proceeds, you feel more and certain of your original prediction. This is why mindset can be such a self-fulling prophecy, of course, but more than that - this process happens on the unconscious level most of the time. 

Of course, this also happens with your body’s motor movements toward the external world, as well. The brain will predict, “I am going to feel the sensations associated with moving my legs,” and then the motor system fires off commands to begin walking, fulfilling the brain’s predictions once more. As a fun aside, this is often referenced as the reason that we cannot tickle ourselves: the brain sends a prediction of how our body will feel once a motor movement (tickling) will occur, hence the sensory receptors come to expect the sensations inherent in the tickling and no prediction error occurs - instead the sensations are “explained away” as minimal prediction error. 

Precision 

Of course, not all prior beliefs or simulations, and not all sensory data, are created equal. Some sense data is very noisy, and other sense data is very clear. Some beliefs deserve high confidence, and some do not. The degree of confidence in either case is referred to as precision. Precision weighting refers to the neural mechanism of turning the volume on neurons carrying prediction errors up the hierarchy. Take as an example, a loud room where you’re attempting to hear your friend talk - you want to turn up the volume - increase the precision weight - of the auditory information coming in. Or, when driving down a dark, rainy road, you want to avoid the risk of an accident by increasing the precision weighting on your visual inputs more than when driving on a clear, sunny day. 

An interesting feature of the predictive processing model is that it explains the mechanism of attention. Attention is conceptualized as how much prediction error is let through - almost as though the volume is turned up on the neurons that carry prediction error in a particular sensory stream. 

Another interesting outcome is that there is an inverse relationship between precision of the top down belief (or prediction, or piece of simulation) and the precision error. Truly paying attention to any incoming data means relinquishing some certainty or conviction in our hypotheses.

Networks

We already know that our brain is capable of constructing entirely believable scenes - we dream every single night, and we can all attest to the fact that dreams can feel quite real. The same architecture that constructs these believable dreams is busy creating your simulated reality all day - what neuroscientist Anil Seth calls “controlled hallucination.” Unlike dreams, our waking simulations mirror reality, and hence they are realistic and follow the rules of the external world. 

There is a huge number of components that construct these simulations. First, all of our external senses and the position of our body in space, which is referred to as proprioception. Additionally, there are two more major components: thoughts, and interoception. We may not always think of our thoughts as sensory (though, fun fact: the Buddhists consider thought the sixth sense!) but they integrate within our simulations and have profound effects on our experiences - if you’ve ever engaged in cognitive behavioral therapy, which helps people see the relationship between their negative thinking and their moods, you know this quite well. Or, think of the example above about the annoying acquaintance. The second major component is interoception, which refers to information from within our body (e.g. your breathing, digestive system, blood pressure, etc.) Our interoceptive system is probably the most influential on how we feel in any given moment. It is strongly related to our emotions, sitting at the core of emotional experience through what is termed affect - the basic positive, negative, or neutral feelings we all have in our bodies at all times, also discussed above.

Brain activity occurs within core networks that connect anatomically distant parts and allow for collaboration. Different networks specialize in the sense of self, emotions, interoception, action outside the body, memory processing, sensory processing, decision-making and cognition, etc. At any given time, most of your networks are active in one way or another, but the specifics of the activity determines your experience. There is a reinforcing feedback loop to network activity: when specific network activity occurs with frequency, it becomes more likely to occur again in the future because the connections literally strengthen. This is the reason that our minds tend to fall into familiar states of being - the ways we feel have gotten bootstrapped and refined over our entire lives. 

Priors and hyperpriors

As already discussed, our simulations are shaped not only by sense data but by our priors: accumulated experiences that we’ve had over the course of our lives, as well as our cognitive beliefs. And those priors are subject to priors as well, which are called hyperpriors. That is, we carry probabilistic beliefs about our beliefs in hierarchical form. Some of these hyperpriors are inborn and based on our biology, such as aversion to certain tastes due to toxicity, avoidance of pain, the understanding that light comes from above, or the fact that we expect faces to look not be concave (leading to illusions like this one). 

However, hyperpriors come in purely conceptual, abstract flavors also. In psychology, we talk about core schemas, for example, “the world is a safe place,” or “people are untrustworthy,” or “I am a likable person,” which flavor the kind of thoughts one is likely to experience and believe, the kinds of feelings one is likely to experience, and the kinds of interpretations one is likely to make about their experiences.  

Conclusion

In summary, convincing evidence has led the scientific community to conclude that our brains are predicting machines that simulate our reality moment by moment. The aim of the brain is to model its environment, which includes both the body and what is happening outside of the body, as accurately as it can. It constantly updates its model by comparing it to incoming data from the body and its environment and tweaking when necessary. Moreover, it performs active inference, meaning that it actively seeks to confirm its hypotheses, both by moving the body in ways that bring about its predictions (like eye movements to find new evidence, or motor movements that fulfill its predictions) and through interoception and thought (by constructing the emotional and cognitive experiences it expects). 

How certain are we about this? Pretty certain. Predictive processing has swept neuroscience and been accepted as the grand unifying theory that explains everything from psychopathology like psychosis, depression, and autism, to cognitive biases, to the placebo effect, to how babies learn about their environment.  Some neuroscientists are arguing about the details of the algorithm, and much of the cellular level mechanisms have still not been worked out. However, the unshakeable truth is that we don’t directly experience reality but rather our brain’s simulation of reality; we know that because we know that all experience requires the brain, and that we only experience that which is transduced and interpreted by the brain Do you mean our experience is subjected to our own subjective perception? The word subjectivity is important and missing here. . We have considerable evidence for the effects of top-down simulation modeling on our daily experience, both within single sensory domains and across integrated, real-world scenarios. We also have evidence that suggests that what we thought was sensory processing, in many cases, is the processing of prediction error (or, unexpected stimulus). On the whole, more and more evidence emerges in support of predictive processing constantly. 

This paper is meant to provide an explainer for understanding the basics of predictive processing. In future writing, I aim to dig into the fascinating insights that predictive processing can provide us for understanding the human experience, primarily surrounding consciousness and the self, and personal psychology.