Do we see reality as it is?
I discuss this question with Donald Hoffman who is professor emeritus at the University of California, Irvine. He studies consciousness and perception from an evolutionary point of view. His research has led him to make a bold claim that while we do not yet know what the underlying reality could be like. Rather, reality as we know it now – including space, time, and objects – is a useful fiction that evolution invented for us.
His TED talk on our perception of reality has been watched over 3 million times, and in his recent pop-science book for the wider audience, The Case Against Reality, he makes a convincing case on how our perceived reality is an illusion that has evolved to help us survive and reproduce.
What we talk about
1:46 – Towards building a mathematical understanding perception
5:12 – Evolutionary game theory to help understand perception
6:26 – Computing truth is expensive, so evolution get rids of it
8:00 – Mathematical proof that evolution doesn’t tune us to see the truth
9:56 – David Marr’s ideas and inverse optics
11:30 – What are fitness payoffs? Why do they matter? How are they determined?
16:23 – Maximizing for truth vs maximizing for winning
18:40 – Illusions that show physical reality doesn’t exist
23:21 – Physical reality not being fundamental as a scientific claim
24:23 – Existing physics points to spacetime not being fundamental
30:25 – The basics of the interface theory of perception
34:40 – We have to guess mathematically the deeper structure of reality
38:00 – Model-free planning in reinforcement learning
44:45 – Why should we care about what is objective reality?
49:20 – How can we use science to go beyond physical reality?
56:57 – Reality is a huge social network of conscious agents
1:03:41 – Predictions from the mathematics of conscious agents
1:09:26 – “We will be able to warp space and time”
1:09:35 – Bridging conscious agents to physical reality
1:15:32 – How far along have we come in his journey of building a fundamental theory of reality based on consciousness?
1:17:31 – Is mathematics part of the underlying reality or is it a useful fiction too?
1:30:12 – Does your research on reality and consciousness impact your personal life and beliefs?
Notes and Key Insights
1/ Illusions show that we do not see reality as it is. For example, patches A and B in the image below are of the same color (which means the photons arriving from these two patches are of the same wavelength).
2/ You can verify that the colors in the two patches are exactly the same via a color picker app. Yet our mind sees them as completely different colors. This clearly demonstrates that, at least in this case, we’re not seeing reality as it is.
3/ The reason this illusion works is because our vision isn’t like a camera. It cannot be because a) unlike a camera sensor, we have a big hole in the middle of our retina (blind spot) that we never notice); b) incoming photons pass through many obstructions (most notably neurons) that we never notice.
4/ A true perception of reality should have a big black hole in the middle of the vision. Moreover, become we saccade our gaze several times a second, we should see constantly shifting images rather than the stable vision we perceive.
But we don’t.
5/ The reason our vision is oblivious to rapid eye saccades and the blind spot is that our senses haven’t evolved to perceive reality as it is. Rather, our senses have evolved to help us to survive and reproduce.
6/ In terms of calories and energy, it’s costly to process information. Take the housefly, for example. It has a tiny brain and needs fewer calories than us. It has eyes that are used to find food and mates. But given the size of its brain and calories its brain has for processing information, do you suppose the fly has a complete, rich, and accurate perception of the world around it?
7/ Well, from an evolution’s point of view, as long as the fly is able to find food and mates, it’s preferable to have an incomplete and distorted but calorically cheap representation of reality than an accurate perception of reality.
8/ In fact, that is what Donald Hoffman and his colleagues prove via a mathematical theorem called Fitness Beats Truth. Put simply, it suggests that the probability that our senses have evolved to report the truth about reality is zero.
9/ They suggest that instead of reporting the truth, our senses report fitness payoffs that our estimates of the potential for acquiring resources, attracting mates, and getting ahead of others.
In short, fitness payoffs are our estimates of reproductive potential.
10/ You can imagine fitness payoffs as points in a game. The more points you accumulate, the longer you can play the game.
In the game, if there are two players: one focused on acquiring as many points as possible (fitness) and another focused on understanding how the game works (truth), guess who will win?
11/ Of course, fitness depends on truth but it isn’t mapped to it one to one.
Take the example of oxygen – too little or too much of it will kill us. We just need the right amount of it.
This means an organism evolved to report the true amount of oxygen will get out-competed by an organism evolved to report the right amount of oxygen.
Reverse engineering from fitness to the truth isn’t trivial. Oxygen wasn’t discovered until 1771.
12/ In summary, the claim is that our physical reality is like virtual reality. We do not know what true reality is but we can be sure that whatever we perceive isn’t likely to be it.
There are a few objections one can raise against this idea.
13/ OBJECTION 1 -> Okay, so if we don’t perceive reality as it is, why not jump in front of a train?
Donald Hoffman suggests that the reason you don’t jump in front of a train is similar to the reason you don’t drag your important files on your computer to the trash bin icon. Nobody believes that the trash bin icon is real (it’s implemented behind the scenes as transistors and electric signals), yet interaction with it has real consequences for you.
14/ Similarly, even though we don’t perceive true reality, it doesn’t mean that our perceptions are arbitrary. Our actions within the perceived reality have real consequences.
15/ OBJECTION 2 -> If we are in virtual reality, how do we explain the success of science and engineering? We have sent people to the moon, so surely we must be understand something about what reality is?
16/ Donald Hoffman says that our engineering success suggests an increased mastery over the virtual reality we’re embedded in. Just like a player can get better at a game by studying game mechanics, we can master our perceived reality better and better.
But just like a gamer doesn’t understand how the game is implemented under the hood, we don’t necessarily understand what the actual reality is.
17/ Modern physics actually agrees with the interpretation that the actual reality could be much stranger than our perceptions. Bell’s theorem suggests that reality is not local and that quantum entanglement over large distances is a real thing. The holographic principle suggests our 3D universe may actually be a projection from 2D.
18/ As a theme in physics, reality as we perceive has become less and less tenable. From this perspective, the claim that we don’t perceive true reality is not surprising.
19/ Although it should be recognized that our perception isn’t completely arbitrary. We do not have direct access to reality and our only access to it is via our senses. So when things change out there, our perception changes accordingly.
20/ For example, we perceive many objects as round because they share something that causes roundness to emerge in our perception for all of them. Similarly, different red things share something (a particular wavelength of light) which causes us to see red.
21/ In fact, finding out invariants of the world might be is what science is all about. Coming back to the example of a game, an expert player might master game mechanics better and better, but you know what will be even more effective at winning the game? Reverse engineering how the game works and directly interfering with its source code.
22/ We do not yet have a complete understanding of reality and I’m not even sure if we will ever have it. But the success of science in predicting more and more phenomena over time suggests that we’re understanding the game mechanics of how reality works better and better.
23/ All this sounds nice and logical, but how do we explain conscious experiences? Any description of reality must accommodate the subjective experiences we have of seeing red or smelling coffee. Where do these subjective experiences come from?
24/ An incomplete but important answer is that these perceptions are compressed representations of reality that got evolved for adaptive behavior. Color vision helped our ancestors pick ripe fruits from unripe ones. Fragrances told them in an instant what to avoid (bitter) and what to crave (sweet).
25/ That’s good but it appears that what we subjectively experience represents reality in a completely arbitrary way. (Although it’s an ongoing area of research and there may be some structure).
For example, try to guess which of the molecules is vanilla and which one is thyme.
26/ Can’t guess? That’s okay.
Thankfully, to operate in the world you don’t have to remember the molecular structure. Our brain directly translates the signals originating corresponding smelling these molecules into a distinct subjective experience which you can use to take action.
27/ It does this translation because the molecular structure of these molecules was irrelevant for our survival and reproduction. For success in an evolutionary competition, what matters is our actions and this compressed signal that there’s vanilla or thyme around is good enough for informing such actions.
27/ Donald Hoffman suggests that even the experience of space, time, and objects are adaptations honed by evolution. Perhaps, we perceive two things near in space if, given everything else being equal, they have similar (reproductive) fitness benefits for us? Perhaps our experience of time is a way to keep track of changes in fitness payoffs?
28/ Physical reality being an evolved illusion is what the theory Fitness beats Truth predicts.
It explains why we don’t perceive reality and how our perceptions might arise, but it doesn’t explain perception itself.
For example, it doesn’t explain why we must feel anything? From the inside, why does the pain feel real?
29/ Donald Hoffman has another theory for explaining qualia, which he calls conscious realism. He suggests that since the only thing we can be sure of is our inner consciousness, maybe what’s out there is just consciousness.
Maybe reality itself is comprised of only conscious agents? Maybe space, time, and everything else we perceive is a result of the dynamics of other conscious agents?
30/ This theory is a radical reinterpretation of reality and he and his fellow researchers are still developing it. Their present goal is to assume that simple consciousness agents connect with each other in a social network and explore if the emergent dynamics from their connection resemble the behavior of elementary particles such as gluons.
31/ Once they make one or few connections from the network of conscious agents to elementary particles (and perhaps even spacetime), connections to the rest of the physics will be simpler because most of modern physics is built on these fundamental concepts.
32/ The theory at first glance may sound nuts and perhaps even pseudo-science. But the theory will have testable predictions which we can use to prove or disprove it.
It is also worth remembering that almost all theories that we take for granted today sounded nuts when they were first proposed.
33/ I’m personally VERY excited about the promise for scientific viewpoints that connect “objective” reality and “subjective” experiences. With consciousness such a big mystery, I’m glad researchers like Donald Hoffman exist to push our understanding on these lines forward.
Join 150k+ followers
Follow @paraschopra