Volatility
It took me way too long to wrap my head around the concept of variance.
In the simplest terms, variance provides a measurement on how far a set of numbers are spread out. It effectively compresses the differences - whether you have one thousand or one million numbers, the variance is always one number.
Many years ago, Archimedes took up a challenge to count the grains of sand necessary to fill the earth. In tackling this problem, he came up with his own bounded number system:
"He used his system to calculate the number of grains of sand that would be needed to fill the entire celestial sphere. This suggests that he, and ancient Greek culture in general, may not have had the concept of an abstract number at all, so that, for them, numerals could refer only to objects – if only objects of the imagination. In that case universality would have been a difficult property to grasp, let alone to aspire to. Or maybe he merely felt that he had to avoid aspiring to infinite reach in order to make a convincing case."
All number systems are a tool that allows us to better reason about the world. And they are never perfect, it's clear that there were things wrong with Archimedes' number system: numbers aren't really bounded, but it was useful nonetheless.
Variance is built on top of many of our other tools, aka abstractions, already. Its necessity is a bit of a contradiction - humans have a difficult time reasoning about large sets of numbers, so we've come up with average and variance and other forms of compression. But the contradiction is that every layer we've built on top of each abstraction actually brings us further from reality: there is no true "average", there is no perfect square, and the number pi doesn't really exist.
This makes it so that the variance of a large population and the relevant prerequisite knowledge (expectation and random variables) are so far removed from a physical analogue.
My first encounter with variance was through a statistics course. The mathematical definition is:
This is the difference between the squared expectation and expectation squared. Expectation is just a fancy way of saying the average accounting for the probability of each number happening. School does a poor job helping students decipher this - exams emphasize rote mechanical work (calculate the variance of so-and-so population) or even more abstract concepts (prove that variance isn't a linear operator). I recall spending hours doing practice problems and memorizing proofs. Looking back, a deeper understanding would probably have made the memorization easier.
As a tangent, this complaint applies with many other math courses, and in particular with linear algebra. Eigenvalues and diagonalization meant nothing at the time but if I saw how they were used in recommendation systems, I probably would have paid much more attention.
Most of us are exposed to variance much earlier in life. When we do anything or we make any decision, there's always the possibility that something goes wrong. Humans, as the risk averse and loss averse creatures we are, place more emphasis on things going wrong. We call this risk, which is really just variance in the unwanted direction. Since the beginning of time, we've developed heuristics to manage risk: bring an umbrella in case it rains, save more than you spend. Risk has a simple solution: optionality. Optionality gives us choices for whatever we want whether that's leaving that job that isn't fulfilling anymore or choosing to take our vacation a week later. And once we have optionality, variance is a big positive, we can selectively choose to be only exposed to the upside. High variance is polarizing - you either love it or you hate it without much of a middle ground.
In my life, I've had a number of things go terribly wrong. The potential risks behind major decisions I've made have occurred. Alfred Krozybski, a Polish scientist once said "the map is not the territory", and I often reflect on this when something does go wrong. Variance, risk, and bad outcomes are a product of some process: whether that's rolling a dice, the macroeconomic cycles, or tectonic plates shifting. The map in this case is our understanding of the different factors and the territory is the reality - sometimes there's a sharp disconnect, and that's where things go wrong. Sometimes we don't have a grasp on every relevant variable or the process itself, which is why probability and random variables can be so powerful. A generalization is that the more complicated the concept, the further it takes us from reality, and the more caveats there are to piece the map with the territory. Science allows us to understand the factors, and probability allows us to do our best when the factors aren't available.
To illustrate this notion of probability's utility when factors are unknown, think about what the odds of heads are when you flip a coin: 50/50. But what if you knew the direction of the wind, the vector and magnitude of the flip itself, the surface it lands on and all the other relevant factors? The flip itself would be more deterministic: you'd know more precisely whether it would land heads or tails.
The understanding of how inputs and processes relate to the outcome is science. Science allows processes to converge to be deterministic: if we buy stocks and treasuries, our portfolio will be much less volatile since their prices are generally inversely correlated. Agricultural science has progressed to where we can have consistent crop yields regardless of the forces of nature. Society continues to research health and well being to limit the worst case for any given disease or condition.
Variance is an abstraction - it is a generalization around a set of numbers. Perhaps the goal of formal education is to provide you with the tools and the prerequisite knowledge to think through abstract concepts. You can give a man a map, and you can test him on it. But you can't make him explore the territory.