This blog is about random, independent and probably highly insignificant, non normal non-stationary statistical meanderings. The last I calculated it, the probability that anyone could finish reading such random stuff was about 0.00034.
So, there I was. In the midst of that terrific traffic chaos in Pune. 6:00 p.m. People, tired from the day’s work in office, wanting to get home. Mothers, with two little ones on scooty peps. School kids chattering dime to dozen on their bicycles, barely noticing the smog and laughing delightedly in a way that only a “school-is-over” event can bring on. In the rickshaw next to my car was an old man looking agitatedly at his watch and urging the rickshaw wallah to hurry. Hope nothing’s wrong, I thought. Hawkers selling roses and kiddie toys looking shrewd and snappy about the traffic jam; an extra 4 minutes at the signal means brisk business. The signal turned green. And the traffic moved.
In different moods and colors, to different destinations from different start points, with a sense of happiness and with irritation, the traffic moved as one. One rhythm, one direction, one sense of purpose. Radio Mirchi squawked “Traffic moving slowly and normally through Karve Road”…Eh? What was that? Normally?
Why is traffic normal, when there’s nothing normal about the individuals who are its components? A gentle tap on the window and I saw those twinkling eyes giving me my answer. Oh my God, are you really George Polya? Yes, he nodded at me gently. And then said the sentence that absolutely put to rest any doubt I could ever have about his identity. “I thought I am not good enough for Physics and too good for Philosophy. Mathematics is somewhere in between.” As Polya settled down in my car, his usual spry animated self, the answers dawned on my mind. The Central Limit.
I mean, look at all around you. You find those large number of random, independent variables, each with their own distribution, each going through the motions of love, happiness, stresses and tensions. People. Random variables. And yet, the summation of the populace, life in general, moving normally. Wow, I breathed softly, hats off! Polya, you crazy genius, in which fit of madness did you coin the words “Central Limit Theorem” to explain the most fundamental theorem in probability statistics. Its not just probability statistics; time and again I connect to the Central Limit and the Law of Large Numbers as theorems that hold the explanation to the process of life itself.
As life passes by, as the days go by, we find that we all tend to an average way of life, some average thoughts, some average standard of living. Sure, each person defines his own average differently. But the fact is that as the days collect into our kitty, we start living the Law of Large numbers. On a particular day, life may treat us rather well or unkindly; but the average life of the average person is on an average…average! How “mean”, you may well say, but there you are! Such is life; defined by the Law of Large Numbers.
I like to think of life as a time series; mostly non-stationary. It rejects almost all attempts to stationarize it. Should you manage to, it mostly turns out to be an ARMA process, part Auto-Regressive (anthropologists love to call this genes), part made up of a Moving Averages of past errors (and the psychologists call this environmental adaptation). Predicting the future (how rum is it that the master of forecasting should be named Box; this is one guy who is definitely out-of-box!) requires an artistic, atheoretical and unique approach and mostly is riddled with crazy levels of failure. And yet, most of us, wizards to muggles, want to study and understand the mad art and science of forecasting. They call it divination, and we, astrology. “Mars is bright tonight.” We try to fit patterns to the lines on the forehead and assess the statistical probability of combinations of stars creating situations of a particular type for us. By the way, did you know that the Ordinary Least Squares as a method of minimizing errors was suggested by Gauss in his work on movement of celestial objects? There you go. All statisticians have been enamored by it. The movement of the stars. Their effect on our lives.
Gauss. The other genius. He is always around in my life too. The Gaussian wrackspurt, if you will….
The other day in college, end-of-the-session babble broke out in the classroom as I wound up the discussion. And suddenly a student asked me a question. Such was the babble that I couldn’t really hear him. And merely guessing what he was saying, tried to give a rational answer to his question. His look told me that I had inferred wrong. A gentle tap on the door .“Inference. It always goes wrong when there’s way too much noise around you”. I swung around to see him walking into my class. The babble was still on; but my mind had turned into a silent zone. There was total silence. Till he decided to talk. Gauss. The Greatest Mathematician since Antiquity.
“To hear the question and give the answer: That’s estimation, my dear. But, to interpret the question correctly. Ah, that is the art and science of inference. If the model errors are too vocal, exhibiting auto-correlation or heteroscedasticity noise, the inference on the estimates goes wrong.”
How did you know, my mind was asking. How does the solution come to you…Does it come in a flash of lightning, or is it a slow, prolonged process.. “I have had my results for a long time: but I do not yet know how I am to arrive at them.”
A good poem and a rainy day. Can’t get better than that. A poem titled “Yevgeny Onegin”by Pushkin. Hmmm, on the continuation of life even as death takes us over individually. And what’s this? Markov’s commentary on Onegin? Andrei Markov. Really? I mean, could this be THE A. Markov? As in, the Markov chains guy? What in the world does HE have to do with Pushkin now? Quite a lot, as the story goes.
We go back in time to meet Pushkin and Markov. In the tsarist Russia of late 1890s. At that time, statistics had developed to a point where probabilities of sequences of events were being estimated as the product of the two events happening independently. So, if a day can be sunny or rainy, the P(Day is sunny) = 0.5 and P(Day is sunny when earlier day was sunny) = 0.5*0.5 = 0.25. But, events are usually dependent, aren’t they? So, if today is sunny, doesn’t it impact the probability of tomorrow being sunny? Isn’t it true that probabilistic chains necessarily need an intertemporal dimension? The probability of an event today HAS to depend on the event occurring in the most recent past, a little lesser into more distant past and so on. And therein lies the genesis of Markov chains.
Markov was a man possessed in proving the interdependence of chains of events in a temporal sequence. His choice of a sequence to prove interdependence? The poem “Onegin”by Pushkin. Oh, this is sublime!
He chose the first 20,000 letters in the poem arranging them without punctuation or breaks, counting 8,638 vowels and 11,362 consonants in the process. There were 1104 vowel-vowel pairs, wherein a vowel follows another. Now, if occurrence of a vowel in the sequence is independent of what occurs earlier, then, the P(vowel-vowel pair occuring) = (8638/20000) * (8638/20000) = 0.19. That implies that in 20,000 letters, such combinations should have occurred 3731 times, nearly thrice the 1104 number of times that they occurred!
Wow, wow and more wow! This necessarily means that independence of vowels stands rejected! Thus, letters in a word are NOT independent you see; there is an overwhelming tendency for vowels to alternate with consonants in a language! The same logic has hence been used in applications as diverse as understanding motions of gas particles to creating the Google algorithm.
Onegin may be a poem par excellence; but it will go down in the annals of history as the literary masterpiece that served as a workhorse for a uber creative mathematician. How befitting that Onegin should talk about continuity; because if there is one theme that Markov explored, it was one of continuity and interdependence…A gentle tap on my shoulder…I need to go. It’s Markov.