Saturday, January 14, 2012

The marvels and the flaws of expert intuition: story of Ramanujan’s first letter to Hardy

Malcolm Gladwell’s “Blink” presents the marvels of expert intuition and Nassim Taleb’s “The Black Swan” highlights the flaws of expert intuition. Well, Srinivas Ramanujan’s first letter to Hardy exemplifies both – the marvels and the flaws of expert intuition. Let’s see how.

Genius or fraud? “I beg to introduce myself to you as a clerk in the Accounts Department of the Port Trust Office at Madras on a salary of only Rs. 20 per annum.” Thus began the letter sent by 25 year old Srinivasa Ramanujan to Hardy, the then renowned Cambridge professor of mathematics. It was dated 16th January 1913. The nine page letter contained around fifty mathematical results claimed by the Indian clerk. Ramanujan had explained in the letter “I have had no University education but I have undergone the ordinary school course.” What struck Hardy at the first glance was the strangeness of the theorems, not their brilliance. His first reaction was: is he a genius or a fraud?

The letter: The letter contained theorems in number theory, calculus, infinite series etc. Some of the formulae were familiar to Hardy, while others “seemed scarcely possible to believe”. For some reason, Hardy decided to consult his fellow mathematician Littlewood to reach a verdict on whether the author is a genius or a fraud. The duo spent three hours – from 9pm till midnight – going through the manuscript. At the end they concluded that its author is “a mathematician of the highest quality, a man altogether exceptional originality and power”. Some of the results Ramanujan sent to Hardy in his letter belonged to breakthrough category – Hardy was to conclude later. Hardy persuaded Ramanujan and the government to bring him to Cambridge.

The marvel of intuition: Ramanujan’s Indian friend in Cambridge, Mahalonobis who later went on to found the Indian Statistical Institute once asked him how he got a startling result. Ramanujan said, “Immediately [after] I heard the problem it was clear that the solution should obviously be a continued fraction; I then thought, which continued fraction? And the answer came to my mind.” The answer came to my mind sums up how Ramanujan got many of his results. Question is: How did answers come to Ramanujan? Did they get handed down by the goddess Namagiri of Namakkal in his dreams as some believed?

Ask yourself “What is 2 x 2 = “ and now ask yourself “What is 19 x 37 =” An answer would come to your mind immediately in the former case while nothing comes to your mind immediately in the latter case. Daniel Kahneman explains in “Thinking, fast & slow” that there are two different modes of thinking: a fast mode and a slow mode. The fast mode (called system-1) is intuitive, effortless and automatic while the slower mode (called system-2) is rule-based, effortful and controlled. You could choose to start the computation of “19 x 37” using the rules of multiplication learnt in school and also stop in between if you wish to do so (slow mode). You had no such control for “2 x 2”. The answer came to mind automatically (fast mode).

For an expert, such as a chess master, when he looks at a board position, several patterns and associated offensive and defensive moves come to mind immediately (like “2 x 2”). They come from a vocabulary of 50,000 to 100,000 patterns stored in memory. Psychologists believe that it takes 10,000 hours (about 6 years of playing chess 5 hours a day) to reach this level of expertise (or active vocabulary). Had Ramanujan done this kind of practice?

By 1900 (age 12) Ramanujan had mastered S. L. Loney’s Trigonometry. However, the year 1903 was a turning point when he found a book written by Carr containing 5000 equations related to algebra, trigonometry, calculus, analytical geometry, differential equations. From this point, life of Ramanujan who graduated from school that year with a reputation of being “off-scale” went off-balance. He didn’t do anything other than mathematics. He failed four attempts to pass in college in 1904, 1905, 1906 and 1907, lost scholarship and didn’t do well even as a math tutor. And yet, this five year period of 1904 to 1909 was the most productive time in his life because, in all likelihood, he was at home manipulating equations in all of his waking hours – on a slate, on a paper and in the mind. It is also quite possible that his fast mode of thinking was so active that during sleep it continued to work on the half-solved problems. The fast mode is known to be working even when we are not conscious. Ramanujan had easily finished his quota of 10,000 hours well before 1909.

Did his intuition ever trick him? Yes, it did, occasionally and it showed in his first letter to Hardy as well.

The flaw of intuition: In his first letter, Ramanujan had written that he had found a function which exactly represents the number of prime numbers less than x. After Hardy demanded a proof, Ramanujan sent one. And Littlewood figured out that it was wrong. Well, Ramanujan’s formula wasn’t bad. It was off by only 53 for calculating primes up to first nine million (which are 602,489). However, the difference became bigger as numbers grew larger. And this is what Ramanujan failed to see.

Checking a sketchy proof rigorously, at least partly, is a responsibility of the slow thinking mode. As Kahneman observes that the slow thinking mode is inherently lazy. If an answer given by the fast mode looks fine, it just endorses the verdict. In Ramanujan’s case the slow mode was not just lazy, it was weak. Because he never formally learnt how to write a rigorous proof until he reached Cambridge. The upside of this characteristic was that the fast mode was freerer than usual in its meanderings – making him highly creative in his approach.

What was interesting about Ramanujan was not that his intuition was sometimes wrong. Even a great mathematician like Andrew Wiles also had a serious hole in his first proof of the Fermat’s Last Theorem. What was different about Ramanujan before coming to Cambridge was that he was equally confident when he was right and when he was wrong.

Two key learnings from this story: One, a rich vocabulary of several hundreds of thousands of patterns resulting from a prolonged practice and timely invocation of appropriate patterns by the fast mode of thinking creates expert intuition. Two, knowing when not to trust the instincts is an important characteristic of a true expert. Ask yourself, "Which are the possible situations in which my instincts could be on a slippery ground?"

Sources:

The man who knew infinity by Robert Kanigel, Washington Square Press, 1991. (Thanks to my friend Ramprasad Moudgalya for recommending the book to me).

Conditions for intuitive expertise, Daniel Kahneman & Gary Klein, American Psychologist, Sept 2009.

Related articles:

My most favorite YouTube video and the marvels and the flaws of intuition, Sept 19, 2010 (On Kahneman's lecture at Berkeley on the same topic).

Impossible problems and successful approaches: story of Fermat's Last Theorem, Oct 28, 2008 (On Andrew Wile's successful attempt at solving 350 old most famous open problem).

4 comments:

  1. Very interesting article.

    The problem you have hinted at is faced not just by experts but, even more so, by those who hire experts. How do we know when an expert is outside their circle of competence and hence we should ignore them?

    Since there is a big halo effect around experts (think: nobel prize winners), anything they say - even outside their area of immediate expertise - is taken as gospel truth.

    Your thoughts?

    ReplyDelete
  2. Ravi,

    For hiring decisions, here is what Kahneman has to say and I like it (from “Thinking, fast & slow", chapter 21, pg 232). It is quite rare for Kahneman to write “This is what you should do” kind of advice:

    --- Kahneman's advice ------

    Suppose that you need to hire a sales representative for your firm. If you are serious about hiring the best possible person for the job, this is what you should do. First, select a few traits that are prerequisites for success in this position (technical proficiency, engaging personality, reliability, and so on). Don’t overdo it - six dimensions is a good number. The traits you choose should be as independent as possible from each other, and you should feel that you can assess them reliably by asking a few factual questions. Next, make a list of those questions for each trait and think about how you will score it, say on a 1-5 scale. You should have an idea of what you will call “very weak” and “very strong”.

    These preparations should take you half an hour or so, a small investment that can make a significant difference in the quality of the people you hire. To avoid halo effects, you must collect the information on one trait at a time, scoring each before you move on to the next one. Do not skip around. To evaluate each candidate, add up the six scores. Firmly resolve that you will hire the candidate whose final score is highest, even if there is another one whom you like better.

    You are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go into the interview unprepared and to make choices by an overall intuitive judgment such as “I looked in his eyes and liked what I saw”.

    ReplyDelete
  3. Found this article by one of my favorite authors Chip & Dan Heath Why It May Be Wiser To Hire People Without Meeting Them. They use the term "interview illusion" to refer to the certainty we feel about learning about the person when we interview him/her. The comments followed by the article are informative too.

    ReplyDelete
  4. Applause for this lovely text which I found by searching for Ramanujan and the 10,000 hour rule. Hmmm I am going to estimate the hours spent being taught religion and math in my religious elementary school, and consider the consequences thereof. Both subjects have been volcanoes in my life for decades, in which I have worked at exporting religion and importing math, post-elementary school.

    ReplyDelete