How to be a Bayesian

Just finished this book on the history of Bayes theorem and I highly recommend it.

In case you’re wondering what is it, keep reading.

(A thread on Bayes theorem)

1/ Statistics is all about calculating probabilities, and there are two camps who interpret probability differently.

Frequentists = frequency of events over multiple trials

Bayesians = subjective belief of the outcome of events

2/ This philosophical divide informs what these two camps usually bother with.

Frequentists = probability of data, given a model (of how data could have been generated)

Bayesians = probability of model, given the data

3/ Most often we care about the latter question.

E.g we want to know given that the mammography test is positive, what is the probability of having breast cancer.

And not given breast cancer, the probability of test being positive.

4/ These two questions sound similar but have different answers.  ...  Read the entire post →

How to think about risk

1/ The most common mistake with risk is NOT differentiating:

personal, unique risk

FROM

collective, average risk

2/ Personal, unique risk arises from things that are unique to you.

For example, personal financial risk is what incurs to you because of your peculiar investments. Maybe you picked certain stocks or invested in a “hot” property

3/ Contrast this to the collective and average risk of the entire nation’s economy tanking or a bank tanking and wiping savings of millions of people like you.

4/ Usually we end up conflating these two types of risks in our minds.

Many people don’t invest in equities because it’s too risky.

Yes, it’s risky but in a collective and average way, not a personal and unique way

5/ Protecting against collective risks is an exercise in vain because the collective risk is never eliminated, it’s simply gets shifted somewhere else

6/ Collective risks are less worrisome because collectives of people want status quo

So on the realization of such risks, you can take comfort in the fact that there are millions of other concerned folks who will protect the negative impacts of risk for you

7/ Case in point: the 2008 financial crash in the US.

At that time, everyone thought the world economy was in ruts and that people lost money.

Fast forward to today and thanks to bailouts, after the dip, here’s the US stock market performance. ...  Read the entire post →

On the strangeness of giving advice

Calvin never gives free advice! (Credits: Bill Watterson)

1/ Giving advice is a strange thing.

2/ First of all, let’s get this right off the bat: the advice-giver accrues MORE benefit from giving advice than the one who’s receiving it.

3/ When we give advice, our half-formed thoughts crystallize and tell us clearly and reinforce what we believe in.

The receiver, on the other hand, has the tough job of figuring out what we mean and then making changes to his/her life based on the few bits of info we give out.

4/ Giving advice also helps us find our tribe. People who give similar advice band together. This is why SF/VC culture is a cult.

5/ Giving advice legitimizes our weirdness.

If enough people give advice about saving time by drinking your meals, it’s no longer weird.

6/ Our rate of giving unsolicited advice >> rate of giving solicited advice.

Why do we poke people and ask them to change? It’s mainly driven by FOMO. By giving advice to others to live their life like we live, we want to ensure that they’re not living a superior life.

7/ Thanks to Twitter, our rate of giving unsolicited advice to complete strangers >> rate of giving advice to near and dear ones.

It’s as if by tweeting, we’re telling things to ourselves and hoping a fellow tribalist finds us so that we both can reinforce our views.

8/ Giving unsolicited advice to strangers was a job once limited to sages or madmen.

Now, it’s everybody’s business.

9/ It’s also interesting that we often give advice and move on. Unless we’re personally attached to someone, we rarely have skin the game to ensure the advice receiver changes.

The GIVE ADVICE -> MOVE ON -> GIVE ADVICE pattern helps us feel smart and helpful.

10/ That’s it.

Of course, this essay WAS unsolicited and I expect to benefit from it, one way or another. You’ll most likely forget about it in an hour and move on, but that’s OK.

Remember: giving advice benefits oneself more than it benefits the other 🙂 ...  Read the entire post →

Perils of binary thinking

Not winning does not imply losing.

(a thread on perils of binary thinking)

1/ We recently ran an A/B test and here were the results.

In the test results we still don’t have 95% statistical confidence (probability to be the best) but we’re going ahead and implementing the variation on homepage.

Why?

2/ Because, and here’s the key idea.

Not getting a winner is different than having a loser!

Allow me to unpack..

3/ Whenever we talk about test results, our immediate hunch is to jump and categorize into binary categories: either something is a winner or a loser.

But in real world, there are no binary categories.

4/ There’s no magic threshold at which a variation suddenly becomes a winner.

The 95% confidence threshold is arbitrary and whether you act based on it (or some other threshold) depends on your goals.

5/ So, even though the variation may not be good, we know that it’s likely that it’s not bad.

Rather than getting trapped into discussions about winner/loser, what matters is what is likely impact of implementing this on website.

6/ In best case, variation actually turns out to be good.

In the worst case, it is similar to existing one.

(In the extreme worst case, it can be a catastrophe but if you see the range of uplift, it is much more likely to be better than worse)

7/ With data, it’s easy to fall into the trap of binary decisions about data while what matters is likely impact.

KNOW the difference between probability and impact.

Read first chapter of this book.

8/ TLDR: there’s negligible probability that that we’d have a catastrophic super-solarflare that will fry up our world’s electronics and internet.

But if that happens, it’ll be really, really bad.

9/ That’s it!

Hope you liked this mini-thread.

The take-home message is to not get fooled by probabilities by always keeping focus on consequences of probabilities. ...  Read the entire post →

Twyman’s law

Twyman’s law states that any data or figure that looks interesting or different is usually wrong.

Sounds unbelievable, isn’t it? 

But, it’s true. I saw this in action recently and wanted to share that story with you.

In June, we ran a test on our homepage and while I was looking at conversion rate by segments, I noticed that users from Windows had a 400% higher signup rate for VWO free trial as compared to users using Mac OS X.  

Now, that’s baffling and our team spent a good deal of time trying to understand why was that happening. Someone in marketing hypothesized that perhaps Mac OS X users have a better design aesthetic and our homepage wasn’t appealing to them. Was it true? ...  Read the entire post →

Horizon based thinking

Whenever you’re at a crossroads and are wondering what to do next, here’s how to think.

1/ The first thing to ensure is that you acknowledge that a lot of your decision-making will be driven by how you’re feeling *right now* and not how you’ll feel once you choose a path.

The need to escape can make us choose things that we won’t like long term.

2/ The way to keep emotion-driven decision at bay is to *force* yourself to think in three horizons:

1. Short term goals
2. Long term goals
3. The path between the two

3/ Yes, I know this sounds trivial but far too often:

– We’re either hyper-focused on short term and optimize our way to getting stuck into a local-optima
– Or, we’re hyper-focused on the long term and forget that present actions is how we get to the future ...  Read the entire post →