All posts by Paras Chopra

Not winning does not imply losing.

(a thread on perils of binary thinking)

1/ We recently ran an A/B test and here were the results.

In the test results we still don’t have 95% statistical confidence (probability to be the best) but we’re going ahead and implementing the variation on homepage.

Why?

2/ Because, and here’s the key idea.

Not getting a winner is different than having a loser!

Allow me to unpack..

3/ Whenever we talk about test results, our immediate hunch is to jump and categorize into binary categories: either something is a winner or a loser.

But in real world, there are no binary categories.

4/ There’s no magic threshold at which a variation suddenly becomes a winner.

The 95% confidence threshold is arbitrary and whether you act based on it (or some other threshold) depends on your goals.

5/ So, even though the variation may not be good, we know that it’s likely that it’s not bad.

Rather than getting trapped into discussions about winner/loser, what matters is what is likely impact of implementing this on website.

6/ In best case, variation actually turns out to be good.

In the worst case, it is similar to existing one.

(In the extreme worst case, it can be a catastrophe but if you see the range of uplift, it is much more likely to be better than worse)

7/ With data, it’s easy to fall into the trap of binary decisions about data while what matters is likely impact.

KNOW the difference between probability and impact.

Read first chapter of this book.

8/ TLDR: there’s negligible probability that that we’d have a catastrophic super-solarflare that will fry up our world’s electronics and internet.

But if that happens, it’ll be really, really bad.

9/ That’s it!

Hope you liked this mini-thread.

The take-home message is to not get fooled by probabilities by always keeping focus on consequences of probabilities. ...  Read the entire post →

Whenever you’re at a crossroads and are wondering what to do next, here’s how to think.

1/ The first thing to ensure is that you acknowledge that a lot of your decision-making will be driven by how you’re feeling *right now* and not how you’ll feel once you choose a path.

The need to escape can make us choose things that we won’t like long term.

2/ The way to keep emotion-driven decision at bay is to *force* yourself to think in three horizons:

1. Short term goals
2. Long term goals
3. The path between the two

3/ Yes, I know this sounds trivial but far too often:

– We’re either hyper-focused on short term and optimize our way to getting stuck into a local-optima
– Or, we’re hyper-focused on the long term and forget that present actions is how we get to the future ...  Read the entire post →