Know why: How to beat marketing survivorship bias

Ian Lurie Nov 18 2015

survivorship bias jupiter and io

Why does marketing advice suck?

We all look for marketing tactics we can repeat. You know what I mean: Some guy writes an article about marketing strategy, and boom, everyone’s off to the races.

And 95% of the time, the tactics out of context fall flat on their metaphorical faces.

Using the latest marketing how-to is a lot like sex. Know the risks or bad things happen and bits fall off. Exercise proper precautions and the worst case is things go wrong and you learn something.

I wrote this while on my fifth long plane trip in 5 weeks. I’m a little punchy. My jet lag has jet lag. So if I just alienated/disgusted half my audience, I apologize. The guy next to me just read this over my shoulder and edged away. He gave me the armrest. WORTH IT.

Wow. Sounds like I’m saying “Ignore all marketing advice!!!!”

I’m not. I’m saying it’s easy to pull the wrong tactics from a marketing success story.

If you want to extract the right tactics take a step beyond “What do I do?” and ask “Why, really, did this work?”

The problem: Survivorship bias

There are lots of reasons marketing blueprints can get you into trouble. The one I most worry about is survivorship bias.

If we get all scientific about it, survivorship bias is our tendency to focus on and compare to techniques, people, etc. that ‘survive.’ We miss the failures because they’re less visible. It’s a common driver of the ‘correlation does not equal causation’ cliche.

In marketing: We rarely document work that deposits large, stinky turds on the Internet. We only write about the winners.

If we only write about the winners, then you only read about tactics as part of success stories.

Say I write about a retargeting campaign that worked really well for me. In it, I point out that I used orange buttons instead of green ones. Because of that, someone decides to use orange buttons in their retargeting campaigns.

But what if 99 other people ran retargeting campaigns with orange buttons and failed. No one reads about those. They only read about the survivor: My success story. They see the orange buttons and latch on to them as the most obvious tactic.

That isn’t necessarily bad. A strategy with a one percent success rate is better than a zero percent success rate. If you know what really made it work.

Survivorship bias often makes the wrong tactics look right. We see those tactics and stop digging.

So keep digging. Always ask ‘Why, really, did this work?’

Now, I’m going to demonstrate this whole theory by kicking the living feces out of my least favorite tactical cliches.

Infographics don’t work just because they’re infographics

Marketing campaigns now ooze infographics. Marketers puke them out all over the Internet. Why?

Our infographic addiction is based on survivorship bias: We see lots of successful infographics.

So we decided that they worked because they were infographics. But we’re only looking at the successes. Lots fail (this diagram is completely made up, and probably too generous):

Successful Infographics: Hard to spot

Successful Infographics: Hard to spot

We’ve been wasting budget dollars on them ever since. Every now and then, one succeeds and we all yell “OMG INFOGRAPHICS” and do even more.

Avoid the trap by asking “Why, really, did this work?” Marketing campaigns don’t succeed because they use infographics. They succeed for all sorts of crazy reasons:

  • Relevant data
  • Great design
  • Good writing
  • Good promotion and marketing

Ask “Why were these campaigns successful?” and you learn pretty quickly that the success stories are based on the cornerstones of great, successful content, not one specific type of content. You get at the tactics and techniques that really made it all work.

Knowing the Golden Ratio doesn’t make you a designer

I’m about as good at design as I am at disco. I use the Golden Ratio all the time to try to salvage the crap I crank out on a weekly basis. I use it for typography, layout and anything else I can.

My visual creative still makes babies cry.

Capybara Kingdom

See! I used the Golden Ratio!!!!

Why? Every project I hear about that uses the Golden Ratio is a huge success.

Survivorship bias. Every project I hear about that uses the Golden Ratio is a huge success. The other 999,999,999 projects look about as good as mine.

Which is why my work still looks like the world’s ugliest dog, only less cute.

You can avoid my mistake by looking at why some of those projects using the Golden Ratio succeeded:

  • Talent
  • Sense of color
  • Talent
  • Audience understanding
  • Talent
  • UX design

Asking “Why?” helps you dodge survivorship bias. It points you in the right direction: Instead of trying to replace design skill with the Golden Ratio, get a skilled designer. Then they can pick the right tool.

Ruby on Rails doesn’t make great web sites

In non geek-speak Ruby on Rails is a set of tools, built using a programming language called Ruby. It’s designed to build web sites.

When Ruby on Rails hit the development world, I ran to it. I practically drooled. I loved Python, but it was a terrible web site tool. I hated PHP, and sucked at it. I saw lots of beautiful sites built with this new toolset.

So I used Ruby on Rails to create a dynamic web site.

Weird. It still sucked.

The Ruby on Rails website showed the great sites built using the toolset. They didn’t show the other 99.99999% of sites so hideously worthless they could sterilize guinea pigs a mile away. Survivorship bias.

Turns out, all those Ruby on Rails sites worked well because great web developers created them. Not because they used Ruby on Rails.

If you see a bunch of successful sites built on a new development toolset, ask “Why, really, was this site successful?” Avoid my humiliation.

Before you flame me about testing and data

Multivariate and A/B testing helps us figure out which tactics really work. But don’t stake your campaign on those results. Use them as a guide.

Test a lot.

Collect lots of ‘passive’ data, too: Traffic, behavior, sales, whatever you can. Use it when you ask “Why did this work?”

Question why something worked, and dig well past the data.

Data can lead to survivorship bias, too, because we’re interpreting it. Never, ever forget that.

Before you flame me about failure stories

Some of us do write about spectacular marketing failures. Survivorship bias isn’t 100%. It’s a bias, not an absolute.

We don’t examine failures as closely as we should, though. So ask “Why, really, did this work?”

That’s all I’m sayin’.

Be a smart user of tactics

This all may sound hypocritical. I write lots of tactical posts. I read them and use them, too. They’re fantastic resources.

Just remember: We generally write about successes, not failures. That tactic you want to use may have nothing to do with the success story. Before you use it, ask “Why did this really work?” Avoid survivorship bias.

Required reading

To learn more about survivorship bias and other fine things, read Fooled by Randomness by Nassim Taleb.

I borrowed a copy from Portent’s resident genius, Blake. It’s now tattered after my dragging it around the world. I’ve read some chapters two or three times.

If you’re serious about marketing, or investing, or history, or reality, read this book.

 

 

Get the Best of Portent’s Marketing Blog

Join thousands of fellow marketers!
Get Portent’s latest how-to, strategy, and training articles by email

5 Comments

  1. Thanks for the read. It’s always a good thing to be sobering up about “the next big thing”. I recently read an article about how SaaS startups need to schedule at least 24 months before any success is made along with some sound case studies, and one person commented “surely we can do it faster”. Yes, surely we can, but the bottom line is that this mentality of “we can do it faster” is most likely built upon what you’ve written: the survivorship bias. If you read about 58 SaaS startups’s successes over the last year but never get involved in their daily running, it’s understandable that you’d carry a belief that it didn’t take them all an average of 24 months to build up to that success.

    • I’m reading that book to death. It’s amazing.

  2. This book is truly awesome :)

  3. Marc Gulinski

    Marc Gulinski

    This is a nuanced and refreshingly useful insight. Hope to see more book references (so much more productive then having to sift through endless marketing blogs). Thanks!

Comments are closed.