Know why: How to beat marketing survivorship bias
Ian Lurie Nov 18 2015
Why does marketing advice suck?
We all look for marketing tactics we can repeat. You know what I mean: Some guy writes an article about marketing strategy, and boom, everyone’s off to the races.
And 95% of the time, the tactics out of context fall flat on their metaphorical faces.
Using the latest marketing how-to is a lot like sex. Know the risks or bad things happen and bits fall off. Exercise proper precautions and the worst case is things go wrong and you learn something.
I wrote this while on my fifth long plane trip in 5 weeks. I’m a little punchy. My jet lag has jet lag. So if I just alienated/disgusted half my audience, I apologize. The guy next to me just read this over my shoulder and edged away. He gave me the armrest. WORTH IT.
Wow. Sounds like I’m saying “Ignore all marketing advice!!!!”
I’m not. I’m saying it’s easy to pull the wrong tactics from a marketing success story.
If you want to extract the right tactics take a step beyond “What do I do?” and ask “Why, really, did this work?”
The problem: Survivorship bias
There are lots of reasons marketing blueprints can get you into trouble. The one I most worry about is survivorship bias.
If we get all scientific about it, survivorship bias is our tendency to focus on and compare to techniques, people, etc. that ‘survive.’ We miss the failures because they’re less visible. It’s a common driver of the ‘correlation does not equal causation’ cliche.
In marketing: We rarely document work that deposits large, stinky turds on the Internet. We only write about the winners.
If we only write about the winners, then you only read about tactics as part of success stories.
Say I write about a retargeting campaign that worked really well for me. In it, I point out that I used orange buttons instead of green ones. Because of that, someone decides to use orange buttons in their retargeting campaigns.
But what if 99 other people ran retargeting campaigns with orange buttons and failed. No one reads about those. They only read about the survivor: My success story. They see the orange buttons and latch on to them as the most obvious tactic.
That isn’t necessarily bad. A strategy with a one percent success rate is better than a zero percent success rate. If you know what really made it work.
Survivorship bias often makes the wrong tactics look right. We see those tactics and stop digging.
So keep digging. Always ask ‘Why, really, did this work?’
Now, I’m going to demonstrate this whole theory by kicking the living feces out of my least favorite tactical cliches.
Infographics don’t work just because they’re infographics
Marketing campaigns now ooze infographics. Marketers puke them out all over the Internet. Why?
Our infographic addiction is based on survivorship bias: We see lots of successful infographics.
So we decided that they worked because they were infographics. But we’re only looking at the successes. Lots fail (this diagram is completely made up, and probably too generous):
We’ve been wasting budget dollars on them ever since. Every now and then, one succeeds and we all yell “OMG INFOGRAPHICS” and do even more.
Avoid the trap by asking “Why, really, did this work?” Marketing campaigns don’t succeed because they use infographics. They succeed for all sorts of crazy reasons:
- Relevant data
- Great design
- Good writing
- Good promotion and marketing
Ask “Why were these campaigns successful?” and you learn pretty quickly that the success stories are based on the cornerstones of great, successful content, not one specific type of content. You get at the tactics and techniques that really made it all work.
Knowing the Golden Ratio doesn’t make you a designer
I’m about as good at design as I am at disco. I use the Golden Ratio all the time to try to salvage the crap I crank out on a weekly basis. I use it for typography, layout and anything else I can.
My visual creative still makes babies cry.
Why? Every project I hear about that uses the Golden Ratio is a huge success.
Survivorship bias. Every project I hear about that uses the Golden Ratio is a huge success. The other 999,999,999 projects look about as good as mine.
Which is why my work still looks like the world’s ugliest dog, only less cute.
You can avoid my mistake by looking at why some of those projects using the Golden Ratio succeeded:
- Sense of color
- Audience understanding
- UX design
Asking “Why?” helps you dodge survivorship bias. It points you in the right direction: Instead of trying to replace design skill with the Golden Ratio, get a skilled designer. Then they can pick the right tool.
Ruby on Rails doesn’t make great web sites
In non geek-speak Ruby on Rails is a set of tools, built using a programming language called Ruby. It’s designed to build web sites.
When Ruby on Rails hit the development world, I ran to it. I practically drooled. I loved Python, but it was a terrible web site tool. I hated PHP, and sucked at it. I saw lots of beautiful sites built with this new toolset.
So I used Ruby on Rails to create a dynamic web site.
Weird. It still sucked.
The Ruby on Rails website showed the great sites built using the toolset. They didn’t show the other 99.99999% of sites so hideously worthless they could sterilize guinea pigs a mile away. Survivorship bias.
Turns out, all those Ruby on Rails sites worked well because great web developers created them. Not because they used Ruby on Rails.
If you see a bunch of successful sites built on a new development toolset, ask “Why, really, was this site successful?” Avoid my humiliation.
Before you flame me about testing and data
Multivariate and A/B testing helps us figure out which tactics really work. But don’t stake your campaign on those results. Use them as a guide.
Test a lot.
Collect lots of ‘passive’ data, too: Traffic, behavior, sales, whatever you can. Use it when you ask “Why did this work?”
Question why something worked, and dig well past the data.
Data can lead to survivorship bias, too, because we’re interpreting it. Never, ever forget that.
Before you flame me about failure stories
Some of us do write about spectacular marketing failures. Survivorship bias isn’t 100%. It’s a bias, not an absolute.
We don’t examine failures as closely as we should, though. So ask “Why, really, did this work?”
That’s all I’m sayin’.
Be a smart user of tactics
This all may sound hypocritical. I write lots of tactical posts. I read them and use them, too. They’re fantastic resources.
Just remember: We generally write about successes, not failures. That tactic you want to use may have nothing to do with the success story. Before you use it, ask “Why did this really work?” Avoid survivorship bias.
I borrowed a copy from Portent’s resident genius, Blake. It’s now tattered after my dragging it around the world. I’ve read some chapters two or three times.
If you’re serious about marketing, or investing, or history, or reality, read this book.
Get the Best of Portent’s Marketing Blog
Join thousands of fellow marketers!
Get Portent’s latest how-to, strategy, and training articles by email
CEO & Founder
Ian Lurie is CEO and founder of Portent Inc. He's recorded training for Lynda.com, writes regularly for the Portent Blog and has been published on AllThingsD, Forbes.com and TechCrunch. Ian speaks at conferences around the world, including SearchLove, MozCon, SIC and ad:Tech. Follow him on Twitter at portentint. He also just published a book about strategy for services businesses: One Trick Ponies Get Shot, available on Kindle. Read More