Just Measure It: How Measuring and A/B Testing can Unfuck your Marketing

Transcription

Siobhan: Welcome to the only actionable podcast to help you unfuck your marketing and run a business that gives a shit. I'm your host Siobhan and this is Marketing Unfucked.

Today's conversation is all about A/B testing and the lack thereof with Dennis van der Heijden. Let’s do this!

Hi Dennis, how do we unfuck marketing?

Dennis: We are going to measure everything from now on.

Siobhan: And how is that going to help us?

Dennis: Because the idea is that if you don't measure it, you don't know what you're doing at all and you're just going blind.

Siobhan: Okay, so we're going blind…Are you implying that most of us are going blind or that a lot of us are seeing the wrong things?

Dennis: Most of the marketing you do is hard to measure, and it is terribly complicated to measure things. But you should really try and do it. If you don't do it, you must accept that you're going blind and not pretend that you know what you're doing… because you don't.

Siobhan: So, the problem is that we are pretending?

Dennis: I think we all have too much of an ego and we need to be humble. Like, we all don't know what we're doing. We're trying to do our best.

Siobhan: Okay, so what are some examples of going blind or where we pretend that we aren't?

Dennis: I took a course recently from somebody that works at Microsoft. And Ronnie, the course leader, said they were testing a lot - like thousands of experiments a month. That's the stuff that's happening at Microsoft. A/B testing (for people who don't know), is when you do an experiment where you’ve changed one or multiple items on your website (or apps or whatever the case may be) and you're trying to guess which of the things that you're doing will work.

That's why we hired very smart people that know conversion optimisation. It's an industry that you also know quite a bit about. In that space, one out of 10 of these experiments is winning. That's it. That's it. There were tests that nobody looked at. There was like ‘Can I get a small change on an ad format in Bing on the right-hand side?’ Nobody wanted to take it. I think it was something about bolding the items on the line. It's kind of ridiculous. Then the engineers, after several months, said, ‘I'm looking at the backlog of seeing this little thing, I'll pick it up, it's like an hour work so let me go and do that.’

That's a 12% lift in their revenue. That’s $100 million in the US only. So basically, you're having 70 smart people in Bing working on the experimentation team. Everybody thinks this thing is not going to work. And you're like, ‘Oh my god, we have to look for winners like that, like these golden gems,’ and stuff like that.

I wanted to say here… you don't really know what's working. You, me, and the other person on the other side of this podcast, probably doesn't know what's going to work.

Siobhan: Well, we don’t know what’s going to work.

Dennis: Exactly, you don't know what's going to work. You don't have to throw spaghetti at the wall all the time. But you have an idea… and that's okay. But with that idea, you need to run an A/B test.

If you can do that, go for it. I don't want to rant about the whole thing of billboards, but I think we crossed the time where we had billboards on the road that was sold by very slick people that said they would convert because they could measure it - and they sold a billboard. I think we passed that time quite a long time ago. So, we have to measure it and we have to be humble, we have to keep trying, and you have to learn from things.

If you don't, you're going to lose out on the game. I think marketing by throwing spaghetti in the wall is and at the same time, thinking that you know what you're doing - both approaches are wrong, I think.

Siobhan: So then which approach is right? Because going blind or in action is wrong, but thinking you know what you're doing and having an ego is wrong. So, what's the happy medium?

Dennis: Ah, we should all chill and go to Bahamas! We should change profession.

I think you must have a hunch, right? That’s what you and I do, we have a hunch. It can be based on something we've seen somewhere else, something you read, something that your call centre told you, something you heard in the survey, that's the base, it’s enough to get started.

Siobhan: But isn't that measurement? If you start looking at user surveys, and you start looking at data to get your hunch, isn't that essentially measurement? Don't you need to apply some human aspect to measurement to make sense of it?

Dennis: Yeah, this is called quantitative measurement. You have to start somewhere. I can't quantify it that. You can't say that interview, customer, or complaint was 70% in the right direction. We have a direction, we just don't know if it’s right because you don't know if that one customer represents many, or maybe the whole world, but you must try.

You have a gut feeling that this is going to work, and with gut feelings, you can start. I think most people have a good business sense about their industry. That doesn't mean that's the truth. You should go and apply it but have something we call the roll back. Have something that is not applying it. And then you have a part of people that see that new version, or that new thing that you think is going to work, and another group that doesn't see it. And then you have a controlled experiment. That's why booking is winning. That’s why Netflix is winning and that's why Amazon is winging. They do a lot of other things too, that’s not the only thing.

They have the approach right where you roll something out, checking it against your original and, if it doesn't work, (which is usually the case), you roll it back. Then, you analyse and learn something. So, I think that the fine detail is should we do in action? No, we should move forward. It’s okay to have a hunch. Maybe some people's hunches are better than others. But I think then we should A/B test it. But you know what, I sell A/B testing software so I'm pledging I'm pledging my own product. But you don't have to buy my product, by the way, there's tonnes out there.

Siobhan: There are tonnes of alternatives. But Amazon and Netflix - these are mega companies. they've got the traffic to test. How does a smaller business measure?

We all know that you need a certain amount of traffic to measure and sure, you can get creative. But I'm not sure you'd call that measurement anymore. So, what do you do if you aren't there yet, but you need to measure and not be running blind?

Dennis: Well, I think there's another way to measure. If you look at time, and you have actions, and you can go to your Google Analytics and make a pin on the moment you took a particular action. Then, you go and look two weeks later and see if that action improved something. Is that pure? No. Lots of external events can happen, right? You can have another campaign running and email marketing and stuff like that. But if you look at source data, what's the source of that traffic, you can get an idea that this might work. Until you're bigger, you can start doing it like this.

I guess this is what entrepreneurs are doing anyway if they're not having traffic. You’re trying to find out what the next best thing is with the least amount of budget (or the highest amount of outcome), and you rank those in your head. If people are curious, there's lots of frameworks out there where you can put prioritisation in your marketing efforts. They measure the potential outcome and the effort you need to make and then you give them a number and you rank them. It’s easy to move the highest things to the top. But, looking at your historical data and pinning things that you've done in that period can get you a hunch.

Siobhan: But you're still working with a hunch and on top of that, you have to make sure that data is actually correct. We all know that a lot of these implementations aren't actually accurate or useful.

Dennis: If you don't have the money or the resources to do things, you’ve got to start somewhere, even with inaccurate data. You have to start doing something. In time, you'll have the resources to hire the best and correct Google Analytics to find the problems. But even if you're looking at Google Analytics, and you're looking for something like pageviews, or conversions are sufficient to get started.

Siobhan: Okay, so we get started, we annotate things in Google Analytics with the side of it caused a lift or didn't over a certain time. Great. How do we take it from there? Because you're going into measurement now, you’re going to start A/B testing…

Dennis: No, you don't start A/B testing. I don't think that's the first approach ever.

Siobhan: Okay, so what's the first approach after annotating?

Dennis: What you can do is simple things like user testing. Invite a couple of people in your shop if you have a physical shop. If you don't have a shop, try to go to a conference or somewhere where users are, have your laptop open, and ask them to do certain things. Things that you think are easy such as buying something from your shop. Ask to record that session on the screen. I think that's the first thing everybody should do.

If you don't have those people, maybe ask somebody in your family that's 65 to go buy something on your site. If you don't have somebody like that, ask anybody that can write and read (maybe an 8 or 10-year-old) to do the same. This is how we started. Go and do user studies. It’s cheap, it’s fun, and super enlightening. You can find lots of things to improve in just one session or with help from five people. I'm sure you have a list of at least 25 - 30 things you think you can do better just from there.

Siobhan: And then we'll prioritise them and implement them and annotate to measure for uplift?

Dennis: Exactly. You don't A/B test it because you don't have the terrific, so just go for it.

Siobhan: So then once you have the traffic, we A/B test, is that really useful? Is it useful just to A/B test? Or is there this element of needing to get your KPIs sorted and needing to know what you're measuring for because this is what I'm seeing. I see plenty of companies who are A/B testing for the sake of A/b testing.

Dennis: Yeah, I see it as well. We just looked at an account of a very famous marketing company, which I'm not going to name, but they’re huge. They pretend to be an amazing company, and, on the inside, I go and look, and we implemented a limit of 50 goals recently – and they have 450 goals.

Siobhan: What? How do they keep track of that?

Dennis: Exactly. I was like… why is that? And they said that with every A/B test, they have something that they want to improve, like something different. So, I asked them - how does this make sense in a year plan? Like, who came up with this? They replied, “Oh, we wanted to have distributed A/B testing. We wanted to have as many people A/B testing as we can.”

Like, okay… did you all agree on one or two specific goals you wanted to achieve with something called the guardrail metrics on the side maybe? It was all duplicates of everything. It was an incredible mess.

If you're aiming for return on advertising spend, you have the multiplier that works for you, you can break that down into micro metrics like conversion rate times average order value and the cost of acquisition and stuff like that. So, you have one metric that you measure against, with maybe two metrics that you can influence that are your levers you can pull. So yeah, it is important to have that big goal before you start A/B testing.

Siobhan: So then before you A/B test and measure, you need a strategy?

Dennis: I think the strategy can be as simple as - do we know what we're going to do this year? That could be the start. So, what are we trying to improve this year? Maybe alongside, you can refine that because it's very hackable (that number).

I remember a search engine saying, “We want people to do more search queries.” That make sense, right? And then, because of a bug in the A/B test, they had to keep doing searches because they didn't get the results they expected. So, metrics was going up really well so you're gonna have to think about your metrics a little bit because you do want people to find something on the search engine and be happy about it. Coming back each time because your search engine doesn't work is not what you're trying to optimise for.

It's okay to get your first time metric wrong. Very big companies get that wrong in the beginning. You can fine tune it a little bit in time. But having something to start with and benchmark against is important.

Siobhan: So, let’s go back to what you were saying before that there's this lack of action, forward movement, and measurement. We've talked a lot about how you can do it, who has done and who has done it wrong.

But why are you saying there's a lack of it? Are you seeing that there are a lot of agencies and companies that are just taking action without measuring it? I mean, where's that initial thought coming from?

Dennis: My biggest frustration is that people are chasing winners. It's like… if I don't get a winner, the software is not good. It shows a little lack of maturity in that sense, right? So, all my good ideas, I have maybe one every couple of years, come from all my past failures. That's the only way that it works. I'm open to new things because I've tried it and I've read it. So, when people fail at their A/B tests, they take as a personal loss, right? And that hurts.

You're laughing because I know you felt that at one point in time. It’s like I told him it was a great idea, and then I implemented it, and it didn't work, and it will backfire on you. Your ego is crushed and the customer opinion of you drops.

The whole thing that I'm seeing inside my company is that the forward motion is what counts, right? And then taking the learning from that is - if you've lost something, and it didn't work out, just segmenting that by device, traffic source, country or by whatever metric you have available. Just do that post analysis. I bet you there's insights in there. I mean, you could lose overall and then suddenly, you're saying, hey, on iOS, I have something here that's not as bad as the other one. Why is that not as bad as the other one? Based on that, maybe go ask some colleagues and discover that, for example, maybe the menu should go five pixels down, so it doesn't cover this or that. It’s a great insight.

Do that for the next test. I think that's what is possibly there. And it's a pity because I'm fighting against a market that's not yet so mature.

Siobhan: You don't think that the market is mature yet? I think it's going up.

Dennis: Sorry, the market? Yeah, maybe our market.

Where we are with our current pricing, slightly under £1000 a month is still very affordable in this space. So, that market probably isn’t a full blown experimentation team. In that market, we may have some people that are still learning. I mean, that's where we positioned ourselves, there’s a lot of education, which is what I’m doing right now.

Siobhan: I think it's a good thing though because you're not harping on the fact that you need to A/B test or measure. You're saying – do what you can right now and keep on moving forward. Just like in your example, if you fail at a test, (we all fail at testing, I can attest to that) is that you can find learnings from them.

I will say that you cannot find learnings and then implement it just for that device. This is what I get a lot. “But we can just implement it for desktop if it worked on desktop” - like no, we could retest it.

So, why do you think a lot of people aren't measuring, or they are not taking action? Is it coming from somewhere? Because you know, we've got our winners, we've got the people who are chasing winners, we've got the big guys who are testing phenomenally, and we've also got smaller companies who are starting to enter that field with experimentation and working with agencies, etc.

But you were insinuating (in the beginning) that we need to measure and there's a lack of action? Where is that coming from? Who is not measuring? Or what are we not measuring other than A/B tests?

Dennis: I think the scale of zero to $10,000 a month for your ecommerce or your SaaS company…I don't think anybody's measuring. It’s not the time for it. You're still looking for product market fit and things like that. So having a Google Analytics is probably fine and having the monthly recurring revenue, that's all fine.

I don't think you need to start measuring at that stage. It's just surviving and getting to product market fit. Then 1000s to 100,000, I think most people are still not measuring all that much. I think that's the space for lots of customer interviews, user sessions, long onboarding, like take your customers by the hand because you're looking for the messaging and scalability and things like that. I think that's the moment where you should start implementing a correct analytics tool. If that's Google Analytics, or anything you have. You can spend $1000’s getting somebody in there and setting it up correctly. But I don't think you need to look at weekly. It will help you in the future if you get that right now.

Once you hit $100,000, and going for that million, I still think most people don't need to A/B test. But then the measuring starts with your ad funnels and paid traffic and things like that. Start measuring very clearly what your return on ad spend is and what those conversion rates are on the landing pages that you have. I think once you hit a million dollars in annual revenue, and for most people that actually is monthly revenue, when things are getting really serious, you should start thinking about getting a team of three people together, of which one is more of a user focused person, one more analytics person and one more developer type.

You should have a team of three trying structural experiments. And that could be still ads, it doesn't have to be on page or in app A/B tests. But that's the moment I think you would be missing out on significant growth if you don't have three people in that space. If you don't do that, you're still A/B testing but you just don't know the outcomes of this. So, you're learning nothing. You’re still trying things and you have nothing to segment, you have nothing learned. You don't even know if you won or if you lost. So, you're basically flying blind.

Siobhan: In closing, what are we losing if we don't measure, especially once you've hit that $1 million mark? What are you losing?

Dennis: I think you're losing out on understanding your customers. You don't know what they want,

Siobhan: Which is the most important thing, isn't it?

Dennis: I think you can shove things through their mouths by pushing ads, but eventually that's going to catch up on you. You really need to invest in understanding who is your buyer and I think that's what you will learn

Siobhan: Great, thank you so much Dennis.

Dennis: You’re welcome.

Siobhan: Thank you for listening to Marketing Unfucked. Make sure to leave us a review. See you in two weeks.