Skip to content
Optimization Testing Tested: Validity Threats - Beyond Sample Size episode artwork

Aug 3, 2007 48 min

Optimization Testing Tested: Validity Threats - Beyond Sample Size

from MarketingExperiments.com Web Clinic Podcasts Hosted by Mec Labs

Through online optimization testing, we strive to discover which webpage or email message will perform best by trying each version with a random sample of target prospects.We decide during test design how much difference we must see to warrant making a change, and what level of confidence we need to consider the data "valid" using statistical methods. But is sample size the only factor that should be considered when assessing the validity of our test results? In this clinic, we will identify the four greatest threats to test validity, and look at some case studies that reveal the dangers of relying on a "statistically valid" sample alone.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Optimization Testing Tested: Validity Threats - Beyond Sample Size

0:00 48:21
of matches

Transcript Auto-generated, may contain errors

There's a lot of content in today's session, and so it will be necessary for us to move through each piece. We're going to look at optimization testing, and we want to talk about testing and the validity threats associated with testing. Some of you who may be students in our certification programs have had extensive teaching on this. It really is an important topic, and it's important because in so many cases we conduct tests and base decisions on phone data. I see it over and over and over again. I saw it yesterday I was talking with one of the largest retailers on the Internet, and I saw...

and suspect that I discovered significant validity threats in the way that the data is being sampled. The danger here is that when I talk about validity and confidence in results, almost everyone thinks immediately about the statistics class that they took in college and the need to understand what can be some complex math regarding probabilities. And clearly, the sample size is an important part of ascertaining what is valid in terms of your test results. But it's not the only threat. It's not the most common threat. And it's not the one that...

most people immediately detect when there's a problem in their data. Most of us understand the bell curve and its impact on confidence and the way we ascribe some level of probability to a series of tests. And if you don't, please don't feel insecure. That's information you can learn. We teach it in our programs, and it's something that you can have calculated for you by some of the tools that we generate or that you may be able to ascertain somewhere else. You don't have to be a mathematician to do solid, effective testing. But...

You do need to understand validity. And if you don't, you are jeopardizing your future decision capacity. And so what I'd like to do is begin to ask ourselves the question of what do I need to be concerned about when I'm designing my next test? What is the potential danger that can derail me and could yield a false set of results? Now, I should go on, but there's one more point I want to make for you. It's particularly dangerous to run a test that's invalid. It's particularly dangerous because if you just speculate and you don't test...

you probably have less confidence in your decision than if you speculate and run a poor test. And I think it's like the person who takes a four-hour self-defense course and rushes into a tough part of town feeling overconfident that they're prepared. They'd be better off having not taken the course at all. And so with that in mind, I'd like to drill down and talk about validity. And we should start immediately with an actual test or case. And so let's look at a new test design. We conducted a seven-day experiment for a large industrial parts company. Their market is predominantly midsize in large businesses, and the company's largest volume paid search traffic sources at Google.

And our primary goal for this test was to reduce the bounce rate. I remember I personally engaged the leadership in this organization as we thought about this test design. And we needed to reduce the bounce rate for the company's homepage. They felt like if they could reduce the bounce rate, if they could get more people into the site, they could have an increase in revenue. And so off the same marketing span, they'd see a greater yield.

So we put together a smart research team, and one of the research managers was Brad Workens. And one of the key people who really oversaw everything in this was, we call her Gabby here. I think her name is Anna Gabriela. Gabby, you correct me if I'm wrong, Diaz, but she's an IPA. And I think, Gabby, you're from Venezuela originally, is that correct? Correct. And she works here in our labs at Marketing Experiments in our Jacksonville Beach offices. And I may ask you some questions from time to time regarding this, Gabby, so if you'll just kind of stand by. Sure. We tested the control.

against a different page design, and you can kind of see the difference. The controls on your left, treatment one here, and treatment two. Treatment one is a more or less product-oriented page, and treatment two feels like a directory. So if you'll look again closely, you'll see control, treatment one, treatment two. And what I'd like you to do is think about that and ask yourselves, as a marketing professional, which one of these pages do you think would produce the highest click-through, the least bounce? So I want you to type in, using your Q&A section, if you would, either C,

or one or two. And I'm going to just kind of watch your numbers come in for just a moment. So I see two, one, two, one, two, one, two, two, C, two, one, two, two, one. It's about evenly split between the twos and ones. Good. One, two, one. Melanie just voted one. And I'm going to step up here for a second, John. See, just to get a sense of what people are thinking. Wanda, you're at two. It looks like Yanni, you're at one. David at two. Sam.

He says, why is there no sound? Someone from the technical group will help you. Two is Melissa. C is Mike. Andrew says two. Steve says one. Another Stephen. I recognize you, Stephen. You seem to be on every one of these. It's good to have you back. Stephen Diebold, that's number two. He voted two. Chase, William, Deborah. Okay, let's get to results. And then we have Jimmy Ellis, who's a panelist working on our team, and the director of our scientist group, who actually voted, and he put down four. I'm literally looking at his entry, and I think that...

You know we have a problem when the director of science is, number one, can't count, and number two, doesn't understand the difference between controls and treatments. Jimmy, I want you to know when I'm delivering a tough lecture like this, it's always comforting to have you back there ridiculing me. I just figured the higher the treatment, the better the results, right? The higher the treatment, the better the results. Now, there's logic. Thank you, Jimmy. Jimmy, you probably actually worked on these treatments to some extent with Gabby, didn't you? You got it. Okay, so you can help us also. What I want you to do now is we're going to just look at some same issue-related validation. So after one week...

A large enough sample had accumulated to satisfy the test design threshold criteria of at least 5% difference in bounce rate with a, you know, your standard 95% level of confidence. Let's see what we discovered. All right, so here's how the treatment's performed. At least this is what it appears to be. The control was at 22.6%. Treatment 1 was at 20.2%. Treatment 2 was at 12.9%. Jimmy, I'm going to think that you did treatment 2 and Gabby probably did treatment 1. Yeah, say that now. I know where you're going.

Apparently, the control is outperforming both, and almost no one put C. But let's look deeper. The directory style page yielded a 9% lower bounce rate over the control page and a 75.6% relative production. Now, we looked at those results, and the goal of a test is essentially to discover an answer. Now, you need to understand that. The goal of a test is not to move the needle or improve ROI. If you think that way, you won't design your test right. If you get the right answer,

then the test was successful. The key is not getting the right answer. The key is asking the right question. Most of us don't ask the right question, and I would say 85%, if not 90%, of the tests I see are not asking the right question. It's particularly rampant in multivariable testing, by the way. But even in single-factorial design, A and B split test, the right question is not being asked. So let's just look at this. Using the outcome of, you know, validated tests, it might have been natural to stop gathering data for the test to commence with implementation of treatment to and or follow-on research. However,

While post-test analysis and planning continued, we kept the test running for an additional week. Now notice this. After a second week, before closing out the test, we extracted a fresh set of reports expecting further confirmation of the prior outcome. What we found instead is quite different. Let's look at the new table. Now this is different. Treatment 1 is at 19.7. Treatment 2 is at 20.7. The difference between the control and treatment 2 is a... That doesn't look like the right difference between control and treatment 2 to me. I'm looking at my own chart.

Yes, Bob, go ahead. And I'm sorry, the title may not be clear. There's a difference in treatment two between week one and week two. All right, so I'm looking at the table, and I thought that I caught Bob who oversees all of this, had made a mistake, but clearly Bob never makes mistakes. There are many opportunities to come. Anyway, it's...

The difference you see in red is the difference between the prior table and this table. And what you need to understand is pretty clear below there. In week two, the control and treatment one pages performed much as they had in week one, but the treatment two page bounce rate soared to 20.7%, exceeding both other pages. The relative increase in bounce rate from week one to week two is nearly 38%. Now that's wrong. Something's wrong. Gabby, what was going through your mind when you got back these test results?

Well, it clearly was a big difference. And the first thing, I mean, we look at many different variables, but one of the things that you will and actually everybody will see at the end of the brief, it's related to the traffic and the relevance of the traffic that was going to all the treatments. All right, well, let's look at that then, all right? So since only the radical redesign treatment was significantly impacted, and you may have, if you've taken our certification classes, you know that we educate something called a radical redesign and you do it with what we call a variable cluster, but that's when you change many elements at once. where you drill down and test single elements, which is what we're taught we're supposed to do. Actually, you should be doing that only in certain cases. But since only the radical redesign treatment was significantly impacted, our first thought was that many returning visitors who were accustomed to seeing the familiar control page may have bounced by clicking back to the search results page to verify or by manually entering the URL. In other words, we thought that we might have an issue with the returning visitors, people who are already familiar with site.

So to test this hypothesis, we extracted performance data separately for new and for returning visitors. Now, that's pretty significant. So I think that's clear to everyone. If you have a question about that, you go ahead and fire your question into us right now. I'm watching to see if we're being clear. Let's look at what we discovered when we broke out the kinds of traffic. So go ahead. When the returning visitors are filtered out and only new visitors are included, the results are similar. The bounce rate for the control page changed only by 2.4%, but the bounce rate for treatment 2 soared from 13.5% to 24%.

The relative increase in bounce rate from week one to week two was nearly 79%. Now, I think we're being clear. It's a little confusing, but do you get the basics of what's going on here? We're not able to understand why we have this massive differential. So we're drilling down, and one of the questions we ask is, well, is it a difference in the source of traffic? Is it a difference between returning visitors and new visitors? And so we continue to test. Jimmy, do you want to make a comment on what you were thinking at this point, either you or Gabby? I'm not sure. I know you're the head of sciences, and I know you oversaw this experiment, but do you recall what was happening at this point?

Well, in terms of those pages, you know, it's just trying to run the highest quality tests that we can run. New from returning, can we standardize the quality and get better, more sophisticated? And then just trying to discover. So how do we resolve all this? What are we doing? I mean, right now, we're trying to understand the issue. The big problem is trying to make certain that the results we're getting are valid. Gabby, where did all this go in the end, or where is it all going now? Because I know you've been overseeing this set of experiments.

Definitely. I mean, it's important to also remark that nothing in the pages changed. I mean, all the treatments stayed exactly the same, and we maintained the same source of traffic, but we saw fluctuations in traffic between week one and week two. And what we further kept, I mean, we kept testing the treatment that won in the first week against a couple of changes then and further with additional changes, and then actually found a much stable. bounce rate with the winner's style, but with a couple of changes. With which style? With which style? The directory style. The directory style. Yes. All right. So before I go further with Gabby, if you're online, you're saying, okay, so how does this relate to me and what is this about? I'll tell you what it's really going to come down to is what could have caused the swing or the reversal? Since this was a significant and unexpected change, we began, you know, pretty significant forensics here, understand what might be mitigating the data.

In doing so, we had to consider four primary threats, and I want you to understand those threats. Now, listen, this slide is worth the time you're spending right now. You need to understand what I'm showing you at this present moment because it nails down some of the most common areas I've discovered that are occurring as we try to accomplish tests on the Internet. We're going to look at them, and as we look at them, I want to teach you about those, but I just want to make a point to you. And that is that when the Internet was first launched, hardly anyone took advantage of its capabilities. We didn't understand what we could do and couldn't do, and so there were lots of poor websites.

Lots of mistakes, and there's still problems today. As marketers, it became clearer and clearer that this is a perfect medium for testing. So I remember talking about it. We were testing way back when almost no one was. And I remember discussing it, and we eventually patented the approach to online testing. I don't know if we got the patents or applied for the patents and got patent pending. In any event, I've watched it change. And now if you go to a conference, you may know that somebody on the call know that we own Marketing Sherpa also. And when I go to one of our Sherpa conferences, it's amazing to me. Everybody's talking about testing. So you might say we're in phase two of this.

But what I'm discovering is we're testing wrong, and we don't even know it. It's amazing to me how dangerous it is out there right now with decisions being thrown around and data being thrown around with a degree of confidence that this doesn't deserve. And here's four reasons why this is occurring. It's four things that you want to protect yourself against. One is history effect. And that's something we formally teach, but it's the effect on a test variable. This is the academic definition by an extraneous variable.

and associated with passage of time. It just means that essentially the length of your test created a problem, and that is because during the passage of time, something you couldn't control changed. We'll give you an example coming up. And then there's instrumentation effects. By the way, if you're on this line and you've had certification, and you've been certified in our program, shoot me a Q&A. Let me know who you are so I can see who you are. I'd expect you to be an expert. Stephen, you have. Congratulations. You may have input, friend. I'm glad that you completed that. Instrumentation, others are coming in. Good.

Instrumentation effects. Now, these are the effects on the test variable caused by a variable external to an experiment. It's associated typically with a change in the measurement instrument. We just had this with Google, believe it or not, Google Analytics. We had a significant instrumentation. We had...

We had a test that was, I hesitate to say this because we're supposed to be the experts. It was a complete and total disaster as far as I'm concerned. And it was a disaster because Google Analytics, we had to get, we had to work, and Google Analytics is a good program. It's a good product, but we could not get accurate reporting. It was not installed properly, and Google thought it was, and everyone else thought it was, and we had to go through it. And Gina, I think you're on the line. Gina's one of our other analysts. Gina, are you there? I think she's on the line, but she's probably happened to unmute the line. If you're on, Gina, when you get on, just interrupt me. Anyway, you've got to be careful because the tool you're measuring with can skew your results.

Selection effects, you can see the technical definition. It's the effect on a test variable by an extraneous variable associated with different types of subjects not being evenly distributed in criminal treatments. But the reality is what we're talking about is that you test one category of subjects, and that category of subjects doesn't actually match the category of subjects you're going to be trying to improve or get your result with. It would be, for instance, running a test with pay-per-click traffic or natural search traffic and expecting the conversion you discovered with natural search to hold up in a different channel like pay-per-click. They will not be the same.

Sampling distortion effects, that's the one we are all familiar with, and it's simply the effect on the test outcome caused by failing to collect a sufficient number of observations. It's not running the test long enough or collecting enough data to get confidence. Now, with those four effects that you need to be aware of in mind, let's look at some examples and drill down deeper and talk about how to prevent them. I'm aware, now listen, you may be a PhD, we have those types of people on these calls, but you may also be a new marketer who's running your first test probably using third-party software. And if that's the case, I'm going to try to help you, too. I'm not trying to make this – we're not trying to deliver content that's only for a very select few group of experts. I hope this can be actionable for you. And that's why I want to look at some examples, okay? So how do you prevent history effect? You need to ask a question. Did anything happen in the external environment during the test that could significantly influence results? Were there significant industry events, holidays, seasonal factors, company, industry, or news-related events? Those kinds of things can skew all of your results.

Let me just take a moment and move to an example. So here's me. And Gina, I see your note, by the way. I know you're online. We're going to keep going for now, but apparently Gina was online listening, but she wasn't. We hadn't given her authorization to interrupt the lecture mode, so that's been taken care of, and I may come back to you, Gina, if there's an opportunity. In fact, when I talk about that particular effect, I may do so. But look with me at test design. Here we conducted a seven-day experiment for a site to provide search and mapping services to a nationwide database of registered sex offenders.

I'm looking at the way that's read. The search and mapping services are not provided to the Nationwide Database of Registered Sex Offenders. They are not the customer, for those of you who are writers of this particular slide. They are actually the registrants in the database. And the objective here of the entire site is to inform concerned parents and guardians of the location of a sexual offender that may be in your neighborhood. The objective was to determine in this test, by the way, which ad headline would have the highest click-through rate. And we had a real issue here. So here's the problem.

We run this test, and if you look at the headlines, Child Predator Registry, predators in your area, find child predators, is your child safe? So let's just look at that for a moment and do the vote. I want to hear 1, 2, 3, 4 from you, if you could. 2 from David. 4 from Franz. 4, 4, Chase. 4, 3, 2. 4, 4, 4, 2. 2, 2, 4, 3, 4, 2. Lots of... There we go.

Yep. Brandy at 2, 3, Eric at 3. Okay, I'm seeing more. Good. All right, there's a standout here. We have a Cindy who voted for 1. And there's one more of us to me that voted for 1. There may be some more. They're still coming in. We're watching and tallying. All right, so here you have the four headlines. And we ran the test. And what I want to do is I want you to just skip through the next slide, John, and go to the slide after that. Next slide. There we go. So you should look here, and let's look at what we discovered. Just look at the predator in your area. Click-through rate was 6.7%, pretty high. Child predator registry, 4.4.

Find child predators, 4.4. Is your child safe, 2.9. All right? Now, the difference between first and last, 133.5%. So kudos to you who looked at number one, but we do have an issue, and I'm about to show you what that issue is. Back up, John. Here's the problem. During this test period, the nationally syndicated NBC television program, Bateline, aired a special call to catch a predator, and we had no idea that was going to happen. It's a good thing we were monitoring for, you know, this kind of history effect, because the program was viewed by approximately 10 million individuals, and many of them concerned parents.

And throughout this program, sex offenders were referred to as predators. This word was used in some, but not all, the headlines being tested. And what happened? It skewed the results. Go back to the data set, John. In the 48 hours following the Dateline special, there was a dramatic spike in overall click-through to the site. The click-through rates aligned in descending order of the prominence of the word predator. The best-performing ad performed 133% better than the headline without the word predator. Now, think about that for a moment, marketer. It ought to frighten you. Because among the ways that have been invented by smart mathematicians to extract money from your pockets,

Google ranks right up there behind casinos. Now, I may have Google on the phone, and you know I respect what you've done, and we try to have a friendship with Google, so I'm not picking on the organization and trying to fleece you. But anyone who's ever worked at Google can understand that the efficiency of that market are designed to essentially capture the most revenue possible. That's how they serve their shareholders, and there is genius in the design of their system. And if you don't monitor carefully, you can lose money fast.

You know the difference between Google and television advertising? Is when you lose money with Google, your metrics can tell you exactly how much you've lost. When you lose money in television advertising, you just have a vague sense that it didn't go well. But both can be bottomless pits. Now, again, my only point is...

You might have been the marketer who set up this test and then made an important marketing decision about which headline would work and then poured money into a budget based on significant click-through rates of 6.7% and appropriately high conversion rates as well, never having known about the Dateline Special. And the next thing you know, your campaign is skewed out of shape and you are losing money. The bad news is you can't monitor this every day unless that's your sole job. And you can go 30, 60. I've seen our number one major retailer of fitness equipment who over a period of 90 days lost enormous amounts of money because he just didn't have the time.

He was about a $10 million a year Internet group, and he just didn't have the time to go in and monitor everything. He was counting on some people to do it who did not do it well. All right, so that's the problem you can have with history effects. Let's look at another. All right, so now we're talking about instrumentation effects. Now, by the way, in a moment I'm going to take questions. I'm getting to the point where I'm going to let you fire your question at me. So you've got a big team on here, and you may have thoughts about your own tests, and to the extent that we can answer them on this call, we're going to try to.

All right, so I am here. I'm looking at instrumentation effects. And, Gina, we just went through this. I just had a long conversation today about this. And I'm not trying to impugn anyone in the call. And you didn't know I was going to ask you this. I understand. But what happened in our implementation of Google Analytics that skewed our numbers in that test that you worked on? Don't reveal the name of the company, Gina, but give us a sense.

Can you hear me, Flint? Yes, I can. Okay. Basically, what had happened was the Google Analytics code wasn't implemented correctly because it needed to be implemented using the subdomain tracking. And I actually just got confirmation from the Google Analytics rep today on how that needs to be implemented correctly. But we had several calls going back and forth between the partner, myself, and Google. And essentially, what was it doing? What was the mistake? What was it causing in terms of error on our part? What did it look like if it was wrong?

It was calculating the bounce rate incorrectly. Okay. All right. We could see that because we had another metric in place. We were also using a monitor tracking. Okay. So now, Gina just brings out a point that's vital. And it's the professional way to approach this. We had two sets of metrics in place. And because of those two metrics, someone was able to see that the numbers weren't accurate. And I suggest you always have two metrics reporting systems in place to catch instrumentation errors like this.

So that's instrumentation effects, and thank you, Gina. And I know how hard you've worked to get that right, too, and I'm grateful for your efforts. Let me go on to the instrumentation effects as it relates to test design. We conducted a multivariable landing page optimization experiment. Now, this is, you know, multivariable testing. Jamie, I'm going to have you help me with this in just a moment. The goal was to increase conversion by finding the optimal combination of page elements. And the test treatment was rendered by rotating the different values of each of these test variables, even among arriving visitors. Most of you know the difference between single factorial, which is AB, sometimes ABC, but we're testing.

essentially one element or one cluster of elements. And then there's multivariable testing where you're testing many things at once. If you look at the next page, I'll show you an example of that. Let's go forward again. So here's a whole group of elements that are being tested simultaneously. There's six variables set up to rotate and then highlight it. Now we had a problem, and John, let's go back and look at that. We discovered that in the testing software, a fail-safe feature was enabled specifying that if for any reason the test was not running correctly, the page would default back to the control.

As such, the testing system would deliver to the browser not only the treatment page and its values, but also those for the control page. And the browser would then render only the page corresponding to the test condition. Now, that's kind of hard to understand. The best way I can explain it is that it added lots of page load time because it was loading many more values or variables or pages than, you know, the control. And I'm going to let Jimmy just comment on that because, Jimmy, you oversaw this, and I remember that you were the one working on it. Do you want to give us kind of an explanation of what was happening? Basically, it's just that it would load both code was on the page, but you wouldn't necessarily.

So what happened in the entire phase was the elements were being swapped out, so showing up, but the load times worked. You understand, this was using a professional third-party product, just like many of you on this line might be using. You know, this was using the kind of product that you would hope would be reliable, and I'm not saying they're not a reliable provider. I'm just saying that there was an error, and you might not think that you need to check on these kinds of errors because you're trusting a third party, but the reality is you do. Here's what we discovered. Look at the load time.

Now, we don't need to spend a lot of time wading through this chart, but if you just look at it, what you'll notice is that on a 56K modem, it's an extra 9.56 seconds. And by the way, many of the customer profile that would come to this particular site would be using a 9.56K modem. It's an extra 9.5 seconds. That's significant, and that will clearly mitigate against conversion, which will, of course, hurt the entire experiment. And, Flint, I can add a little bit to this. I was recently looking at Google Analytics for a few different of our partners.

the connection speed to conversion rates. Interesting thing that we recently noticed is that that portion of customers that are on dial-up typically are the highest converting customers that they have compared to those who are using dial-up. It was over, like, for example, the base conversion was about 1.5% for this one company. Larger than these, significantly reduced. I think their load time for a 56K modem was over two minutes, whereas their coding can load faster and impact that. That's exactly right. And why do you think that is? I'm asking my audience, and if you'd like to just take a moment, use your Q&A and talk to me about this.

Jimmy's made a keen observation. Why do you think you got higher conversion from the customers on the 56K Modems? Would anyone like to think through what that might be? Good, good. I'm seeing some answers coming in. Wayne, I'm going to read yours in a moment. Ryan, John. I'm going to have to disagree with Andy. He says they're less educated. I guess perhaps the product is a bad product.

I'm going to continue. Here's some of the things I'm seeing. They're logging on intentionally, learned patience. They were motivated. They don't want to waste time browsing. They're more innocent. They don't aimlessly browse. They have no money. I'm sure. But here's the thing I want to point out for you. If you study the marketing experiments conversion index, the highest, most important variable, we call it M on that formula, C equals 4M plus, it goes down a long list, is motivation.

Typically, someone coming into the process with a 56K modem and getting engaged is quite intentional about the process. I think you're dealing with a better qualified prospect. Would you agree with that, Jimmy? Yes, yes. And there's one more thing. We need to move on, but here's something that would help everyone that's on the line.

Many of us put together our treatment, but we haven't really tested it in the appropriate browser combinations. There are sites on the Internet you can go to that you just plug in your URL, and they will generate for you a report showing that page in 70 or 80 different combinations of browsers and operating systems. Don't underestimate the significance of doing browser compatibility testing. The reality is there are often flaws on your new treatment page that you don't know about. And so, I mean, if 20% of the Internet is not seeing that page right, it's going to skew all your results. We typically run a quick browser test right away, and that gives us a sense that we have a page that's displaying cleanly.

Selection effects are another issue, and you need to ask a simple question. Just like we had a question that you needed to ask regarding instrumentation and history effects, here's the question you need to ask regarding selection effects. Did anything happen to the testing environment that may have caused the nature of the incoming traffic to be different from the treatment? Now, that's an important question. And I think that if you ask it and monitor things carefully, you may discover that there's been changes in the characteristics of traffic. You need to make certain your traffic flow is even. I've seen this repeatedly. I used to see this on the small pay-per-click engines back when there were more small pay-per-click engines trying to compete aggressively with what used to be called GoTo as opposed to Overture. What happened was...

They would make a new partnership and get a surge of traffic, which would then translate to click-throughs on the pages of advertisers, but the traffic was absolutely worthless. And so the advertiser would suddenly have, you know, many, many clicks to their page, but really poor conversion. It would panic the advertiser, and they had no idea what was going on. Moreover, most of the time, the smaller pay-per-click engines would conceal what was taking place. I actually had to get online with the chief technical officer for one of them with indisputable data and call his bluff. Because they make money on those clicks, and they need that money. But all clicks are not equal, and we have to keep that in mind.

All right, so I'm going to stop for a moment to take questions. There's a lot more about this. In fact, I have many slides that I'm not even going to get into right now. Part of this, we teach in the certification programs. Part of this will be, some of these things will be in the briefings that we're going to be doing with you. But let me just take a moment and let you ask questions about your own testing. Let us know how we can help you right now with answers that might make a difference. So go ahead, fire a question at us. Someone said off the topic, but what are some of those sites that show you how your page appears in different browsers? We can try to help you with that. Jimmy, what's one of the sites we use?

One of the sites is browsercam.com. Okay. I haven't used that recently. A lot of times what we'll do, and if it's not, you know, if you don't have a native operating system in browser a couple percent, a lot of times depending on what those are, they're less than a couple percent. Okay, is it worth our time to actually optimize for these combinations, or do we just focus on the big stuff? And so we'll get everything above that typically on our own. And so we have a bunch of detailed ones serving on 20 sites through the thing and get, I think it's about, it's not a perfect service, but gosh darn it, it's really handy.

One point, just for any of you that aren't sure, Market Experiments has no advertisers, and we're not affiliates of anybody like this that we're talking about. So when Jimmy brings up a distinct company like that, I just need to make an editorial disclaimer just so that you all know we don't benefit anything by bringing up these people, and they have no idea that we just mentioned them either. So just so you know. Now, there's some other questions coming in, and one was just a comment from Ryan who says, he says it's scary, that's his quote here, how folks rely on tools but never test the tools. You do need to test your tools, guys.

And you can come away from this call almost feeling insecure and throw up your hands and say, look, it's just too hard to conduct a valid test, and I don't know if it's even worth it. Believe me, it's worth it, and yes, you can conduct a valid test. And frankly, if you just identify these four validity effects and watch for them, you can do fine. And if you're still insecure, get certified. This isn't a commercial. This briefing, we do these twice a month. We don't charge for them, and we don't sell products in them. But I can tell you, it just doesn't matter. I think with just confidence, at the risk of sounding like I'm selling something, get certified.

In seven weeks, you learn what to do and do it right and do it well. It's just you've got too much money on the line not to know how to do this well. Here's some other questions. Someone asked how many days were measured for each test. And the answer is absolutely variable, and it depends on a calculation regarding how many actions we need to measure. And that calculation is part of the math formula. We build it into a test protocol and calculate it automatically. You don't have to do a lot of probability calculations, but you need to know what tools to use if you're going to get good validity. Somebody says,

Somebody said, what was the reason discrepancy between week one and week two for the treatment in your initial example? I think I know the answer, but to be very candid with you, at the risk of sounding, well, just for the sake of being transparent, we're still not sure. We did go on and test and get a significant improvement for them, but we're still trying to understand what went wrong with these tests. And I have to say that sometimes we just don't know. And we want to know, and we're trying, but the reality is it's a mistake to go on and pretend you do when you don't. You'll get further if you just admit it.

Let me see if there's another question I can answer. Yeah, someone's talked about the business cycles. It just depends on the length of your business cycle, and that's why there's two ways to run a test. One's in sequence. Try this page for a week. Try that page for a week. That's a sequential single variable test. I've done that. In some cases, it's your only way to go. In some cases, it gets a great result. Give him a second. I'll probably be right back on. If you're sitting by, I just was just in the middle of my conversation. I'm not sure if that was a sign or if we just had a technical problem, so I've just moved to a fixed zone, and so I'll continue. I see that you did lose me. I was gone.

Somebody's asking about multivariate testing products. There are a number on the market. We're actually in the middle of a lot of testing to see which ones are most effective. We've worked with different ones, and I'm not really prepared to tell you which one to go with yet until I get better data. But I would also suggest that you can do most of what you need with single-factoral design, certainly initially. You've got to be very careful that you don't depend too heavily on a multivariable design because...

And I'm not trying to take away from everybody in that business, but you can have some difficulties with your results if you don't understand the limitations and you know how to use it well. There's some free information on our website about how to conduct A-B tests, how to conduct multivariable tests. There's briefings. Go to our site and go to the archives or go to our site and do a search. Listen, there is millions of dollars worth of free research on that site. And they're full reports. They're not, I mean, they're briefings. They tell the information. Get in there and use them. That's what they're for. All right. I think I'm going to...

I think I'm going to bring things to a close now. And what I'd like to do is simply invite you to take a few moments and let us know if this has been helpful. I've got a large team on the phone. I probably have 20 people on the call, I'm guessing. Maybe it's 10 people on the call. And we're working hard with our overall staff here of people that are working through the entire process of trying to discover what works. But we really value your suggestions and your comments. Let us know if you found this helpful, and we will carefully monitor your feedback. So use the Q&A feature, if you would, and write to it. And the other thing is, tell some friends about it.

The call is free, but we really just rely on you telling other people about the website and the calls, and hopefully as a community we can all learn together the most effective way to conduct our campaigns. I appreciate your participation today, and I'm going to sign off and then just watch this screen to see your comments as they come in. Thank you so much.

French Your Way Jessica: Native French teacher founder of French Your Way Boost your French listening skills and test your comprehension with this one of a kind series of podcasts. Get the chance to listen to a real conversation between native speakers talking at normal speed AND customise your learning experience through carefully designed sets of questions (2 levels of difficulty) available for download at www.frenchvoicespodcast.com. All interviews also come with the transcript. French teacher Jessica interviews native speakers of French from around the world who share a bit of their life and passion. Where else would you meet in one same place a French yoga teacher based in Melbourne, a soap manufacturer from Provence, or a couple cycling around the world? Growth Triggers: Content Marketing & AI Luis Diaz Discover how to use your podcast to generate leads, sales, and influence in any market. On this show Luis shares his best tested and proven secrets, tips, and strategies from launching over 300 podcasts in over a dozen different niches and generating over 59 Million Downloads for his clients. Learn how to increase your podcast listenership, create multiple new revenue streams with your podcast, sell more of your core offer and build a show that gives you leverage and authority in your space. From picking the right monetization strategy for your podcast to learning how to position your podcast in a crowded market on Apple and Spotify; this is show is the ultimate marketing and sales resource for online business owners who want to know what works with podcasting. Get in touch over on Instagram <a target="_blank" rel="noopener noreferrer nofollow" class="link" href="https://www.instagram.com/luisr

Frequently asked questions

How long is this episode of MarketingExperiments.com Web Clinic Podcasts?

This episode is 48 minutes long.

When was this MarketingExperiments.com Web Clinic Podcasts episode published?

This episode was published on August 3, 2007.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this MarketingExperiments.com Web Clinic Podcasts episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.