Hello, my name is Sarah, and I'm the moderator for today's call. The topic for today's call is landing page conversion, getting significant improvements even when you can't complete your test. And now I'd like to introduce Dr. Flint McLaughlin. Okay, Sarah, thank you very much. We have a lot of content that we need to review today, so we're going to get started immediately. I shall announce again that this is about landing page conversion, but it's unique.
closely connected to a problem that I have seen across the testing community, and that is what can you do to get significant improvements or even to learn in those situations where you actually can't complete your test or where you have a validity issue, particularly a validity issue connected to the size of your sample. If you've been through our certification program in online testing, you know that sample distortion is a significant factor, and you have to protect against that validity threat. But when you run a test, and you're not able to actually get a result you're truly confident in, do you simply set aside the data and start over, or is there a way for you to glean useful learnings? Now, I can tell you this. We are working on a Bayes theorem solution. I've announced it a couple times in the past, and we think we're getting closer to a methodology that would allow you to get confidence with a much smaller sample size. This should revolutionize online experimentation and offline, for that matter.
I'm not talking about using a new way to approach testing in terms of statistical relevance, but rather a new way to analyze the data. So let's get started, and the question becomes, what do you do when the test when conversion rates don't validate? We've been there before, and it's particularly difficult when the differences in conversion between the control and the treatments are minuscule, so minuscule that the test results actually don't validate. What do you do then? Is all the time and energy just a loss?
we're going to scrutinize recent research findings to determine when it's possible to draw valid and valuable conclusions from invalid test results. To do that, I need to take you to a simple formula. I'm not teaching this. It's part of our certification class, but you do need to understand something important about it. We developed and patented this approach to testing. It's part of our test protocol documents. The operative letter in this formula for now is the letter U, because it is more important to conduct a useful test than to conduct a successful one. And I've said this, and I've taught this, and if you've been through our training, you've probably already heard it. However, I need to talk about it in kind of an inverse fashion today. The reality is conducting a successful test just makes...
you able to conclude that your research question was answered with a high degree of probability. On the other hand, that doesn't necessarily do anything to help you improve your numbers, because if the test is not designed properly, even though it's successful, it may not be useful. In fact, if the research question isn't the right research question, then it matters not that you ran a successful test. You ran, at that point, a useless but successful test. Now, you've heard me teach this if you've been through our training, and the formula here, the formula we designed, is to try to help you actually achieve what we would consider to be a useful test. U stands for useful if connected to utility. However, I'd like to turn the formula on end, and for just a moment suggest to you that possibly there's a way you can conduct a test so that even if the test itself is not successful, you still might find it at least somewhat useful. That's the topic for today. How can we get useful results?
from an unsuccessful test. With that being the case, we need to think about two ways to think. Most of you know or some of you may know I'm trained as a philosopher, and one of my specialties is understanding the concept of authority as it relates to reason. And there are two, I actually argue in a paper against Klein, if you guys are familiar with him, related to whether there really is such a thing as induction or deduction. But let's use those categories now as they're useful and can help us understand. There are two kinds of logic.
And with induction, we're talking about any form of reasoning in which the conclusion, though supported by the premises, does not follow from them necessarily. With deduction, we're talking about a process of reasoning in which the conclusion follows necessarily from the premises. It's like Aristotle's syllogisms. Now, if I'm losing you, stand by, because we're going to get practical. You just need to understand that what we're talking about doing here is using the process of induction to glean value from your test. So, keeping that in mind, let's please look at...
Test design, case study, number one. In this test, we conducted a 26-day experiment for a non-profit foundation. They raised money for Alzheimer's research. The goal was to increase conversion and consequently increase the total donation amount. Pretty simple. Look at two treatments and you'll get a sense of what we did. Here was the original on the left-hand side. That's your control. On the right-hand side is our treatment. Obviously, the two pages look a little bit different in part because we're taking a longer page and...
putting it at a ratio where you can see it on the slide. What's important really here is that we reduced a number of steps and did several critical changes in the process. We're already redesigning this to improve it yet again. But if you'll look at the two pages, I'd like to share with you a problem that we experienced. Here's what we discovered. Essentially, to gain validity with this particular test, with the relative difference that we're averaging at current, we're going to have to run this test for 4,000 days. Let's look. Control is 4.20. Treatment is 4.06. Difference is 3.33. This creates a significant problem for us.
It's not that you could expect a site that receives donations for a particular illness that has such high volume in traffic and transactions that you could immediately conduct fast tests, particularly with minor differentials. So it shouldn't come as a shock to anyone on the phone that this test might take longer than, say, the test in a major e-commerce site that has 10,000 transactions a day. However, even we didn't expect a 4,000-day problem. This is the value of this protocol, by the way, that we teach and use because it helps you predict how long the test is going to take, and it checks the validity and all that you do. This protocol is kind of a specialized Excel spreadsheet that we've built that calculates validity and moves you through the actual formula I talked about. What you need to understand is that our researchers were in a position where they had to know if they should continue the test or if there was something else coming that could be more effective in terms of design. So look with me, if you would. I think there's a slight difference in...
My slide, this is just a note to my team and the slide I'm looking at. But let's just look for just a moment. So here you see the bell curve. One of the first things we thought about is, what can we do? Well, we noticed this. The treatment page had a substantially higher average donation. Now, that's a vital point. Notice the control's average donation was 67, but the experimental treatment's donation was 237. That's a differential of 254%. It's dramatic. So if we eliminate the outliers by filtering out all the donations beyond the two standard deviations from the mean,
Now, if you look at the chart with the bell curve at the bottom, you can see those deviations. They come in the form of lines, vertical lines. And if we eliminate those and get to the heart of the test, can we then establish validity by changing the criterion to actual dollar amounts? I want to stop there for a moment. Bob Kimper, I hope you're on the phone. I know that some of you right now are probably interested in this topic, but still trying to clarify how you're going to be able to apply this to the problem you have right now or the problem you may have as you run your own test. Please bear with me. We're going to get really practical as we go. We also know that not everyone on this phone call is an expert in statistics, and the best marketers I know of in the world really would need to use a tool to calculate the validation or the 95% interval. So what I want you to do is not get lost on the bell curve in this diagram, but get a sense.
that there's two changes we're talking about making. One is throwing out the high and the low, and the other is analyzing this with a different unit, instead thinking about the difference in the actual dollar amounts. With that in mind, I'm going to try to move forward. If you still are confused, we're going to stop at various points and help you get your questions answered, and we'll walk through it. So let's just look. I think as we continue through this one, you'll start to see what we're trying to accomplish here. With the high and low, the outliers removed, what we noticed was that the treatment page still had a much higher average of information.
So here we are now. We're trying to figure out if there's any value we can glean from this tab. We cannot run it for 4,000 days. So what we do is we narrow the scope of our analysis, we remove anything on the outside edges of the curve in terms of results, and then we come back in and we start analyzing the dollar amount. And even when we take those out, we see a percent difference of 254%. After we make the changes, it's still 176%. So taking up the out-teach streams, we still see a really significant difference.
So the question becomes, well, can we wisely substitute the dollars for the conversion rate? Now, think about this, because the test was originally measuring the conversion rate. But now, is it possible for us to measure the difference in dollars? Can we change the success unit? And look at the chart. It'll help you see this. This is actually an extra from the real test protocol document. If we substitute average contribution per donor in place of the number of donors. Now, for those of you that have been through our course, this is something you would actually just type into your spreadsheet. And if we change those two blue numbers,
Because if you look at how this is designed, and for those of you that are still a little bit confused, this kind of does it for you. You don't have to be concerned if you're not a math expert. All you would do if you were using our protocol is type in one of these two numbers. And in this case, you're typing in what we could call a success for treatment one and success for treatment two. What we've done is we've changed those two blue numbers and instead moved it to a different issue. We've put in the successes and we've measured it with relation to the dollar amounts. And look down at the bottom. We're asking, is it significant? It shows us the standard deviation is 1.205.
And then it calculates significance threshold, and it says yes. If you want a next statistic, all you need to do is put in two numbers and look for the yes at the bottom. I'm going to move on in just a second. Bob, you're on the phone. Is there anything you'd like to add to clarify this? You did a lot of this analysis.
Not at this point, other than simply saying the question at hand is because we were not able to validate using the primary success measure, which was conversion rate. And conversion rate, we're asking the question, can we use dollars instead? Because there is such a significant difference between the treatments in dollars, would it be valid to use dollars instead? And simply putting the numbers in for dollars instead of relative success rates. it would validate if it's valid to do that. Now, some people are asking some basic questions like, why would it take 4,000 days? And I'm in this very difficult situation in that I have a vast audience on the line right now, and some of you are experts in testing, and some of you have been certified by us, and some of you have not. I can't go down to the very basic level, but I can simply say this, that in order to make certain that the probability of your results, or in order to feel as though the results that you got from your test are reliable, you have to have a certain sample size and a certain number of successes that will allow you to
glean this this condition. And if the simple differential is small, it takes more actions. You have to measure more instances. And we predicted, based on the current volume coming into their site, the number of transactions per day, et cetera, is 4,000 days. And in fact, we'll be addressing that in a little bit more detail a little bit later on. Yeah, so bear with us. Let's keep going, because if you hang in here, you're going to learn something today. Even if all you run is simple A-B split test with a third-party tool. So here we are right now. If we can establish validity using dollar amounts, then we might come to certain conclusions about this test. It could be helpful. For one thing,
We might determine that placing the 94% goes to research image. Do you see that image down to the right? That's an image we placed down there. The donation amount could have reduced anxiety, which in turn might have increased the amount of giving. We might wonder if eliminating the radial buttons for the large suggested donation amounts may have reduced intimidation or otherwise favorably affected average donation amount. Look at the left of the control. The buttons go up to 5,000 there. Here, 500, and then we have you write the number in.
We also wonder if changing the font color or wording of the minimum allowable donation amount may have had a favorable effect on average donation amount. Now, that's something that you can start to question, and there are ways you can try to determine if that's what the factors were. You can run other tests, but you can also look at the history of tests you've already conducted with that product or with similar products. But here's the problem. We cannot say that it's valid to simply substitute dollars for donors.
Still, deeper analysis of the data does appear to indicate this very consistent pattern of higher contribution amounts for the treatment to page than for the control page. Now remember, if you are the organization we're working with, just by running the test, your donations are up 254% on the treatment, and they maintain that over an extended period of time. You're not, if you're running this organization, an academic trying to convince colleagues that your methodology is worthy so that you can achieve yet another promotion in the great system. Rather, you're just wanting to make... or generate more revenue so that you can contribute more. And in this particular case, that's clearly what seems to be happening. If you were to drill down deeply on these numbers, you can see the big difference. You can see a major increase. And those numbers are there at the bottom. On the left, a major set of columns is the control, and on the right is the treatment. On the left, even though there were 38 donations, the total was $2,546. On the right, with less than that, with 32 donations, there was $7,502.26. That's significant.
Let's keep going. Now, remember something. These are just informed deductions slash inductions, Bob. We can only observe that while the page changes did not substantially increase the conversion rate of arriving visitors to donors, for the sample, the average amount of money donated by each visitor donor increased by more than 170%, between that 170 and 254. Splitting the testing period, this is something else, into multiple shorter time intervals. We're looking for a pattern.
So we ran separate tests, taking that full duration of the period and looking at it in shorter time intervals. If you don't understand that, if you don't know how to do that, get certified and we'll explain that to you, but it's something you don't necessarily have to do at all times. So I just want to suggest that by doing this, we still saw a consistent pattern of higher contribution amounts. Then producing a contribution amount, what's called frequency distribution, for the entire test period and also for the spur test sub-intervals, again showed this pattern. All of that seems to indicate that the difference is...
either due to outliers in the time or in the contribution amount. We're hoping it's the contribution amount as we continue. Any thoughts you want to add to that, Bob? Only that when, you know, as we look at the overall data for the entire test period, and we... And we are not completely satisfied with, you know, with the result that is, you know, we wonder if there are additional insights. One thing that we can do is look at, you know, dig a little bit more deeply into the nature of the data and look for patterns. And one way to do that is just what we have done here, which is to look, number one, at the frequency distribution. Is it normal in the sense that, you know, the mean, they're roughly the same number. And another way is to, another way is to, which means,
different sub-intervals and see if they look a lot different from one another or if you see a bunch of mirror images or if you see a bunch of little cases that look very much like the larger case. In each of those circumstances, you do something different. You can clue different things, but by looking at the data in that way, you can gain additional insights and identify some important things about the reliability of the data in the test. And we'll talk more about that. We're going to talk more, and if you're still not clear, hang in there. We're going to help you. We're going to help you either by letting you contact us after this with more questions, but we're going to show you other cases. And these are interesting cases that will continue to show you this from a retail standpoint in lots of different ways. Please look at what we did in this particular case. Here's a summary of the methods that we employed. First, we looked at characteristics beyond just the primary success measure. That's, you know, the results of your single factorial test. And in particular, we were looking in-to-end conversion.
But we changed that, and we also looked at something else. We looked at total dollars donated for each treatment. And we looked at the average donation amount. So that's something you can do. You can drill down and look at related but different sets of numbers. Then we just looked for anomalies.
That's something that just throws everything off. For instance, what if one person donated $4,000? Well, that would make it feel like you had a much greater success, but by one donation of $4,000 within that particular set of numbers that you're using for your sample, it could skew everything. That's why, by the way, why the test needs to run for a long time so that you get a long enough interval to account for such anomalies. In this case, we just removed anomalies, and we looked at that to see if they might impact the required sample size. Then finally, we looked for patterns of behavior. Now, that's a really good thing. In fact, it's really quite smart. What Bob is saying is that just instead of looking at the test and Bob...
I'm making up the timing here, but instead of looking at the samples over a 30-day period, we might look at them over five, six-day periods or subsets within those periods and see, does the general pattern look the same? Does this look like something that's going to continue to repeat itself? This is the way we did that. And so let's look at how it might apply to another case. Now, Bob, interrupt me if there's something critical you want to suggest on this. I know you worked on that. Jimmy, I believe you worked on this particular case. Bob's involved also. But this is a 25-day experiment for an e-commerce site that sells wheelchairs and medical equipment to businesses and consumers, and the objective was to increase the site's rate of conversion. This is a pure retail play. So what we're going to do is just look at the test design associated with this. So here we are. Next slide, please. Here's the control, and here's two different treatments. Now, Jimmy, do you want to say anything to set this up?
Sure. Basically, there's a few different strategies we're working with here. The page on the left, the control page, is what I would consider a typical directory-style page that many e-commerce... Speak a little louder, Jimmy, or get that phone closer, friend. Sure. Is this a little better? Much better. Yeah, so the control page is what I consider a traditional e-commerce category-style or directory-style category page that we're testing. The middle one, our strategy was to reduce the number of... I'll simplify the choice for the customer for three product lines, which constitute the majority of their sales. And the third treatment is more of a configurator.
which we've tested many times and has performed extraordinarily well, we wanted to do a low microtest with that option as well. So that's what kind of sets us up for, you know, the next. Jimmy, where did this configurator go? I don't recall this myself. Was it on two or was it on the control or was it a standalone page? No, it's a standalone page. It's a standalone page. I see it's there, but it's a standalone page. All right, it's much shorter than the others, but this is not a subset of the other pages. Obviously, it's a separate treatment completely. All right, so this is what we were trying to do. Let's see what happens. All right, control did 2.71. Treatment two did 3.06. Treatment three did 2.85%.
The difference between treatment two and the control is 12.65%. Here we go with another problem. They get real validity because the difference is important. 12% is a lot of money, but it's still not 80%. We often see differences of 50%, 150%, 200%. We see some really dramatic differences. The larger the difference, the easier it is to reach validity. By the way, that's why if you've had training with us, variable clusters are so important because they can help you hit a sudden spike in conversion that shortens the test cycle.
Here we are. To gain validity with a relative difference in conversion of only 0.35%, we would have had to run the test for over 179 days. Treatment 2 had a slightly better conversion rate, but we cannot yet conclude that it's a better performing design because we don't have time to wait. This merchant is in the business of generating profit. He can't wait for 179 days for this test. So what do you do next? Well, you can either rerun the test, or you can run a radical new redesign variable cluster.
For those of you that have trained with us, that simply means that we change a cluster of variables all at once with an absolutely radical redesign. You're attempting to jump categories to get away from an incremental improvement of the existing style, the existing category, and to do something so dramatically different that you more or less experiment with what could have significant impact. So here you are trying to challenge the control with a treatment that gives you a significant, hopefully positive difference. But there's something else. is there anything of value that we can learn from a test before we run the rerun or before we develop the radical redesign? Is there anything of value we can learn from what we've already done? If nothing else, this might inform us as we're trying to improve the results in the next variation. So could we gain some insights from attributes beyond simply the overall test-long conversion rate? All right. Good. Now, I see some good... Andrew, I'm going to continue, but that's good input that we have from you. I'm watching everyone's input, and let's continue. Is there anything of value...
We can learn? Well, let's look at something different. Let's look at the revenue per order. This is different than conversion. Both with and without the outliers, everyone on the line here now should remember the outliers, those are the extremes. The control page had a much higher average revenue per order than the other two pages, especially the page with the highest conversion rate in the pest sample. And that should tell you something. It should tell you a lot of things, and I'm sure we're going to let some of that unfold as we continue to talk with you. But I would also remind you that...
It tells me that, is it possible that we might be getting more orders with the other pages and more volume or more revenue per order with a control page? And what if I could do something to get the control pages high average order value matched to my test pages increased number of transactions or higher conversion? That's the kind of question that starts to emerge in my mind when I look at a data set like this. So let's go on. How could treatments with such similar conversion rates vary so widely in revenue per order?
Really, that's a why question. It starts with a how, but it's really asking why. Why did these treatments with such similar conversion rates vary so widely? Sometimes your answer to how, as in how can I get more profit out of my store, sometimes the answer to how is why, or at least it begins with why. When we analyze the test results on a product level, looking at the specific products, we notice two subtle but very important patterns. The featured product was one of the top converters for the control page.
The control page sold 175% more accessories than the treatment pages. So why was there a big difference in the average order price? The difference was that the control page was selling way more accessories, and that's critical. Let's look. Someone's asking about these slides available on a link. These slides themselves are not, but you'll be sent this brief. It goes out to all of our subscribers, and if you're in this clinic, you should be a subscriber, and in it you should receive all of this. So it's coming. Let's continue. Featured items were the key point of sale for the control page. That's what made the difference.
The presence of product accessories on the landing page increased overall accessory cells in terms of cells volume for the entire site. The first accessory on the landing page had this foldable drink holder, and it had the highest sales volume. So that tells us we should consider testing other accessories here to determine the one with the maximum impact. Let's keep testing accessories until we discover those that actually produce the highest possible yield. We should also test the impact of including the relayed accessories along with all major product categories on the same page. What else can we learn from that? We learn that when we feature these products around the core product, we're much more likely to sell or around the core offering on that page, the core action we're asking to take. And it makes us pay attention to the role of accessories. In fact, it might be better if you could have tested the control page against the treatments in such a way that treatments have the exact same accessories featured or weighted almost the same, but actually changed only the way we talked about the core action we were trying to take as we pushed them towards conversion.
In other words, I would want to almost keep the consistency of the control pages presentation of treatments as much as possible. I'm sorry, presentation of accessories as much as possible so I could drill down deeper on the individual components of the page. So what do we do? This is another review. This is what do we learn from this that will help you in your own work? Well, first of all, we looked at characteristics beyond just the primary success measure. Again, this is the same thing we did in the last one. Here we looked at average revenue per order for each treatment.
Similar to what we did before where we looked at average, you know, donation per donor. Here, we also looked at the mix of products ordered for each page. We studied the actual products associated with the cells when we looked at each conversion. That brings me to the second thing. In this, we analyzed the amount of variation in each variable by looking at measures of variance. Greater variance translates the need for larger samples. It's also an indicator of possible hidden sources of difference during the test period. Now, that sounds confusing to me. So let me try to explain to you what we're saying. We looked at the variations.
or the variation, either word is appropriate, as it related to the individual components, in this case the products. They were an indicator that there might be something different taking place on a product level. And we wanted to drill down deeper and understand those differences. Jimmy, you want to say anything about that, or Bob? Jimmy, you there? Yeah, I'm here. I don't have anything to add to what you're already talking about. Okay. Then we looked at the page performance data for each of the attributes measured and looked for connections between page design attributes and performance. Bob, explain that.
Sure. We looked at the page performance data and looked for connections across time as well as across different product groups. So we looked at the control page or the directory style page, which had many product categories and just a lot of information. The rationale, and Jimmy, please feel free to join in since you're designing the test treatments. The rationale for treatments two and three were based upon experiences with eCommerce sites where if you...
focus the attention of visitors on either top-selling products or ones that are specifically based upon a specific page, based upon the landing page from a search term that are relevant to those particular searches. So if we focus the attention of visitors, that often will increase conversion, increase the performance of the page in general. So as we look at test data, we have hypotheses. We go in and design the treatments with something specific in mind, something we expect to have happen. And in interpreting the test results, when looking at the data, we compare...
the test results with our expectations, with the hypothesis we had in designing treatment. And then we look for connections between the performance criteria, not just conversion to sale, which is the primary success measure here, but also the performance of which product sold and does it look like there was an impact from the test design and did the things we had expected to see actually occur. And if the answer is yes, then wonderful, we've learned something. If the answer is no, if something completely unexpected happens, then we learned something. And that gives us insights for subsequent testing. So those are the inductive insights.
I have two more case studies in front of me. I'm so conscious of the fact that we want you to learn from this, and we don't want to overwhelm you with the level of expertise or detail. And I know there's people on the phone who follow this immediately, and some of it's more challenging. But I want to stop for a moment and let you fire questions. Do you have questions for us about what we've said thus far? I've got a number of experts from our scientist group on the line, and Andrew said keep up the expertise, and I appreciate that, Andrew. We will. Andrew also brought up something that would be very good if you can do it, and that is drill down and look at the profile of the customer that's placing the orders in relation to each of the pages. Sometimes that's possible. Sometimes it's not.
But if you can discover that, it makes a huge difference. And we teach that when we talk about tracing the motivation of the customer. It's in our conversion index. Motivation is the highest impact variable in our formula. Any other questions? Someone said, can we get these Excel spreadsheets? Yes, we actually lay those out for you. It's part of what we do in our certification, teach you how to use them. They're pretty helpful. Any other key questions? All right, I'm going to continue moving forward for those of you that are standing by. All right, so let's keep going.
So if you were in our last clinic on validation, you'll remember the case study in which it tested validated the first week. Then by the end of the second week, it didn't. I mean, it looked absolutely right in the first week, and it looked completely different in the second. What in the world was happening? I can tell you before we even get any further, I still don't know, but I can tell you what we're learning. Go on. Here are the pages we tested. On the left is the control. Then you see treatment one. Then you see treatment two. And back up, John. Here's the control and treatment initially. Control is 21, treatment is 19, and treatment two is 16. 21.38, 19.94, and 16.88. Now go forward. Go to the next slide.
One of the ways of gaining more insights is to disaggregate it by different attributes. Look at the data with a different set of attributes. So here, if you split the data into two distinct weeks, instead of looking at it at a two-week period but as two different weeks, you see that the picture looks very different. That's very important. Look at week one at treatment two. It's at 12.9% in terms of conversion. But look at week two. It's at 20.7%. It's a 60.5% difference. Now, the control page and treatment one perform very consistently, but treatment two's bounce rate changed dramatically. It went from 12.9% to 20.7%.
Now, I think I was saying conversion. We're talking bounce rates. What caused this? And I want to just ask you, and I'm going to really test you for a moment. Take just a moment and tell me, as you look at this particular test, what you would be questioning about this difference between 12.9% and 20.7%. Based on what we said thus far, what's one of the first things you check? Change in sample. That's from Zineb. I would question how many people back out since it was a new page experience. Someone said, what is bounce rate? Tony, that's the number of people come to the page and leave without going deeper into the site. Someone said the motivation of the client. Someone else said, I'd look at motivation.
And let's just take a moment and think about this for a moment. What is one of the first things? Uniques versus returning visitors. That was a very intelligent suggestion. New versus returning, came from David. Good, we've got some people on the phone who are thinking. Traffic source, which is related to the last two suggestions. One of the things that I'd be asking is just this. I'm moving your questions so I can see more of your responses. Our staff, try to answer some of these questions that I'm not getting to. Bob, if you can help me with some of the questions like...
Is there a template for landing pages, et cetera? That's something you might be able to answer. But let me just try to bring you back to this. What would I do if I saw the 12.9% and the 20.7%? Well, the first thing I'd probably do, and we've talked about this, is I'd go back and look for any extremes. Outliers is the terminology we've been using. I know that we're looking at a much larger sampling size here, but I still want to know what is happening. If I saw, for instance, if I saw, for instance,
A serious problem with this on a particular day of the week or a particular short interval, a major burst there. I wonder, all right, did we have a problem with the history effect in our test? Was something going on that was different? Did our instrumentation fail us? In other words, did something go wrong on our website or on our testing mechanism? I'd want to go back and try to look for something that doesn't seem consistent in terms of pattern. In fact, in a sense, what you're really doing is...
stratifying the data. Let's take a look at some things about that. What can cause these performance differences? One is seasonality. Weekdays versus weekends, but even particular days of the week. I mean, there are huge swings based on these kinds of issues. The other is, and by the way, time of day is significant. But the other thing is validity threats. We teach about these in our certification, but a history effect is something having external, taking place external to your testing environment during the duration of the test. That might, for instance, let's suppose that you are
testing for a week and on the third week there's a television special about the same product or offering that you have, that television special might skew the numbers because many, many foreign people could be searching for that particular product or just the opposite could be frightened away from that particular product. We're seeing that happen in real estate based on a lot of negative publicity right now taking place in relation to the mortgage business. So you're looking for patterns and almost most importantly, you're looking for interrupts to patterns because they are cues to you that something may not be right. And let's keep moving, all right? So what do we do here? We analyze the amount of variation. I'm trying to review. I'm trying to, again, bring this back to something you can apply in the test data and found it to be high. There was a big swing. That's what we're saying. And this, once again, suggests the possibility of some hidden difference, something that occurred that we don't know about. Whenever we see a wide swing...
I'm going to try to bring this home to a very simple example that you may have experienced in the past. In fact, write me if you've experienced this. Write it right now into the questions if you have. You may have had a pay-per-click campaign and went one morning to check it and discovered that overnight or within the last 24 hours, you suddenly had a real spike in click-throughs. And then went back to your order process and much to your chagrin discovered that you didn't really have a spike in orders. You just had a lot more people clicking through the pay-per-click ad to your site.
What happened then is you got a high pay-per-click bill but no return on that investment. Suddenly your bill spikes because you're paying by the click. But when you look closer, it didn't really translate into revenue for you. Well, it's that variation. It's that spike. It's that difference in pattern that's a sign for you something is wrong. I was engaged in a test one time and that happened. Jimmy, do you recall that? Small pay-per-click search engines. There was a huge spike in one of the small pay-per-click search engines in traffic. And I looked at my partner that we were studying, a research partner, and saw this high...
amount of traffic or click-through and absolutely no impact on sales. That meant a major bill coming to the partner. Well, we contacted the search engine, and they said everything was working fine. What they didn't know is that we had a number of research partners from around the country using that search engine, and we were studying, and all of them had the same spike at the same time. I had to get on the phone with the chief medical officer and say, look, you can't hold out on us. Something went wrong. And if it hadn't went wrong, we wouldn't see this pattern across these various sites that we're monitoring.
Does that make sense to you to see how that can help you realize that something isn't right? Now, I have someone named Rafiq who just said, I agree with this with a large apostrophe. And that's exactly right. And I also had someone talk about what are these instrumentation effects. That's when you get a problem with your actual, with the actual tool you're using to measure. So it might be the third-party provider. There are several of them out there you could be using. Or it might be something on your website. It might be a problem with the code you're using. It might be a problem with the server. But those things can really affect your validity, and you've got to watch for them.
We teach this in great depth in our certification class. All right, so let's keep moving. So here's another case study that I think you'll find really interesting. Really, this is case study four, Bob. I think that should be notated on the slides. Here we received an email from a longtime student with a question that dealt specifically with the interpretation of results. So here we have a student, been through our classes, sends us a tough problem. It's a problem they're experiencing, and they're saying, can you please help us understand what's going on? Now, they're very capable, this student, and they were working very hard to run valid tests, and to their credit, they were monitoring things carefully.
So they had a four-treatment test that showed a significant difference among the pages tested, but they couldn't see if the difference was significant enough that they could absolutely say one was a winner over the other. By the way, be careful in single-factorial design. Running many treatments really requires you to have a much larger sample size sometimes. It means the test could take much longer. Also, be careful with these new multivariable test approaches. There's some real weaknesses there. I mean, I just saw a study where they did over a billion variations. I was just discussing it with one of my scientists group.
And there was a multivariable test with a billion variations, but the actual gain was insignificant. Most of us don't have enough traffic to test some of these major multivariable tests or, you know, to run these kinds of multivariable tests. And even when we do, there's a lot of issues with actually gaining a significant conversion victory. And there's reasons that we'll talk about later. So let's go back to this. So here we are with the treatment. It's a red car. This is the first treatment with a customary square start box. The second was a family in a car. The third was men falling off a log. And the fourth was a red car with a round start box.
So those were the four, and we were testing them. The idea was which one would increase click-through. So four graphics on a landing page, and we're trying to improve click-through. Here's what we discovered. The red car square was 17.79%. Time in the car was 17.84%. Min falling was 17.52%. Red car round was 17.01%. I want to stop you right there. Before we drill down with this, I'd like to just point something out for you. Good testing is a science or a skill.
Knowing how to test well is a science. Knowing what to test is an art. The problem that you're looking at here really is that there's no significant category jumps. All four of these ads, even though they look different, are still the same category. Correction? Just tell me on the phone. I don't mind. I know it. Yeah, that I'm aware of. Bob came in here to make sure I knew something. We did not run this test. This is what the students submitted, but I am aware of that. Thank you, Bob. My point is that...
that even if you learn how to test, what's most important is that you discover the art of knowing what to test. That's where the conversion index and all the things we talk about regarding landing pages, the way you build your treatments. Jimmy Ellis is on the phone, and he is one of the best in the world at looking at a site and figuring out what the treatment is that should be tested. Jimmy, do you have any thoughts before I go on? No, no, at this point. No, no, you're doing a great job. I mean, you have to frame it, but there's not too much to add right there. Okay, that's fine, friend. Feel free if you have something. Jimmy's the director of optimization in our group. Now, if you look at what this data set, here's what you need to understand.
There was such a huge sample size, because this is a high-volume major site, even though there's not that large a difference between these four, there was a big enough sample size to indicate a significant difference among the performance of the four treatments. But there's a problem, because if the first treatment is the control page, then the sample size is not big enough to say that the second treatment is significantly better than the control. In fact, because of the difference between the first and the second being so small, it would take five years to accumulate a special sample size to conclude this with a 95% level of confidence. Now look at that. That's a problem.
It just wouldn't be a problem right now. Just for those of you that are new to this, if the red car square was 70.79% and the family car was 28%, we wouldn't have the problem we're experiencing right now. It would be such a difference in terms of performance that we could use the current sample size in terms of traffic and say, clearly, this one is a winner. Number two, family in a car. However, that's not what happens. So what would you do? Now, by the way, lest I lose any of you right now, this is really common. You can be in this problem tomorrow if you're testing. So what do you do next? Well, let's go to the next slide.
So we couldn't conclusively say that the first three treatments are better than the other using the data the student has collected the date. But when we took a closer look at treatment four, we saw something very important. And it showed a difference between it and all the other treatments. And there was one significant difference. Let's look at it. Treatments one, two, and three were squared. Treatment four was rounded. Now, are you telling me that whether or not the corners are round or square can impact conversion? You bet it can. I've changed two words on an entire page and seen conversion spike. Two words. Jimmy, you recall the test we did with the New York Times and those two words at the bottom? Absolutely. And we saw a huge gain.
So here we are. We're looking at this and presuming that the red car image is the same one between treatment one and treatment four. Then the shape of the start box might have caused some drop or a significant drop in, quote, starts, which is very important. Quote, starts are the people that begin the process. We were measuring end-to-end, weren't we, or did it just click through? Were they measuring end-to-end? Okay. Both were measured, but what the primary test was actually preparing in terms of its protocol was people come to the page and start the quote. That was the key. Arrival to start. So it may well be that the square start box
is superior. Because if you look, treatment 4 had the lowest response of them all. And if you throw out treatment 4, then you've got really tight differences between these various... you know, these four, the remaining three. So the point you need to understand, if you're on the line, you say, okay, so what does this mean to me? It means that if I was in your position, if this were you, the first thing you'd want to do is design these tests now using the square box and consider that there may be something with this round box that makes the difference, or drop to a single factorial test and test the red car square against the red car round and use just those two again. Moreover, you might actually change other parts of the treatment but feature the difference between the square and the round and see if you see any difference. That's if you're willing to gamble that another test might...
provide a larger lift or that you might run the test and not have a significant data set or know why because of the size of your variable cluster. All right, guys. You've been superb. It's 5 o'clock. I have a list of things. I'm going to show them to you. We're going to mail these to you. In fact, here's what we'll do. Bob, because it's 5 o'clock and we should stop, and I know they're going to get the journal issue, mail this whole set that you have, this slide and the next slide, in an email body to them so they don't have to wait to get these conclusions. What you're looking at right here is the key things that you can do.
These are the things you can learn from what we've been teaching you. And if you look at the next slide, there's more. Just skip ahead for a moment. Let's back up. One is, you know, you need to value the null conclusion, and we explain that in the paragraph that follows. The other thing is you want to look for patterns. We talked about that. The third thing you can do is compare secondary or non-test design measures, new sets of numbers that you didn't originally plan to test. The fourth thing you can do is look for connections between the treatment's attributes and patterns of purchase behavior. We talked about that. The fifth thing you can do is look for patterns of seasonal performance difference among the treatments based on time.
And that means seasonal from a statistical standpoint, by the way, not necessarily from a calendar standpoint in terms of holidays, et cetera. Number five, search for and protect yourself against validity threats. We talked about three kinds of validity threats, really four. And six, keep careful historical records because going back and comparing this test with things you've done in the past can help you draw certain inferences, certain assumptions, and that's an important piece. We will probably try to mail these six steps out to all of you that came into this call so that you can take more time to read them. I need your help. Don't log off yet because here's my issue.
This has been a fairly complex session, and I'm afraid I might have been too complex or not enough. Could you give me feedback? I really need to hear from you. I don't know if we lost you as we went through this or we helped you, and all we want to know is the truth so we can keep serving you better and better. So if you'll take a moment to give us some feedback on whether you found this helpful or not, I think that would be splendid. And we'll study it, and then we'll prepare if we think about how we present our next clinic. If you did find this difficult, bear in mind you're going to get it all via the brief that we mail from the journal, and you'll be able to take your time, study it, and go through it in more detail.
I thank everyone online for their trust. If you did find this helpful, we're going to be posting the actual audio on our website, and you'll be able to follow that. You can recommend a friend. If you can tell a friend about these particular briefings, we would think we would be very grateful. They're always free. Take care.