The analyst that was involved was Jalali Hartman, and Jalali has, of course, been with us for a long time. He also runs e-growth labs and has been deeply engaged in research for many years. And with Jalali online, I'm going to be sharing with him at times the chair as he speaks back and forth and helps me. He is intimate with the partners associated here and has been involved in this research in depth. So, Jalali, welcome to the call.
And in the meantime, there are other team members we have. In the old days, we introduced all of them to you, but it takes too long now, and there are so many. We're just going to continue and allow them to jump in as they need to if they have points to add. So let's get started right away. And the topic has been announced and is a vital topic, and essentially it addresses conversion issues that result from the technology barrier that many of us experience. Due little anomalies such as bugs or database exceptions, lost emails, server timeouts, or changed email addresses will significantly reduce revenue.
We're going to look at some data sets. We're going to look at a 60-day experiment with a major subscription website. And we're going to discover some interesting points regarding the capacity for technology to either serve us or to hinder us. And it may sound unusual at first, but we have discovered that our technology can actually get in the way when it comes to conversion. And I'm not just talking about technology problems. I'm talking about the associated breakthroughs in technology that allow us to present more options and accomplish more than we could in the past.
And so Jalali will be working with us as we look at the data sets. We often forget that the number one objective of a website, in most cases, is to somehow move a customer to the place where they're prepared to buy. Now, if it's a lead generation site, that may be only acquiring enough information in order to complete a transaction offline. If it is a retailer, it may simply be moving them off the homepage into the level where they can actually see the buy button and click. But I can tell you right now that in most sites that I review, there is serious confusion when you approach the page, and there are what we call conflated objectives. And those objectives...
get in the way of each other. And often this happens because of turf wars and over the site real estate, or it happens because a site was originally developed to accomplish one objective, and then later as it grew, competing objectives were introduced until the eye path itself is convoluted and until we start to see a performance decay, but we can't really put our finger as to what the issue is. In this case, if we're crystal clear regarding our objective, then we need to realize something. Sometimes the architecture we need to manage our business is really not the same architecture we need to market our business. Well, that's an important quote. I didn't write it, but I like it. I think Jalali wrote it. Sometimes the architecture we need to manage our business is not the same architecture we need to market our business. And essentially running a scalable, secure, integrated web infrastructure is very challenging. And a good CTO does not want to have to change order processes every week. But because marketing is testing, marketing departments are often appealing to technology departments for greater flexibility and agility. And the solution really is to create some type of testing ecosystem where we can perfect our websites and microtest.
I have often said that the technology spend should be considered a marketing spend. I've said that the customer service spend should be considered a marketing spend. Now, for those of you with financial training or MBAs, I'm really talking about the difference between cost accounting or general gap accounting and actual management accounting for the purposes of smart management. And with management accounting, there's much more flexibility to how you look at things. On one level, you may separate these, and you may set them apart in terms of their various cost centers as individual business units or contributors to your net differential. But I can tell you right now, when I'm thinking about how to grow an organization,
there's a large chunk of the technology span that I consider marketing span. The problem is, when you set these departments up individually, you often have competition between marketing and technology, and this can create problems, as we've seen. We're going to look at a case study. I'm just going to ask Jalali to describe that as he describes the actual test that we designed and what we went to work with. As we're moving towards that, I just will point out something that we have here, and that is that much of our common online marketing verbiage comes from technology roofs. And I think it's interesting. Herman Glenn worked with Netscape to implement the first.
NewYorkTimes.com site, and Jalali, I'll let you mention a bit about that, but we've discovered that often that the submit button itself originated from a programming statement in SQL or SQL. So why is it used so widely today? It's submit really a strong marketing word. I think most of you know it isn't if you've done testing. And in fact, it can mitigate conversion. I've looked at that in the past, and I want to just point it out now in the context of this particular study, and I'm going to invite Jalali now to describe the case we're going to be looking at in a few moments. Okay, thanks, Lynn. Well, there's probably a couple of things that everybody has in common on the phone right now.
Things have evolved and changed, and how we all scramble to get our websites up and get them in a better situation. If you're in marketing, you're probably waiting right now for technology to do something for you in order to get you some, if you're in technology, trying to hit a deadline and trying to make what you may sort of feel like is an insignificant change to a website for somebody in the marketing group. What we're going to go over today is essentially two companies where we ran into some significant revenue, not because of a technical, they're originally designed. And when we looked into it, we found just that it was designed for management of our business, and they weren't designed for flexibility.
and for us to market easily and what's happened to make it easy to buy. You identify a major where you're losing just tons of customers. The larger the company, the more things seem to get. The 60-day test where we actually created a mirror of basic forms. And what we found in that process by going through would have experienced how they got live site, which are areas where people are having issues that you just don't know. They just sort of get lost to your website. So their product is a subscription product. They're selling information. And actually a pretty good registration funnel, the registration process.
essentially gave you access to another level of content. It was based on an email address and a password. The numbers ended up getting the majority of people trying to sign up for this paid product. And of those, 225 came through our dummy form, thought they submitted valid payment information, thought they had successfully completed it. But when we went to actually try to run them through, there was some issue with their credit card, there was some issue with their username or password already being on file for another account. And essentially, we were unable to get them signed up or resolve it. And about 22% loss in revenue strength, these business rules that the company had set up.
thinking it would make for easier management of customers and single sign-ons and that kind of thing was actually preventing people from signing up and they'd lost revenue. And I just want to interject a couple of things about this. Jalali's going to drill down. We're going to talk a little bit more deeply because I think most of us on the line understand that technology can get in the way. I think most of us on the line probably experience some frustration with our technology and its limitations. We're going to actually show you some things you may not have thought of before as to what could be causing problems in your own site, and we're also going to talk about some remedies. Moreover, I just want to point out that Jalali is particularly...
qualified to talk about this. And I'm going to tell you some things about him that he won't tell you about himself. I just think it will help you to understand where he's going and what experience he's working out of when he begins to make these suggestions. For one, in his 20s, he co-founded a dot-com company, which has received more than $13 million in BC and is, I think, profitable and is the major player in its niche now. It's a solid group. He's been with us and our research group. He worked full-time despite his ownership in the original group throughout 2006 and has now launched eGrowth Labs, which is working in cooperation with us on a number of research projects. He's overseen the work with the New York Times. He's overseen the work with...
with TheStreet.com and other major groups and has been deeply engaged in solving these issues. I've seen him take conversion in some of these groups up by several hundred percentile. And so as we were planning this research project and when I saw the data that he was assimilating, I was really quite impressed because I found out that what he was doing was getting underneath the system itself. So many times these calls, the kinds of calls we're having right now, are about the obvious. I mean, they seem obvious to many of you who have learned birth practices, and that is, you know, improve this button, change this headline.
You know, work here to improve the flow, reduce the number of steps. We're all starting to learn that. I've watched the IQ, more or less, go up across the net when it comes to this particular area of conversion. But I think we're moving into an area where you need a more sophisticated understanding in order to continue to scale up the kinds of conversion increases we're seeing. I can tell you that with the times we saw over 1,000% increase in conversion over a period of 18 months and then testing. And you're not going to achieve that unless you do what Jalali did here. And if you were trying to follow this test and imagine how it might have looked if it was on your own system, I was just listening to you, Jalali, and I thought I would clarify a couple of things that might not be clear to them.
With this ecosystem he built, he simply created another platform. We call it Growth Engine, and the Growth Engine allows us to test a mirrored process on our own rather tight, sophisticated system that gives us flawless numbers and is incredibly reliable. We watched the orders come in, but rather than processing them automatically, we were manually processing the orders. And it is when we went back to process them properly through the normal channel or system that they would have been done automatically had they been on the research partner's own platform that we discovered we couldn't even process these. This helped us experience firsthand the frustration that the customer would be going through, and that's where we began to learn of the kinds of problems we're discussing now. So, Jolly, I haven't kind of given a bit of that, and I probably want to talk a little bit when we get to this conversion formula coming. I'm just going to let you go back and continue, and we'll just walk through this back and forth together. Okay, sure. Well, basically, this is what we're finding. I think Flint mentioned it before that marketing is testing, and it may seem sometimes like marketing experiments because that's what we do. But what we're finding is what your competitor is doing is probably not really working, and they're really...
and there are certain things that you know work better than others. But honestly, the only way to move through these issues and increase your conversion is just try different things and test different things and understand what has a goal. That's what we were trying to do here is just really understand, well, 22% of the people aren't making it through the system. Why is it? And what can we do? Obviously, we can do all kinds of changes in pricing and our advertising and put different headlines on the product, but it's hard to get a 22% increase right off the bat. So we started a bit a little deeper in terms of what three to four people per day from having this. This was not an easy test run. We had someone work.
answering phones and keying in orders and logging error messages. But if you go to the next slide, Mike, and I think this would be an example of a friction. If you just take where, you know, look at your numbers and look at where you're losing everybody, and it's very likely that you lose people either in just a navigation or in a registration process is where you're to enter a username and password, enter some, and so on. So if you want to give a little background on this.
This is one of our indexes. We have a whole group of formulas that are patent pending as part of the research that we do. We have used this over and over again to grow companies like Britannica, Encyclopedia Britannica, and others. We've seen multi-hundred percentile increases. And I'm not going to explain it in depth because I think it will take us off task today. Some of you have heard us talk about this. If you've been in our certification program where we teach you how to conduct tests.
We teach this in depth, and along with it, we teach you the testing formula that we have developed, and it's proprietary, and when you utilize it, it helps you get a very valid result, and it's for single variable testing and also variable cluster, which is an approximation of multivariable and much simpler. Bottom line is you can get that information in certification, and we teach that program. Some of you are probably already aware of the fact that we're just now booking up in one of the classes that we have. We're also giving away an A-B split testing tool for those people who register for that particular program and that are participating in this clinic. It's rare that I do something like this in a clinic. It probably is the first time I've ever brought it up. But I can tell you that I was given a note. I'm looking at the paper in front of me as I got ready to speak, reminding people that we're giving this A-B split testing tool away and that you can register today on our site at marketingexperiments.com for the certification program and online testing in a pay-per-click. Really, the formula was here for this reason. Most of us, when we apply a formula like this,
are making a certain assumption. And that's as if we were trying to improve the motivation of the user, the clarity of the value proposition, trying to reduce incentives and friction and reduce anxiety in a vacuum, as if we were operating on a perfect system. But there really is no perfect system. And so there comes a point when you've done all of this and you still can't get the best result. And we found, essentially, that there were three types of errors in this project. We're going to show you another one in a moment, but let me let Jolly come in and talk about this research test and the errors we discovered. Okay, well, the first error is a common one, and most of us, most of you,
analysis of their credit card charges, especially if you have large recurring revenue streams coming from credit card billings. Credit card issues are probably the number one technical issue that we run into, where people have about or think they do, but for some reason it's not working, they can't get an order placed. And usually what happens, if you look at that, the error messages associated are written by the developers. A lot of times they don't make any sense. They're not confirmation to try to figure out what's actually. So they run into this issue with their credit card. There's no clear solution. And usually what most people do is they have so many checks and balances because they don't want to from a development perspective.
From an engineering perspective, you don't want to have all of these exceptions. You sort of want people to get their information into the system right on the front end so you don't have to deal with all of this incorrect data in your database. But what we're finding is that if you can run a test and just let people go through, essentially turn off validation and let people go through and just look at your conversion and look how many people are actually putting bogus information. A test that I did recently was exactly that. I took validation off the card, and you would think you would just get tons of fraudulent orders. So there's no validation. As far as the person knows, they're signed up. They can enter any credit card number they want as long as it was. You know, honestly, out of two or three where people are pissed, but most people, confirmation.
or exceptions where there was an issue, it gets spit out, and somebody actually physically calls them. In this particular case, the lifetime value of a customer could have, literally, you could have a customer service, and also it gets a $10 incentive for everyone that they can recover. And you'd be amazed at how many of those you can recover and how it adds up. The $100,000 in, it sounds like a lot of revenue, but it's only 3.75 orders a day. So if you can recover that, I use name and password for the site. And so when they went to answer the same email address that they've been using, you have them in the process, and then you send them into this loop where they're having to check their email and recover a password. People just never, you do not want them closing the browser down or opening their email. Let them through the process.
The final one we saw in this case was there was a business rule that had been set where the potential in their account, even if, you know, essentially they were trying to come back and pay again and sign up again, this was really set up to take free trials over and over. I think it was something that probably came out of a requirement of people sitting around in a meeting somewhere saying, well, we need to put this in place to prevent fraud. But when you really look at it, this is a way to keep the product. It just, you know, it wasn't happening. You might get the occasional person doing it. So all these things are established. Usually, you know, as architects and as developers and engineers, you're building the best system that you can, but you don't necessarily always take, you know, most development teams, most CTOs are responsible for it.
a functional site and, you know, respective, not necessarily. You don't want any manual interface. I think, you know, essentially if you don't get anything from this conference, the math is done on a piece of paper and just do the math on what is each customer worth to, you know, how much do you make to pay someone full-time? Let me, if I can, Jalali, well, go ahead, let me make sure you finish that. Yeah, I'm done with those three points. Go ahead. I want to just do a couple things for those. You know, I'm watching questions that are coming in and looking ahead at what we've got to cover.
There's another case study coming up. I found this one really fascinating. It engaged multiple tracking tools, and we actually compared the analytics and discovered where a lot of money was being lost. So stand by for that. It's coming in just a few moments. Jolly, I think I'm going to move kind of rapidly through the part here that talks about what they can do to prevent this, and then I'm going to get us into the next case study. I think that if you'll notice on the slide coming up here, there's several things that you can do.
First of all, place an order yourself. I'm shocked at how many marketing companies don't do this and don't do it frequently enough, how many marketing departments are not truly monitoring your system. I'd like to dwell on this point. I am amazed at how out of touch some people are with their own system. But I won't dwell on it except to say, place an order yourself, do it frequently. Call your customer service plan and pretend you're having an issue. We all know how to do that, but it's important. Use the forget password feature. I've done that with many companies, and the disaster that ensued and the confusion was shocking. And I've done an online plan while I was talking to major leaders in the organization, and they have been embarrassed because they didn't realize how horrific their own system was.
The 800 number on each page, recording the types of issues, those are critical, that can help you. Investigating where you lose the most customers during the registration process, study the funnel. In our growth engine, we can see it eight at a time, exactly where they're dropping off, and that's where we focus our attention. Look at the credit card figures and the error messages, and I like number seven, take away all the checks for a little bit and verify. Jolali mentioned this earlier. It's a remarkable way to discover what's really going on in your system. Now, because I think those are not necessarily profound points, but they're useful, we've listed them. I think the next study, however, is quite interesting, and it talks about how inaccurate reporting nearly cost us thousands of dollars in lost advertising.
And essentially what we discovered was that registration errors were not the only source of confusion. And sometimes web metrics are creating a significant problem for us, and I'm going to let Jalali discuss that. There's other problems that aren't even in this report, Jalali, I may mention at the end regarding web metrics and how they can invalidate, you know, how they can invalidate the instrumentation that you're using to complete the process. But why don't you tell them about this study because I know you saw this one, Tor saw this one as well. Okay. Well, essentially one of the biggest problems now, it used to be that they didn't have metrics. What I'm saying a lot of is that people end up getting steered in the wrong, and honestly the only reason we even looked into it, you know, inaccurate conversion,
and then only to be disappointed later. So we didn't want the executive team to think, okay, this is a huge success. We started looking at the numbers. We did a three-way validation. We did three types of tracking on the same site and sort of compared things. Go to the next slide, Mike. And this is what we found. So this was a, again, this was a secret, but we played a period that gave us 17. Our in-house tracking, so what we've been using all along, showed 800 unique visitors. 1.9% conversion. That seemed more realistic to us. So we just sort of assumed that Google Analytics must be wrong and found 1,016 unique dollars. Wait a minute. Now we've been using stock, not wanting to run information.
There's a lot of confusion about it's actually a simple fix of how you're treating. If you're counting, especially if you're counting unique visitors or page views, these are all going to be completely different numbers. It doesn't necessarily matter which one you use. We use unique visits or unique visitors, I'm sorry, because I want to actually saw and offer how many people. But the most important thing is comparing two data sets. One will be unique visits and one will be unique visitors. And there's a huge difference using the actual numbers. When we first looked at it, 8,500 people, 106 people purchased. That was a reasonable conversion rate. What we decided to do again was to look a little bit deeper.
set of invalid clicks that was bought or something like a common occurrence. Usually they do a good job of reversing some of that stuff back out if you're just basing it on how many people, different tracking systems that we use. And we'll click on a link. And because most of these tracking systems involve me or redirect or something like show a click, we don't show any visit or session. That was one issue. Google was hot. They sent more traffic to us. And actually, the other thing that was when we dug into the log file and... Is it scan alert? Yeah. Yes, actually. All right. Keep going. And it was taking out some of these clicks. So what we had to do, which if you've ever had to do it, download the server, certainly any enterprise level. We were using Rackspace in this.
where you can actually analyze your log files click by click what was happening and who these people were. And we started to see patterns, large groups of users coming from Indian or offshore and so on. So what we did is we went through and started filtering. I never found an exact match, and we were just getting miserable shopping cart abandonment rates. And what we found was that Google and Yahoo were adding every single product to it. We're getting thousands and thousands of people trying to buy, and they're not continuing. What is the issue here? And we poured over to people, and, again, that was those numbers, but it wasn't actually a live person. I'll show you how it all sort of evened. 948 unique visitors and 76.
Excellent. Excellent. Now, Jelani points out something, and Allison, you can take us to the next slide. Some of this ought to frighten you because you realize you're trying to make marketing decisions without knowing for certain whether or not the data you're basing those decisions on is valid. Had we depended on the original data, we would have based our PPC budget estimates on a 1.9% conversion versus a 17.3% conversion. That's 800% higher.
And if we hadn't studied the data, we would assume that we could not possibly spend more than 51 cents per click while the actual break-even point was at $4 a click. In our pay-per-click certification, we lay out a formula for how you accomplish this. And the reality is, no matter what we teach in the pay-per-click certification program, if you don't know, if you're not getting good analytics, if you don't understand how to properly test, you're likely to be making decisions based on faulty data. And there's some things you can do to solve the issue. I mean, for one, you can install Google Analytics. And we're not getting anything to tell you that. It's just something that we've used.
in this particular case, and you may want to do the same. You can look at your log files and compare them to your core site tracking. Are the numbers close? This is tedious work. I remember one time when we had to do it on a huge experiment. We were tracking pay-per-click conversion for all of the shopping engines, and we needed to go down to the log file to actually validate it, and it was a long, hard project. But it's worth it sometimes. Number three, you can make sure you understand the difference between unique visits, page views, sessions, and unique visitors, and make certain you're being consistent, particularly when two departments are talking. Now, there's questions coming in. Feel free to keep sending your questions, because I'm going to...
We're going to try to answer those in just a moment. But I think the big lesson here is that while the Internet is certainly progressing and while our technology is improving, we're at the point now where we're just technology savvy enough to be dangerous in some cases. We can do a lot of things with our sites that we couldn't do two years ago, but some of those things are getting in the way. Or some of the architecture that we designed two or three years ago was designed without a clue as to what the marketing impact would be. you may be basing very important decisions right now based on faulty data. And a lot of times, we need to just spend the extra time to go back and validate our assumptions. It's worth it. In many cases, it can make a significant difference. Now, there's questions coming in, and I need to see what I can do. A lot of those are being answered by our staff online, but if there's a question regarding this actual session, this lecture, then we'll see if we can help you. One of the common questions I keep hearing is, can you get the presentation slides? The entire brief, which contains all the data, it's actually a plain text email document, but it contains all of this and more.
is going to be released to all of our subscribers. You should be getting that soon. It's simply taken after this presentation and it's revised one more time and then it'll be sent out. Make sure you're on our list. If you signed up for this, we assume you're on our subscription list. You know, it's free. You can get the data that way. By the way, there's millions of dollars worth of free data on that site. It's worthwhile to go through marketingexperience.com and get those reports and spend some time with them. Let me see. Are we going to offer a Google Analytics trend in the future?
We're being asked that question right now by Google, so get back with us and we'll let you know. Any concerns about Google learning too much about your business? I don't know how to answer that. That's another question that's come in. It's true. They're learning a great deal about every business right now, it seems. I think that's an individual choice, but they do begin to feel like Big Brother. Hopefully they'll stick with their motto and do no evil. How does core metrics compare to Google as far as filtering and data clicks and capturing good data? You want to answer that, Jalali?
Yeah, I don't see the questions. Could you repeat that? How does core metrics compare to Google as far as filtering invalid clicks and capturing good data? The core metrics? Oh, I have absolutely no idea how they filter. I think the point of the system you're using, you're going to want to do a – what I'm sort of recommending is do it. If you don't have time to do the audit, get somebody – do the occasional comparison and just see how it comes out.
Yeah, that's good. And, you know, if we don't answer your question now and you need to email us and get an answer, we'll do our best. There's a lot of questions coming in now. Which program did Jalali use to download server log files? We did this at Rackspace, and they have a simple utility that allows you to do that, and that's all that we did. I think, Jalali, you just imported them into an Excel spreadsheet, correct? Yeah, actually, most servers have, in that case, a common one is a program called Plesk. And within Plesk, there's a... Spell that, Jalali. Spell that. P-L-E-S-K. Okay. Okay. Let me see.
What if we can't wait for your next testing? Is there a way to get into the class in the middle of it? Yeah, you can register for the certification program today. They're closing one off. I think yesterday was the end of the early registration for some of them. But if you just go to marketingexperiments.com, I don't want to use these sessions to plug the course, but I try to answer your question. You can certification at marketingexperiments, and I think there's some classes you can still get into. They fill up, and they fill up capacity very fast. We launched two classes, I think, last week, and there was like 170 students in the first one, and something approaching that in the second.
If you're paying for 3,500 clicks but only 946 you're getting to your website, shouldn't you always be based on the 3,500 clicks? Well, what we're really calculating is the conversion rate in this experiment. We would assume that you're going to get a refund on those clicks from Google. We were capturing and making these adjustments immediately. Google will capture some of these through a variety of methods. We've written on this earlier. If you're going to consistently lose that kind of traffic, then that's going to impact your ROI. But what we're looking for is a projected ROI based on a genuine conversion number, and that's why we have to get down deeper and understand that. Someone says, would it be safe to generalize?
Let's click on that so I can read the full question, Allison. Would it be safe to generalize and say that Google Analytics is better than most if one does not have time to audit what's measuring? I'm not sure of that yet. Ask me after we've done more analysis and we can answer that question for you. Listen, I want to thank everyone. If we haven't answered your question yet, feel free to email us. We're doing our best. Let us know if you think this subject's been useful. Please take a few moments just to use this same chat feature or the question and answer feature and give us feedback. We read every one of these log files, and we care a lot about what you say now. It helps us plan the next session. Also, if there's a topic, someone talked in here about password flow. That may be a topic. We can't discuss it now, but that may be a topic for a future session. And if that's the case, tell us what you'd like us to be studying. We're planning all of the time. Also, just tell us if this has been helpful today for you, and we'll pay careful attention. We're very grateful again. Thank you for your trust.
We'll be back in about two weeks with another brief for you. In the meantime, you should get this brief in your email. Thank you, Jalali. You did a superb job in the whole research project.