If you look back at the Sampa sign up form, it has been through 4 or 5 different versions. At one point we asked you about 15 questions. Then we worked with one local UX expert to simplify a bit, then we worked with a Portland-based UX firm to simplify it even more and we had just 8 inputs:
Name
Email
Confirmation of email
Password
Confirmation of password
Agree to Terms of use
Web address
CAPTCHA
After reading the book "Web Form Design Fillin in the Blanks" (by Luke Wroblewski) over the 4th of July, I decided to take a closer look at our sign up form again.
I am always afraid of making any change to our sign up pipeline since we have such fantastic conversion rates that is easy to assume something, make a change and it to backfire.
Well, there was at least one change I've made that was a no-brainer. The Terms of Use checkbox wasn't checked by default and I changed that.
But that's not the interesting part. On the book, Luke mentions that a lot of the things we ask on forms are limitations of the application itself and that it doesn't add value to the customer ("inside out").
You can clearly see some of that on Sampa. Why do we ask twice for your email? Why do we ask twice for your password? Why do we need your name? Why do we need CAPTCHA?
We have good answers for all of the above, and we A/B tested several scenarios over the last 2 years to know that we are close to the right combination that yields maximum conversion and retention. But one piece was never tested and it was off: CAPTCHA.
Why do we need CAPTCHA? Like any other free web service that you sign up to (Hotmail, GMail, Ticket Master, Blogger, etc.) CAPTCHA is necessary to prevent automated bots from creating hundreds or thousands of fake accounts, either for Spam or Link Farm. So it's a necessary evil that good people have to go through so we can keep the bad people out, or is it?
Well, last Monday at 5:13 PM we removed CAPTCHA from Sampa. It wasn't easy and clean, but we created a set of tests and rules that will make us not display a CAPTCHA about 99% of the time. I can't really disclose the tests we make since that's the secret you want to keep out of the hands of bot creators.
The result: 9.2% improvement on our conversion rate!
The entire team is just shocked that such a huge percentage of users wanted to sign up to Sampa and either didn't pass the CAPTCHA (less likely) or found the form more intimidating with a CAPTCHA (more likely). The third explanation which we've been talking about is that with less vertical space, the "Continue" button is above the fold more often and give users a better sense of how short this form is.
Comments (10) for "A 10% improvement on convers...
Why did I have to enter a CAPTCHA now anyways to post this comment? I am going to try to understand the new logic. Probably has something to do with email domains and whether they seem legit or not.
By Hacker - 7/17/2008 2:31 PM
"The Terms of Use checkbox wasn't checked by default and I changed that."
I am not sure I agree with this entirely. While I can appreciate the effort the streamline the process, I believe that actually clicking on the checkbox is the legal acceptance of the Terms of Use.
If you enabled it on default for me I can very easily argue that I never saw, read or accepted the Terms of Use.
By Harsha - 7/20/2008 10:02 PM
Hmmm, 9.2% isn't 10%.
Never will be to the best of my knowledge.
Even so, a 9.2% increase is impressive for just removing the widget.
Heh, I guess you don't want pernickety little pedants like me signing up - I'm getting a CAPTCHA on the sign-up page. ;o)
By Sainkho - 7/21/2008 12:20 AM
Great data!
Could you post two or three examples of the Captchas used on the site?
I've always said all Captchas are not equal and some ar eharder to read (for people) than others. Just curious how readable your guys's Captcha was...
By Zephyr - 7/21/2008 12:35 AM
We only removed the CAPTCHA from the sign up page. Everything else stayed the same, including CAPTCHAS in comments.
These could be radically different results... and I'm not clear which you mean.
We monitor our captcha failures pretty closely, and we're pretty please with our conversion rates for people who come to our signup funnel. Although we do see captcha failures, they don't appear to represent a significant enough rate to suggest that eliminating 100% of those captcha failures would bump our conversions by +9.2%... But I could see them bumping our conversion rate * 1.092.
Anyway, we've switched off captcha on our signup as an experiment... we'll see what the results are.
Hi Brad, we improved the "rate" by 9.2%, not the conversion, which means our conversion went from about 10% to 10.9% conversion.
Our CAPTCHA was dead easy to pass because it only used english language words, so we don't think users were stuck there. It's just that for our target user, CAPTCHA could be somewhat intimidating, uninviting or an extra step on the form.
Thanks for clarifying... by the way, I think we have seen a similar reaction. Like I said we track our captcha failures pretty closely, and we were definitely getting some... in fact we've seen cases where someone tries really hard to get that Captcha right (maybe 4-6 times before success)... but these are rare. Anyway, so far our experiment is inconclusive, although we have seen a small bump since changing this, it's within the normal fluctuations, so, we need more data.
I agree with you that for some target customers (like those of Sampa and Sweat365) captchas are just intimidating and may cause some funnel abandonment.
Subscribe to this Site
Comments (10) for "A 10% improvement on convers...
- 7/21/2008 12:37 AM
December, 2007