Hamburger vs Menu: The Final AB Test

In case you missed this post about an AB test on mobile menu icons, make sure you check out the comments. There are some very interesting insights about A/B testing and its shortcomings.

The post went a tiny bit viral, and suddenly it wasn’t just my mother reading this blog.

Three things I learned:

1. This icon hamburger has lots of names: hamburger, sandwich, and even hotdog ?! What it actually is, is a list icon. We’ve just co-opted it to mean a navigation menu.

2. When something gets noticed, some people get a little mean (source)
po

3. One commenter said I was the Dunning–Kruger effect in action. This phenomenon is when you try to sound clever but are actually a dumbass.

Thanks for the vote of no-confidence.

In this hyper-connected world full of rockstar developers and super-smart designers, I’m humbled on a minute-by-minute basis. I might need to start attaching positive affirmation stickers on my laptop.

The Final Hamburger A/B Test

I do enjoy A/B testing, but conclude what you want. I’m not an expert, nor am I advising anything, but sharing what happened on a single website.

Using a commercial A/B testing service can get very expensive very quickly, and well beyond the budgets of small-time web designers and developers. So, hopefully, these posts are helpful for some of you.

If you are using social sharing buttons, you might find these tests interesting.

Variation 1

Bordered list icon (hamburger).

menuab2@2x

Variation 2

Bordered word menu.

menuab@2x

Results

240,000 unique mobile visitors were served the A/B test.

Variation Unique Visitors Unique Clicks
Hamburger 120543 1211
Menu 121152 1455

abtest3results

The test was large enough to achieve statistical significance.

The MENU button was clicked by 20% more unique visitors than the HAMBURGER button.

Where things get interesting is when we break down the data a little:

Unique Visitors Hamburger Clicks % Menu Clicks %
iOS 148097 906 0.61% 1143 0.77%
Android 87245 216 0.25% 237 0.27%

There is very little difference in the Android user preference, but their lack of engagement is disturbing.

Interpretations

Hamburger icons may appear to be ubiquitous, but they are not the only option.

There is an issue that is much more important:

Android users are almost 3x less likely to click a navigation button than iOS users.

Why?

Caveats

  • These are the results from one website (see more about demographics here).
  • The test was done using some in-house code, so I cannot guarantee the perfect execution of code across all devices. I do not have time or capacity to rigorously test code like the big commercial AB testing services like Optimizely. Bear in mind that to run this test with Optimizely would have cost $859 (I kid you not).
  • I can’t measure intent with this test. I’m measuring clicks on a webpage. Maybe the user thought menu as a list of food to order. Maybe they wondered what the hamburger icon was and tapped it. Who knows. AB testing cannot tell you this.

Further Reading

James Foster
James makes websites, you can follow him on @exisweb.
Filed under AB Tests
Created: March 21, 2014, updated on May 25, 2014

83 Comments

  1. I *like* your focus on data-driven design, for what it’s worth. You’re doing good work, keep it up! Thanks for the info.

    • Thanks so much, good you found it interesting.

    • Seconded! Having intuitions is one thing, testing them out and reporting the results is another – great stuff!

  2. Good to know. But.. what about the magnifying glass icon?
    Thanks for the article!

    • The magnifying glass icon was tested well by the Nielson group (linked to in this article). It seems that users are very familiar with this.

      • Great work, but it’s Nielsen with an e.

      • I wonder if it’s because people are used to it now, versus this icon which is just a starting trend.

  3. In the late 1990s testing showed that a user first looked for position on the screen for action, then words, and then finally the pretty picture. Words are easier to define then pretty pictures.

    I wonder what engagement would be like if the menu/hamburger button was moved to the right side of the screen.

    • will be the end of the world!

    • Nav buttons tend to be top left or top right. To be honest for a righthanded person, a hamburger button on the right is much easier to tap than on the left.

      • This is one of several good reasons why phone operating systems should allow customers to specify whether they are right-handed or left-handed, and make that information available to apps so they can exploit it.

        • That’s an interesting thought. There is nothing to stop app developers from creating an option for users to set that same preference inside their app, despite an OS not feeding it that data.

  4. for what its worth, Facebook puts the word “more” under the hamburger on its app.

  5. I also want to see the same for apps. My experience otherwise, with actual tests of other parts of the UI, is that websites and apps are perceived differently.

    Love to have a solid guideline, but am not sure it’s universal or just for the Web.

    • Actually, I just realized I am getting a test of some other fundamentals done in (I hope!) the next few weeks. It’s going to be an app, so I’ll try to add this into it since we do have a menu.

  6. I have actually AB tested [MORE] for some mobile apps.
    People usually react to their impulses. I’ve found that our organizing/naming of patterns are not at all what people would consider their next logical move. Good stuff.

  7. For the record, I’ve really enjoyed your posts on this Burger Situation (there are always the folks who get a little too excited riding their soapbox! Sounds like you’ve ran into a few since your post went big)

    Anyhow, Re. Android vs iPhone 3x less likely… I can’t say it with any real evidence, but perhaps it shares some of the reasons mentioned in these posts:

    http://daringfireball.net/2014/03/ecosystem_chess_game
    (page search for word ‘elephant’)
    Also:
    http://branch.com/b/android-vs-ios-usage#E3zYpuD4lEE
    (plus theres loads of other posts about the fact iPhone user spend more than android users for both obvious and less obvious reasons, stuff like http://www.businessinsider.com/apple-android-market-share-ecommerce-2014-1)

    So, while it could be a risky suggestion, is it fair to say that the reason why the menu button gets hit more is that *generally* the average iPhone user is just more engaged/savvy/inquisitive etc than the average Android user?

    • Interesting stuff, and so much assumptions. Certainly lower end android devices are much cheaper than a new iPhone. I think this issue far more important than hamburgers. As the android user base continues to grow, why do they engage less with websites?

  8. Unless piggybacking off of known uses, colors and icons have to be learned. There is a point of ubiquity where that is no longer the case though. Best thing to do is ask a cross-section of people what they perceive for directional feedback. And remember, you are not always your target audience.

  9. “Menu” is English. Unless that’s your primary target audience’s native language, I’d recommend hamburgers.

    • I agree with the suggestion of language, but any global web site has the same problem with the rest of its contents. I dont think that put icons instead of phrases is the solution. Moreover, in this case, “Menu” is also used in Spanish (with a diacritic).

    • “Menu” works in Spanish too… Well, “Menú”

      • And Menü works in Germany.

    • The word “Menu” works in several western languages : english, spanish, french, italian, and so on… A lot of people, isn’t it ? :)

      It depends of your targets, ok. But it works in many many cases.

    • But then your whole site would be in a different language anyway, in which case the navigation would be changed as well.

    • I’d contribute the Japanese (“メニュー”), Simplified Chinese (“菜单”) and Traditional Chinese (“選單”) translations.

  10. Hey, James
    Thanks for a great post.
    Can you talk a bit more about why the testing with Optimizely would cost $859? I thought if you were on a Gold plan you’d qualify for the following visitor volumes:
    200,000 monthly visitors + $5 / thousand additional visitors
    Which in your case should cover the 25K users.

    Thanks!

    • The test had 300,000 visits to it.

      300,000 visitors + $5 x 100,000 = $359 (gold monthly) + $500 ($5 per 1,000) = $859

      Optimizely bills by visitors NOT unique visitors. This test had about 300k visitors to it. However for my results I’ve studied unique visits as this takes out any bias from 1 or 2 users who might repeatedly click one of the buttons.

  11. Could it be that the reason that Android users are less prone to clicking a menu button is that they are used to activating a physical button on the device for their menu, as opposed to iPhone’s screen oriented navigation systems?

    • This has been suggested by other commenters. It’s not a good sign for mobile websites, as it seems to show that Android users are far less likely to engage with the site.

      • I don’t think it has much to do with physical buttons as those were depreciated near the beginning of 2012.

        As for why the android users are clicking less? Who knows.. Maybe the content didn’t drive the traffic for that demographic. Maybe the site didn’t render smoothly. Maybe they hate puppies. (I’m willing to bet the implied classism by earlier commenters isn’t really the issue, although I do see the rampant assumption that “all android users are poor/thieves/uncultured” constantly thrown around in the design world).

        While data driven design is great, I feel like there needs to be interview data on *why* they didn’t want to click the menu. There could be some third unexplored factor that isn’t being caught by your test, and making decisions on incomplete data could lead to alienating a whole user base.

        • I think performance is one aspect. I test with a low-end android phone, and it’s not pretty. Its speed of parsing JavaScript is woefully slow, and scroll speed is an issue also…

          • A lot of Android users also use the sliding option to open menu items, rather than reaching up to click a menu button. If you had the sliding functionality (which is a standard operation in native apps), I wonder if you’d get more views.

  12. Very interesting post. Thanks for sharing! Would be very interesting to have a third variation with the icon and the text. Do you tried that?

  13. Thanks again!

  14. One thing you shouldn’t do when making data driven decisions is make assumptions. You should have reached out to optimizely. I’m sure this free advertising is worth a lot more to them than your subscription cost.

    • Did reach out. No response.

    • Interesting. 9 years on and the ideas are still important. Icon-only controls can frustrate users.

  15. Thank you for sharing these results. I’ve seen the hamburguer menu more and more in lot of websites even in large resolution sceeen. Why?

  16. Have you tried (or possibly could try) testing MENU vs NAV?
    I assume it might work better, as websites are assotiated with navigation, while apps with menu.

    Anyways – very interesting case, thanks for sharing your insights :)

    • Patryk, I’ve not been keen to test nav. I think outside of the developer community, “nav” is confusing. To most people navigation means maps.

      I could be wrong, but given the time taken to set up a test properly, it’s not something I will be testing in the near future.

  17. Here are three sources of data which support your finding on iOS vs Android engagement. There are more Android devices but iOS users are more active and spend more money.

    http://static.lukew.com/onedesign_04092013.pdf (page 47)
    http://techland.time.com/2013/04/16/ios-vs-android/
    http://www.slashgear.com/android-vs-ios-personalities-infographic-15171918/

    Part of this may be the demographics. iOS users tend to be more educated, have a higher income, urban and are more likely to be 35 and above.

    Another theory I have, but I have found no data to confirm of refute it, is that there are many *kinds* of Android devices. A Kindle is an Android device. Kindle Fires are probably active but I assume that black and white kindles don’t get used much for apps and web browsing. Also, I believe there are Android dumb phones with a keypad. Those probably get little web use. I wonder if there are other peripheral devices that run android like set-top boxes, thermostats, GPS devices etc. which are not conducive to web browsing. If anyone has any insight on this let me know. :)

    http://static.lukew.com/onedesign_04092013.pdf page 47.
    http://techland.time.com/2013/04/16/ios-vs-android/
    http://www.slashgear.com/android-vs-ios-personalities-infographic-15171918/

    Mike Ryan

    • Some great sources there.

      Yes looking through analytics at all the different devices that run android is bewildering. It presents a real challenge to UI designers as it is so difficult to test with so many difference viewport sizes, device layouts etc.

  18. I would be interested to see if these stats matched in a year.

  19. Although not quite the same thing, this just reminds me of another ‘all links should be blue’ kind of statement. Maybe there is a more appropriate icon.

  20. Final A/B test? under which context? how is the sample of users? it was an online test? for an IxD maybe the best thing would be to run it with real people watching and listen the users reactions and doubts?? ecc.. ecc… ecc..

    • This was part of a series I was writing as I did larger and larger tests.

  21. those results are like ones of a coin toss, not determinative 50 50 at best

  22. The flaw of these kinds of tests is you only measure if they’re opening the menu and not if they’re actually using it. Are they actually accessing one of the menu items after opening it or closing it because they just wanted to glance what’s there out of curiosity?

    iOS has no menu/hamburger buttons anywhere, while in Android the icon is well known and part of the core experience. IMO iOS users see the icon as an alien and are curious to tap it (“hey what’s this, what’s in there?”), while Android users could care less about it when using your site/app (they know the icon and will only open it when they are really willing to navigate a specific part of your site/app).

    • My latest test point to this (I think). Presented with a button (in this case a MENU button on a fixed header), iOS users will click it. Yet they don’t necessarily engage deeper with the site, or go on to click onto other pages.

      However the up shot is that they click on a button called “MENU” far more than the hamburger.

      A/B tests are difficult, and drawing any kind of conclusion is even more complex.

      • Thanks for replying. :)

  23. Great stuff James! Congrats for this article! So, can I use a “menu” button (forget the hamburger) to activate an off canvas panel, like Facebook, or a tab view on top? What is the best choice?

    • That’s something you would want to test. I suggest reading this excellent post about hamburgers vs tab bars. Hint: the author recommends tab or tool bars. Although it is focused on apps rather than websites.

  24. Google have recently started to push their logo + nav pattern for Android apps:

    https://developer.android.com/design/patterns/navigation-drawer.html

    Users click logos on websites. It seems like Google spotted this (and looked at how Amazon were doing it a couple of years back) and decided to make a mobile guideline.

    Several frequently used Android apps like Gmail and YouTube (both google) have this pattern and it makes sense there. Check out Twitter for a hugely confusing pattern in comparison where they don’t do this.

    Naked Wines use the hamburger but on the right hand side of a clean action bar. In that case, it also seems to make more sense as it’s more like an “overflow”.

    • I have seen this appearing, and it’s like a half-hamburger. Again I would want to test this.

  25. Loving your data-driven approach. In my (brief) experience in mobile, the hamburger option was the most confusing from the users’ point of view. I thought at the time that it could be due to lack of users’ familiarity with it. Tbe niche I was in was insurance.

  26. Great article!
    The big advantage of an icon, though, (any icon, but here we talk about the hamburger) is in metalanguage solutions. Though the word “menu” is the same in many languages, it still varies in some, and if the length is more than 6 letters, it will not work in most mobile designs.

Add a Comment

Your email address will not be published. Required fields are marked *