Hamburger vs Menu: The Final AB Test

In case you missed this post about an AB test on mobile menu icons, make sure you check out the comments. There are some very interesting insights about A/B testing and its shortcomings.

The post went a tiny bit viral, and suddenly it wasn’t just my mother reading this blog.

Three things I learned:

1. This icon hamburger has lots of names: hamburger, sandwich, and even hotdog ?! What it actually is, is a list icon. We’ve just co-opted it to mean a navigation menu.

2. When something gets noticed, some people get a little mean (source)

3. One commenter said I was the Dunning–Kruger effect in action. This phenomenon is when you try to sound clever but are actually a dumbass.

Thanks for the vote of no-confidence.

In this hyper-connected world full of rockstar developers and super-smart designers, I’m humbled on a minute-by-minute basis. I might need to start attaching positive affirmation stickers on my laptop.

The Final Hamburger A/B Test

I do enjoy A/B testing, but conclude what you want. I’m not an expert, nor am I advising anything, but sharing what happened on a single website.

Using a commercial A/B testing service can get very expensive very quickly, and well beyond the budgets of small-time web designers and developers. So, hopefully, these posts are helpful for some of you.

If you are using social sharing buttons, you might find these tests interesting.

Variation 1

Bordered list icon (hamburger).


Variation 2

Bordered word menu.



240,000 unique mobile visitors were served the A/B test.

Variation Unique Visitors Unique Clicks
Hamburger 120543 1211
Menu 121152 1455


The test was large enough to achieve statistical significance.

The MENU button was clicked by 20% more unique visitors than the HAMBURGER button.

Where things get interesting is when we break down the data a little:

Unique Visitors Hamburger Clicks % Menu Clicks %
iOS 148097 906 0.61% 1143 0.77%
Android 87245 216 0.25% 237 0.27%

There is very little difference in the Android user preference, but their lack of engagement is disturbing.


Hamburger icons may appear to be ubiquitous, but they are not the only option.

There is an issue that is much more important:

Android users are almost 3x less likely to click a navigation button than iOS users.



  • These are the results from one website (see more about demographics here).
  • The test was done using some in-house code, so I cannot guarantee the perfect execution of code across all devices. I do not have time or capacity to rigorously test code like the big commercial AB testing services like Optimizely. Bear in mind that to run this test with Optimizely would have cost $859 (I kid you not).
  • I can’t measure intent with this test. I’m measuring clicks on a webpage. Maybe the user thought menu as a list of food to order. Maybe they wondered what the hamburger icon was and tapped it. Who knows. AB testing cannot tell you this.

Further Reading

UPDATE: See the great research by NNGroup. It shows how hidden navigation is (unsurprisingly) less discover able, and therefore used less!

Hi, I'm James, and for the last decade I've made a living by making my own blogs and websites.
Updated: July 5, 2017


  1. Very fascinating. Thanks for sharing!

  2. great test, but looks like according to some of your new posts maybe more of an app like navigation are performing better? priority based nav. That correct?

    • You are correct. As time has gone on, I’ve noticed mobile users are engaging with navigation elements less and less. The less visible it is, the more unlikely the user will click on it. So… toolbars or priority nav.

  3. Hi James.
    Great work! I’m using it as reference.
    When this test was runned? (year)

    • Mid 2014. But be sure to read my follow up test

      This subsequent test (end of 2014) showed that there wasn’t much difference between hamburger and text, but that overall interaction with nav buttons was lower.

  4. Hi James,
    just come across this article. Do you know what the current research suggests on our friend or foe the Humburger ( List icon)?

    Has there at all been a change in the pattern of usage since this article in 2014? It’s a very interesting topic and well written and research article.

    • I did a follow up test

      That showed that the hamburger seemed much more used, however overall engagement with site navigation was lower. I think most people know what the hamburger icon means. They think it means “boring stuff is under here”.

  5. Tx James for this great research, and for allowing me to laugh while reading it.
    PS : My laptop has become very recently full of positive stickers tx to my 4 y.o. niece. You should try it.

    • You’re welcome. Hope you got something useful out of it.

      I’m just not ready for stickers on the laptop. That, and the fact that I lease it.

  6. “3 lines” (it does real hamburgers a dis-service…) should be consigned to the dustbin of design history.
    They are meaningless, dull and nearly invisible.

    Personally I almosts never click on 3 lines as I don’t notice them until I’m about to leave – which is almost always because of other design features that reduce navigation and the UI.
    So I leave for a site easier to navigate and with more content visible at first view rather than hidden.
    Anything that hides content – and forces unnecessary scrolling such as a HUGE photo or monotone panel, grey hard to read text (like here!), lack of textured, coloured buttons as “calls for action” – is a turn-off.

    Why have fewer than 1% even clicked even the word Menu?
    May I suggest that it is because it fails to grab attention.
    How about testing a Menu button with a colour gradient and shading for depth?

  7. thanks for sharing! great information.

  8. Using hamburger brings the benefit that don’t have to alter the control for international markets; That means no design changes in the core UI.
    Consistent look and feel helps branding.
    Plus, I’m tired of pandering to users that don’t want to learn how to use their devices.

  9. I’m willing to bet if you filled in that MENU button to look more like a button instead of leaving it as just a border it would get another 20% more clicks.

  10. I don’t want people to click the navigation button. Everything I put behind the navigation button is filler and information specific. I want to tell a story and I want people to follow the flow of the site to get to a desired end state. Number of clicks is a useless metric without context and is usually the opposite of what I want to do.

  11. James – thanks a million for sharing this.

    A/B testing is obviously helpful, but –alongside 1,000 other undefined scenarios, use cases and objectives– this test seems to assume that the most important next step is to click the menu/hamburger. If engagement is defined as a user clicking on the menu, great. Test complete.

    A lot is said about not making the user choose. Sure, we give them the best and most desirable path forward, but I never want to take the choice away if they want to chart their own path.

    It’s also worth noting that some of the most well regarded interactive/digital agencies use this technique regularly. That doesn’t make it right, but you can bet they do a LOT of A/B, even if they keep the results close to their chests.

    All that said, I know you weren’t pushing an opinion one way or the other. I just wanted to add my 2¢ to the conversation as people are quick to condemn a design element that I have found to be extremely useful time and again.

  12. My hunches confirmed. Thank you for making this data-backed A/B-test publicly available.

  13. It’s fascinating to see this test over and over with the same results. Yet, everyone measures how many users click the menu button. The real goal of a page is not to get users to click the menu, is it? So why do we run tests with that as the success metric?

  14. Thanks for taking the time (and money) to do testing and write an in-depth article. I’m not sure where the exact answer lies, but I think the hamburger icon isn’t as bad as you make it out to be. It’s not just you, either — the really popular idea of the day is for people to bash the hamburger icon. Whenever there’s a super popular idea like that, my initial instinct is to resist the herd. Sometimes that instinct is good, sometimes bad. Anyway, though, here’s a more comprehensive overview of my thoughts on the issue:

    • You’re welcome. I wasn’t out to campaign against the humble burger. I just wrote up what the data was showing.

      I don’t see the hamburger disappearing anytime. In fact it be just become better understood as time goes on.

  15. i admire your testing, keep up. really.

  16. Great article!
    The big advantage of an icon, though, (any icon, but here we talk about the hamburger) is in metalanguage solutions. Though the word “menu” is the same in many languages, it still varies in some, and if the length is more than 6 letters, it will not work in most mobile designs.

  17. Loving your data-driven approach. In my (brief) experience in mobile, the hamburger option was the most confusing from the users’ point of view. I thought at the time that it could be due to lack of users’ familiarity with it. Tbe niche I was in was insurance.

  18. Google have recently started to push their logo + nav pattern for Android apps:

    Users click logos on websites. It seems like Google spotted this (and looked at how Amazon were doing it a couple of years back) and decided to make a mobile guideline.

    Several frequently used Android apps like Gmail and YouTube (both google) have this pattern and it makes sense there. Check out Twitter for a hugely confusing pattern in comparison where they don’t do this.

    Naked Wines use the hamburger but on the right hand side of a clean action bar. In that case, it also seems to make more sense as it’s more like an “overflow”.

    • I have seen this appearing, and it’s like a half-hamburger. Again I would want to test this.

  19. Great stuff James! Congrats for this article! So, can I use a “menu” button (forget the hamburger) to activate an off canvas panel, like Facebook, or a tab view on top? What is the best choice?

    • That’s something you would want to test. I suggest reading this excellent post about hamburgers vs tab bars. Hint: the author recommends tab or tool bars. Although it is focused on apps rather than websites.

  20. The flaw of these kinds of tests is you only measure if they’re opening the menu and not if they’re actually using it. Are they actually accessing one of the menu items after opening it or closing it because they just wanted to glance what’s there out of curiosity?

    iOS has no menu/hamburger buttons anywhere, while in Android the icon is well known and part of the core experience. IMO iOS users see the icon as an alien and are curious to tap it (“hey what’s this, what’s in there?”), while Android users could care less about it when using your site/app (they know the icon and will only open it when they are really willing to navigate a specific part of your site/app).

    • My latest test point to this (I think). Presented with a button (in this case a MENU button on a fixed header), iOS users will click it. Yet they don’t necessarily engage deeper with the site, or go on to click onto other pages.

      However the up shot is that they click on a button called “MENU” far more than the hamburger.

      A/B tests are difficult, and drawing any kind of conclusion is even more complex.

      • Thanks for replying. 🙂

  21. those results are like ones of a coin toss, not determinative 50 50 at best

    • I thought the same thing. The difference overall of engagement as a percentage of the clicks is .2%. So small as to be difficult to determine much without repeating the same test several times.

  22. Final A/B test? under which context? how is the sample of users? it was an online test? for an IxD maybe the best thing would be to run it with real people watching and listen the users reactions and doubts?? ecc.. ecc… ecc..

    • This was part of a series I was writing as I did larger and larger tests.

  23. Although not quite the same thing, this just reminds me of another ‘all links should be blue’ kind of statement. Maybe there is a more appropriate icon.

  24. I would be interested to see if these stats matched in a year.

  25. Here are three sources of data which support your finding on iOS vs Android engagement. There are more Android devices but iOS users are more active and spend more money. (page 47)

    Part of this may be the demographics. iOS users tend to be more educated, have a higher income, urban and are more likely to be 35 and above.

    Another theory I have, but I have found no data to confirm of refute it, is that there are many *kinds* of Android devices. A Kindle is an Android device. Kindle Fires are probably active but I assume that black and white kindles don’t get used much for apps and web browsing. Also, I believe there are Android dumb phones with a keypad. Those probably get little web use. I wonder if there are other peripheral devices that run android like set-top boxes, thermostats, GPS devices etc. which are not conducive to web browsing. If anyone has any insight on this let me know. 🙂 page 47.

    Mike Ryan

    • Some great sources there.

      Yes looking through analytics at all the different devices that run android is bewildering. It presents a real challenge to UI designers as it is so difficult to test with so many difference viewport sizes, device layouts etc.

  26. Have you tried (or possibly could try) testing MENU vs NAV?
    I assume it might work better, as websites are assotiated with navigation, while apps with menu.

    Anyways – very interesting case, thanks for sharing your insights 🙂

    • Patryk, I’ve not been keen to test nav. I think outside of the developer community, “nav” is confusing. To most people navigation means maps.

      I could be wrong, but given the time taken to set up a test properly, it’s not something I will be testing in the near future.

    • This was surprising to me, since I thought it was common knowledge, but I agree with James. “Nav” is not known to the general populace (as I found out when I kept using the term in meetings with business analysts of multiple ages and technical knowledge levels at work and they kept asking me what it meant).

      • Anna, excellent observation! This 66 year old BA gets the same “deer in the headlights” stare from develops and clients when I slip up and use the terms site map and navigation — surprising how this metaphor has such limited traction.

  27. Thank you for sharing these results. I’ve seen the hamburguer menu more and more in lot of websites even in large resolution sceeen. Why?

    • Interesting. 9 years on and the ideas are still important. Icon-only controls can frustrate users.

      • I worked on a NHS website in 2002 that had troubles with “i” icons for information. After loads of failed users tests we went with the word “INFO” in a box with a highlighted yellow background.

        Some things never change. Or maybe they just change very slowly.

  28. One thing you shouldn’t do when making data driven decisions is make assumptions. You should have reached out to optimizely. I’m sure this free advertising is worth a lot more to them than your subscription cost.

    • Did reach out. No response.

  29. Thanks again!

  30. Very interesting post. Thanks for sharing! Would be very interesting to have a third variation with the icon and the text. Do you tried that?

  31. Could it be that the reason that Android users are less prone to clicking a menu button is that they are used to activating a physical button on the device for their menu, as opposed to iPhone’s screen oriented navigation systems?

    • This has been suggested by other commenters. It’s not a good sign for mobile websites, as it seems to show that Android users are far less likely to engage with the site.

      • I don’t think it has much to do with physical buttons as those were depreciated near the beginning of 2012.

        As for why the android users are clicking less? Who knows.. Maybe the content didn’t drive the traffic for that demographic. Maybe the site didn’t render smoothly. Maybe they hate puppies. (I’m willing to bet the implied classism by earlier commenters isn’t really the issue, although I do see the rampant assumption that “all android users are poor/thieves/uncultured” constantly thrown around in the design world).

        While data driven design is great, I feel like there needs to be interview data on *why* they didn’t want to click the menu. There could be some third unexplored factor that isn’t being caught by your test, and making decisions on incomplete data could lead to alienating a whole user base.

        • I think performance is one aspect. I test with a low-end android phone, and it’s not pretty. Its speed of parsing JavaScript is woefully slow, and scroll speed is an issue also…

          • A lot of Android users also use the sliding option to open menu items, rather than reaching up to click a menu button. If you had the sliding functionality (which is a standard operation in native apps), I wonder if you’d get more views.

    • I’d have to agree with this theory!

  32. Hey, James
    Thanks for a great post.
    Can you talk a bit more about why the testing with Optimizely would cost $859? I thought if you were on a Gold plan you’d qualify for the following visitor volumes:
    200,000 monthly visitors + $5 / thousand additional visitors
    Which in your case should cover the 25K users.


    • The test had 300,000 visits to it.

      300,000 visitors + $5 x 100,000 = $359 (gold monthly) + $500 ($5 per 1,000) = $859

      Optimizely bills by visitors NOT unique visitors. This test had about 300k visitors to it. However for my results I’ve studied unique visits as this takes out any bias from 1 or 2 users who might repeatedly click one of the buttons.

  33. “Menu” is English. Unless that’s your primary target audience’s native language, I’d recommend hamburgers.

    • I agree with the suggestion of language, but any global web site has the same problem with the rest of its contents. I dont think that put icons instead of phrases is the solution. Moreover, in this case, “Menu” is also used in Spanish (with a diacritic).

    • “Menu” works in Spanish too… Well, “Menú”

      • And Menü works in Germany.

    • The word “Menu” works in several western languages : english, spanish, french, italian, and so on… A lot of people, isn’t it ? 🙂

      It depends of your targets, ok. But it works in many many cases.

    • But then your whole site would be in a different language anyway, in which case the navigation would be changed as well.

    • I’d contribute the Japanese (“メニュー”), Simplified Chinese (“菜单”) and Traditional Chinese (“選單”) translations.

  34. Unless piggybacking off of known uses, colors and icons have to be learned. There is a point of ubiquity where that is no longer the case though. Best thing to do is ask a cross-section of people what they perceive for directional feedback. And remember, you are not always your target audience.

  35. For the record, I’ve really enjoyed your posts on this Burger Situation (there are always the folks who get a little too excited riding their soapbox! Sounds like you’ve ran into a few since your post went big)

    Anyhow, Re. Android vs iPhone 3x less likely… I can’t say it with any real evidence, but perhaps it shares some of the reasons mentioned in these posts:
    (page search for word ‘elephant’)
    (plus theres loads of other posts about the fact iPhone user spend more than android users for both obvious and less obvious reasons, stuff like

    So, while it could be a risky suggestion, is it fair to say that the reason why the menu button gets hit more is that *generally* the average iPhone user is just more engaged/savvy/inquisitive etc than the average Android user?

    • Interesting stuff, and so much assumptions. Certainly lower end android devices are much cheaper than a new iPhone. I think this issue far more important than hamburgers. As the android user base continues to grow, why do they engage less with websites?

  36. I have actually AB tested [MORE] for some mobile apps.
    People usually react to their impulses. I’ve found that our organizing/naming of patterns are not at all what people would consider their next logical move. Good stuff.

  37. I also want to see the same for apps. My experience otherwise, with actual tests of other parts of the UI, is that websites and apps are perceived differently.

    Love to have a solid guideline, but am not sure it’s universal or just for the Web.

    • Actually, I just realized I am getting a test of some other fundamentals done in (I hope!) the next few weeks. It’s going to be an app, so I’ll try to add this into it since we do have a menu.

  38. for what its worth, Facebook puts the word “more” under the hamburger on its app.

  39. In the late 1990s testing showed that a user first looked for position on the screen for action, then words, and then finally the pretty picture. Words are easier to define then pretty pictures.

    I wonder what engagement would be like if the menu/hamburger button was moved to the right side of the screen.

    • will be the end of the world!

    • Nav buttons tend to be top left or top right. To be honest for a righthanded person, a hamburger button on the right is much easier to tap than on the left.

      • This is one of several good reasons why phone operating systems should allow customers to specify whether they are right-handed or left-handed, and make that information available to apps so they can exploit it.

        • That’s an interesting thought. There is nothing to stop app developers from creating an option for users to set that same preference inside their app, despite an OS not feeding it that data.

        • Or just use the accelerometer/gyroscope determine if they’re holding it in their left or right hand (for most people and most phones, there will be a pretty significant difference in the angle at which a phone rests in the left hand vs the right). I am right handed, but often wield my phone in my left if the right is busy – if I had to specify my handedness and stick to it, I would be very annoyed by how far away all the buttons suddenly are from my thumb.

      • Other studies have shown that left vs. right placement does not influence the user has much as the context.
        Most users are using one hand to hold the device and another to tap (this behaviour was also found true with iPhones).

    • Wich studie is this? I’m interested =)

  40. Good to know. But.. what about the magnifying glass icon?
    Thanks for the article!

    • The magnifying glass icon was tested well by the Nielson group (linked to in this article). It seems that users are very familiar with this.

      • Great work, but it’s Nielsen with an e.

      • I wonder if it’s because people are used to it now, versus this icon which is just a starting trend.

  41. I *like* your focus on data-driven design, for what it’s worth. You’re doing good work, keep it up! Thanks for the info.

    • Thanks so much, good you found it interesting.

    • Seconded! Having intuitions is one thing, testing them out and reporting the results is another – great stuff!

Add a Comment