Don’t Be Afraid of the Hamburger: A/B Test

hamburglarMany months ago I ran a series of A/B tests. The idea was to see how much users engaged with the “hamburger” menu icon (compared with a number of variations).

The outcome freaked me out a little.

The data showed that the hamburger was not as well understood as first thought. Articles sprang up in agreement with this.  It was as if an angry mob with pitchforks suddenly decided to lynch the hamburger icon.

Most people concluded that the hamburger icon was not a good thing, and users needed some additional clues.

That was then. What’s changed in 6-8 months?

Quick Recap – Previous Tests

All these tests were only run on mobiles.

Test 1

hamburgertest1

20,000 visits. Compared hamburger icon variations.

The one with the border performed slightly better.

Test 2

ham-test-2

50,000 visits. Compared hamburger against the word MENU.

The word MENU only with a border performed best.

Test 3

ham-test-3

240,000 visits. Compared hamburger against the word MENU.

The word MENU performed 20% better.

The upshot was that users clicked (or tapped) the word menu more than they did the hamburger icon.

Now is a good time for a caveat:

These tests are conducted on a single website. There may be errors. I am not measuring a users intent, but simply button clicks. Draw your conclusions from the data. If in doubt measure and observe on your own site. I was not measuring an app, but a responsive website.

Glad we got that out of the way.

 What The Data Says Now

The test had 434k mobile visits. Simple test of hamburger against the word MENU.

Variation  Visitors Unique Clicks Clicks
Hamburger 217872 1268 1761
Menu 217824 1365 2320

Hamburger is still weaker than the word MENU. This surprised me, as the hamburger is everywhere now. What’s going on here?

hamtest4

When I compared engagement stats (time on page, bounce, pages per visit) – there was no difference between hamburger and menu!

So, users are clicking MENU more but NOT going any further into the site. Did the word MENU make users more curious? But on clicking they didn’t want to go further? This is further indicated by the higher number of clicks compared to unique clicks (they tapped the MENU again to close the menu).

This kind of A/B test is weak, as my desired goals are increased engagement in the website. This is a nebulous goal to measure compared with this test – that showed an increase in bottom-line revenue (the winner was a colored button with hamburger + menu).

My conclusions: Don’t be afraid of the hamburger. Measure if you can.

One other thing that has changed since the last test: Android users are clicking the hamburger more than MENU.

Hamburgers on Desktops?

I’m increasingly seeing the icon on larger viewport layouts. I have not tested the hamburger on larger layouts as it just didn’t make sense.

My feeling is the icon helps to conceal a list of functions that cannot fit onto a smaller layout (like a smartphone).  Concealing all navigation options on desktop behind a hamburger may support a clean and spartan design, but (in my opinion) does nothing to assist navigation of the site.

If you want to hide all your navigation options – why have them at all?

what-matters

Image from love2dev.com

The Overflow or Ellipses Icon

This icon appears to be growing in popularity. It indicates there are more functions – that presumably don’t fit. This is exactly what an ellipses has indicated for years and years.

However, it may not suit all use cases. This example from iTunes 12 is a case in point:

overflow-itunes

You want to find your apps?

Click the ellipses, THEN look for the apps icon.  You have to do this every time you want to see your podcasts or apps (or whatever else is hidden in the ellipses). I have personally found this a  frustrating aspect of the newer iTunes UI.

The good news is that you can edit the defaults, but this was not obvious at first.

What Next?

The speed at which user interfaces change is breathtaking.

The only answer is to test. Then re-test, and stay with what works.

Hi, I'm James, and for the last decade I've made a living by making my own blogs and websites.
Updated: September 14, 2016

13 Comments

  1. Could it be that Android users, with a greater probability of being outside the apple ecosystem, are more likely to have encountered and become accustomed to the hamburger icon which is a prominent feature of the Google Chrome browser?

    Just an idle theory.

    • That’s my theory too.

  2. Gmail used to do the ellipsis in the compose window, even though there was PLENTY of space. So annoying! Bad trend. Just show all the options. Since when is that so bad? Same for the menu icon IMO: just show the menu. Scrolling is cheap. Lose the huge banner, I know you have one.

    • Fair call. Ellipses can lead to frustration – trying to find what you are looking for.

  3. I really appreciate you sharing these tests over the last year. They’ve been super insightful, but more importantly forced design teams to at least give it a good hard think.

    I convinced the folks I work with to go full hamburger on all viewports. For context, this is a higher-ed library system site that is really a lot of different services kind of bundled together: hardcore research tools, tutorials / subject guides / resource collections, a whole catalog [physical and digital checkouts], study room reservation, digital archive, institutional research repo – etc.

    We found that interaction with the top menu was pretty low once users dropped into their site or service of choice, and over time the process of wayfinding between sections was through site search, Google, bookmarks, or just non-existent. For hamburger-havers like Time, this is a disincentive, but we really don’t have the same context and–for instance–it is definitely okay that users of ABC never interact with XYZ.

    It seemed that in this case a full top menu became a drawback, since each service has its own contextual menu or dashboard, there was just too much going on on-screen. Given the low interaction, we decided to side-drawer the global nav and raise the contextual menu items to the top so that each section has its own dashboard.

    • Sounds like a wise move. One thing that came of this most recent test is that interaction with the menu has dropped (no matter what the icon is). I’m not convinced to go with the hamburger on full-width viewports just yet. It would with a site that has a low proportion of new visitors, but not with something that has high number of new visitors (they just don’t know what else is there).

Add a Comment