Consumer Survey: How do Website Visitors feel about Pop-ups?

 

Most people are aware that intrusive pop-ups may be unpopular with online visitors. This study set out to test how pop-up forms affect the degree to which online visitors feel they can trust a business or buy its products. 

Background


You may have noticed a growing trend in the use of intrusive pop-up forms on websites and blogs persuading you to sign up to an email newsletter.

 

Example of a Standard Pop-up form:

standard pop-up

 

Some websites are now also using “negative opt-out pop-ups” which force visitors to click on a button to either accept or reject the offer before being able to view the page content.

 

Example of a Negative Opt-Out Pop-up form:

neg popup

 

Some marketing “experts” claim that the use of negative opt-out pop-ups significantly increases their newsletter sign-up rate, while then also seeming to contradict themselves by describing such tactics as “scuzzy” on the very same website. So, who are you supposed to believe?

 

There is little doubt that if you make it more difficult for visitors to reject a sign-up form you will (perhaps reluctantly) get more of them to sign up.

I wanted to test what impact these pop-ups have on how visitors feel about your brand. In particular, the vast majority of visitors who do not sign up to the newsletter.

 

I set out to test the the following hypotheses:

1) The use of pop-ups has a negative impact on the degree to which visitors feel they can trust a business.

 

2)  The use of pop-ups has a negative impact on the degree to which visitors feel they would buy from a business.

 

Testing perceptions, not behaviour

The purpose of this study was to explore the perception of online visitors towards the business, not how they might actually behave in a purchasing decision. 

Consumers often make purchasing decisions reluctantly and, only then, until a more favourable option becomes available. A single decision to purchase gives little indication as to the longevity of the relationship or the anticipated Customer Lifetime Value. Both factors of significantly more importance to any legitimate business than whether or not an individual sale has been achieved.       

 

Methodology

Using SurveyMonkey Audience, I enlisted a total 1,000 US-based, respondents divided into three equal groups. Each group was asked to visit a generic online T-shirt store created for the purpose of the experiment.

 

Generic T-shirt website shown to the respondents:

tshirt

 

Group A

Group A (the control) were asked to look at the T-shirt page with no pop-ups but with a simple newsletter sign-up form at the bottom of the page.

 

Group B

Group B were shown the same page but with a standard pop-up form that could be dismissed by clicking the X at the top-right corner of the form, or by clicking anywhere outside of the form on the page.

The pop-up was displayed within two seconds of the visitor landing on the page.

 

Group C

Group C were shown the same site but with a negative opt-out pop-up forcing them to either supply an email address or click on a link stating, “No, I do not wish to receive discounts and promotions”. 

The pop-up was displayed within two seconds of the visitor landing on the page.

 

Avoiding bias

None of the respondents were told which aspect of the website was being tested and pop-ups were not mentioned until after the main survey was completed.

After visiting the site the subjects were then asked the following questions:

 

 

Q1) To what degree do you feel that you would trust this business?

trust

 

Q2) To what degree do you feel that you would buy from this business?

buy

 

After they had completed the survey – and on a separate page – I then asked all respondents to describe how they felt generally about pop-ups on websites.

97% of all respondents felt negatively towards pop-ups. The WordCloud below shows the most frequent comments in the largest font:

 

pop green word clould mid

 

Conclusion:

In “Disrupted,” the enlightening memoir by tech journalist and “Silicon Valley” TV show writer Dan Lyons, he recalls his experience of working at the marketing software company HubSpot. He remembers their blogging strategy this way:

“[HubSpot’s former CMO] has one goal: to get leads. If our software analytics were to indicate that our best conversion rate comes from publishing a blog post that just says the word dogshit over and over again… then [the former CMO] would publish that post…”

A blog post that just repeats “dogshit” over and over might generate more leads — most smart people would agree that it would also be terrible for the organisation’s reputation and branding.

While using intrusive  pop-ups on your website might increase sign-ups, this study suggests that it might also significantly reduce those visitors’ desire to trust or buy your business. Both metrics that you won’t find in Google Analytics.


Intrusive pop-ups are the digital equivalent of that slimeball  you meet at parties who interrupts your conversation, thrusts his (and it’s always a him) business card  in your face and then insists on letting you in on whatever “get rich quick” scheme he happens to be flogging at the time.

Perhaps you will generate a few more sales with your intrusive pop-ups, but do you really want to be “that guy”?






Contributors
Special thanks to the following smart and generous people who helped with the preparation and design of this study:

 

Author: Danny Richman

Danny Richman is an online marketing consultant with over 15 years’ experience providing SEO Training and marketing support to businesses seeking to improve their digital marketing and search engine visibility. Clients include Bank of England, BBC, John Lewis and Marks & Spencer.
  • Thanks for the case study. I would have to agree with you. Using popups can be a great lead capturing device, but it does reduce the value of your brand, products or services. A fail safe scenario for using popups would be if I were going to a site with the motive to signing up to a newsletter. If that were my original intent of visiting that website I would want to find the signup form almost instantly.

    I have seen people many times set their popups to appear at the end of an article or after a certain duration of time. This would be a great idea if your call-to-action just so happened to be what was in the popup. I personally like to refrain from using popups because I want my users to engage with my articles (i.e. leave a comment or ask a question).

    Again, thanks for the read.

    • Cheers Michael. I agree 100%, context is everything.

  • It is worth mentioning that the articles referenced at the top did have two different authors.

  • Hi Danny,

    Great study. I’m curious about exit intent pop-ups. Any study on this? I usually use it a lot, suggesting the readers to sign up for newsletter when they want to leave the website.

    • Thanks Edmond. Not data on this I’m afraid. It would be interesting. Perhaps another day

  • Chris Hutty

    Hi Danny,

    Thanks for sharing that, really Interesting study, however I’d really expect this to be the case for a transactional site. Nobody expects to be hit with in your face marketing as a ‘reward’ for browsing a store

    I’d be really interested to see an experiment like this conducted on a piece of informational content such as a guide, how-to video or blog post. My feeling is that users would be more accepting where they are essentially receiving something for free in return.

    Of course I could be wrong – but I’d love to see the data 🙂

    • My gut instinct (based on my own behaviour and conversations with colleagues) is that it would make very little difference. Whenever I land on a content-based site that forces intrusive pop-ups on me, I either leave the site or it radically changes how I view the reputation and legitimacy of the site owners.

      I agree, however, it would be an interesting topic for another study.

      • “I either leave the site or it radically changes how I view the reputation and legitimacy of the site owners.” — wow that’s a rough statement. So 90% of the sites you visit you leave pretty quickly? Popups are endemic to so much of the content web these days. I don’t look lesser at a site that gives me one. While they are usually brushed aside, sometimes they mean well. It’s the ones that hit me coming in, next page, and exit blockers, and ones that don’t know a mobile and can’t be dismissed that raise my ire.

        Second side of the thought is pop-ups convert mailing list subscribers, and pop-ups on retail with healthy discounts/offers convert sales. If the LTV of a customer acquired from the tactic passes the “I would..” vs the “I did..”, then what people feel in a study holds only a perception difference and not a real world conversion difference.

        • >>> So 90% of the sites you visit you leave pretty quickly
          Nothing like 90% of the sites I visit have obtrusive pop-ups. Those that do, are viewed more negatively my me and the vast majority of visitors.

          >>>> then what people feel in a study holds only a perception difference and not a real world conversion difference
          And this is precisely the problem. Businesses are fixated on conversion rates because it’s easy to measure. They ignore the degree to which they damage their brand for the majority of visitors who didn’t happen to convert on this single visit. Sometimes it’s better to have no data at all than incomplete data that leads to poor decisions.

      • Stephen Leung

        Thanks for the interesting study… I agree that ecomm site should definitely not have popups.

        However, I somewhat agree with Chris’ feelings too though … if it’s informational content, and I liked what I saw, I’d probably want to see more.

        Some shows popup before I even start 2 lines into the content, those are annoying.

        My gut feeling though is, say it’s a somewhat long article or video, if they read half way, or most of the way, then i show the popup, stating the value proposition of subscribing, with clarity (eBook, future articles as good as this one, etc), I could hardly imagine that contributes much negativity, if at all.

        Anyways I could be wrong too … but I do think saying ‘not to use popup for a website’ is a bit strong of a generalization 😉

  • Garry De La Poer

    I generally hate these kinds of pop-ups. But it should be also noted that they can be controlled so they would only appear if someone has added stuff to the shopping basket and makes a mouse gesture that looks like leaving the site (mouse towards top-right of screen) and a popup could be used to prompt them to save the basket for later by entering their email address. Or showing a certain popup if they have viewed x amount of pages, or certain product brands, etc. It is not as simple as treating all popup the same. Having said that, I find them irritating.

  • LMnoP

    I know people find them annoying, I certainly do, but the important thing is not if people find them annoying, but how do they actually impact sales, conversions, signups etc. Also, how is the data different when comparing pop-us that ask for an email for future marketing which might feel spammy to people vs. offering assistance/help now in the form of live chat or similar?

  • Hans Koevoet

    Ok, let’s put this to the extreme.

    This is the idea. If you visit a site, quickly an overlay pops up which asks you to subscribe, and you MUST fill in a valid e-mail address, otherwise the overlay just won’t go away and you can’t read or do nothing. Subscription rate will be a whopping 100%. Good idea?

    • Subscription rate won’t be 100% because most of your visitors will just leave and probably never return.

      Is it a good idea? That depends on whether gaining a few extra signups is more important to you than most of your visitors thinking your business stinks. Choice is yours I guess.

      • Hans Koevoet

        Thx. Disclosure: I wouldn’t do it. But the reason I asked it is that more and more sites are nearing to this extreme. You land on the page, an overlay quickly pops up asking for your mail address, and is hard to close, because it is (intentionally of course) designed in that way. A nice collection of these kinds of sites: http://tabcloseddidntread.com/

  • Hans Koevoet

    Take a look at this one: https://www.invisionapp.com/. Very soon after you landed, an overlay pops up, demanding your email address so you can win a free T-shirt. It’s an offer you can’t refuse: there is no possibility to close it.

    If you do, immediately a second modal pops up, asking you something else (forgot what is was) to double your chance to win the T-shirt. This time, you have the option to say you’re not interested

    Believe it or not, but the moment you closed this second one, a THIRD overlay pops up, this time demanding you to follow Invisionapp on Twitter.

    Ironically, Invisionapp is about… web design.

  • Boris

    Hi Danny,

    Not sure that I concur with your conclusion there about desire to buy from your business. If you were to incorporate metrics like # sessions, conversion rate, AOV and revenue then marry them up with these heuristics that would be a more solid ground to base such a conclusion. Just because a person says something does not necessarily mean that they will do that thing or that they represent other people.

    • In fact, it would be almost impossible to measure this with any degree of accuracy.

      When 97% of your visitors tell you that they hate the pop-ups on your site and when they clearly have a big impact on the way people feel about the degree to which they trust your business or want to buy your products, you might just want pay that some attention.

  • Hi Danny,

    Is there a chance to see what were avg. time per session for each of these landing pages?

    • Hi Marcin

      As the participants in the study were not genuine shoppers we did not record this information. Sorry

      • It makes sense. However, it would be interested to see a study that check how these design changes impact the user engagement (time per session, pages per session, bounce rate, conversions, etc.) which could be interpreted by the search engines as a level of satisfaction with a website that they have suggested to a searcher on a search results page. Anyway, thank you for this really interesting study.

        • Agreed. As I said in the post, this study was a test of attitudes and not behaviour

          • I think, there will be no need to check an impact of pop-ups on user engagement which could result in lower ranking position.

            During our conversation, Google has announced that from January 10, 2017, all websites that show pop-ups and overlays “immediately after the user navigates to a page from the search results, or while they are looking through the page” may not rank as high.

            You can find more information about it in this article: https://webmasters.googleblog.com/2016/08/helping-users-easily-access-content-on.html

  • Rather than argue the merits of popups, exit-intent or otherwise, can anyone share the results of a split-test they’ve run? As marketers, we all have our own notions of buyer behavior and often insert our own preferences–as if we behave in some way similar to our visitors. I’d rather see real data, and it seems it wouldn’t be terribly difficult to test, no?

  • Pingback: 6 Online Marketing Tactics Your Customers HATE - Sacha Doucet()

  • Pingback: The Future of SEO: 7 Changes from Summer 2016 | Austin Bay Austin Bay()

  • Pingback: Lessons Learned in Content Marketing, tactics that do/(do not) work()

  • Pingback: Wishing bog-off to the pop-up – Digital Marketing Blog | Iro+Co()