User Research Surveys at Scale

An important part of Loggly’s User Experience department here is the outreach to users both new and familiar. Yet, it is hard to find the continued voice of the user amongst the constant business of our schedules. The recruitment, screening, and scheduling of one-on-one walkthroughs that will drive a high rate of participation among over five thousand users (and counting) is fairly daunting task. Any parent that has ever tried to schedule a playdate among multiple families knows what I mean. For us UX generalists, if we are to commit a sizable portion of our schedule to outreach for participation on user research efforts, what’s the best way do this at scale?

Determining the Audience

The first component of these efforts to be understood should be, “Who do we want to talk to?” If there is a focus on participants that are new users who have just signed up, then you’ll likely overlap with sales and conversion efforts. From the user’s perspective, this means prompts for different conversations with different people from various groups from your company, all competing for spots on their calendar. Further, many of these sales discussions involve demonstrations around common usage scenarios (which showcase the full features) of the product. As a result, you may find the participant has an unfair advantage as to how much he / she knows before trying to finally use the product themselves. Instead, you want be able to gain the genuine instinct of the uninitiated.

From my earlier post about guerrilla testing at conferences, you can pick up some lessons on how best to recruit participants right off the conference room floor. This is a great way to conduct ‘cold’ sessions with already screened participants – users that have never seen the product before and are trying to resolve interaction metaphors on instinct, rather than learned behaviors. This is a great way of fulfilling what gets “seen” and what parts of the interface or concept get ignored. These participants are great providers of honest, unbiased feedback.

But what about the users that have been using your product for months? Those who have skin in the game, have already put their sweat and dollars behind customization of their experience. Surveying these participants allows us to see where they’ve found both utility and upon what areas Loggly needs to expand. Surveying only the uninitiated won’t provide feedback on any nagging functional roadblocks, those which are found only after repeated use. These seasoned pros are the participants that will provide the most useful feedback, sessions where you can observe the environment that they’ve created for themselves. Making an observational research analogy, a messy room is more telling of the occupants’ personality than an empty one. Given your limitations, how has the participant been forced to find work arounds? The participant has continued to use the product, perhaps not in the way that we expected them to use it – and comparing these two views can be eye-opening.

Gathering of the Minds

How to gain access to this proverbial messy room of the participant? Find your friendly Marketing representative / Sales Engineer at your company and discuss with them the best way to integrate User Experience outreach emails into the overall, post-funnel strategy. For example, post-funnel would be after their trial periods have long since expired and they, the user, is either comfortable in their freemium state or fully paid up. Although this is not prompting a sales call, you should be comfortable on how to avoid (or sometimes suggest) workarounds to optimize the participant’s experience, giving them greater value of use. Six months of active use should be enough time to establish a relationship with a piece of software, whether they love or hate it. If there’s enough insight into the analytics side of the sales process, it might behoove you to also make sure that the user has had a minimum number of logins across the six months (or however long you’ll allow the users to mature). Another, more complex scenario, would be to trigger this UX outreach once a specific set of features have been initiated – giving off the desired signature of an advanced, informed user.

Outreach emails should empower the recipient to make the experience of the product better, both for themselves and their fellow customers. Netflix does a great job of this, in  asking how was the streaming quality or any delays around arrival times of their product. I also recommend asking the users a couple of quantitative and qualitative questions, as this metric something you should be doing for your greater UX efforts already. When engineering this survey, be sure to track which tier of customer, free or paid for example, is filling out these surveys, as both their experience and expectations could be wildly different. Following successful completion of this survey, one which should be designed and advertised to only last 5 minutes in duration, insert a prompt to participate in a compensated user research survey.

It depends on the complexities of your product, but typically 20-30 minutes is enough time to cover at least the main areas of function. Any longer, and you might encounter people not wanting to fit in an entire hour block into their schedule. If these recorded sessions are kept to just a half-hour, I find that a $25 is sufficient compensation for this duration of time, but your results may vary.

Tools of the Trade

I’ve found Calend.ly to be indispensable for mass scheduling like this. The features of gCal syncing, timezone conversion, daily session capping, email reminders, and custom messaging all are imperative to a public-facing scheduling board. Anyone can grab a 30-minute time slot from your calendar with just a URL. In order to really scale this user research effort to the point where it can be automated, you cannot spend the time trying to negotiating mutually-available times, converting timezones and following up with confirmations. As a part of the scheduling flow within Calend.ly, an input field asks the participant for their Skype handle in order to screenshare together, and I’d advise creating a separate account for this effort. With every session participant, you’ll begin to add and add more seemingly random contacts, any semblance of organization and purity for your personal contact list will be gone.

Once the user is on the Skype call, ask for permission to record the call and make sure that you give a disclaimer that their information will be kept private and shared with no one outside the company. You might also add ahead of time that any support questions that come up, you’ll be happy to direct to the proper technicians. Permissions granted, I find that SnagIt is a good tool to capture and record video, especially given the resolution and dimension of the screen share tends to change based upon the participant’s monitor size. Be sure to re-iterate the purpose and goal of the call, and provide them with a license to say whatever they want—good or bad—you want to hear it.

When you’re walking the participant through the paces of the survey, be sure to annotate the time started and any high / lowlights you see along the way. In front of your desktop, a basic note-taking utility application (or even pad and paper) should suffice. This will allow you to go back after the survey is finished and pull quotes for use elsewhere, such as powerpoint presentations or similar. I try to write a running diary of the transcript and a summary at the end just to cover what areas of the application we explored, as well as what feedback we gathered. Summarizing the transcript and posting the relative files should take no more than 10 minutes, which will still keep your total per-participant time to under an hour each. Calend.ly allows for you to cap the number of participants that can grab blocks of your time, so you can keep a set amount per day, preventing complete overload of surveys in your schedule.

Share the Love (or Hate)

I want to make sure that these sessions are able to be referred to by the executive and Product Management team for use in their prioritization strategy. Setting up a instance of MAMP / WordPress on a local box (we’re using one of the Mac Minis that power a dashboard display) which allows me to pass around the link internally and not have to deal with some of the issues around large video file sizes being uploaded, as well as any alleviate permissions concerns with these sessions being out in the wild.

Also important is to meta tag the posts attached to these files when you upload them. This allows faster indexing when trying to find evidence around a certain feature or function. Insert your written summary into the post content, and you’ll be able to better search on memorable quotes that might have been written down. These resources can be very good for motivation internally, especially amongst the Engineers who don’t often get to see people using the product they continually pour themselves into. They’ll also resonate with the Product team, who will see first-hand what’s needed to re-prioritize for the next sprint. After awhile, you’ll start to get a great library of clips that you can draw knowledge from. There’s also a certain satisfaction to seeing the evolution of the product in the interface through these screengrabs. That which was shown as confusing at one time may now be fixed!

Conclusion

User research is imperative to the success and prioritization of any software application (or any product, for that matter). This article summarizes some tools and techniques and strategies that will allow to create an automated workflow of continually-scheduled screenshare surveys with customers. In addition, the methods mentioned in the article above discuss how to best leverage and organize this content internally, so that all might benefit from your process.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.