Myths about Data-Driven Design


There is a lot of buzz about data-driven design, but very little agreement about what data-driven design really means. Even deciding how to define data is difficult for teams with spotty access to data within their organizations, uneven understanding, and little shared language. For any site or app, it’s standard practice to have analyticsA/B tests, surveys, intercepts, benchmarks, scores of usability tests, ethnographic studies, and interviews. So what counts as data? And more importantly, what will inform design in a meaningful way?

Myth 1: Data Means Numbers

A lot of the data about a site or app flows in from analytics, and analytics are basically tallies of who has come to your website, how they got there, how long they stayed, and what they clicked or tapped. Other data may come from how many clicked A and how many clicked B. More tallies. Then there are intercepts and surveys where scored responses are counted. Still more tallies.

Myth 2: Data Is the Objective Truth

Quantitative data typically tallies completed actions, and usually those tallies are compiled by software rather than humans. This makes quantitative data seem like hard fact.

Even if data is big, it does not mean it is objective. Bias is inherent in any dataset. Datasets are created by humans, who interpret them and assign meaning, even if a machine runs the numbers.

Myth 3: Bigger Is Always Better

The hype around Big Data (note the CAPS) suggests it has the power to reveal all the secrets of humanity and accurately predict the future. And OK, sometimes bigger is better when it comes to data. If you are measuring something subjective, like emotional response based on self-reported ratings, more responses give you a greater confidence level in the results.

Myth 4: Data Is for Managers, Not Designers

Data is often used to pass judgment on a site or app (e.g. “The data says that conversions are down after the latest redesign”). Of course, this strikes experience design practitioners as reductionist, opportunistic, or just plain wrong. Certainly, it’s tempting to look for data that proves the wisdom of a decision, whether to solve internal battles, counter a gut-feel approach, or just prove return on investment. Proving a point is only one part of the data story though.

Myth 5: Data Kills Innovation

Data is seen as the antithesis of innovation, in all sorts of ways. Well, actually in three ways:

  1. Most data, whether analytics, survey data, or customer service data, is backward-looking. Although we can discover patterns and trends, it is not easy to make predictions based off of those discoveries.
  2. Data is tactical rather than strategic. Think of Google’s 41 shades of blue testing. Because data-informed design is associated with A/B testing, it seems like a good way to tweak a design element but it’s not so great for creating an amazing experience.
  3. Data, especially analytics, seems to skim the surface. Seeing what people clicked, how much they scrolled, or where they lingered can work well to form a picture about how to market a product. It does not work so well for informing design, because it lacks information about motivations, expectations, perceptions, or emotions.

Myth 6: There Is a Right Way to Use Data to Inform Design

So far, there isn’t one canonical way that works for every team in every organization. There are a few guidelines to start with though.

  • Use data from a variety of sources to inform your design: analytics, A/B tests, social media sentiment, customer service logs, sales data, surveys, interviews, usability tests, contextual research, and other studies.
  • Include numbers and context. Whether you call them quantitative and qualitative, studies and non-studies, or big data and thick data, you need the numbers and the context to tell the real story.
  • Make sure data is sensitive to the complexity of the human experience. Use averages sparingly, infer with caution, corroborate liberally.
  • Use data to track changes over time, explore new patterns, and dig deeper on problems, rather than just to prove who’s right or wrong.
  • Decide on meaningful categories that let you make sense of the data and tell a story about the experience.
  • Develop a way to share and discuss data in your organization, and start by defining the basics together.

Read the full article here


No Responses Yet to “Myths about Data-Driven Design”

  1. Leave a Comment

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: