The real cost of not doing user research
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 561 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44578 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Decision theoryWindowFormal grammarPresentation of a groupMobile appMultiplication signInformation technology consultingComputer animation
00:36
EmailGoogolProjective planePoint (geometry)GoogolShared memoryMereologyInternetworkingInformation technology consultingMultiplication signBitComputer animation
01:51
Multiplication signComputer clusterXMLComputer animation
02:19
XMLProgram flowchart
02:44
GoogolProduct (business)Scaling (geometry)NumberContext awarenessDifferent (Kate Ryan album)Operator (mathematics)TwitterDecision theoryEmail
03:46
SpacetimeBitDecision theoryBit rateVector potentialInternetworkingCore dumpRight angleFitness functionContext awarenessSystem callField (computer science)Shared memoryLevel (video gaming)Validity (statistics)NumberBookmark (World Wide Web)WordDataflowIterationComputer animation
06:49
Software testingProcess (computing)TwitterComputer animation
07:14
CodeSoftware developerUsabilityAxiom of choiceLevel (video gaming)Confidence intervalAreaTask (computing)Software testingCASE <Informatik>BuildingBitSoftware developerContext awarenessInternetworkingMaxima and minimaShooting methodComputer animation
09:35
Figurate numberQuicksortPlanningComputer animation
09:52
1 (number)UsabilityObject (grammar)Axiom of choiceQuicksortBlogCASE <Informatik>Goodness of fitVapor barrierTwitterDecision theoryBuildingScheduling (computing)Time zoneMultiplication signSoftware testingArchaeological field surveyRight angleEmailStatement (computer science)Hidden Markov modelComputer animation
13:06
Slide ruleWritingElectronic program guideUsabilityOrder (biology)Software testingSlide ruleDataflowFormal languageValidity (statistics)Right angleComputer animation
14:32
PrototypeProduct (business)UsabilitySoftware testingSlide ruleTask (computing)Archaeological field surveyFrustrationSoftware testingUsabilityGoodness of fitProduct (business)Video gameIndependence (probability theory)Rule of inferencePlanningArchaeological field surveyControl flowBuildingEuler anglesGoogolDataflowStandard deviationStress (mechanics)DigitizingBitService (economics)Multiplication signFlow separationSoftware maintenanceComputer animationMeeting/Interview
16:31
Multiplication signResultantRight angleSampling (statistics)Perfect groupOnline helpBitType theoryBuildingSlide ruleWebsiteComputer animation
19:07
Euler anglesComputer animation
Transcript: English(auto-generated)
00:05
So, apologies, I don't see my presenter notes. Okay. So, an app that is a nightmare to use, or spending your own time and hours and
00:21
like efforts to develop something that then is not used by people, or just building something and then looking at it and it requires hours to support. So, and this is really the cost of not doing user research. And so, my name is Olena Bulihina, and I am a UX consultant in an agency called
00:41
WebCredible, and I'm doing mostly user research now. And I just wrapped a very intense week of interviewing, so I will be a bit low today. But the point is that I'm going to tell you a story from another time, when the project I was working on suddenly got big. And so, October 2011, two things happened.
01:02
Coldplay released their fifth album, and Google replaced the native sharing in Google Reader with that Google Plus abomination, and I was furious, and I was so sad. Because to me, it was the part of the Internet that was just worth being in. Because when people shared the best bits of their RSS feeds, and
01:24
then you could have a meaningful or somewhat meaningful discussion around that. And the whole population of Iran was me on that one, because for them, it had even larger consequences. They were using RSS as the ways to get unfiltered and uncensored access to the news, and it was just sad.
01:42
Somebody was building somewhat of a replacement by then, but six months in, nothing was happening, I was still sad, and I had this brilliant, brilliant idea, we should build the RSS reader for ourselves. And the first person I recruited for that was my now husband, who happens to be a brilliant infrastructure engineer.
02:01
And he said yes in hopes to never hear about this again. And then the second person I pitched this idea to was my good friend, who was 19 at the time, and he was very upset that he still hasn't achieved greatness yet. So I promised him an amazing project, and through us, off he went. And then next two years, they honestly feel like a blur to me right now.
02:23
But we managed to do some design work, and we set up infrastructure, we get it working, we got our 100 users, and then 500 users, and then we got 5,000 people from Brazil, they're amazing by the way. And then we got to 10,000 users, and then something happened.
02:40
Any idea what was that? Google closed their reader, and suddenly we found ourselves on a very, very different scale of operation. Suddenly nothing we built was supporting the amount of people who came to the product. Suddenly every minute or ten minutes of downtime were accompanied
03:01
by hundreds of tweets, angry emails, things like that. And then a lot of our design decisions, because I was doing the design, were not supporting the scale at which we were operating. This is how our numbers looked roughly before and after. Those are the 10k users I was so proud about.
03:23
And then it was after trying to do that by ourselves for half a year, getting a very severe burnout, the old reader was acquired. And I'm very happy to say that it's thriving as much as a reader can be thriving in 2019. But this is just for the context of the scale we were operating, and
03:42
while user research is even more important in the scale. So our user research approach, to be honest, there was none. And when I say that, what I mean is that if you take any of the existing design or research methodologies and map it against what we did, you will see a lot of white spots everywhere.
04:03
So mistake number one, we didn't do discovery. And discovery is the stage where you explore the problem space, where you learn everything about your users or potential users, the context they operate in. And my discovery approach was to take a look at all the existing readers,
04:22
then maybe speak to ten RSS junkies. You know the people who subscribe to like 2,000 feeds? And then they say they read all of them? And also to then be very, very heavily inspired, as I say, by the Google Reader.
04:40
On one hand, it could be worse. But on the other hand, it wasn't the right approach to take. Because we use the discovery findings for validation, and they should never be used for validation like that. And we just took our assumptions, kind of validated them and progressed with them. But not talking to users in the field,
05:02
by not figuring out what's even the value of RSS. We were never able to do this big strategic decisions that would be so beneficial for us couple of years in when they were needed. And then, mistake number two, we didn't try to understand the why behind the design that we did.
05:23
So we set up a tool called User Voice, where people could submit their ideas and then vote for them, and then kinda used it for prioritization. And also we used Google Analytics to track the flows, to track the bounce rates, the usual stuff, go on stuff, right? And it kinda gave us the picture.
05:41
But again, it wasn't the right approach, because the most essential insights and the most important decisions they do happen behind those whys. And for example, people in User Voice, there was a very vocal minority of hardcore RSS users. And they all voted for a feature called bookmarklets, where you could share
06:01
the bits of the Internet that were not in your RSS feed into your reader, right? And it was voted as the most wanted feature. And we took it, and we spent hours doing that. Iterations are like not getting it right. Lots of developmental effort as well, and it was done by one guy.
06:22
And in the end, you know what? It was used by less than 10% of actual users. Of our core audience, they kinda didn't care. And then, imagine if we would just do it right. And if we just got what is important in the sharing.
06:41
And then we would never start with a bookmarklet. And honestly, it might not even be in our roadmap at all. And finally, and this is the value of user research. So this picture is from Twitter. And you know what? If you have these balls full by the beginning of the day, and by the end of the day, you come to three empty balls,
07:01
are you sure that you understand the process that happens in between? Because most likely you're not. And this is why it's essential to understand the why. It's essential also to test with the real users, which was our next mistake. We did not test with the real users. And looking back, I don't know why.
07:21
So we didn't have the resources. And then we were spreading ourselves so thin that it was never in top of our priorities. But let me just talk what we missed there. We missed usability issues. And we had to fix them on the go. So imagine bad user experience. And then imagine even worse user experience because somewhere
07:42
there's a team of people trying to fix that. This is just not a viable approach to take. And again, testing with users will uncover those little, small, tiny insights that in the end will be built up into this beautiful, beautiful golden nugget of research.
08:01
So by now you probably are persuaded, or like partly persuaded, that user research has some value. Who here knows about this whole shoot designers code thing? Anyone? No, okay. My friends know, good. So a bit of context. Last three years, the internet has a lot of think pieces,
08:21
medium posts, and runs about whether designers should be able to take on some developmental tasks, or just understanding computational thinking is enough, if yes to what level, stuff like that. I don't think that this is an interesting question. This is an interesting question. Should developers research? So I think that everybody who is building something here in this room should.
08:45
And because essentially if you're one person working on something, or if you're a team of three people, two people, five people, you realistically you have only one choice. It's between not doing any research at all, and doing something by yourself.
09:00
And in this case, I will always, always advocate for doing something by yourself. Because essentially user research is not science, even though we kinda use scientific methods. And user research definitely is not art or design, even though both of them are involved a bit. It is risk minimizing. And if you're building something,
09:22
you need to have confidence in the thing you're building. And you do it by research. So even some research is still better than no research at all. So some of the things I wish I knew six years ago. How do we usually do it?
09:40
The plan is super simple. First of all, get the people, sort your research questions, figure out the method, then prepare, run, analyze, and then do it until you get the picture. I'll start with the users. So where could we get the users for research? It's a very easy answer to that. If you're building something, it's for somebody, right?
10:02
Find who your users are, where they hang out, and talk to them there. Record them there. We had user voice. We had Twitter comments. We have Tumblr blog posts. We had emails. We had people just reaching out for us, and we could just use them and schedule small tiny interviews, and then have the clarity that was lacking in a lot of the decisions that we did.
10:22
And what else is there? So find them, try to recruit them. And for those who are super, super new to this, we tend to compensate people for that time. And I understand that nobody has budget for anything. It's 2019, nobody has any money. But it could be Amazon vouchers, small ones.
10:43
It could be charity donations, but careful with suggesting them in one sentence, just like I did. People will opt for charity donations, even if it's not their choice. So then how do you get to the research questions? It's very easy.
11:00
Essentially, it's what are you trying to find out? So what do you want to learn? Then you transform those into research objectives. And then think about how to meet those objectives. It is a very straightforward process, and the more you practice it, the more you will be getting it right. So for example, it could be, what is a good research question?
11:21
An example of a bad research, I'll start with a bad research question, right? So bad research question in our case would be, will our users like the old reader that's like leading and also that's not really a research question. And what would be a good research question for discovery? Hm, who are our users?
11:40
What are the problems they're currently facing? What is the value of RSS? How do people currently read news? And again, what are the problems, obstacles, and barriers they face when they're doing that? And then those could be translated into specific methods. So, methods.
12:01
The golden rule here is that methods should come from the research and objectives and goals, and it's a beautiful, beautiful statement. But reality always disagrees with it. And there's always time, there's always cost involved or scope that end. So you can just never get it perfectly right.
12:21
So we are entering the danger zone of my talk. Because I took all the methods, and then I just left three of them that I think everybody should be able to practice in one sort or another. And those are user interviews, surveys, and usability testing.
12:41
And also, gorilla research is an approach. I couldn't figure out whether to cross it out or not, so I left it. So, let's start with interviews. Those are usually moderated discussions that may involve other methods, but usually it's just you, your user, and you see it, and you have a nice talk that partly feels like an interview.
13:01
But also, in reality, it should feel just like a dialogue. So, if I were to describe them in one slide, I would say that they're great for discovery, they're very useful for usability testing. And if you were doing interview right now, so, first of all, you have to practice listening because the other person will be doing speaking
13:22
the most, and so, to quote Hamilton, it would be like, talk less, smile more. Then, write a guide. It will help you because the interview never goes right. So, write a guide that will have the order of the questions, the flow of the interview, but never use it verbatim,
13:40
because people will see when you're trying to read the questions. Then, also beware of the leading and closed questions. This is what we are absolutely obsessed about in the research community, that you should never ask something like, which button will take you to a checkout, because that is a leading question. But, however, asking something more generic, something more open,
14:03
something like, hmm, if you were to proceed right now, how you would do this, that is not a leading question. Also, watch people's body language. And this is something that is very hard to get right. But people will say one thing, and then they will be very tense. Most likely, it means they mean something else. And also, dive in with the right mindset to the interviews,
14:22
because you're not there to validate people or test them. You're there to learn everything about them. And so usability testing is another, it's like a golden standard of what we do in user research. So prepare everything in advance.
14:40
Always do the pilot, if you're unsure, do several pilots, because this is where most obvious flows and most obvious things will just pop up. Again, the same. Listen much more than you will be speaking, do not prompt people, and maintain impartiality. I cannot stress how important is the last one, because you have to be there a bit like independent.
15:04
And if you're building this thing, you're so usually in love with it, or you hate it so much, then it's very hard to be impartial in this sense. And then the surveys. Now, those are very hard to get right, because they look easy, but in reality, they don't. And what I suggest to you is to use very focused, very small,
15:24
dedicated surveys for early product research. They do wonders, because this is how you can learn about people's attitudes and behaviors, and also try to figure out how to prioritize things you're doing. Now, you will have to do a lot of research on your own,
15:41
which means that, Google, googling. And there is a brilliant approach that other people are taking is Google Ventures and their approach to this. So just find them on Google Ventures medium, and that's all. And I'll skip a bit, because it's five minutes left. So this thing, when running research,
16:03
again, those look like general good rules for life as well. So plan everything in advance, allow yourself more time. Always plan the lunch break. I recently had a research session where I didn't plan it, and I didn't have a lunch break. It was horrible. Then be flexible with your approach, because you never know what will go wrong.
16:21
Everything that might go wrong. But hope for the best, expect the worst, and find a buddy. Again, this is a good rule for life as well. But to quote government digital services, user research is a team sport. You will need somebody to support you and also take notes. So let's recap. Oh, more slides.
16:42
Great. So let's recap what we learned. As how the reader team never did any user research, and how sorry they are about that now, and how you can not follow the steps. And do some of the user research to help yourself get it right.
17:02
So this is my last advice to everybody who may be inspired by this talk to do something by themselves. Allow yourselves not to get it right every time. And also allow yourselves not to get user research right at all. Because when you talk to a researcher, they ask, we will be obsessed with methodology and questions and perfect phrasing.
17:23
And it's very easy then to be obsessed about maybe I'm not doing this right. And this obsession stuck. And then research feels overwhelming. And it doesn't have to be. Because in reality, you're helping yourself. And again, if you come up with an engineering mindset,
17:41
you're mostly thinking about quantitative and sample sizes. And where is the statistical significance in this thing? And again, we are not trying to prove anything there. We are not trying to extend the results on the population. And we are definitely not trying to get published. What we are again trying to do is to minimize risks. And maybe, maybe add some, it's a bit like gambling, honestly.
18:02
If I were to have a metaphor for the user research, it will be gambling. And trying to kind of help yourself from the future. So it's also very hard, but I think we all should take some skills to practice. So if you're building something, if you're thinking of user research, in the end of the day, get out, find some people, and talk to them.
18:22
Show them what you're building, and maybe do a tiny interview. If this approach feels terrifying to you, I assure you, it absolutely is. And I still feel dead inside when I need to do this type of research, when I need to talk to a person and show them some website. I'm in a cafe, but in the end of the day, people are pretty great.
18:44
And everybody will be helpful. So yeah, and having some data, however small or however imperfect it will be, is better than having no research at all.