EVERY SPOKEN WORD
50 min read · 10,104 words- 0:01 – 2:19
Emmett’s startup origin story: why “build first, talk later” failed
- SPSpeaker
All right, uh, good afternoon. Today's guest speaker is Emmett Shear. Emmett is the CEO of Twitch, which was acquired by Amazon, where he now works. Um, and Emmett is going to do a new format of class today, uh, and talk about how to do great user interviews. So this is the talking to users part of starting a startup. Uh, should be really useful. Thank you very much for coming.
- ESEmmett Shear
Thanks, Dan. Context so everyone knows, uh, where I'm coming from, from this. Uh, we started our-- I started my first startup with Justin Kan, uh, right out of college. Um, we started this company called Kiko Calendar. Uh, it didn't go so well. Uh, I mean, it went all right. We, we, we built it. We, uh, we sold it, but we sold it on eBay, uh, so that's not necessarily the end you want for your startup.
- SPSpeaker
[laughs]
- ESEmmett Shear
And, uh, it was, uh, it was a good time. We learned a lot. We learned a lot about programming. Um, we didn't know anything about calendars. Uh, neither of us were users of calendars, nor did we, during the period of time we worked on Kiko, talk to anyone who actually did use a calendar. Uh, so that was, uh, that was not optimal. Uh, we, we got the build stuff part of the, uh, startup down. We did not get the talk to users part. Uh, the second startup we started, we used a very common trick that pr- lets you get away with not talking to users, which was that we were our own consumer. We, uh, we had this idea for a television show, Justin TV, a reality show of Justin Kan's life, and, uh, uh, we built a whole set of s- technology and website around the reality show we wanted to run, um, and so we were the user for that, uh, for that product. And that's actually one way to cheat and get away with not talking to many other users is if you're just building something that literally is just for you, you don't need to talk to anyone else because you know what it is you want, um, and what you need. Uh, but that's actually a really limiting way to start a startup. Most startups are not just built for the person who is, uh, who is using them. And when you do that, every now and then you get really lucky, and you are representative of some huge class of people who all want the exact same thing you do. Uh, but very often also that just turns into a side project that doesn't go anywhere. Uh, so we kept working on Justin TV for a while, and we actually achieved a good deal of success because it turned out that there were people out there who wanted to do the same thing we did, uh, which was broadcast ourselves live on the Internet. But, uh, the issue with Justin TV,
- 2:19 – 3:51
The Justin.tv pivot and the realization that user insight drives growth
- ESEmmett Shear
the thing that, the thing that sort of kept us from achieving greatness is we hadn't figured out yet how to, uh, how to build towards anything beyond that initial TV show. We knew how to bu- we built a great product, actually. If you wanted to run a live twenty-four seven reality television show about your life, we had the website for you. We had exactly what you needed. But w- if we wanted to go do more than that, if we wanted to open it up to a broader spectrum of people, a broader spectrum of use cases, we didn't have, uh, we didn't have the insight to figure that out because we weren't that user. Uh, and so at some point, we decided to pivot Justin TV. We decided we needed to go in a new direction. Uh, we thought we'd built a lot of valuable technology, but we hadn't identified the use case that would let it get really big. Uh, and there were two directions that seemed promising. One of them was mobile, and one of them was gaming. Uh, and I led the gaming initiative inside of the company. And what we did with gaming that was very, very different from what we'd ever done before was we actually went and talked to users because, while I loved watching gaming video, I was very aware that neither I nor anyone else in the company knew anything about broadcasting video games. And so, uh, I was amped about the content. I thought there was a market there. That was sort of the insight that, uh, that I had that I think wasn't common at the time, which is how much fun it was to watch video games. Uh, quick show of hands, people know about watching video games on the Internet here? Okay, I'm just gonna assume the people listening to this also know about it. If you don't know about watching video games on the Internet, you should go read about that, uh, because, uh, it's sort of important context for the stuff I'm gonna talk about. But, uh, the
- 3:51 – 5:52
How Twitch began: interviewing to learn the broadcaster’s world
- ESEmmett Shear
main point is, uh, I thought that was awesome, uh, but I didn't know anything about the side of it that was really important, which is actually acquiring the content, uh, to start broadcasting. So we went out, and we ran a-- actually a very large number of user interviews. We talked to, uh, a lot of people and brought that data back, and that formed the core of all the decision-making that was for the next three years of product features on Twitch, uh, was sort of some of the insights we got from that. And we continued to talk to users and, in fact, built an entire part of the company whose job it is basically to talk to our users, um, which is an, a whole division that we just didn't even have at Justin TV. We had no one at the company whose job it was to talk to our most important users. Um, so, uh, so that was Twitch. And I'm gonna, I wanna give you guys a little bit of a, uh, uh, a little bit of an insight into, uh, with Twitch, what, uh, what that, what that meant going to go talk to users. So we determined that the broadcasters were the most important people. Um, and the reason we determined that was when we went and looked into the market, we went, I, I, we looked into what, what determined why people watched a certain, uh, streamer and went to a certain website. They would just follow the content, right? You had a, you had a piece of content you loved, uh, and the broadcaster would come with you. Um, and that's actually the one really important point about user interviews, which is that who you talk to is as important as what questions you ask and what you pull away from it. Because if you go and talk to a set of users, if we'd gone and talked to viewers only, we would have gotten a completely different set of feedback than talking to the broadcasters. And talking to the broadcasters gave us insight into how to build something for them. Um, and that turned out to be strategically correct. Uh, I wish I could tell you the recipe for figuring out who the target user is, uh, for your product and who your target user should be, but there isn't a recipe. Uh, it comes down to think really hard and, and use your, use your judgment to figure out who you're really building this for. Um, so, uh, what I wanna do
- 5:52 – 8:59
Exercise setup: choosing the right users for a lecture note-taking app
- ESEmmett Shear
is a little bit, something a little bit interactive now, which is, uh, we're, I'm gonna, I got a bunch of ideas, uh, from, from you guys actually, so sort of suggested ideas. Um, and I'm, I'm gonna pick one of them, and, uh, I want everyone to sort of sit down and do, do step one of this process for me right now, which is think aboutWho would you go ask about this? Like, which people, where would you go to find the people you needed to talk to about this, uh, in order to, uh, in order to learn about what you should build? Um, and so the idea we're gonna use is, let me see here, of these ideas. So here, it's a lecture-focused note-taking app. The idea is, I don't think that the state-of-the-art for note-taking is good enough yet. Um, and I wanna make a note-taking app that, uh, impr- you know, improves that experience, makes taking notes in class better, um, or taking notes while listening to a lecture online better. Um, so you know, maybe it has collaboration features, uh, maybe it, like, helps you focus better somehow. It has m- multimedia enhancements. I don't know, right? It, all sorts of possible features. But that's the, that's the idea. So take, like, take 120 seconds right now and think about not what you would ask or what the right features for this app is, but who would you talk to? Who should the... Who, who is gonna give you that feedback that's gonna tell you, uh, whether this is good or not? I actually mean it right now. Take your laptop out, like, type, write some stuff down. Think, think about, like, the... You can, it's, it's good enough to, like, think that in your head. But actually, like, if you actually just write it down, and, like, just come up with the five people you'd talk to or the five types of people you'd talk to, um, and who you think the most important one was. Like, actually do it because there's nothing like actually running through a practice of something and trying to do it to actually get it into your head the right way to do it. Um, I'm gratified to s- hear clicking in the c- of keyboards now. Um, if you're following along at home, pause, actually do it. Think about who you, who would you talk to? Um, because, uh, that's a, that is the first question, uh, for almost any startup that you need to answer is, like, who is my user and, and where am I gonna find them? All right. Uh, that's, like, way shorter than you'd normally use to think about this problem. It's actually a really tricky problem, and, like, figuring out where to source the people is pretty hard. But, uh, uh-
- SPSpeaker
[coughs]
- ESEmmett Shear
... we're going to move along anyways in the, in this highly abbreviated version of learning how to build a product and run a user interview. So, um, can, uh, can I, can I get one volunteer from the audience to come up and, uh, tell, tell us what, uh, who you would talk to, and we'll talk about it? You guys are all preselected. Here you go.
- SPSpeaker
[laughs]
- ESEmmett Shear
I don't know how to turn this thing on. Here we go. So who do you talk to? Yeah.
- 8:59 – 11:47
Audience share-out: students aren’t the only “user” who matters
- SPSpeaker
Um, I would definitely talk to college students first, obviously, because we sit in a lot of lectures. And specifically, I would wanna talk to college students studying different subjects to see if maybe, um, you know, if you're an English major, if that makes a difference versus you're studying, um, math or computer science in terms of how you wanna take notes in different lectures.
- ESEmmett Shear
Um, and, uh, so you, so you're gonna talk to, talk to a bunch of college students. Would you pick any particular subset of college students? Like, sounds like you really, we don't wanna talk to all college students or, like, a broader array.
- SPSpeaker
I, I would wanna talk to college students, um, like, and break down the divisions by, like, people who study different areas maybe. And then also maybe it would make sense for people who have, like, different study techniques 'cause some people take a lot of notes, some people don't take that many notes but still jot stuff down.
- ESEmmett Shear
Right. So I mean, that, that's a really good start. Like, that's, that is actually, obviously a group of users you wanna go talk to, especially if you're targeting something at, you know, at college students, uh, as the consumer. Um, and if you're talking to college students as a consumer, uh, the, you're gonna get a lot out of students about what their current note-taking habits are and, you know, what they would be excited about. Um, one of the problems with selling things to college students is that college students don't actually spend very much money. Um, it's, it's really hard to get you guys to open your wallets, especially, uh, if you want them to pay for a school-related thing. I mean, people don't even wanna buy textbooks, right? Uh, I think you've probably, probably all used Chegg or that, you know, borrow it from your friend or whatever. Uh, and so, uh, o- one of the, like, one of the things w- that I think you'd be missing if, if you go after just the students, right, is you want to figure out who, who is the most important person, uh, to this, to this app? And if you actually had a note-taking app, my guess is for colleges, the people most likely to actually buy a note-taking app that you guys would use would be college IT, right? I mean, presumably, for most, for the most part, if you wanna sell s- software to students, like, the people who have to get bought into that is usually the school administrators. So that would be, that would be one approach. If you like thought, "Okay, well," you presumably go talk to the college students, and you find out, uh, they don't actually buy any note-taking software right now at all. I mean, likely. Uh, it's possible they do, um, in which case I'm, I'm completely wrong. And this is why you actually have to go talk to the users. But, uh, you then have to try to maybe try other, other, uh, groups, right? So I would talk to college I- I would talk to IT administrators as well. Um, I think that's a, that's another, uh, area that's really promising. You might l- talk to parents, right? Who, who, who spends money on their kids' education and is, like, willing to pull their wallet out? Like, the, you know, parents of kids. Um, parents of kids who are freshmen or going off to college for the first time. You need this app to make your kid productive so that they don't fail out of college. Um, and, and there's actually a lot of groups that are potential, that aren't necessarily the obvious user, but who are critical, critical to your app's success, potentially. Um, and you, when you're, when you're s- at the very beginning of a startup like this when you're like, you have this idea that you think is awesome,
- 11:47 – 16:17
Live user interview demo: uncovering behavior, not feature requests
- ESEmmett Shear
uh, you wanna have that broadest group you possibly can. You don't just wanna talk to one type of person and, and, and learn that. You wanna get familiar with the space. You wanna get familiar with the various kinds of people who could be contributing. All right. So, uh, uh, let's, uh, let's have someone come up, and we're gonna, we're gonna pretend, uh, we're gonna run this user interview. So we're gonna talk to a college student, um, and try to find out, uh, what we should build, you know, what we should get, uh, into this note-taking app. So, so some, another volunteer, please, uh, for, for running an interviewYes. All right, so, uh, hello.
- SPSpeaker
Hi.
- ESEmmett Shear
S-
- SPSpeaker
I'm Stephanie.
- ESEmmett Shear
Hi, Stephanie.
- SPSpeaker
Nice to meet you.
- ESEmmett Shear
Um, welcome. Thank you for agreeing to do this user interview with us.
- SPSpeaker
[laughs]
- ESEmmett Shear
So, uh, I wanted to hear from you about, you know, what are your note taking habits. Like, how do you take notes today?
- SPSpeaker
Sure. So, um, I take notes in a variety of ways. I like to, um, now because of speed and efficiency and just to come back to it later, it's easy for me to just take notes on my laptop. Um, and so a lot of those notes will be primarily text-based. Um, but in certain classes, so for example, if I'm taking a history class, most of it will be in text, but if I'm taking a, taking a physics class, for example, there are gonna be more complex diagrams, different angles that I have to draw, and so that's a little harder, harder for me to both get the-
- ESEmmett Shear
What software do you use for this stuff today?
- SPSpeaker
I just do pen and paper for that.
- ESEmmett Shear
You do pen and paper. So you do a combination. You take notes with pen and paper.
- SPSpeaker
Exactly.
- ESEmmett Shear
You take notes with your computer sometimes.
- SPSpeaker
Yeah.
- ESEmmett Shear
Um, and, uh, when you take notes with, uh, when you take all these notes, at the end of it, like do you actually review them? Like do you... Be honest.
- SPSpeaker
Mm-hmm.
- ESEmmett Shear
Do you actually go back and ever actually look at these notes?
- SPSpeaker
The pen and paper, not so much, but yes to the, um, software based, because it's more easy to access, and it's easier for me to, to share and collaborate and maybe like even merge notes with other classmates and friends.
- ESEmmett Shear
So what do you, uh, what do you use to take notes today on your computer?
- SPSpeaker
Um, Google Docs, um, and Evernote.
- ESEmmett Shear
Google Docs and Evernote.
- SPSpeaker
Mm-hmm.
- ESEmmett Shear
And, uh-
- SPSpeaker
And-
- ESEmmett Shear
Tell me more about like why two things at the same time?
- SPSpeaker
Um, so Evernote is easy if I'm trying to just collect it for myself, I think. Um, and yes, you can share, but I think Google Docs for me is, um, easier to share, and it depends also if, you know, a friend has already created a folder, for example, on Google Docs, and I just have to add to that folder if it's a group project, for example, versus if it's for my personal use, I, I tend to, um, go more towards Evernote.
- ESEmmett Shear
So it sounds like you do a lot of like note taking collaboration.
- SPSpeaker
Yeah.
- 16:17 – 18:53
Avoiding the feature trap: learning problems beneath the “horseless carriage”
- ESEmmett Shear
I appreciate that. So, uh, so that, that's like, that kind of stuff, you notice we're not talking about the actual content of the app at all. Like I'm not, I'm not, I'm not really interested in features. I don't really want to know about, uh, what they, the specific feature set in Google Docs or Evernote. I might start digging in a little bit more into which features actually get used, like if she's actively collaborating, you know, is, how does that work? I heard some interesting things about, oh, we have, we use folders. That's interesting to me. Uh, but the main thing you're trying to do when you're running these first set of interviews is not necessarily get... Like, questions about like user flows and like optimizing that, or questions about, uh, like the specifics of, um, of, of any of that stuff, uh, kind of can be distracting because users think they know what they want. But like you, you get the, uh, you get the horseless carriage effect where you're, you're, uh, uh, you're n- you're getting asked for a faster horse instead of trying to design the actual, uh, real solutions to the problem, if you start asking people about features. So you want to stay as far away from features as possible because the things they tell you wind up almost feeling overwhelmingly real. Um, when you have a real user asking you for a feature, it's almos- it's very hard to say no to them, uh, because, uh, here's a real person who really has this problem, and they, they're saying, "Build me this feature." But as you start to talk to lots of people and really get a sense for what, what their problems are, you figure out if this is actually a promising area or not. And like based on what I heard there, it's like starting from that user interview, I'm not necessarily positive there is a problem, or there's at least there's a, there's a big enough problem that it's worth building a whole new product for, uh, because I didn't hear a lot of like things where I'm, where, where there was a, a big blocker, where there's something really wrong with the way it, the way it, uh, it was working. And unless I had some big idea, uh, I would take that as a, you know, maybe a negative sign. Um, but it doesn't necessarily mean that you can't, uh, you can't move forward and keep talking to more people because just because you talk to the first person and you don't get anything out of it, doesn't mean there's not going to be, uh, a ton more people who actually have a problem. And you, once you've talked to about six, seven, eight people, uh, you're usually about done. Uh, it's unlikely you're gonna discover a bunch of new information there. Um-Which is why it's important to talk to different extremes of people, right? Go, go find people who are at different, uh, different points because if you talk to six or s- six or seven Stanford college students, you're gonna get a very different response than if you talk to six or seven, uh, high school students or six to seven pa- six to seven parents. Um, all right, once again, look at the... So,
- 18:53 – 21:43
From interview to hypothesis: propose one ‘quantum improvement’
- ESEmmett Shear
um, based on that though, right, uh, I think the, I think it's possible you could come up with a set of ideas, right? You have this information about how someone takes notes. You've, you've come up with, uh, potentially when you came up with this idea, you had, you had some ideas as to, when you heard this idea, you had some ideas as to like how you could build something cool. Um, and so if you're gonna build just one feature on top of Google Docs, uh, what would that feature be, right? And that's, for, for, for a new product like this, it might be a good way to like get started thinking about where to go, which is, uh, okay, so they're extensively using this thing right now. How could we make that experience just one quantum better? Something that would be really exciting to this person to be one, uh, one step ahead. And so why don't you take two minutes right now and think about what that feature might be. Uh, actually like try to, try to come up with, uh, what, uh, what you might do based on what you heard from, uh, from Stephanie that could convince her to switch away from her current collaborative, multi-person, all working together workflow on Google Docs to your new, your new thing that is, has all the features of Google Docs plus this one special thing that's like going to make it, uh, that's gonna make it more b- uh, more useful and, and convince them to stop using the thing they're already using. Awesome. All right. So I'm gonna invite our, our third guest. If you, if you, if you have something up. Uh, I, I don't wanna put you on the spot if you feel like you're not, you're not sure, but, uh-
- SPSpeaker
Sorry.
- ESEmmett Shear
Yeah.
- SPSpeaker
So what I, what I... Is it on?
- ESEmmett Shear
Yes.
- SPSpeaker
Okay. What I thought about was like the, the reason she uses Evernote is because like, of like sticky note type notes, like, like more thoughts and like details. So I feel like Google Docs has like documents and not like smaller notes. So I feel like a feature that would be like super, like a mobile version of Drive that doesn't like, isn't that clunky and like doesn't make you make real documents could be like really useful.
- 21:43 – 25:26
Validating ideas without building the whole product (and the money test)
- ESEmmett Shear
Awesome. So right, that's a, that's a good insight, right? That's exactly, well, that's one of the things you get out of that, that user interview. And now you have this idea, right? You've gotten this, uh, I guess user fed- feedback, you got this idea. What if we had a, uh, Google Docs that had the collaborative aspects and the group aspects of that, but where you, uh, you could pull in more little one-off notes and it was, it was designed more around note-taking. And so the question is n- now, once you have this idea, which I think it, it's a, it's a actually probably reasonable approach, is this enough? Is this something people would actually switch, uh, just to have? And the way to validate, there's two ways to validate that. One is if you're a, a quicker programmer, you can literally just go build it and throw it out in the world and see what happens. Uh, and that's, uh, that's great, and if that, when that works, that's, uh, that's an excellent way to approach it. But a lot of the time, that one little thing that's just, just a little bit better, uh, might take you three months to actually build something that's worthy of actually using. And so you actually wanna go out and validate that idea further before you go ahead and start building it. Um, and so you might take that idea and you might go back, uh, go back out and, you know, you can sit down with, uh, with diagrams. You can, you can draw what the, uh, what it looks like, um, draw the workflow, and go bring that in front of people. But, uh, the one thing you really don't wanna do is ask them, uh, this is, this is just a sort of a trap I wanna warn you against doing, is just don't go out and say, if you come up with a feature idea and go out and ask people, "Are you, you know, I got this idea, great idea for a feature. Are you excited about it?" Because the, the feedback you get from users if you tell them about a feature and ask them, "Is this feature good?" Is often, "Oh, yeah, that's great. Like, that sounds like such a good idea." Uh, but when you actually take that in front of people, uh, and you actually build it, you then find out that while they thought it was such a clever idea, no one actually like cares to switch to get it. And so the one question you can't ask is, "Is this feature actually good or not?" Yeah, Sam.
- SPSpeaker
What is the minimum that you can do in your experience to actually build it on that question of asking, you, you know, between asking and actually building the whole thing?
- ESEmmett Shear
Yeah. So Sam's asking if, what's the, what's the minimum you can actually, uh, get away with to validate given that you can't actually just go and ask them, "Is this good or not?" Um, and it's, it's highly dependent, uh, the answer to that is highly dependent on the particular feature. But usually, the, the best thing you can do is, uh, is, is really just hack something together, right? It's you find, if your, if your idea is to build something on top of Google Docs, don't for your V1 go rebuild, uh, a, an awesome Google Documents but for note-taking application. Uh, find a way to write a browser extension that, that, that stuffs just that little bit of incremental feature in and, and see if it's actually useful for people. Um, go like act- go, go-Find a way to cheat is what it comes down to because if you can't actually put it in front of people, um, it's really, really hard to, uh, to find that out. For bigger things where you're actually trying to, uh, s- get people to spend money, it actually gets a lot easier. So if you're selling it, uh, it's great actually. Sales is this cure-all for this problem. Get people to put, give you their credit card, and I guarantee you they're actually interested in the feature. Uh, it's, uh, it's one of the most validating things you can do for a product is go out there and actually get them to commit to pay you up front. And the problem is when you're working on a student note-taking app, that's gonna be relatively hard, uh, because you probably, unless your idea is that you're actually gonna sell it, it's probably something where you're thinking at least the, the trial version's free and you're not necessarily gonna learn, uh, that much by trying to charge people money. But if you go out there and you c- if you can get people to say, "Hey, I'm gonna, I'm gonna give you money," the money test is amazing. It really does, uh, clarify
- 25:26 – 27:58
Twitch case study: what current users asked for vs. what mattered
- ESEmmett Shear
whether or not they're really excited about it or not because if you're not five dollars excited about it, you're probably not very excited about it. Um, so the last thing I wanted to do is actually work through with you guys, uh, what happened at Twitch. Um, so I, I brought some slides of feedback, um, that I'd like to get put up. That's my, my only slides for the, uh, for the thing and we're, it's, it's, what it is, is it's, it's, it's representative excerpts of Twitch feedback. I had a whole like 26-page document full of all the feedback, and then I realized that reading that was gonna be a little bit tedious, and there was no way I'd make it through it in a lecture. So pretend that like this is, stuff is all representative of, uh, like lots of people said this kind of thing out to, to us when we asked them questions, and I've already pre-condensed it for you into the real feedback you got. So when we were working on Twitch to go launch it, we, uh, we went and we talked to a bunch of existing Justin TV broadcasters and asked them about their experience broadcasting, what they liked about broadcasting, why they broadcasted, what they broadcasted, what else was going on in their life. And the interesting thing is when you talk to users of your product who are, who are detailed users of your product, they come back to you with actually very detail things about features 'cause they actually get mired in the feature, and you have to sort of read between the lines. But, um, they asked for, for us things like, "I want to be able a way to clear the ban list in my chat room." Like this, that was actually a very common request, uh, because there was a particular issue with how our chat rooms worked. People would ask for the ability to edit the, uh, titles or highlights after creating them. And, and it's, it's, th- this was like, this stuff was really consistent. As you talk to broadcasters, you probably talked to 12, fi- fo-14, something like that, uh, broadcasters on the Justin TV gaming platform. Um, we got, we got all this feedback and, you know, what else do we have? We have your, your competitors have all these cool features like polls and scrolling tags. You c- I can personalize chat there. And, uh, uh, so we have some positive feedback. They're like, "Oh, you guys don't have ads. That's great. Um, I need to be able to ban trolls." So a bunch of stuff about chat, a bunch of stuff around, uh, interactivity with, with, uh, uh, interactivity with the, uh, with their viewers, and that was all really interesting, right? So this is what the, this is what the Justin TV broadcasters, uh, wanted us to build, and this is what they, what, what, where they felt pain using the, using the product. And, uh, so if you thought that what we did was go and address these problems, you would be wrong because actually people who are using your service already and are willing to put up with all these issues kind of, kind of means that these are probably not actually the biggest
- 27:58 – 29:28
Competitor users reveal adoption blockers: stability, monetization, reach
- ESEmmett Shear
problems. Because if you're willing to ignore the fact that you can't edit the ban list and titles are uneditable, and there's no way to get trolls out of your channel, and you're using the service anyways, maybe those aren't huge problems. And so that sort of brings up another really important point, which is you have to compare, uh, you actually have to compare groups of people and compare the level at which they, uh, uh, they argue with each other. So if you go to the next slide, um... Yes. Nice. Uh, we got competitor broadcaster feedback, which was really interesting. So this is stuff that you, we heard a lot from people who are using other broadcast platforms. Um, they wanted to be able to switch multiple people onto their channel at the same time. Uh, they, they complained about us, us not having a rev share program, um, or they talked a lot about how they were trying to make a living. They really wanted to make money, uh, pursuing this, uh, pursuing this gaming broadcasting thing. And they talked about, a lot about video stability. Our ser- service wasn't good in Europe, um, specifically, but, but just globally, video stability was this huge, huge issue for them. Um, and if you compare and contrast, actually it was really different. Like the things that people who didn't use our service said about what they cared about was completely different than the things that people who were using the service cared about. Um, and we focused on this stuff because this was the stuff where it was so bad they weren't even willing to use our service because of it. Um, and most of them actually had thought about it 'cause we were, our user base happened to be a very, uh, well-educated user base in this, in the area who knew about all their options for, for, for this. And they would, they actually,
- 29:28 – 31:30
Non-users expand the market: removing barriers to new behavior
- ESEmmett Shear
you know, reaching out to them, uh, meant that they, they'd probably already tried all four services and actually had an opinion. It's great when you can get users who are that, that informed and that, that they understand the space that well. Um, and, uh, and if you go to the... I'm just gonna go to the next slide. Yeah, here we go. Um, the other big thing we did that I thought was really important was we talked to non-broadcasters. Um, so we went out there, and we talked to all the people who weren't using us or competitors. Um, and in many ways, those are the most important people, right? Talking to your competitors, that's a short-term win, right? If someone's using a competing, piece of competing software, um, unless your, your piece of competing software is something like Google, which is a search engine which everyone uses, okay, maybe then, then there are no non-users to convert. But in the case of gaming broadcasting, almost everyone's a non-user, right? The, the majority, and this is true for most new products, the majority of people you're com- competing with are non-users. Um, they're, they're people who have never used your service before, and what they say is actually the most important. What they say is, is the thing that blocks you from expanding from, uh, uh, expanding the size of the market with your features, right? If you, all you do is look at your competitors, uh, and yourself, and all you do is talk to your com- your, you know, people who use your competitors' products, people who use your products, you can never expand-- well, not never, but you're not learning the things that help you expand the size of the market. You want to talk to people who aren't even trying to use one of these things yet.Who, who've thought about it maybe, but who aren't, uh, who aren't into it. So what did they say? Um, "My computer isn't fast enough. Uh, I'm focused on training 12 hours a day for the next tournament. Um, I like making the perfect video and, like, editing it, uh, and so I just upload things to YouTube. I don't do live streaming. I don't, I, I, uh, I have no desire to, to go into that space." Um, or, uh, or this is actually, particularly in Korea, this was a big problem. Uh, "Once our strategy gets broadcast in a major tournament, we have to start over. We have to, like, come up with an entirely new strategy. And so the last thing we ever would wanna do would be s- s- broadcast our practice sessions. Are you crazy? Uh, that's gonna hurt us in the next big tournament." And so this became, this became a big outreach program for us, trying to figure out how we can get people over this. And we bought people computers.
- 31:30 – 34:58
Turning interviews into strategy: solve for goals, then win trust internally
- ESEmmett Shear
We, uh, uh, we worked really closely with gaming broadcast, uh, software companies to help the people who made the broadcasting software to make that better. Um, we started building broadcasting into games and into platforms. Like, we built broadcasting into the Xbox. We built broadcasting into PlayStation 4, uh, because we wanted, we needed to overcome this issue that, like, it was too hard. Uh, broadcasting wasn't, uh, wasn't possible. And so you sort of combine these ... For us, these are the b- three, three big groups we looked at for broadcasting. Uh, and you combine that feedback, um, and what it tells you is not the features to build, right? Because the, the features they asked for, uh, things like polls, um, things like a, uh, um, you know, the, the ability to have child account, like, child accounts on your account. We haven't built most of that stuff. Um, but what was important were, were the f- were the, the issues, like, the goals they were trying to accomplish there. People wanted money. People wanted stability and quality. Um, people wanted universal access for viewers all around the world, um, to be able to watch them. And so that became our focus, actually, and we dumped almost all of our resources into things that, uh, none of, no one ever mentioned, uh, in an interview. But those are the things that actually address the problem. And the way you could tell that it worked is w- as we, we would build these things, then we'd go back to the exact same people we interviewed and we'd say, "Hey, you told us you really cared a lot about making money. Well, we've built you this subscription program that will let you make money." And, uh, it, i- it's, it's astonishing, 'cause most people aren't, have never had that experience, actually. They've never s- talked to someone and, uh, said, "It would be really great if your p- product had feature X," and then, and then, like, two months later or a month later, your product actually has feature X, um, or at the very least a feature that addresses the problem th- they brought up. And so, uh, it was actually, the, the people we converted first to our product were the people that we'd talked to about user research. They were the ones who were actually the most impressed, which is kind of fun. Uh, but it really worked because those, we picked people who were representative, and we picked big broadcasters, small ones, medium ones, and we, we made sure we were addressing their concerns. And that, that was completely different from how we'd approached the problem on Justin TV. Because on Justin TV, when we'd try to do this, we'd, we'd sat down, we'd, we trawled through huge amounts of data. Like, we, we spent tons of time looking at Google Analytics, looking at Mixpanel, looking at in-house analytics tools, figuring out how people use the service, looking at where our traffic came from, uh, completion rates on flows. We spent all this time doing that, and that, that's good. I mean, you can learn things from that. I'm not telling you not to look at your, uh, at your data. But, uh, it doesn't tell you where you need to go. It doesn't tell you where, what the problems are you need to address. Um, and so we would just sort of invent these ideas in Justin TV, and then nine times out of 10, without talking to someone, that idea turns out to be bad. Um, and that's actually one of the most disappointing things about doing user interviews and user feedback, which is why I think so many people don't do it, which is that you're gonna get negative in- news about your, your favorite pet feature most of the time. Like, you're gonna have this great idea, and you're gonna talk to a user, and it's gonna turn out that, uh, that nobody actually wants that. Like, no, no one's actually ... They're actually completely concerned about completely different things, and they don't care about what you thought was important at all. And, uh, and that's a little bit sad, but just, just think about how sad you'd be in four months, uh, when you launch that feature and it turns out no one actually wants to use it. So, um, I think that's about it for my, the lecture section of what we're, we're talking about. I wanna take some questions from, from the audience.
- 34:58 – 46:24
Q&A: common interview mistakes, recording, channels, and evolving user sets
- SPSpeaker
What do you see startups get most wrong about doing a user interview? Like, most startups don't do them at all, but the ones that do, what are the most common mistakes?
- ESEmmett Shear
Um, I'd say the, the most common mistakes are showing people your product. Um, don't, don't show them your product. It's, it's sort of like telling them about a feature. Um, you wanna learn about what's already in their heads. Your, you wanna avoid putting things there. The other thing is, uh, asking about your, your pet feature direction. So if you think you wanna add, add subscriptions to your product, uh, going and asking people, "Would you pay for a subscription?" Going and asking them, "Would you use this feature?" Um, and I'd say the, uh, the other big mistake people make is talking to who's available rather than talking to who they need to talk to. There's certain users that are really easy to get at because they are, uh, say, members of your, uh, forum already, right? You have, you have some product forum, and you go and you, you talk to the users on that forum because they're, they're easy to get access to. Um, we, we spent, like, weeks digging for identity information and figuring out who these people were, uh, so we could contact them, so we could talk to them. Uh, because a lot of these people weren't, it wasn't obvious. They were just some user on a, on a site, and if that site didn't support messaging, there was, like, no obvious way to interact with them. Um, and so we spent a bunch of time trying to network and, and find those users and bring them on. Because if you, if you just talk to who's easy to talk to, you're not really getting, uh, uh, getting the best data. The, the fortunate side there is that almost everyone is flattered to be asked what they think. Uh, and so, uh, most of them will actually talk to you and tell you things. Yeah.
- SPSpeaker
How hard was it to get buy-in from the rest of your company? I mean, like, you can go and be like, "Whatever, I'm in charge, so you're doing what I say," but that's probably not the best way of doing it.
- ESEmmett Shear
[laughs]
- SPSpeaker
So how did you get them to-
- ESEmmett Shear
That's a good question. So the, the question is, uh, how hard is it to get buy-in from the rest of the company, and how do you do it? Um, getting buy-in, if you just go to them and say, "Uh, I figure, I talked to the users, I figured it out. Uh, we have to build this"Uh, is really hard, um, because people don't trust you. Uh, there's something magic about showing them the interview though, so I really recommend you record interviews. Um, recording interviews is like magic. A, it stops you from taking notes in the middle. Um, and taking notes is a little bit disruptive. It, it makes it hard for you to feel like you're actually engaged in the conversation. Um, and B, you can then play that recording for people. So when, they don't have to be there for the entirety of all the interviews, but when you wanna make a point about what, what, what we should be building and why, you can just play back for the rest of the company that interview, and it, it's like magic the influence it has on people's, uh, thoughts on what's, uh, what the right thing to build is. Yes?
- SPSpeaker
So did you, um, since you mentioned recording, did you, uh, try to insist on doing Skype interviews rather than over email or-
- ESEmmett Shear
Um-
- SPSpeaker
What was your impression of the difference?
- ESEmmett Shear
Yeah, so you definitely wanna do Skype, uh, or ... Sorry, the question was, um, did we insist on Skype interviews for recording? Um, you don't wanna do interviews over email if you can avoid it, uh, because interviews over email are non-interactive. And the most interesting things you learn in interviews come from the, "Interesting, tell me more." Uh, 'cause the instant that you h- you hope it's vein, and they'll say something you didn't expect. And the instant they say something you didn't expect or didn't already know, you should drop into detective mode, and detective mode is, "Huh, that's interesting. Can you tell me more about that?" Um, and people don't like silence, so they'll keep talking to fill the void. Uh, and the best part about doing it over Skype or doing it in person is you have that interactive, uh, feedback, and you can actually pull a lot more out of people. Email interviews are, they're okay, but they're, they're basically useless. If you're in person or over Skype, they're actually also e- easy to record. Um, make sure you ask them if it's okay to record it. Uh, it's not polite to record people without their consent. But if they're willing to, like, give you an i- user interview, they're probably willing for you to record it as well.
- SPSpeaker
Sorry, but what about the international market? Like, you mentioned it, that you had, uh, a lot of users in Korea and, I don't know, like, maybe they didn't feel comfortable speaking English or ...
- ESEmmett Shear
Yeah. Um, so, uh, the question is, like, what about people in the international market where you're trying to do user interviews with people who don't speak your language. That's just really hard, and actually to this day, Twitch works way better in English-speaking countries than it does in non-English-speaking countries. And I think a big part of that is we are much better at talking to people in English-speaking countries and learning what their needs are, and we're not as good at it in other countries. We've tried to address that by hiring people who speak Korean, um, having them translate. We've tried to address that by, uh, finding representative people in those countries who speak both English and Korean and reaching out to them. But the problem with that is, like, uh, the, you're not actually getting a representative sample no matter how hard you try. The very fact that they are a fluent English speaker means they're not representative of all the people who don't speak fluent English. Um, it's just a hard problem. Uh, it's why companies find it easier to w- build markets that win in their home, in their home country, uh, much more easily than abroad, uh, because it's really hard to talk to users abroad. Yes.
- SPSpeaker
Um, what channels did you use to reach out to them, and did you ever compensate them?
- ESEmmett Shear
Um, so the channels we used to re- what channels did we use to reach out to them and did we ever compensate them? The channels we used to reach out to them, uh, were, uh, onsite messaging systems. So, like, if you're, most site websites have some way to contact a user, so if they're a visible user of another website, you use that site's messaging system and say, "Hey, I was watching your stream," or whatever this person was doing on the site. Uh, "I'd love to ask you some questions about, uh, your use. Um, would you mind hopping on a Skype call?" And as for, the other thing we do is we'd find out who peoples were and we'd send them email. Um, we'd like run into them at events, 'cause a lot of these people go to the same events, and we like would go to the events and, like, get, we wouldn't run the user interview at the event, but you get to know them, you exchange business cards or, you know, whatever it is you actually do now that aren't, isn't business cards. And, uh, uh, and you, you'd get in touch with them. We tended not to compensate people. Uh, I think that if you, if no, if people don't care enough about the problem to, like, talk to someone who's trying to solve it, uh, y- you're probably barking up the wrong tree. We never had any trouble getting people to talk to us without paying them.
- SPSpeaker
What about onsite, uh, user feedback tools? Do you get a lot of feedback from that?
- ESEmmett Shear
So, so there's this whole second set of user feedback that's really important, um, that I should talk about. Um, the question was, what about, like, onsite user feedback tools? Um, and I think the stuff you're talking about is where you, you have like a, uh, a new product and you wanna see how pe- if it's actually gonna work or not, and so you put it in front of people and you see how they use it or not. That's really important. That kind of work is super, super important, and it can tell you lots of things about where you went wrong building something before you launch it, uh, which is great. It doesn't tell you what to build. It, it helps you iron out the s- the kinks and edges of the thing you did build. But generally speaking, we, uh, that wasn't the kind of user feedback we were getting. I mean, it, that stuff's good. It's good. It's like a, it's much more similar though to the, to the data-driven approach, right? You're finding out, why are people dropping off on this flow? You're not finding out, what problem should I really be solving for them, and what, what do they care about as a human? And for this kind of, like, really early stage user interview, which is the kind of user interview that crucial startups do, uh, that's the, that's where you wanna focus. So we didn't bring anyone on site actually. It was almost all over phone or Skype. Yes.
- SPSpeaker
So for the three different groups of people that have different kinds of feedback, so as a startup you have limited time and resources. Is there a group that you will focus on first?
- ESEmmett Shear
Yeah. So with the three different kinds of people, uh, did we focus on one of them, uh, given that we had very limited resources? Yes. Uh, we focused on the competing, uh, people using competing products because we knew that they already were interested in the behavior that we needed and, uh, they were willing to do it at all. And therefore, all we had to do is convince them to switch, which is a much easier thing to do than to try to create a new behavior where none existed before. Um, and we had to do that because we had to get some quick wins because my gaming project inside of Justin TV would've been killed if it wasn't showing 25% month over month growth every single month. So, uh, we did, and that meant focusing on short-term, get the people in right now. And that turned out to be good in general because, uh-It turns out that building something that some people want generally generalizes. Uh, and so I wanna bring in people who weren't even users of the service as well. Yes.
- SPSpeaker
Twitch has been around from the beginning to like build up, for example, the video game industry. And in the beginning, this industry was very like decentralized, like there wasn't a lot of cohesion with like, you know, different video game companies consolidating where tournaments are and stuff, but now that's very different. So you said originally you spoke to like broadcasters and, um, you know, streamers themselves. How has that changed when like, for example, like Riot has, you know, banned users or professional players from streaming their own stuff? Have you tried to, you know, gain leverage with that or-
- ESEmmett Shear
Yeah, so the question is, what about the game publishers basically, right? The game publishers are these huge important people in the space. Um, A, the game publishers, and any big company for that matter, isn't gonna give you the time of day as a small startup, um, which is both good and bad. Uh, it means you don't really need, need to talk to them because they're, uh, they're not interested in you, but it means you actually just can't talk to them. I mean, we tried, but no one wanted to talk to us. And, uh, uh, they did once we started getting some traction and, and becoming a little bit, slightly bit of a player in the space. I don't wanna like talk that bad about them 'cause they, they, they were nice about, enough about it. It's just that, you know, when you're, when you're a tiny little startup, there's lots of tiny little startups and they, they don't have the time to talk to all of you. Um, as we've gotten bigger actually, uh, the point that, you know, game publishers have become an increasingly important, uh, constituency for us. And if I was to talk about who Twitch does user interviews with now, who we, uh, who we pulled information from now, uh, it would include game publishers, definitely. Uh, because they bec- they've become much more active in the space. It was something that they weren't, uh, particularly active three or four years ago as much as they are now. And, uh, uh, that's another really important point about user interviews in general, which is that the pool of people you care about is gonna shift over time. Uh, the people who get you started, like the crucial people to get your product started for the first six months, are not who will be using it three years later. And it's very important you keep doing this stuff because one of the things that's really easy to do is do a little bit of it in the beginning and, and achieve some level of success, and then just sort of stop talking to new people. Um, and that's a good way to make the n- the next set of features you build be not as good as the first ones.
- SPSpeaker
How about one more question?
- ESEmmett Shear
Yeah. Yes.
- SPSpeaker
How do you give good user feedback if you're a user?
- ESEmmett Shear
Um, so how do you give good user feedback? It's a really good question. So, uh, I think what I, what I want a user to do is I want a user to tell me about, uh, what they, like what they're really thinking, right? And what, what, what their problems really are, and to just sort of ramble. Like, I want someone to like just t- tell me about stuff in their life because the, the more you learn about them as a person and sort of them- the- their, what's going on in the context of what they're doing, uh, the easier it is to understand why they want the things they want, and that's really the critical question. So I'd say like, you know, what I'm looking for in a, in someone who, when I'm doing a user interview, is someone who's going to, uh, be willing to talk a lot and be willing to, to really give me a full, give me a full picture. Um, so that's what, I guess on the flip side, if you wanna be a good-- if you wanna help people out with good user interview feedback, uh, uh, ramble. Like be, just, just talk about stuff, uh, and everything. All right, great. Well, thank you very much.
- SPSpeaker
Thank you very much. [audience applauding]
Episode duration: 46:27
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode qAws7eXItMk
