Dollars to Donuts: Recent Episodes
Steve Portigal
Sam Ladner and I have a new podcast, called Off the Path with Sam and Steve. It’s a limited series, where we talk about what a research career looks like, even when it goes “off the path.” Our first three episodes are live.
Show Links* Off the Path with Sam and Steve on Apple Podcasts * Off the Path with Sam and Steve on YouTube * Off the Path with Sam and Steve on Spotify * Episode 1: The Good Enough Career (Apple) * Episode 1: The Good Enough Career (YouTube) * Episode 1: The Good Enough Career (Spotify) * Episode 2: What Do You Do When You Want To Grow (Apple) * Episode 2: What Do You Do When You Want To Grow (YouTube) * Episode 2: What Do You Do When You Want To Grow (Spotify) * Episode 3 Good Interviewing (Apple) * Episode 3: Good Interviewing (YouTube) * Episode 3: Good Interviewing (Spotify) * Sam Ladner
The post Announcing “Off the Path with Sam and Steve” first appeared on Portigal Consulting.
Tomer Sharon passed away last week. I’m reposting his appearance on Dollars to Donuts from 2019.
Show Links* Episode transcript * Episode (and show links) from 2019 * From LinkedIn, announcement of Tomer’s passing The post 53. Remembering Tomer Sharon first appeared on Portigal Consulting.
It’s the first anniversary of the second edition of Interviewing Users, and it’s now available as an audiobook. Listen to this episode for a sample.
Show Links* Interviewing Users on Audible * Interviewing Users, 2nd Edition (print and eBook)
The post Interviewing Users anniversary and new audiobook! first appeared on Portigal Consulting.
In this episode of Dollars to Donuts, I talk with Emily Sun, the head of Design and Research at Hipcamp. We discuss staying engaged in work, designers doing their own research, and research at a small, growing company.
There’s actually a big opportunity with smaller companies. At small startups, you are much closer to the people who are making the long term vision for what the company is going to be. Because we have access to that level of leadership, there is a lot that can be influenced through research. – Emily Sun
Show Links* Episode transcript * Make Things That Matter — Steve Portigal: Improving your user research process * Interviewing Users, second edition * Emily Sun on LinkedIn * Hipcamp * The Fun Scale * Alyssa Ravasio, Founder + CEO of Hipcamp * Sifteo * CHI conference * 3DR
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
The post 52. Emily Sun of Hipcamp first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my conversation with Tamara Hale, the Director of Product Experience – Research & Insights at Splunk. We talk about the long tail of impact, being an anthropologist of work, and having a creative practice.
The ‘doing the research’ bit is only about a quarter of your job. The rest of it is all the other stuff that goes around it. It’s about storytelling and influence and developing a vision and creating alignment around who the customers are and creating alignment on what actually are the business goals. It’s your stakeholder mapping. It’s your internal research. It’s your knowledge management. It’s improving how we work. All that stuff is part of research, and if you only think of your job as that quarter, you’re missing out on some of the most interesting and also trickiest parts of the job. – Tamara Hale
Show Links* Episode transcript * Tamara Hale * Splunk * Tracey Lovejoy * Catalyst Constellations * Blurring the Lines: Building a collaborative dialogue from the intersection of creative practice, ethnography and business
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
The post 51. Tamara Hale of Splunk first appeared on Portigal Consulting.
in this latest episode of Dollars to Donuts, I talk with Vanessa Arango Garcia, Director UX Research & Research Operations at Delivery Hero. We discuss creating an engaged research community across a global organization, being accountable for impact, and how today’s challenges provide an opportunity for the research progression to grow.
We care a lot about our craft and we need to keep the quality up, but we also need to be pragmatic in how we are able to optimize that process of doing research to focus in the next stages. We dedicate too much in doing the research, delivering that report. And later, sometimes it’s very difficult to dedicate time to following up, connecting with the team, sitting together, ideating, thinking about the roadmap, because we don’t have time. We are jumping from research to research to research because every research takes time. – Vanessa Arango Garcia
Show Links* Episode transcript * Rally AMA with Steve Portigal on what makes a research practice mature * Interviewing Users, second edition * Vanessa Arango Garcia * Delivery Hero * PedidosYa * EuroCopa * The new researcher: Navigating the evolving landscape of UX Research
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
The post 50. Vanessa Arango Garcia of Delivery Hero first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my conversation with Sarah Gregory, Director of User Research for Consumer at Coinbase. We talk about research comms, archiving user research, and doing research that no one is yet asking for.
Our email is designed for one very specific leadership stakeholder, and it is tailored to how that person likes to consume information. There’s a different stakeholder that hates email. That person, I use Slack. Another stakeholder tends to listen very well when they’re live in a regularly recurring monthly meeting. And so I make sure that research always has one or two slides in that meeting. You have to know exactly who you want to be listening, and you have to change your techniques depending on who that is. Which is really just understanding your user, right? – Sarah Gregory
Show Notes* Rally AMA with Steve Portigal on what makes a research practice mature * Interviewing Users, second edition * Sarah Gregory * Coinbase * The Basics about Cryptocurrency * Bitcoin * Blockchain * Ethereum * McDonald’s Theory * Sian Townsend * Jobs To Be Done * Human-Computer Interaction * Becoming a Level 1 Sommelier * Drops of God * NFT
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal.
I was recently spoke with Rally’s Lauren Gibson for an Ask Me Anything session about user research maturity. I’ll link to the detailed writeup and the recording of the full conversation, but here’s a short clip.
Lauren Gibson: I’d really quick, like to go back to the skills point you were mentioning. Are there any group or team skills, like say a team is like, we’re trying to upscale or like hiring staffing. What skills would you say would be good ground zero ones to add to your team or have your team focus on to kind of increase that maturity?
Steve: I have a bias here. My bias is about the work that I do and the stuff that I write about and that I teach. So I’ll own that, but I wouldn’t say interviewing skills, just like I wouldn’t say survey writing skills or Qualtrics skills. I think there’s that piece above it that makes us better as researchers kind of method aside, domain aside. And those are things like, and these are big words, which you have to chew on a little bit to get to where they’re meaningful. But, you know, we talk about words like empathy and we talk about words like curiosity and listening. I think there’s a big skill around sort of not just knowing yourself, but kind of hearing yourself. We work in fast paced environments. We are asked to be experts. And so developing comfort with not knowing and being able to be confident and curious to be able to honestly ask a question that we think we’re supposed to know the answers to. Like these are, these are about knowing ourselves and kind of hearing ourselves and being that person in the meeting that asks the question that no one’s willing to ask, being able to say something. So this is maybe about storytelling, but there’s an empathy and compassion aspect to it as well. Being able to do some research and bring it back in a way. And I don’t mean what is your deliverable look like? I mean, how do you kind of set yourself and how do you talk to a person that you have a relationship with so that you help them hear something that’s new or that’s slightly new, but that’s impactful and significant and might suck for them to hear, right. We’ve learned that the thing that we’re doing is not going to work. And there’s another new problem to solve. Like that’s a great thing to find out, but presented without compassion and without some nuance can be seen as harmful.
And so these are like emotional maturity, hearing your own discomfort, being sensitive to other people’s discomfort, being a good storyteller. These are all in service of relationships. I guess that’s maybe what it kind of gets down to the relationships that we can build with people. And sometimes they’re like at a distance relationships, kind of like we’re having with everybody today. We don’t all know each other and we haven’t spent weeks and weeks in the same room and kind of shared ownership, but we’re all trying to connect and share information and help each other learn and draw from this. So we’re in a lot of environments and we work in different ways where we want to use these kinds of self-knowledge and emotional maturity to kind of build relationships because that’s how these things that we’re trying to accomplish. And change. So, yeah, if I want to talk about upskilling, I think those are the things I would kind of work on because they pay off across the board. And you can do that explicitly. Like I have taught a storytelling workshop. Like you can say, lean into that, but these are also side effects from practicing any of the more technical skills of research. You want to practice survey writing and trying to go to survey. You’re going to learn humility. You’re going to learn empathy, right? It is baked into everything, I think that we do, if we are reflective about our own learning.
Check out the whole session, if you like. And why not buy your nail salon worker and your tax preparation specialist their very own copies of the second edition of Interviewing Users. If you really wanna help me out, write a very short review of Interviewing Users on Amazon.
As always, I’d love to talk with you about the challenges your team is facing and how I can help.
Okay, let’s get to today’s episode with Sarah Gregory. She’s the Director of User Research for Consumer at Coinbase. Sarah, it’s great to have you on the podcast. Thanks for being on Dollars to Donuts.
Sarah Gregory: Thank you, it’s an honor to be here.
Steve: It’s an honor to have you. Can we start off with an intro from you?
Sarah: Yeah, absolutely.
Steve: Are there other directors of research?
Sarah: My name is Sarah Gregory and I am a director of user research at Coinbase, specifically working on our consumer and retail products.
Steve: There are currently no other directors.
Sarah: I am the only one, but we do have different product groups that could have a director at some point. Right now, I’m technically the only one.
Steve: And what is Coinbase?
Sarah: So we are a cryptocurrency company. So it can be anything from where you first bought your first Bitcoin. A lot of people tell us that their first Bitcoin ever they bought on Coinbase. But since then, we also have institutional products for institutions that want to custody cryptocurrency. We have developer products that for people who are actually building with blockchains. So we have all of those, which is the product groups that I mentioned. So I specifically focus on our retail and consumer presence, which is Coinbase.com, the big blue website, the big blue app, our retail customer.
Steve: If that’s retail, does that mean that’s consumers?
Sarah: Yes, consumers.
Steve: I hesitate to sort of open the lid on Pandora’s box here, but for the context of this conversation, what’s the minimum viable explanation of crypto Bitcoin blockchain that you can provide to people like me that know what you’re talking about, but don’t know what you’re talking about? Sorry.
Sarah: Oh, my goodness. I wondered if this one was coming. I hope I do it justice. No, it’s OK. I’ll do my best. So it’s the most famous cryptocurrency is Bitcoin. But there are others. And you can think of it as programmable money. So there are some cryptocurrencies that are just trying to be money, new money. You could use them for paying people. You can trade with them. You can do all kinds of money things with them. So you can have a programmable, borderless money that is not controlled by a nation state. Or there are different other things you can build with some cryptos. You can actually create like different kinds of economic situations. You can do things that are unrelated to money. Not censorable, so you can build something that allows you to ostensibly tweet on a blockchain that could never be removed because it is on blockchain. So there’s all kinds of different things that you can do. So yeah, that’s what I got.
Steve: And how long have you been working at Coinbase?
Sarah: Over six years now, which is crazy. I never would have thought I would last that long. If you had asked me six years ago if I would still be working in crypto in six years, I would have said you’re crazy. You know, it’s just a very fast changing environment. It’s a fast moving industry. I didn’t know very much about it when I started. So I would have thought that there would be some reason to move on and do something different. But it turns out it’s really fun. So there you go. Well, I certainly didn’t know very much when I started.
Steve: Going back over those six years, do you recall the journey, if it was a journey, just to sort of understand the specifics of the landscape and be able to understand it and talk about it?
Sarah: I’d heard of Bitcoin. I had vaguely heard of Ethereum, which is the second most widely used cryptocurrency. But I didn’t know anything beyond that. And they said, don’t worry, you don’t have to know anything. We really just want a researcher to build out a team. And I said, OK. Later wondered if that was a horrible mistake, because that sounded scary and I had no idea what I was getting into. But I knew that I was going to learn along the way. I knew that if I stayed curious and just anthropological, I suppose, in being able to observe and learn about a new emerging technology, then it was going to be OK. And it was. And emerging technology is fascinating. I mean, it’s never the same. You wake up every day, it’s different innovations, different things happening in the greater environment. And so to study that from a research perspective and different communities and cultures that are popping up and different things that are suddenly hitting the scene and trends taking off, there is nothing really like it. Oh, yeah. Yeah, it does. I think that I mean, it’s it’s three times longer than I’ve ever been at any workplace.
Steve: You made that comment about at the beginning of the six years, you wouldn’t have imagined that it would be sustainable. Not your words exactly, but I’m getting that because it has changed, it continues to be, I don’t know, interesting, new.
Sarah: I think that’s probably why I wouldn’t have thought that I would stick around just because every job I’d had prior, it’s like you kind of hit that two year cycle of like, this was fun. I feel that I mastered some things about this. Am I bored now? And sometimes the answer is yes. And in this case, the answer just kept being no. I felt like I did different tours of duty. The role changed as I went and I felt like I was always learning. So focusing on consumers, we’re focused on regular people who are not companies.
Steve: So you’re focused on the retail part of the business. Can you paint a picture of what research looks like? You know, anything about sort of structure, activities, team?
Sarah: They’re not developers building apps. They are just regular people who are interested in buying and owning and using cryptocurrency. So because we’re responsible for Coinbase.com and the app, we’re also responsible for Coinbase Wallet, which is our slightly more intermediate crypto app. I’m happy to explain that if that’s interesting, but just, you know, people who are interested in engaging in the crypto space, sometimes as an investment. Sometimes they’re doing other interesting things with it. That is our retail customer. So it’s everything from the super early funnel of, okay, how many people own crypto? Who’s interested and what do they know about it? Where are they starting from both in the US and internationally? And then all the way down to, hey, we’re about to release a staking feature. Okay. Like what features should it have? Who’s going to use this and what are the requirements that they’re interested in for this feature? So it can get pretty deep. We have a pretty diverse user base. You can imagine that there’s a really big difference between the really advanced crypto users who are super into this stuff from like a hobbyist perspective, all the way to people who are just hearing about Bitcoin. They got interested in it. They figured they wanted to own or buy some. And what does the experience look like for them and everybody in between? There’s a large range. So we’re doing some foundational and strategic. We’re doing some long term stuff. We’re doing a lot of tactical stuff. We are doing a lot of concept testing.
Steve: What does your team look like?
Sarah: Working on retail and consumer right now, there’s six, seven, I suppose, including myself, eight, if you’re counting our lovely intern. And then there are a handful of other researchers working in the different product groups, not on retail and consumer. And so I believe it’s 10 or 11.
Steve: Before you said, oh, those groups could have directors of research but don’t.
Sarah: Right, exactly.
Steve: That’s those other researchers.
Sarah: It sort of gets into how we’re organized, which I know different research teams may or may not report into design or product or, you know, are you centralized? Where are you located in the business? We are in product reporting through whatever is the specific area of product managers that are our stakeholders. So we have our consumer product group, which is focused on retail and consumer. So that is headed up by a head of product who then has a senior director of design and research in our case, his name is Jeff. And then I report to him. So we have, we report into design, I guess, if you wanted to put it that way, who reports into product for retail and consumer. And that would be true of institutional, that’s true of developer. And so it’s interesting because, yes, we report into design, but we have a very close relationship with product as a stakeholder. We used to be centralized. There were previous iterations of the company where we had a VP of design and then, you know, we had a director of research who was across all the different product groups. And that came with its own challenges, as I’m sure anyone who’s worked on a centralized team would tell you. You’re kind of expected to cover everything and everything is your responsibility. Whereas when you are reporting into the product group that is also your stakeholders, obviously that comes with its own limitations and challenges. But at the very least, you’re close to where the decisions are being made. You’re close to the people that are, you know, working on the thing. And so that’s true of, we’re very close to our design stakeholders. We’re very close to our product stakeholders. And so we have a friendly and happy communal relationship with the other researchers and the other product groups. But fundamentally they are studying different audiences. They are working on different products. And so we have like a shared tool stack. We have a shared budget and we engage in some practices together like research crit and things like that. But oftentimes we’ll be working on different things.
Steve: For your folks, what’s the mechanism for determining what you’re going to work on?
Sarah: So obviously we’re very close to whatever are going to be the priorities of the product group that we’re in. They have OKRs, very common setup, but we’re not necessarily married to exactly what’s on their roadmap. I tend to think that a healthy diet for a researcher is some stuff supporting their roadmap where either they’ve specifically requested it or we have gone in as experts in subject matter experts and user research and said, “Hey, we think you should really do this research.” And then also having our own proactive ideas about maybe the things that nobody’s asking for. So we own our roadmaps. Product does not write them for us. We generally say based on what we’re observing of your priorities and what you’re working on, here’s our quarterly roadmap and what we’d like to do. And maybe they have some feedback or suggestions, but generally they trust us to say, “Here’s the research we think you should have.”
Steve: Yeah, can we talk a little about the research that you’re proposing that’s not directly tied to what the product’s roadmap is?
Sarah: So I think it’s a little bit of keeping your ear to the ground of what are the inklings that you’re starting to hear. They’re not asking for a project, but maybe there’s a particular topic that seems to be coming up a lot or a question that you hear leadership kind of grappling with. And you’re like, “I’m not sure if they know that research can help with this, but I know that research can help with this.” And so we’re going to proactively reserve some room on our roadmap to really give this a shot. And oftentimes, they’re grateful that we did it, even though they didn’t ask for it because you never know when they’re going to react really positively. It’s like, “Oh, I didn’t know you were working on that project, but I’m so grateful that you did. That’s super interesting.” Or maybe they do ask for it six months later. You’re like, “Wow, I’m really glad that six months ago I thought to invest in this research because you correctly anticipated the thing that they would be interested in.”
Steve: We could probably draw some lovely two by two because who doesn’t love a lovely two by two? But if there’s these categories of, you know, research that’s directly responding to things that are being done like OKRs and things that are about keeping your ear to the ground on one axis, I’m thinking about the other axis because you talked about foundational or tactical and sort of other ways of dividing up the research. I’m curious kind of how that maps out and maybe that’s not a fair question but to me it’s very different, I would guess, to say, oh, we’re going to, you know, on our own go evaluate some design directions or some interaction mechanism versus we’re going to go understand some motivations or some set of behaviors. OK, so I drew with my hands the two by two but I’m not even sure that’s the right way to think about it. Can you say a little more about, you know, how you might characterize or break down these studies that you’re identifying the need for?
Sarah: It’s definitely in the latter that you mentioned, which is better understanding a particularly emergent behavior or a motivation. Maybe there’s a competitor name or a particular space that we continue to hear coming up a lot and we’re like, hey, we don’t really have a lot of information about that or I can tell that there are some strategic discussions or decisions that it sounds like they’re kind of getting stopped up on. So it’s very much in the foundational category that we are proactively suggesting things. There are times that we go out there and we say, hey, this is a designer, a feature that hasn’t been evaluated in a while and we think that it’s important to go look at this. But I would say more often it’s in the foundational strategic. For example, one that we’re considering right now is there’s this particular trend that we’re seeing happen and we’re trying to figure out whether or not that’s a space that Coinbase wants to play in. This is something new that we’re seeing people do with crypto. We’d like to better understand exactly who’s doing this, why are they doing this? Is this an area where we think we have a competitive strategic angle or is this something that we could support in some different way? Or do we just really want to better understand this and now six months from now when a stakeholder comes and asks me, hey, what do we know about this? And I can say, oh, we looked into it.
Steve: Well, that leads me to a follow-up in terms of what happens with this research. It’s not responding to a request or it’s not tied to the — I’m sort of floundering even, like, what’s the label for the kind of research we’re talking about?
Sarah: I’ve heard it labeled many things because I have heard other research leaders talk about this often f like, what do you call it? Is it forward looking pathfinder, horizontal? I mean, it’s like there’s so many different buzzwords you could throw around. Like we all deal with this, right? We all deal with what do you do when you’re trying to get out in front of a question? So I’m not even really sure we know what we call it, but we certainly know it when we see it. And you can also see different research leaders have different strategies for how do you do that? Do you have a team that is specifically not dedicated to a certain group of stakeholders? They’re kind of like a centralized team. Do you get contractors for that? Do you pay vendors for that? Like how generally do you deal with it? And what I will say is that even though it can be hard to know what are the shots to take, like when do you take your roadmap and reserve a big part of it for this thing that you’re not even really sure if it’s going to land impact or lead to something? Every time we’ve done it, I’ve been so grateful that we really like scraped by and managed to find the space and time to do it in whatever way, because inevitably what happens. So here’s something about crypto. Crypto is cyclical. So the price of Bitcoin goes up. You get a whole bunch of new people who are very interested. They’re curious about this space. They’re hearing it in the news. They may not know a lot, but they’re excited to engage. And so then your doors are flooded with all of these people who are in this like beginner mindset. Then the price of Bitcoin goes down, as it always does.
And then suddenly you still have very active users, but it’s the people who are really hardcore in the space. They’re very hobbyist crypto people who never… They came a long time ago. They’re never going to leave. Like they’re just really interested in this space and they stuck around. But there’s fewer of them. They definitely engage with the product in a very different way. They have a very different set of needs. But inevitably, because, you know, last I checked, crypto has not died yet, even though they like to say Bitcoin is dead or whatever they say, it has not died. So inevitably the price goes back up again. Well, now, I mean, all the research that you’ve been doing, that user that you’ve been looking into and serving for the past year or two or however long it takes, now you’re flooded with the new users again. And so you actually, we invest a lot in lit reviews and summarizing past research, often as like the first step to a research study of, you know, like what have we looked into before? What is relevant? What is evergreen from the last time that we entered the cycle? And the time that we entered the cycle before, you can start to see longitudinal patterns. And that can be a very helpful way to guide where to go next, because you can’t always predict exactly how the market is going to evolve and what’s going to change. But you can say, we’ve been here before and here’s what we think might become relevant later. So you can try to call your shots. Sometimes, yeah, I would say we tend to do them in one of two cases.
Steve: Are you doing lit reviews to look for what those shots might be or what those speculative projects might be?
Sarah: The first is when a stakeholder comes to us and says, hey, what, you know, we want to do a research project on X. What do we know about X? It’s like, well, let’s have the very step number one be to check the archive and say, what do we already know? And sometimes we know a lot, sometimes we know a little, but the very least we can check the box and say, hey, maybe we don’t even need to do research. We actually already have a perspective on this. We’re able to save ourselves a lot of time and have a much more comprehensive strategic point of view for you. And 10 out of 10 times, the stakeholder is like, great, awesome. Like, I’m so glad that’s even better than the thing that I asked for. And we’re able to just machete that request away and move on. And then there’s the kind of more long-term cadence where maybe once or twice a year, I’d say, we’re doing it, where we’re genuinely looking across all the research that we did over the last six months and sometimes even prior and saying, what have we been learning? What do we know? What are the themes that are coming out? Let’s get together as a team. Let’s run that workshop. And we take a different approach every time. It doesn’t always look the same. We kind of experiment with different brainstorm formats or different structures for it, I guess you could say. But then the outcome is, here’s the trends we’re seeing. Here’s what we’ve been learning, big themes. And sometimes it identifies, here’s a big area that we don’t know about. It actually resulted in this recommendation for ourselves of this trend that now we want to go follow with our next six months. We revealed a knowledge gap for ourselves. So that, I’d say, we do maybe once or twice a year.
Steve: I want to ask just a very tactical facet of that because, I mean, this is a challenge for so many teams. What information do we have? Where is it? Who knows about it? How do we access it? I think you called it the archive.
Sarah: Yes.
Steve: How have you organized your information such that you can do this?
Sarah: I feel very passionately on this topic. I think that my team, if you were to ask them, what are the things I’m best known for? One of them is aggressively making people document in the archive their studies, because I’m such a big believer in pulling out past studies to not only save time for yourself, but also invest in those longitudinal data points and perspectives. It’s literally just a spreadsheet. I think a lot of people have this misconception that it needs to be some fancy tool. We’ve tried that. It didn’t work out because then we lost budget for fancy tool. Fancy tool went away. Now you’re scrambling to try to migrate the entirety of your archive, which in my case, goes back over six years, right, that we’ve been working in this space. And so we literally just use a glorified spreadsheet. And you’d actually be amazed at how many other research leaders I’ve met who also say, yeah, it’s basically just a glorified spreadsheet. It’s like all these tools that have their merits for sure, and yet for what you need, how much better is the glorified spreadsheet, right, Google Docs? And so we do have our glorified spreadsheet.
And I think that no matter what tool or what format you’re using, the most important thing is regular accountability for updating it. I think that that’s where a lot of archives fail is you have a nice one, but is it consistent with making sure that you’re putting new stuff in it and then tagging that stuff and making it easy for researchers to go back and look through? And so at the end of every month, we look across what are all the studies that happened this month? And everybody has to update the archive. And there are various ways that we keep ourselves accountable for that. And then we hold each other accountable for when there’s a new study request. Did you check the archive? Have we looked at this before? I think that it’s more of a cultural practice than it is any kind of magic bullet tool. Oh, my gosh, let’s pull it up right now, shall we?
Steve: What are some of the columns in the glorified spreadsheet?
Sarah: So I think that we’ve had a number of org changes, so I actually don’t pay a lot of attention to like what team or, you know, because team names may change, teams may change. I think what’s more important that we have is what was the top key insight? The single top key insight, not a list, the single top key insight and recommendation. And that can give you like your two to three sentence summary of what we found in this study. And if you’re interested, then you can click the link to learn more. Usually that contains all the keywords that you need. There is a spot to add additional keywords if your two to three sentences of your top key insight did not include those keywords. But also the DRIs, sorry, excuse me, that’s such a corporate term, the directly responsible individual who did this study, who was involved. Literally, those are the only things that you need. Yes. Yes, that is the researcher that did the study.
Steve: It sounds like very few columns. The DRI is the person that led the study, that did the research?
Sarah: Sometimes there’s multiple names listed. We also have the month and the year and the title of the study with the link to the report. And that’s it.
Steve: Is that archive just for researchers?
Sarah: The primary audience for the archive is researchers, primarily because I like it best when a stakeholder is coming to us saying, what do we think about X? Right? I mean, sure, we could give them the link to the archive and they could search it themselves. But that’s not the same thing as coming up with a research informed point of view, which I find is actually the primary value that we provide and the primary thing that they actually would like from us. Sure, they can go and search for past studies, but oftentimes they want to know, what do we think? Do we think we should move forward with this? What do we think the opportunity is? Which comes from a human researcher who has access to the archive more so than a PM or a designer running around looking for their own studies.
Steve: I was in a conversation recently and talking to people, they were running basically a mentorship matching organization. And so we were trying to understand how did that work? And they described so much about what kind of data was captured, all these different fields that they had, but then they said that most often their ability to match was that so-and-so who did intake for somebody remembered that they had just talked to somebody else. Like there was institutional knowledge or knowledge held among the people as opposed to the data itself. I’m thinking about them as you’re talking about this, like these tools invite a certain usage like you’ve captured everything so you can dive into it, but people work on stuff and they tell stories like culture kind of carries a lot of this. What kind of things happen when a stakeholder comes and says, “Oh, I have this,” is as I’m kind of surmising recall and interactive social, cultural stuff. Does that impact the sense of what we know versus maybe a more cold-blooded look at data and fields and reports and so on?
Sarah: The short answer is yes, I do think it does. I think there’s no replacement for the people who remember the last time and the last time that we did this study and the people who were on the receiving end of that request. And one nice thing, this is just luck, I suppose, is our research team is still very small. And it’s also a lot of people who have worked a long time in this space and at the company. And so they do remember the last time. And I think in those situations, having as a practice to rely on the fellow researchers and have a strong community of researchers who will phone a friend and say, hey, has anyone ever looked into X? Or does anyone remember a time when a stakeholder asked about X? And in addition to also combing the archive and seeing what exists, I think both are equally powerful and important in their own ways. And that a good comprehensive look at what we know is probably going to involve both. And not just researchers, right? I mean, what about also talking to the other important stakeholders and functions, PMs, designers, data people in our lives? And so I absolutely am a believer in that institutional knowledge. And I’ve seen what it looks like when you lose that institutional knowledge.
So I mentioned that our industry is very cyclical. Price goes up, price goes down. The user base changes. Well, also the size of the company has changed and the people working at the company have changed. I am absolutely a dinosaur by Coinbase and crypto standards. And so I remember the last time, am I the only person in the room that remembers when we discussed this, right? And we were in a period of hypergrowth where the company’s strategy at the time to address the growth was to just hire, just relentlessly hire. And it was like every time you talk to a person, that person was new. Their manager was new. Their team was new. They didn’t know anything. And it feels like Groundhog Day. You know, like I’m snapping here. It’s like Groundhog Day. Like you end up just rinsing and repeating the same things. And in those moments, I think that you’re not doing your job if you’re not looking at what we might have looked at before. You’d end up executing the same study every three months because every three months, you know, a stakeholder shows up and asks the same question. Where are your personas? That’s a common one. So I think that that’s where we really learned how much time we could save by keeping track of the last time that we did it. And fortunately, we are — I mean, fortunately, not fortunately, I don’t know. Non-normatively, we are not in a state of hypergrowth at this time. So the nice thing is there’s just less turnover with human beings. And so there’s less thrash with like the same types of questions and, you know, a lot of that chaos.
Steve: You said a bunch of interesting things, but one reaction is that phoning a friend, I tend to think of that, my own bias there is that’s kind of a failure, but you’re characterizing it as a success. That’s a valid way to tap into what we as a collective group of people who work together know.
Sarah: Yeah, absolutely. Because I think that a lot of knowledge is still in human beings. And we can try to automate away. We can try to make it factoids and bullets. But I think that our knowledge is cumulative. Like as researchers, we are anthropologists in this space. We are deeply putting together everything in our heads and having a point of view. And that point of view is what we offer. I mean, in 10 years, when all of our jobs have been automated away with all these AI tools or whatever, I mean, the last thing I think to go will be. But you, as a researcher, yes, I know all the findings, all the data points, all the studies. But what do you think we should do? Like what is your strategic recommendation? That is the thing that is hardest to automate away. And I think that that is the value that we most provide to the company and to our stakeholders and to each other.
Steve: I wonder if the discipline that you’ve created around updating the spreadsheet, it’s like why we take notes about things and then we remember them better. You know, as you’re talking about these two approaches, I think you’re saying they’re interrelated in an interesting way.
Sarah: Yeah.
Steve: So, yes, if you have a discipline of updating the database, the glorified spreadsheet, then that might make you better at recall when someone says, hey, what do we know about X?
Sarah: Absolutely. I mean, even if I were the person who conducted all of the studies for the last few years on a particular topic and I put them in the archive, I still might be going through and using the archive to bring all of those studies that I have done before to kind of bring them all out, look at them, think about them, come up with a take, maybe do some affinity mapping if you’re like a Miro enthusiast or something like that. And so it is helping me put together the data points that I have collected, regardless of who collected them. You’re kind of engaging in this coming up with the perspective. Okay.
Steve: Maybe we’ll go back to the kind of research that has no name. You gave the scenario a couple of times of, you know, someone, a stakeholder approaches you and says, you know, you’re able to say, like, well, we’ve actually done that research and here it is. And so that makes me think about, oh, you’ve got sort of the stuff kind of held back. And I was wondering when you do the research that I’m not effectively categorizing, yes,
Sarah: The research that shall not be named.
Steve: when you’re doing the research that shall not be named and you get to some point, you’re, quote, done with it. And you’re not really done, but, you know, wrapping that up, what kind of things happen with it?
Sarah: I think that in the ideal perfect world, somebody says this is revelatory and life changing and we’re going to go do everything you said we should do. And I think we all know that that doesn’t always happen. Sometimes it’s the opposite where it kind of is the tree that falls in the forest and there were people around to hear it, but they were fairly unmoved and it just wasn’t the right time, you know, or maybe there were some unanswered questions. It informed a V2, but it appeared to just stop where it was and it went into the archive and nothing happened. And I think that’s the thing that you kind of have to do is kind of put a flavor in second lives and third lives and fourth lives of research that you were kind of bummed because maybe it didn’t have the revelatory life changing impact that you wanted it to have at the time. And then you bring it out again and you say, actually, this is exactly what I needed. And maybe it just was a little bit ahead of its time. We’ve seen that happen a lot. Big believer in evergreen research. A lot of jobs to be done research is like this, I find. And that’s like a whole category because then you also have to help stakeholders understand exactly what to do with that. But a lot of that stuff, I mean, I cannot tell you how many times we have pulled it. We all have our favorite studies, right, where we looked at behaviors in banking. I mean, completely outside of crypto, just unrelated. How do people deal with their regular money? What’s important to them in banking? At the time, it was like, how relevant is this? I mean, most people aren’t seeing crypto as their bank. It’s very different. But the number of times that we’ve gone back and said, here are the canonical behaviors that we observe with people and money has been, I would do that study all over again. It was a great investment. And we just didn’t realize it necessarily one month, two months after we finished it.
Steve: You’re painting a picture a little bit of, you know, we talked about the second lives, then that first life, the research is completed and doesn’t get the uptake that it might that first go around. How does it get published or shared or socialized when it’s something that is on your roadmap and not on somebody else’s?
Sarah: We share it in largely the same way. We have certain established channels, cadences, ways that we summarize recent research. And it doesn’t really matter where it originated or even sometimes who on the team did it. We’re still going to do the roll up of like, here’s the biggest insights that came out this month. We have a channel where we share our completed work. And we would share it the same way as anything else. And at that point, it might be like, oh, my gosh, this is amazing. Or at that point, it might be crickets. And sometimes you have to go further beyond that. It’s like, what live channels do we have access to? Is there like a regular leadership monthly where we can present one or two insights from this? Or are we going to signal boost in different channel? I mean, obviously, the way that we work digitally and remotely impacts this a lot. We’ve had to think a lot about research communications in a variety of different ways. We’re never not thinking about research comms, actually, if I’m honest. And so you really try to make sure that you’re signal boosting it to the right audiences in the channels that they pay attention to in a variety of different formats in every flavor. And then if it still feels like it’s not generating conversation, then you file it away. And it’s amazing how suddenly in six to 12 months, they start asking again, and you’re really grateful you did it.
Steve: This to me gets at a complaint or an anxiety that researchers have. And I don’t know if this correlates to sort of the overall research maturity. You know, I can imagine someone listening to this and saying like, wow, you guys are going off and doing research that you think is important. And you’re like, I wish I had time for that. But let’s say I did have time for it. I did that.
Sarah: Right.
Steve: Well, no one wants it and no one cares about it. And so it would just be crickets. But you’re describing a larger strategy. As you said, you never are not thinking about research comms. So assuming relevance, assuming that, you know, everybody can understand the relevant research, you have a built out structure for comms, that kind of a baseline. And then you’re also thinking about on a piece of material or a set of insights specifically, how to go about that. And then I think, as you said, you know, you bide your time as well, because you have a handle on what the information is. It doesn’t have to sync. It can hang around because you have a way of retrieving that and saying, oh, yes, we do know something about X. But it’s something about Y.
Sarah: For sure. And I think you have to know the shots to take. I’m not saying people should just go do all this research and be OK if it doesn’t go anywhere. I’m a big believer in trying to make sure that you’re taking the most impactful shots that you can take based on your knowledge about the organization that you’re working in, like the space. And if putting together all the information, the best information you have at the time and saying, look, I really think this is a knowledge gap that strategically matters to our business and to what we’re doing. And if you’re right about that, then I do think it’s just a matter of time that somebody comes back and is glad that you did the thing. And there is no magic formula. I mean, it really is like you have to sometimes beg, borrow, and steal your time of like, yes, we all have a million requests that are coming from stakeholders all the time. And how do you still reserve the space? It’s hard. It’s hard for everybody. I wouldn’t say we’ve found a magic formula. Although, if I did have one tip of something that has worked, if there is something that you really want to prioritize that is proactively initiated and not a request from somewhere outside of your research team, what are you going to deprioritize? Are you good at deprioritizing something else? And that is a muscle that we’ve been able to work over time of like, okay, I really want to make sure that we have room for this foundational research this quarter. Can I make sure that my requests from stakeholders are not eating up my entire roadmap? Are there things that I feel that they don’t need as much as they’re saying or that aren’t going to be as impactful as they’re making it sound or that I think can be answered in other more lightweight ways? And if you’re good at doing that, then you’ll always be able to reserve your time for the other things. But that is hard to do. That’s hard for everyone. I do think that the best no’s sound like yes’s.
Steve: And you didn’t use the phrase saying no to things. That is often the label that’s put on that. What are best practices or guidance you might have in the deprioritizing and telling someone that whatever you have to tell them as a result of that?
Sarah: Because at the end of the day, I think it’s important to remember why your stakeholder is coming to you. They’re coming to you because they have a question or there’s a decision that they need to make. And there are all kinds of different ways to help them with that goal that don’t necessarily involve dropping everything you’re doing and running a study. It can be, well, can we run an A/B test? Isn’t that the better way to handle this? Or here’s an educated guess we can make based on this other study that finished recently. Or we have a process where certain very tactical, not super high stakes research, it’s a form of democratization we call partner-led research. So it’s not them doing it themselves. They are leading it as the partner. The partner is PM or design. And we are helping them, but we’re making sure that it’s like the proper method and tool for the job. Like we have a way of helping them execute it themselves when it’s something that’s fairly well scoped to just their area. And so there are all kinds of different tricks of like if they’re coming to you because they have a question or they have a decision they need to make, how do you help them answer that and make the decision in ways that don’t necessarily involve a study? There’s all kinds. And then they’re grateful because they get what they need. And it usually doesn’t sound like a no, we’re not going to do that. It’s like, thank you so much for coming to me. What an interesting question. Let me try to find the best way to answer that for you. And the best way could look a bunch of different ways.
Steve: So you’re helping. It’s not “Go away.”
Sarah: Yeah, I think it’s a solutions-orientedness in terms of they’re coming to you because they want a solution. And so whatever you do, it doesn’t necessarily have to fall within the bounds of research. Sometimes they really just need help reframing their question in the right way and actually realizing that it’s a data question, like a data science question. Or it’s talking through the options and realizing that actually we can probably just make a guess about this, like use our design instinct. It’s actually like just a design — how should I say this? We should use our instincts as professional designers to make a call. Like are we really going to run a study about whether the button should be red or green? Probably not. But they may not know that. And I can’t necessarily expect the stakeholder to always know when is the right moment to do research. That’s my job to know when is the right moment to do research. So if they’re asking me something, that is always a conversation that I’m ready to have. And it doesn’t matter whether it results in a study or not. I think in terms of roadmap, I think that they are the experts in their own domain, right?
Steve: How does it work with the researchers that are elsewhere in the organization that don’t have the same kind of leadership structure in terms of managing priorities and managing responsibilities?
Sarah: And so I trust that they are running their roadmap in a way that works for them. I’m not necessarily going to go try to micromanage that for them. But, of course, we are here if they ever want to bounce ideas or get a take. We also are very transparent with what we’re working on. They’re transparent with what they’re working on. Like we have a very collegial relationship. So I would say that that works with smaller teams because they can always just say, “Hey, has anyone looked at this?” But we also do certain shared practices. Like I mentioned, we have a shared crit where twice a week we get together. And any researcher from any part of the organization can bring a challenge or a topic or something they’d like some feedback on. We have a shared channel. And so if they’re really like, “Hey, I’m not sure whether or not I should prioritize this. A stakeholder is coming to me asking this, and I’m not really sure how I should think about it.” There is a community of researchers, myself included, who are really open to that. And so we have a shared community of researchers, myself included, who can always help bounce ideas.
Steve: So, crit might include what approach to take about a question.
Sarah: Yeah, absolutely. It can be anything. I mean, crit, I suppose, makes it sound like we’re always critiquing research plans or reports of findings. But actually it’s mostly just like a shared discussion forum where sometimes people come and they have a very challenging stakeholder management question. Like, “How should I deal with this?” Or, “What’s the deal with this tool?” Or, “How do I do this thing?” It’s really just an open forum for researchers supporting researchers.
Steve: Do you have any other research comms?
Sarah: Yes, always. Always research comms.
Steve: Do you have some examples of things that you’ve tried that didn’t work and things that you’ve tried that did work?
Sarah: I think you have to be very careful with regular newsletters. We’ve tried a couple of different iterations. I think we still have one monthly, but we really had to iterate many times to get it to be read by the right people. Something that’s very top of mind is having strong, crisp product recommendations, which I’m told is actually different than a lot of environments. I mean, I guess it’s been over six years since I worked in any other environment. But researchers are saying, “Hey, at other companies, oftentimes they want the insights. They don’t want me to tell them what they should do. They don’t want me to be specific about what products should do and what design should do. And I think you should build this, and I think you should make X into Y.” That is something we absolutely do. And not only do we do it, we’re held responsible for doing it. Like, we’re perceived as not doing our jobs if we are not having a strong point of view. And it doesn’t always come naturally to a researcher. We’re data people. We like hanging out at insights. We like the “how might we” statements. We kind of like the collaborative energy. And definitely there’s a way to do that. I wouldn’t throw it out. But at the end of the day, especially for a certain level of leadership, what they’re looking for is, “But what do you recommend?” And I want it to be three bullets, and I don’t want to have to read it very much. And I want you to be very specific, not just make X easier. You know, how should I be making it easier? What is the specific change? And that means there’s a lot of trust that our stakeholders have in us. Like, I don’t get in a lot of debates about methodology, for example. I don’t have a lot of people who want to sit through an hour-long presentation just so they can question whether I talked to the right audience or not. You know, like, they trust that we did our jobs right. What they want is to dedicate five seconds to reading my bulleted list of three things that I think product or design should do. And the more that we have put that at the front and center, the TL;DR is what we call it. It’s very internet lingo. The TL;DR at the top of the research newsletter, which I mentioned we still have. And really the three main TL;DRs from that month. Not everything, not extensive, very short, easy to consume, and it took a while to find the right recipe. And to identify who is the right person to read that, I don’t necessarily expect that everybody responds to email. Actually, our email is designed for one very specific leadership stakeholder, and it is tailored to how that person likes to consume information, just based on what we’ve observed about how that person requests information to be delivered by other teams, like how they like to receive updates. There’s a different stakeholder where we actually don’t expect that that person is going to read. That person hates email. That person, I use Slack. Another leadership stakeholder I want to make sure is listening, tends to listen very well when they’re live in a regularly recurring monthly meeting. And so I make sure that research always has one or two slides in that meeting. So I think you have to know exactly who you want to be listening, and you have to change your techniques depending on who that is. Which ultimately at the end of the day is really just understanding your user, right? Like except that in this case, the product is us as a team and what we provide, and the user is a specific stakeholder.
Steve: I want to follow up a bit on the specific recommendations part. I am always anxious about finding a balance because I think here’s the thing that we learned is no one does anything with it, but if you go too far without the context in here’s how we think you should address it or how we think we should address it. I want to set up a dynamic where someone can say yes and to that because they know the engineering constraints, they know some piece of software they’ve bought or they know what rolls out uphill versus downhill or they know all kinds of stuff. There’s other decisions being made and other expertise that they’re bringing. When I say we learned X and we think we should do Y, what I don’t want is to get shot down, no Y wouldn’t work and then we’re at the end of it. If I have to defend sort of how might we questions, I think it’s to try to lead to the generation of the best solution with what everybody knows. I totally get the more you can bring people closer to taking action on the research, the more impact you’re going to have.
Sarah: No, I completely understand where you’re coming from, because I think if you talk to every person on our research team, they’d say the exact same thing. They don’t want to get shot down because they don’t want it to be the end of the conversation. We had to completely change how we think about that. We had to become okay with getting shut down and to realize that that actually was not necessarily a worst case scenario. I’ll give you an anecdote that jumps to mind. We just did one of those regular roll-ups that I was talking about where we kind of look across the last six months of research and say, “These are big themes, and this is what we think you should do,” and we presented it to a very large leadership audience in a large forum. And we knew we were taking a big swing where we were being very opinionated with, “Here are the three things we think you should do.” And we get into that leadership. It was a presentation. We did this one synchronously live. And I’m like, “Okay, here’s the three major themes. Let’s talk about this first theme. What are people’s thoughts?” And literally a person raises their hand and says, “I think this is a terrible idea. I do not think that we should do this.” And then the next person said, “You know, I mean, maybe we shouldn’t do it in that way, but I actually think we could do it if we did it in this different way.” And then the next person chimes in and says, “You know, actually, I’m somewhere in between you two. Like, I see the value in this, but maybe…”
And it is so funny because we have been beating our heads against the wall for years because we felt like our stakeholders were constantly being like, “You need to be more specific with your recommendations. You need more of a TL;DR. You need to have more of a point of view.” And we were like, “We don’t know what you mean. Like, how are we not being specific?” And we realized that it’s because we were afraid to get shot down. And what we didn’t realize is that they will perceive us to be not doing our jobs if we don’t come out with a spicy opinion. They want the spicy opinion. And if we are disagreed with, then that actually is going to lead them to other different ways of coming up with different solutions way faster than a “how might we” statement would. It was crazy. And I think that this could be a way in which the Coinbase environment is different than other environments. Like, it turns out that after all that time, the best way to get them to ideate different solutions was to throw out an idea and a recommendation that they hated. And that did more than a “how might we” statement ever did. It was unbelievable. And so now that’s been really encouraging the whole team to be like, “It’s okay to get shot down. It’s okay to get disagreed with. It’s okay for people to think that that isn’t what we should do.” Because to express a point of view, a strongly held opinion, is better than not expressing one at all. Because if we don’t express one at all, they’re going to be like, “Oh, I just wish our research team had stronger recommendations.”
Steve: You reminded me of this article which I’ll link to in the show notes. This guy is basically describing, I think it’s the McDonald’s theory. And he’s working on site with people in their pre-times and having this conversation every day at lunch. And he says, “Do you want to go for lunch?” “I don’t know, I don’t know, I don’t know.” And he decides to facilitate the resolution of that by proposing McDonald’s. And then everyone says, “No, let’s go…” They come up with specific choices which they weren’t doing beforehand. And he says basically the same thing that you do. People will come up with good ideas to fend off bad ideas. And so I don’t think you’re proposing recommendations that you think are bad ideas. But you’re proposing a solution to a problem. And so that’s what the audience does in that example. And they are working to resolve them.
Sarah: Yeah, that absolutely resonates where they want to know what we really think. And sometimes they’ll find a report and the report has a very clear, you know, like, here’s what we found and, you know, therefore here’s our recommendations and whatever. And that stakeholder will actually DM me and say, what do you think? Like, even if what I think is actually some version of what the person said, you know, like what the researcher put, what they want is what I think. That is the value that I provide, even if I’m really just kind of like restating what they said. And so I think it’s a good reminder of like, they don’t have to agree with me, but they do need to know what I think. And if I think something that they don’t agree with, at the very least, they know what I think.
Steve: What does shot down mean? That they disagreed with the recommendation in that meeting. But there was a robust conversation about what happens next.
Sarah: Totally. I mean, that, that to me is a success. I would say the failure state is where they just don’t trust that the research is valid. And fortunately, that’s not a situation we ever, I mean, I’m not saying we never find ourselves in that situation, but it is, it is rare. They trust that our research is valid and now it’s an equally valid discussion about what to do with it. And there could be a variety of different opinions and a variety of different inputs, right? We’re not always just going to go do what users said we should do. Right. I think that’s where it starts to become what’s the strategic opportunity. What’s the business opportunity? What’s, you know, what are the other data sources that we have? What about instinct? What about all these other factors that ultimately lead us to a good decision as a company with research and user insights just being one component of that?
Steve: I’m gonna go back to something you said early on that you’re talking about these six years. And you’ve described changes in crypto and then the changes in the company as a response to that. And you said you’ve done these different tours of duty that your role has changed over those six years. What are some moments or eras that are different than the one that we’ve been talking about?
Sarah: Well, there was the era of Sian Townsend. Hi, Sian. I hope you’re listening. We love you. There was a time when I mentioned that we had a centralized research team and there was a leader of that centralized team back when the team was like closer to 30. I lost count. I don’t even know. It’s much larger than it is now. And the leader of that research team was named Sian Townsend. And that was the first and only time that I have ever reported to another research person. I’d only ever reported to design managers, people in products. And that’s great. But ultimately, you’re not talking to somebody who deeply understands your practice and your function. And so for me, I got so much out of that couple of years where I learned how to be a research leader in ways because I’ve only ever worked at small companies. I will admit I’m a little bit of a bootstrapped case where I’ve worked at smaller companies and/or on smaller research teams that just don’t have the level of structure and process. Like I’ve never worked at Facebook or Google or some of these larger research teams where you just learn the gold standard of how a lot of these things are done. And a lot of these problems that sometimes small company people encounter every day, those problems have actually been solved in a different way. But that’s why I think sometimes you go to these research meetups and you see a bunch of research teams of one or people who maybe just have one or two other researchers they work with. And they’re like, “Hi, how do you address this problem? Please help me.” And there’s such a community amongst researchers where we struggle with these challenging problems.
And to see the way that she had the tools in her arsenal to be able to handle these conversations at a level that I had not previously been exposed to was the best mentorship that, I mean money can’t buy that kind of mentorship. So this is like a big love letter Sian, but that is an era, and that was also an era when the team and by the team I mean the research team, but I also mean Coinbase as a company, we were scaling enormously. I mean we were in that period of hyper growth I mentioned where it was like we were three-xing by the end of the year and it was just constant hiring and constant new people. And that was intentional on the part of the company to really try to scale hugely to meet the opportunity, but that comes with its own drawbacks and challenges and it was just a very different time where we were a much bigger team, we dealt with much bigger team issues. Right now we’re back to being fairly small, which is nice, it’s like comfort zone for me, but I learned a lot in that period. What is the difference between a job to be done and a job to be done?
Steve: Is there anything you’ve done recently as a research leader that you can trace back to something you learned from working with Sian?
Sarah: I had been familiar with Jobs To Be Done, but I had not actually lived it and breathed it and we still reference that stuff today. I think that we no longer necessarily do the roadshow of like really actively trying to educate our stakeholders on jobs to be done, we tend to bring it up more in context of like, “Hey, this actually sounds like let’s talk about this in terms of jobs to be done.” So we tend to talk about it more at a small scale, but that’s because we’re no longer educating the company, the company now kind of understands and now we’re just like teaching them like ways and times to like bring it out and use it and how to apply it in various situations. So that was like the first thing she did, she showed up and she was like, “You all need to read the jobs to be done playbook and I’m buying it for you all.” So we literally did a book club and read it, but we still use a lot of that stuff today. So that’s definitely one that jumps to mind.
Steve: You just made note of having worked primarily in small organizations before Coinbase. And maybe just to riff on that a bit, could you talk about your path? How did you find user research and what did that look like to bring you up to Coinbase?
Sarah: It’s always fun to relive this. I majored in cognitive science with a concentration in linguistics in college and undergrad. And I had a fellowship as a lab assistant in a speech production lab and great people, but I just really did not enjoy the academic environment. So that led me to believe that grad school was probably not the right path. I just wasn’t really sure it was out there. I didn’t really know what I wanted to do. I just knew that it was a major that I had enjoyed, but I set it aside for a while and said, “I’ll just go out there and get a jobby job because I don’t really know.” And then I spent the first two years of my career being a data analyst and wow, I have so much respect for data analysts, people who are in there doing the work of pulling the SQL, understanding the table structures, creating the dashboards, the charts. That was me and that is a rough job. I have so much respect for those people. And I was like, “Oh, it’s definitely not that. I definitely do not want to do this anymore.” So I quit that and I was like, “What do I want to do? Do I want to do HR, PR? What do I want to do?” And I thought back to undergrad where it was not like it is now where you have master’s programs for UX design. I mean, it was around, but I wasn’t exposed to it. It didn’t have such a strong presence in undergrad, but I’d taken one class in human-computer interaction. It’s just like an elective. I didn’t even know what it was. And I actually loved it. We worked on a project. We actually won best project. And the award, this is so ironic in retrospect, the award for best project was you get a paid internship doing essentially user research. And I had declined it because I had already committed to join this data analyst full-time role after graduation. And so I was like, “Maybe I should have done that paid internship. Maybe I got it all wrong.” And so that’s when I started thinking maybe this human-computer interaction thing, maybe this is a job that you can have. Maybe this is a career. And I just was finding anyone who would take a chance on me. I was in Chicago at the time, so a smaller, more nascent startup scene. And so I was just knocking on doors of smaller companies. I think I was even trying to get hired as a project manager just because then I could do an internal hop or something. And I found a very small agency that had a research contract, a research project from one of their regular clients that they didn’t know how they were going to fulfill. And they were like, “Oh, you want to do this? We’ll hire you to do this contract.” And I was like, “Yes, please.” And I’m so eternally grateful to that agency for giving me that chance because I’ve been a researcher ever since and I love it. I’m so glad that I am lucky enough to have found a job that I love.
Steve: I also came through a different era, but human-computer interaction was my toe in the water or peek through the keyhole if there was this thing.
Sarah: Yeah. It’s funny to watch how the terms evolve over time. Human-computer interaction, HCI, I still see classes that are called that. You still have academic texts that are called that, but no one would say that they would work in HCI anymore or anything like that. But I think a lot of us, that was one of the first words we heard to describe this thing we thought was addictive and wonderful.
Steve: Is there anything that you’d want to share about yourself maybe that’s outside the professional realm that would give us a little picture of you?
Sarah: Yeah. Well, we just bought a house. So first time homeowners. I’m based in Seattle and it’s fun to do all this stuff. I waited a while to own a house, I guess. I was very cool being a renter, but getting to know all kinds of things like researching lawnmowers and things, although mostly that’s my fiancé. I have to credit him. He’s doing most of the researching on lawnmowers. Well, here’s the thing. We are level one sommeliers, actually. We have done the training in the court of master sommeliers. We’ve done the wine testing, taken the tests and we love wine. And so that’s a hobby we have on the side. He works in food and beverage. And so that’s a thing we really like to do. We like to go to wine country and we like to do wine travel in other countries. So yeah, I have my little certificate right here, my sommelier certificate. I mean, short of becoming a professional sommelier, mostly just like, I don’t know, impress people with it. I don’t know. He might study for his level too, because I think at a certain point it does start to matter. But I think being a professional sommelier is not necessarily the most common job in the world. I think you use it to plan trips and be a nerd and I don’t know, impress your friends. That’s what I do with it.
Steve: And you did that certification process together.
Sarah: Yeah, we did. It was actually fairly early in our relationship a couple years ago. And I had mentioned that it was something I was always interested in. Having lived in San Francisco prior, I spent a lot of time in wine country and got curious about the words that I was hearing and what I was drinking. And so when I met him, I had already had an interest in doing it. And he was like, “Oh my gosh, what a coincidence. Me too. Let’s do it together.” And I was like, “Well, I guess we only just started dating, but okay.” And actually next year when we get married, we’re getting married in wine country at a winery. So I guess it’s just what we do now. Yes, we love it. I love that show. I would
Steve: There’s this television show, and I wonder if you’ve seen it, if you have an opinion about it, called “Drops of God.”
Sarah: Yes, we love it. I love that show.
Steve: Yeah, how would you describe that show?
Sarah: describe it as very well researched. I mean, I’m not like the world’s foremost wine expert or anything, but I really enjoyed it. And I found that the wine details were dead on. It’s also just, it has that melodramatic flair that is a guilty pleasure for a lot of people. We really enjoyed it. We’re super excited for season two.
Steve: To me, and I think that show was about incredible expertise, right? It’s about a woman whose strange father dies and is some kind of just global expert in wine with this amazing book. And he’s left France. She’s French. And he’s been in Japan, and he’s mentored this Japanese man. And when he dies, he leaves everything to one of the two of them, and there’s this competition that they have to go through. And the competition hinges on this incredible level of expertise that they have in tasting, identifying, and there the drama unfolds.
Sarah: Oh yeah. I love that philosophical question of like, she was like born into it, right? Like she has this instinct of she hasn’t looked at wine in a while, but she just has this great palate. And then he is the trained, the mentored, he’s very well studied. Like he worked really hard to get where he is. And so it’s a little bit like, is it innate? Can you learn it? Like it’s sort of asking these questions about the knowledge that we have is one more legitimate, is one better. So I won’t give away the twist, but people should watch it.
Steve: That’s a great framing on it, because I don’t personally care about anything about wine, but I loved that show because it was about knowledge and passion for a topic.
Sarah: Yeah, absolutely. It’s like, what can you own and claim as like, I have the right to this. I deserve to win this competition. Particularly with something that a lot of people don’t know a lot about wine. A lot of people are kind of intimidated by wine. That’s a reaction that I often get is like, oh, well, I don’t know anything. I really, I don’t. And it’s like, I mean, you know, like you just drink what you like. I’m not going to judge anyone’s wine taste. Like I love stuff you can get at a grocery store or whatever. It doesn’t need to be fancy. And I think particularly because people feel that they don’t know a lot about it. It was like a fascinating topic area to choose for that question.
Steve: Is there anything you think we should have talked about today that we didn’t get to?
Sarah: I mean, you didn’t ask me to sell you on cryptocurrency. My explanation was horribly basic. And you’re not just going to say, oh my gosh, please explain to me, extol the virtues of blockchain.
Steve: I actually can’t tell if you’re being sarcastic or serious right now.
Sarah: No, no, no. I mean, I think that people, I love talking to people about crypto and blockchain because I realized that it can be kind of like a weirdo topic for some folks. Equally intimidating like wine, actually, similar to what we were talking about. But some people are like, you know what, I will ask you if I’m curious about the details of how blockchain works. But maybe not everybody wants to be hearing about it constantly all the time. And I don’t necessarily want to. I don’t know, like if people want to go there, that’s great. But if they don’t, they don’t. I mean, you can imagine at a cocktail party, it used to be easy. I used to work for Ancestry. And people were like, oh, what do you do? I work for Ancestry. Oh, my grandma loves Ancestry. My aunt, you know, like I love it. I did my family tree. Oh, I work for Coinbase. What’s that? A cryptocurrency company. A lot of people are like, oh, like you can tell immediately the people who are like, please explain blockchain to me versus the people who are like, and I’m going to change the topic. I wish you just said you were an accountant. So, you know, I try to just I never know how deep to go, but I do find it a very interesting space.
Steve: I mean, to be transparent, I was, you know, when we talked about it at the beginning, I’m thinking about what would I ask you versus what would you and I ask people listening to this to listen to? So, yes, how much to ask, how far to go.
Sarah: Yeah, I didn’t know what to what to expect. I mean, I didn’t know if this was going to be like, so, you know, like explain blockchain. And it’s great. We actually talked mostly about research stuff, and it’s amazing how industries can be different. But research as a practice stays very much the same. But crypto has a weird brand associated with it. I think that it can there’s just a lot of like weird PR that makes it feel a little like ooky. And I can speak from a first person perspective is very off putting when you’re you’re first in it and you’re like, who are these crazy people that would want to work in this space? Like, is this even legit? And I can understand like, for a lot of folks, they’re just kind of like, yeah, if I really wanted to be like, there are plenty of people running around extolling the virtues of blockchain. And I’d be more than happy to be one of those. But I realize that sometimes it’s like, let’s just talk about the research.
Steve: Well, maybe you can take us out on a nugget or theory or something blockchainy, since you’ve kind of teased us now. Maybe we can wrap on that.
Sarah: I think most people when they first get into crypto, it is an investment. And that’s okay, because investing is not bad. Investing is great. And, you know, that’s very cool when people are interested in investing in it. And we have a lot of like financially related, you know, features and things. But crypto is not just about making money. And those are the parts that are fascinating and kind of change often and are still super early and nascent. And to a lot of people are like, what are you talking about? Like, that’s craziness. Like the idea that I said that it was like programmable money. They’re like, what even is that? But it can be dense, but you don’t have to be an engineer or a very aggressive, like financial person. There are parts of it that are equally interesting. So anyway, but like, I encourage people to be curious about all the different things that can be done with blockchain that are interesting to a lot of different people. No, I don’t. I don’t think so. And like, it doesn’t always evolve in the ways that
Steve: It kind of loops back to your point from much earlier that, you know, there’s a reason that you’ve stuck around for six years, your whole am I bored yet? And you’re describing a technology that you’re involved in that has potential that it hasn’t come anywhere near reaching.
Sarah: you think it’s going to evolve, right? Like, do I think that Bitcoin is going to replace the US dollar tomorrow? No, nor do I necessarily think it should. I don’t know. Maybe that’s a debatable topic. But I mean, different things, like people got really sick of hearing about NFTs. Understandably, they were everywhere. And people are like, why would I even? This is like a JPEG. I do not get this. Like, why is this constantly everywhere? Like, I get it. I get how it feels. But there’s cool things that you can do with NFTs. Like, I won’t bore you, but like, what if there was a way to verify your identity without having to share any personally identifiable information? Like, it’s basically an attestation token of attesting to your identity without having to actually share information. I mean, are a ton of people doing that today? No, some are. But I think that that’s where there’s still a lot of interesting technological development in the space. So is it an empty promise? Is it going to be the thing that we all do tomorrow? Is it somewhere in between? I think that’s where you kind of have to stay tuned to find out.
Steve: Sarah, thanks for a great conversation. We covered a lot of stuff and went into a few different nooks and crannies. It was really great to have you on Dollars to Donuts, so thank you for the time today.
Sarah: Thank you. It was great to be here. Thank you.
Steve: There ya go, whaddya know, that’s the show! Find Dollars to Donuts wherever you get podcasts, and at Portigal dot com slash podcast for all of the episodes with show notes and transcripts. I would love love love for you to rate and review Dollars to Donuts on Apple Podcasts. Our theme music is by Bruce Todd.
The post 49. Sarah Gregory of Coinbase first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Jamika Burge, the head of research for Data and AI at Capital One. We talk about her journey through academia, discovering user research, and intersectionality.
Doing good – for me, as a researcher, and as someone who wants to do good in the world, it means understanding people’s needs in context and providing opportunities for them to succeed. That’s what that means for me. Success can mean different things to different people. I can guess what success means from a business perspective. I can even guess what success means from a researcher perspective, but ultimately it’s that end user who tells us whether or not we got it right. I want that person to feel as an end user, free to share with us when we got it wrong, but also when we got it right. – Jamika Burge
Show Notes* Interviewing Users, second edition * Steve Portigal on the UX Podcast * Jamika Burge on LinkedIn * What is the Positive and Negative Affect Schedule? (PANAS) * Center for Human-Computer Interaction at Virginia Tech * IBM’s Thomas J. Watson Research Center * John M. Carroll at Penn State * Mary Beth Rosson at Penn State * Office of Naval Intelligence * DARPA * Spelman College * blackcomputeHER * #blackcomputeHER conference * National Academies * Capital One
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal.
I was recently a guest on the aptly named UX Podcast, where in reflecting on the 10 years since the first edition of Interviewing Users and the new second edition, we chatted about the changes in the user research field I’ll link to the whole episode, but here’s a clip.
Per Axbom: As you were describing this shift to in-house teams, I almost felt this sense of jealousy, in that in-house teams then get to work with research over time in a way that I as a consultant cannot. And they have a shared experience. And they can even reflect back on research they did two years ago, based around the same product or service. So that has to be a lot different, I guess, when you when you talk to people about research and how you’re doing the same type of research around the same type of product service for a long time.
Steve: Right. And, I’m a consultant myself. I see this with my clients, that they live with some space, they live with some product topics, set of stakeholders, set of users. Some have talked to researchers that meet with the same set of users over years, and they they build these kind of longitudinal relationships. And I’m jealous of them, too. I’m also personally happier where I am, I think I bring value not being in that…there’s a rut, I think that is easy to fall into. And the rut has all these wonderful attributes, like knowing that people and their limitations, like if you have their limitations, their strengths, their preferences, if you’re working with a bunch of different stakeholders, and they have different communication styles, or they have different ways that they engage or different availability, you the in-house person do not have to figure that out every time.
I mean, yes, as the team changes, you’re going to be constantly adapting your styles. And there’s even like a mention of this in the book, I think, because one of the people I quoted describes how they try to have a communication approach that is inclusive, but spread enough to deal with the different needs and expectations of everybody. And so yeah, right, as consultants, we come in, and we don’t have the lay of the land. And I think sometimes that’s an advantage. This may be arrogant sounding, but I think we have the responsibility to speak truth to power, we also have the opportunity, we’re a little less constrained by risk as consultants.
Per: And not as biased maybe.
Steve: Yeah. You know my compensation isn’t tied to the success of the product. Those models create all sorts of interesting incentives. You’re part of an organization, you’re part of a corporation, if that’s the domain that you work in. And your success is tied to success. And as a consultant, it’s not. Yes, the company does well, they hire us back, we want things to do well, but I don’t think we have the same kind of incentive model. So I’m glad there still are consultants, because I think there’s a nice triangulation, or a nice partnership that can happen. I love working with somebody that has the long view inside and they give me the highlights of who and what and I don’t have to do everything my way, but we can negotiate the kinds of approaches and practices. And I can be that unbiased voice just by the fact that I’m not part of it.
James Royal-Lawson: Yeah, exactly. You’ve got that, you know, the fresh eyed approach that you can offer as a consultant coming in. But then we’ve got the opposite edge of that. And you mentioned this in, in chapter 10, Making an Impact, that internal research organizations, they need to keep track of what they know, and what they don’t know. And, what have we already researched? And and that opens up a whole different aspect of historical record keeping, I guess, you could say, which as a consultant, maybe we didn’t have to deal with that. It wasn’t an assignment you were given. And then you delivered.
Steve: I will say and I’ve gotta imagine this has happened to both of you, though, where as the consultant you serve as the offshored institutional memory, where somebody writes you and you haven’t worked with them for a really long time, and they’ll say, Hey, didn’t we do a project about this or that? Do you know what it was? Do you still have the thing that happened to me like, I don’t know, six weeks ago, someone had an anecdotal memory of something and they weren’t involved. And no one was left working there. So, that organization wasn’t doing a great job of documenting whatever different initiatives and so on, and that they were…I could put my finger on it, like I actually had the document quickly. So, yeah, I think there’s an ideal of whatever knowledge management, institutional memory, what have we researched? What are we learned from it? Yeah, I think it’s a hot topic that people are working to try to define.
What I get nervous about is where that problem, which is an organizational one, and an institutional learning one is hoped to be addressed with a software solution. Without asking some larger questions like, what information do we need to save? Like, is it the existence of that report? Is it the report? Is it the person who you know, could talk you through it? Is it the raw data? Is it the decisions that we make? Is it the recommendations? That’s just me riffing on like, what do we want to track? And who’s going to query that information? Is it somebody that is quote a researcher? Or is it someone that wants to ‘hey, do we know anything about x?’ and discover it themselves? So I think there’s just a huge amount of challenges there, like, are you creating different things to archive? Are you archiving them in a way that they’re retrievable by who at what point?
A number of years ago, people in research organizations that were growing, were talking about bringing in somebody who’s a reference librarian, whose job it is, in other contexts, is to be a human being that interfaces between people who need information and storage of information? Yeah. And that’s very different than self-service. So this is like a huge kind of culturally based –any two organizations are going to deal with this differently – what they expect people to be able to do and who there is to do it. There’s a technical need here. But there’s also just a process and understanding of what information, why do we want to look back to what end and building those use cases in? And I think it’s turning out to be trickier, maybe than software vendors had promised us.
Again, that was from the UX podcast. Now onto our episode. I had the pleasure of speaking with Jamika Burge, the head of research for data and AI at Capital One. Jamika, thank you so much for coming on Dollars to Donuts. It’s just so lovely to get to speak with you today.
Jamika Burge: And so good to be here with you, Steve. Thanks for the invitation. I’ve been looking forward to this. Happy to be here.
Steve: I want to start off by asking you how you found user research.
Jamika: That’s a really good question, and I find myself asking myself the same thing, because I don’t think I came into it knowing that it was user research. And as I look back on my experience, you know, I started off as a computer scientist. In fact, even before that, I started as a business major in college. I want to study this area called business communication. I didn’t know what it was, but I knew that I really liked English, and I liked my AP classes, English and biology, in high school. And so it gave me an idea of how I approached learning and problem solving and presentation of what I knew in the world. And so I thought business and communications, okay, I’ll give it a shot. But then I got to college and had a work study assignment my first year there, and it was in the computer science department. So work study is for those of us who needed a little extra money paid to us for working on campus to supplement our education.
And for me, it was in the computer science department. And because I showed up to work every day or every other day, whatever my schedule was, I learned a lot about what technology was. I had never taken a tech course. I’d never programmed. But it for me was an opportunity to learn something new. And so I spent my first year developing my university’s first website. This was like in the mid 90s. And I’m learning about the internet, and I’m learning about what to do to create experiences on the internet, which at the time was web pages, or maybe some Flash, right? Adding a photo or an image online, which were all new things for me and even for us in the field. And so that was my entry into understanding the tech space more broadly, but only in the context of the internet. And after my first year, I changed my major from business communications, which at my college was really business, to computer science. Again, not really knowing more about the field than having spent my first year as a work study student in the computer science lab.
But at the end of my first year, I also had my first internship and was able to come back my second year in college as a full on computer science major. And I spent the rest of my college career as a computer science student, struggling, enjoying, celebrating, rethinking my role in the world, because computer science is not the easiest topic at all, but understood that there was so much to do in the space and so much more that I wanted to do. And even by the time I graduated, I had not learned about human centered experiences or human centered design, or even human computer interaction, which in computer science is the closest thing to user experience and user research. I had, however, learned a little bit more about AI and artificial intelligence.
And so I knew that was interesting and thought maybe I wanted to do more and decided to get my master’s degree and focused on agent based planning and understanding how to create and model agents and experiences using deontic logic, which was, I’m remembering now, the logic of prohibitions and epistemic knowledge, logic, the logic of knowledge. How do we know what we know and what does that mean? So I’m building these models and understanding model based design. And then I learned through an article that I read about human computer interaction, which is not just how to create artificially intelligent systems, but how to ensure that whatever systems we create, intelligent or not, are actually usable. And I was fascinated.
And one of the classes that I took that introduced me to that was called usability engineering, where we could actually test any technology that we created from a user point of view. It wasn’t from a tech point of view. It wasn’t from a performance point of view, but it was from a user point of view. And that excited me. And so that’s how I learned. This is my first year in a master’s program in computer science, where I’m learning about human computer interaction through usability engineering. And that was what started me on my way. I was already on my way towards digging into research more broadly in the computing space.
But here is when I realized, you know what, I could better understand how to create, build, sustain technologies in the context of those who are using them. And those who are using them don’t have to know how to develop them. And so that was my first time learning more about user research from the lens of a computer scientist. And I think I was certainly in graduate school, maybe three years later, when I decided, you know what, I’m going to focus on this thing called human computer interaction. And for a researcher in that space, that absolutely meant that it was user research as an opportunity to really broaden my perspectives about what it means to create experiences for users, technical, non-technical, whoever they were. And that was my first introduction into user research as a graduate student, technically.
Steve: Was there a point at which some of the work you were doing in grad school looked more like user research, like from a 2024 lens?
Jamika: I think so, because in my program, and I went to Virginia Tech, which had, and still has, a strong program in human computer interaction in different areas of the practice, like visualization and human computer interaction models and frameworks. But I think for me, in the truest sense of user research, I learned not just what it was, but how to practice it through my first project in a course that I took, where we were supposed to gather user feedback for a solution that we thought needed to be explored. And I think for this project, I was partnering with one of my colleagues in the program, and we were in a virtual reality class, and we were studying the impact of depth perception using in a cave environment. And a cave environment is, for virtual reality, is sort of this pseudo real experience that feels real because you’re putting on the headgear and you’re in a space that’s projected in front of you, and it’s immersive.
And it was also new for me, but it was an opportunity to investigate, okay, how are people really engaging in this space? And that, for me, was not just an opportunity to understand and see user research, but to embody it. Like, wow, this really is an opportunity to get real feedback from another person in a way that’s meaningful and in a way that helps me and my partner and the whole field understand more or understand something we don’t already know.
And that, for me, was fascinating because what I didn’t realize is that I’ve always been a researcher. Ask my mom. I’ve always asked questions and always asked why. And so, for me, it was a really nice way to see, oh, I know what this is, and I can apply it to something that I am right now in the moment getting trained to do. And it was the first time I think I’d ever really understood the value of loving what you are trained to do. Like, what is a career? Oh, it can be things that are fun and that are really interesting and that are also challenging in ways that I didn’t expect. But yeah, I think that was my first real introduction to user research in a hands-on way through a course that I took.
Steve: And describing it as a lightbulb moment or maybe a series of lightbulb moments sounds very powerful in the way you’re describing it.
Jamika: For sure. And we think about a graduate program and even a PhD as being research. User research is different, and it is much more applied and provides a lot more of what I call real-world perspective in a way that is truly unmatched. And so, for me, that was the appeal. And it led to even my research being a little unorthodox, but it was all about, for my dissertation work, but it was all about, okay, what are real people experiencing and how do we better understand that for the greater good? So yeah, that series of lightbulb moments, I think, really started me on the path that I find myself on even today.
Steve: What was unorthodox about your dissertation work?
Jamika: Well, I struggled. I’ll tell you this first. I struggled with my topic because I was in a computer science department, but my advisor was and still is a psychologist. And I was doing work that was highly integrated with bits of psychology, sociology, and anthropology. And so I struggled with, well, can I do this kind of work in a computer science department? And I was quickly assuaged. My fears were assuaged when my committee said, well, as long as we agree, of course, you can do whatever you need, you know, whatever. And that was all I needed to hear.
And so it took me on a path of really understanding how people share high-stakes emotions across tech media and tech media and channels like email and phone and the internet, IM, for example, and even face-to-face. That’s a medium. And so I brought couples in relationships into the lab and effectively had them argue. That was my culminating academic user research project, if you will. And the context around that was at the time, this was the early to mid-aughts, and there was an influx of technology options to support remote interaction. So we were seeing a lot more IMs pop up. We were seeing instant messaging systems pop up. We were seeing a lot of people engaging in social media before we called it social media, right? There were lots of those activities and even people who were engaging in relationships and even maintaining relationships via email, right? We didn’t really have video at the time the way we do now. And so people had to stay connected, especially people who were away from their loved ones. It was important to do that.
And so I found myself trying to understand, well, even for me in graduate school, I was away from my family and benefiting from understanding ways to get through some pretty heavy experiences and maybe even to resolve high stakes interactions like arguments and even extreme excitement were important. And so it led to my thinking about, well, what are the opportunities for using communications media effectively to support communications for people in relationships? And the value there in relationships is that if people are vested with each other, then we can stress test the tech medium a bit to get at things like, well, how were they able to resolve the conversation or what were the extremes of the conversations that emoted the most or what were some barriers in the conversation that kept them from meeting a resolution or what were some interesting indicators, depending on the medium, for example, where they couldn’t see each other, where one person in the couple behaved a certain way and the other person didn’t get the benefit of seeing that.
Those are some all interesting questions. And while broadly the opportunity of understanding how we use media, tech media to connect was interesting, but the underlying real question was, can we actually get them to argue in a lab? And we were able to do that, but it unlocked a lot of learnings about people, about how we communicate, about what we think about what we communicate to others and the ability to resolve any kind of conflict, especially when it’s not face to face using technology.
Steve: So when couples came into this lab, were they having these interactions mediated through some tech platform?
Jamika: Yeah. So they come in to the lab not knowing at all what they were going to do, which is by design of course, and they’re told what they’re going to do, how they’re going to be spending their time, and they are put into one of three groups. They’re told that they’re either going to be in a face to face condition, an IM, an instant messaging condition, or the phone condition. And if they are in the instant messaging or the phone condition, one person will be moved to another room where they could actually engage with each other separate from each other. And in the face to face condition, they were both in the same room.
Now there’s some pre and post things that are happening. Like before we actually get started, there are course signing informed consent forms that we can talk later about how I learned the value of communicating consent, but also doing no harm, because this work is really important to understanding that when you’re having people come into the lab and argue, we’re sort of creating a level of stress. And I’ll note here, probably appropriately, that we partnered with the University Counseling Center and we gave our participants pointers and paraphernalia that if they needed support as a result of this conversation, or any time in their conversation, the Counseling Center was available for them. So that’s important. But that said, part of the preconditioning and pre activities were I wanted to understand what their current state of mood was.
So I used a PANAS inventory, which is a, wow, I’m really digging back here. It was an inventory that measured people’s mood, their level of happiness. And there were some 50 to 100 questions I had to answer, just so we can get a baseline of how their moods were. So we did that before and after the experiment. And we also did a depression inventory to understand, you know, how are people feeling just in general? Again, what are their baselines? And how might our understanding of their mental state, at least at rest, if you will, change in the course of these conversations, which also helped us to determine, you know, whether or not they were actually arguing and able to get to some level of resolution. So we did that at the beginning, and we did not take depression inventory at the end.
And before we started, we also had people talk about, you know, what are things that you generally argue about? You know, and there’s some common things, we can get into that in a minute. I think you asked me, you know, what was the process of determining what tech media, but all this is before they’re actually separated or assigned to their condition. But once we’ve taken care of all that, they were ready to go. And we’re then separated into the different, according to the tech media that they would be using.
Steve: So you did this research, and then this is for your dissertation. What is your, I mean, it’s a terrible question to ask someone, but like, what is your dissertation conclude? You know, I think I’m sure you’re talking about weekly nowadays.
Jamika: Well, I’ll tell you, I actually talked about this with a colleague a couple of days ago. So it’s not as unusual to talk about it as we might think. And I have some conjectures about why that is. And I’ll come back to that. But there were a lot of things, actually, that were the results of this research. And for those who are a bit more versed and probably closer to this than I am now, know that this research is an area of even media uses and gratifications theory, which means that the technology that we use can help us to deliver news with different kinds of results. And an example is maybe if you’re feeling uncomfortable about having a conversation with somebody, a friend or otherwise, but mostly a friend, you might, depending on how uncomfortable you are with the conversation, choose the communication media based on that feeling. So if I’m telling someone that I, you know, I’m sorry, I’m not going to make the party tonight and I know it’s a big deal for you, but I just I can’t make it tonight, then to keep from being overly uncomfortable, you might not call, but you might text. It’s not the good thing to do, right? It’s just it’s not good, at least socially.
But personally, the pressure of having to tell somebody that you can’t do something that you said you were going to do or that is personally uncomfortable means that folks will likely choose the technology based on what goal they’re trying to accomplish and how it makes them feel in doing that. And so for me, it helped me to see that as one of the results that we found was that people are able to argue, first of all. So they do argue.
But it also because we also let me back up and say this, we recorded them as well in the context of these arguments and that they were able to argue in front of a camera was also very fascinating. And it’s because of the camera in the room that enabled us to learn some other things. And one of the things that we learned is that, you know, people will self-soothe in arguments even when their partners can’t see. And it’s that self-soothing that can aid in the conversation or might give them cause to check out. And as an example, there were a few times where the men in the conversation, particularly in the phone condition, would touch their heads with their foreheads and sort of shake, you know, massage their temples as though they were in pain and trying to massage themselves because they were distressed in the moment. Obviously they’re arguing. But they really weren’t communicating that distress. But they were showing it on video knowingly to some extent because there was one person even in an IM condition when he wrote something back to his wife, he said, she’s not going to like that. And he looked right at the camera. Right? Like those kinds of things were really fascinating. Not that they necessarily needed to be witnessed by the partner, but that they were part of self-soothing for those who were experiencing or emoting in that way. So that was one of the major things.
And the other thing was not only were people able to argue, they were able to resolve their conversation, resolve those arguments irrespective of the medium. The difference was the speed, obviously. People in face-to-face conditions were able to resolve their arguments or even the time they spent in the argument was shorter than those who use different media. And that is not surprising. Obviously it takes more time to type. And certainly the less rich the medium, the more time it takes to communicate certain feelings that may be more obvious by looking at people, obviously. So that was interesting that the medium didn’t really have an outsize effect on people’s ability to resolve the conflict. But seeing how people managed in the moment was really fascinating, especially when they weren’t face-to-face. And I think that for me was really interesting, particularly given the range of participants, the range of experiences they had in their relationships, and that they didn’t know what they would be doing when they came into this experiment.
Steve: So where did you take things after this program?
Jamika: Yeah. And I wondered what was next for me. During my graduate school career, I interned at IBM Research and got a fellowship through IBM and knew, “Okay, I’m going to go to IBM Research and be there.” But it didn’t quite work out, I think in part because while I knew I didn’t really want to take the tenure track, I think I knew I didn’t really want to go into the research industry either. And part of it was I felt that there was so much more for me to learn. I’ve always felt, and I think in many ways that’s why I continued my education, there’s so much more to learn, there’s so much I don’t know. How can I grow my thinking and my skills in ways that make me worthy to do more of this work without realizing that I actually had a good handle on things and was pretty good all along, but what did I know?
And so it was thinking about, “Well, if I’m not going to be a faculty member someplace and if I’m not going to go into a research scientist role, what else is there? I didn’t know.” So I actually decided to do a postdoc and I worked with Jack Carroll at Penn State. At the time, he and Mary Beth Rosson, actually, who was my initial advisor, Mary Beth Rosson and Jack went to Penn State. So I ended up going to Penn State. Again, it was not planned, but I worked with Jack. And in that space, I was working on understanding, now that I knew that computer science could be a part of the human experience in ways that I didn’t realize before I started my graduate program, I thought, “Well, what else can I do?”
And so with Jack, I worked on understanding how nonprofits might leverage wireless technology. And this was in the time where free Wi-Fi or walled gardens in cities was taking off. How might we get access to free Wi-Fi so that people could be connected to the internet and it helps to increase access to technology? And so I spent a year and a half, a couple of years, connecting with nonprofits locally and understanding what their technology needs were in the context of internet and access. And it was a really interesting and even fulfilling body of work because it helped me to understand that while I understand and am trained in technology and all the opportunities that there are, there are so many that don’t have that access. And forget understanding, it’s access. And that for me reminded me, okay, what else might I do in this world that helps us knowers of the technology and of its capacity and its capabilities? How do we bridge the gap between those who know and who have access and those who don’t or who rely on technology in ways that maybe don’t have a choice or for whom we can make those experiences easier? And so for me, it was an opportunity to learn, well, what does giving back look like? And what does doing good look like? And that was my next journey after finishing my PhD.
Steve: I want to pick that up. I also want to go back first though, if I could. It seems like a large contrast to me as you’re describing it between people in labs, one on one, very experimental communication. It seems very like, oh, this is academic research. And to go from that to, I don’t know, social problems and entities and institutions and large social forces. And I don’t know if service design is part of the vocabulary there, but something that seems ethnography-y, the aesthetics, if you will, from the two different major studies that you’ve described, it seems like a really significant transition to hear you describe it. And with that, was it even a transition for you? How did you go from that dissertation world to the postdoc approach?
Jamika: Yeah, that’s a really good question because to me, it felt natural. It felt like an obvious point of transition in my story where I could go from learning about how to do the work to actually doing the work in the field, in practice. And I think that’s what shifted my thinking about the kind of work that was possible for me. And it absolutely is a shift, right, from being highly academic, very much theory-based and driven, and showing our proof of concept, for example, in experiments and questions that we might ask ourselves that help us to get closer to answering our question, the big questions posed in our dissertation, to, okay, now how do we make this real? How do I make this real? And what am I bringing to bear in this work in the community? They don’t care about my dissertation. They don’t care that I did this other work. They don’t even care about the underlying theories. They’re just trying to feed the homeless, or they’re trying to support youth in the community. So for me, it was, okay, now I can go from understanding the academics to applying those learnings in a practical way.
And I think that’s when I realized, you know what, I am much more interested in applied research. How do I practice what I know in real-world experiences? Because that’s, I think, how I can do the most good. Not that I can’t do good in the academic space, or writing a paper, or doing research that changes hearts and minds, right? Whatever that means. I think it was, for me, an opportunity to, if not forsake one, then embrace them both. And that, I think, has been my purview since as well. I’ve never abandoned my academic roots, but I also recognize that practicing the work of research, or even user research, is best when we understand the underlying frameworks and methodologies that help us to get the most, the best, the most useful results, right? So I think they go hand in hand, for sure. So it is a jump, but I try to do both. And it isn’t always easy, but I think it’s important.
Steve: You were starting to bring up that this idea of giving back and doing good came out of the things you learned about access, and that that led you towards the next stages. What was that?
Jamika: Yeah. Well, I love the way you’re setting up the story, Steve, because I’m going back down my own memory lane. And I am appreciating that, at the time, I didn’t like my journey. I thought, I mean, I got my PhD in, I think, ’07, ’08, and it was right around the housing crash. And so many of us who were finishing our programs were looking for work and couldn’t find them, because it was also, in many ways, the crash of the dot-coms. And so all of us were sort of competing for smallish roles, or at least small numbers of roles, not smallish roles, but maybe there were small roles. But there wasn’t a lot of work to go around, at least for those of us who were looking for very specific kinds of research roles. And so in my postdoc, not only was it an opportunity for me to figure out what I wanted to do when I grow up, it also helped me to find another area of research, and even applied research, which wasn’t obvious for me. And that was government research. And so that was what was next for me. I went from my postdoc to doing government contracting work, where I worked at the Office of Naval Intelligence and at DARPA, the research arm for the Pentagon, and did some fascinating research that, in my mind, was the next level, or maybe another opportunity to apply my learnings in some real-world situations to hopefully do some good.
And that was my next journey, going into government research, which completely eluded me my entire educational career. I did not realize, “Wow, I could do this research? I could do this kind of work?” It didn’t occur to me. And so I think for those couple of years that I was at Penn State, it helped me to get in touch with what I really enjoyed, and not be so constricted by what I was expected to do in my career. As a newly-minted PhD, we’re told, “Oh, you should go pursue academic posts and be a faculty member.” But I knew that that really wasn’t what I wanted to do.
And so giving myself the opportunity to explore really helped me to think more about what can I do and what makes sense for me. And so that led me to DARPA, where I did some pretty cool stuff in two different domains. The first was supporting military service personnel who were suffering from traumatic brain injuries or PTSD and developing technologies that could help support them, to developing game-based learning experiences for K through third graders to teach things like calculus and Newtonian physics, but through games.
And for me, that was a nice connection between the human-centered design tech world of supporting the development of technologies that could support our military personnel to the outreach piece, the doing good, to helping our young people see math and science as a career option. And to use games through that was exciting. So that was my next chapter, if you will, of applied research.
Steve: You said that at the time, you didn’t like your journey. It’s hard to reconcile because you’re telling it as a retrospective, but in the way that you’re telling it now, I can’t see what it is that you didn’t like about it at the time.
Jamika: You’re right. There is the benefit of hindsight. At the time, I felt like my path should have been straighter because I’m already jumping at this point. To back up a bit, I got my BS in computer science, went straight to a master’s program. The summer after I graduated from my master’s program, I actually interned at IBM Watson, which is in Westchester County, New York, an IBM lab, and again, studying HCI, whatever that means, right? I don’t even remember the project. But I enjoyed it so much that I considered staying. But I also knew that there would likely be more opportunities for me if I pursued a PhD.
And so instead of staying at IBM Research, I was at Yorktown Heights, for those who might know the area. I decided to actually teach. I taught for two years at Spelman College, an HBCU, an historically Black college or university in Atlanta, Georgia, for women. And so I was there for two years. And the reason I took that stint was I figured, well, if I’m teaching and if I’m close to a university experience, I’m probably going to be more likely to go back to school for myself. And I’m not sure why I thought that, but two years later, that’s exactly where I found myself. I found myself enrolling at Virginia Tech and was there for the next five years. So already my path is a little crooked.
And in many ways, I was behind my peers who decided to keep going to grad school. And so for me, again, in my mind, my experience, we’re always told, go straight through. And there’s a process here. So I’m already breaking the mold. I’m not doing the things that made sense. Finished my PhD for being there five years. And then I’m not getting a job, as many of my friends would say. You don’t have a real job being in grad school, but I beg to differ. But then finished that and then went on to a postdoc, which was in many ways additional education for me.
And so again, a little crooked path, not quite straight. But for me, even looking back, it helped me to really figure out what made sense for me and to be okay that the path was a little broken. Or in my mind, it seemed broken. So I finished my postdoc and then go into government research and thought, okay, well, I could get used to this, but there’s got to be more. And even there, I thought, well, okay, it’s a little crooked still, because I didn’t directly move into working at DARPA when I finished my PhD or my postdoc. I did a little bit of extra contracting work where I was at the Office of Intelligence and some other places. And so that felt a little crooked too.
And again, I didn’t know the industry. I didn’t know that that was what research looked like outside of the academy and outside of the research industry. It just was. And so I had to get my head around understanding that, first of all, it’s my journey and it’s totally fine that it’s not all the way straight. That’s just life and that’s okay. But then also, I’m doing things, a lot of my peers and a lot of the folks whom I called mentors who came before me didn’t do. So I’m kind of creating my own path. And at the time, it doesn’t feel that way. It just feels different. And that’s what I remember. I remember that as part of my journey, but I also look back and appreciate that my role was different because it meant that I was meant to do different things. And that’s what you’re hearing. I’m even reconciling those differences.
And the cool stuff that I can look back on having done, but in the moment I thought, “Okay, well, this is great for now, but what’s going to be next?” I never knew what was going to be next for me from graduates, from being an undergrad. Every point next happened because the stars aligned, not because I planned it. And I think that was part of what I was also feeling in the moment, not really knowing what was next, but not really appreciating what was now.
Steve: It seems like that’s part of how we grow up is there, I think many of us go on these crooked paths and many of us are not told that that’s how it’s supposed to be. And so I totally relate to you feeling like, “Well, I must be doing something wrong because it’s something wrong. What I have exposure to says this is not what to do.” And you can see other people “succeeding.” And at some point, hopefully for all of us, at some point we realize, “Oh, this is actually the way it goes.”
Jamika: That’s right. That’s right. And looking back, I’d have it no other way. But yeah, we’re not told in the moment. Steady as you go, it’ll work out. And this is exactly the way it needs to be. Completely agree.
Steve: So what happened after DARPA?
Jamika: What was great about that work is that not only did I get to participate in some really great work and interesting work and learn more about what DARPA is, high risk, high reward projects that shape our world. Like Siri started off as a DARPA project. The internet started off as a DARPA project. And so these opportunities to impact at a different level were new to me. I knew what the academic route was for even understanding research and experimentation and user experience. And the frameworks that govern those experiences and the knowledge of those experiences. I even knew what the applied world looks like, particularly in the context of social good and community service.
But now I was starting to learn you could actually drive research and research programs in a way that can actually change lives at scale. That’s what I learned in the government work. And that set me on a couple of different trajectories actually, which I think are the same, but it got me to thinking in my mind about dual track opportunities for myself. It was when I decided, you know what? I’m going to keep doing this user research thing. I’m going to keep building and growing my skills and learning more, learning more about the community and all these things, especially through my work, my day job.
Then I also thought, you know what? I think I want to be an entrepreneur too. I want to start my own business. I want to do more of this social good in the context of tech in a way that allows me to grow in different ways. And so that was when I continued to do research. I continued to be part of organizations that enabled me to use those skills. And I also started my own tech nonprofit where I was able to apply my academic research background in doing research, but also developing programs and experiences to develop capacity for tech capacity in particular for black women and girls through my organization called blackcomputeHER. And so there’s blackcomputeHER. That’s a part of my world now, or at least after government work. And then I’m also doing research as part of my day job and negotiating the two because there’s a day job and then there’s the after job work, which is a part of the work of an entrepreneur anyway. And so that was next for me. How might I take my learnings and go to yet another level that enables me to use my skills meaningfully and to explore what else was out there, perhaps in ways that others of my peers or even others who’d gone before me hadn’t done. So again, I’m thinking, okay, I’m going to give it a shot. I don’t know what it’s going to look like, but I’m going to go for it. And that was what was next for me, being an entrepreneurial minded researcher who even now is learning to blend the two in everything that I do. So that was what started for me as, okay, let me figure out what this entrepreneurship thing is and what’s possible.
Steve: I’m struck by the fact that, you know, in describing your choice and your motivation to launch something like blackcomputeHER, and you’ve mentioned a few times being interested in giving back and doing good for the world, but you identified one or one and a half other motivations for you, which I hadn’t really thought about. Like why do people do things that are good for the world? And you said this is also a way to develop your own skills, is to learn and keep growing as a person and as a professional and as a researcher. And then even the entrepreneur aspect of it, which I think maybe straddles between the two. It’s interesting. And maybe it’s, I guess it wasn’t obvious until you said it, and maybe it should be obvious that motivations that we have for phrases like giving back are so loaded in motherhood apple pie type terms, but it’s not incompatible to our own selfish pursuits that we can do things that are good for the world, but also we’re hungry for because of what we’re interested in, in our own development.
Jamika: That’s right. Both things can be true, right? And this was also at the time, and I think it’s less amplified these days, but, you know, it was a big deal eight, 10 years ago to have as a skillset for a major tech company, entrepreneurial thinking, right? What does that mean? How do you bring big ideas to bear for the organizations in which you work? And so that was also on my mind at the time as I’m thinking back and reflecting on what you’re saying here, because I thought, well, I think that way, I’m sure, because you have to think that way from an academic perspective. That’s what professors do. They are entrepreneurs of their own labs, of their own research groups. And it was just a different way of thinking about it.
And while I did an adjunct role for a few years and I was able to write grants and secure funding for some of my nonprofit work, the nonprofit became the vehicle for more, right? More learning, more capacity building for others, more expansion of who I am as an individual, how I think and what I wanted to do and contribute in the world. And that doesn’t have to just be about where I work, right? I’m all the things. And I think that’s exactly a good way of thinking about it, Steve. You know, both things can be true, starting your own business or learning can merge in ways that you didn’t expect. But for me, I’ve learned, I think we’re quite necessary for me to get to wherever I’m going next. And I’m not sure where that is yet, but, you know, it’s all a journey.
Steve: When you started blackcomputeHER, what were some of the things that you were doing?
Jamika: Well, when we started, our mission was to, and still is, to provide professional development and capacity building in tech for Black women and girls. And the reason that matters is because Black women in tech are one of the least represented or Black women are one among the least represented in tech careers. And design isn’t so different if we think about, you know, where user research often falls in our career spaces. So as a person who identifies as a Black woman and having experienced differentiated experiences as a Black woman in tech, I thought, well, what else can we do? And I have two cofounders who are also Black women, and we thought, well, what can we do to, if not change the conversation, we can at least create community?
And we’ve been able to do both. So we’ve created community by starting out with a conference for Black women and girls and allies. I mean, everyone is invited. But we’re very clear that if we want to provide parity in the field, in the professional workforce, then we have to ensure that there’s parity of people who are in the workforce for the skills that they bring to bear. And so how do we start? Well, let’s figure out who the community is. So we started by having yearly conferences where we grew from about 12 as our first initial planning experience to over 200. We’ve had over 200 in our meetings.
And the value of the community is that we get to know each other. We can point to others who are also in the field. And it ensures and reminds us that we’re not alone, you know, in our growth and in our advancement, and in our pursuits of our careers in tech and design. And the other thing that it’s done is helped to change the conversation and change what we know about women, women of color in tech, and Black women in particular.
And my work with blackcomputeHER also led to my being part of the National Academies Committee where we actually focused on that very thing. How do we change the trajectories of women of color in tech, knowing that there isn’t equity, there isn’t parity, but there’s more that we can do to change the conversation. And so that, I think, has been a really great opportunity for me to keep learning, keep pushing, keep changing the conversation, especially through things like intersectionality. How do we acknowledge that there are different experiences for different people, particularly given the different kinds of discriminations that they feel, but also the way that they identify? And how do we bring that level of awareness to the work that we do, to work that I do?
And I can say that now that I’m at Capital One, if I go along that journey of, you know, after DARPA, I came to Capital One and have since, if I can pull the blackcomputeHER face to the Capital One work, I started our intersectional symposium, the first that we’ve ever had, where we talked about our experiences as people coming from different identities and from different dimensions of experience. It all matters. And that actually is a really great way not to separate us, but to actually connect us, because I can have more in common, you know, with a white guy, with you, Steve, than I do with other Black women, because of my experiences, because of my journey, because of my growth opportunities that I’ve had. And that doesn’t make me better, it just makes my experiences different.
Steve: What have you seen people who have participated in the blackcomputeHER events go off and do?
Jamika: One of our staple programs that we have through blackcomputeHER is our Fellows Program. And our Fellows Program is different from a traditional fellowship. We don’t provide funding, but we do provide an experience. We provide with a cohort of those who apply. This is an application experience, so folks have to apply. We invite a cohort to participate over the course of the year, where they participate in webinars that feature subject matter experts across a range of tech areas, even executive coaching expertise and executives in the field. So we provide access to experts in a way that our fellows who are early to mid-career professionals don’t usually get. So we provide that level of exposure.
And I’ll say that as an example of a good news story, and there are a few of them, but a good news story that I have is that one of our first fellows from our first cohort — we’re moving into our seventh or eighth year now, so we’ve been doing this for a while. And one of our fellows who joined our program was in transition. She didn’t really have a role in tech, but was interested in starting, but had done very well in a STEM career. Maybe it was chemistry or something. I don’t remember what it was. But she was interested in moving into a tech career and knew that to do that, she believed that she needed to go back to graduate school. And so she participated in our program for a year, was exposed not just to our mentors, our experts, but also to other fellows in the program. But a couple years later, she sent me a note saying, hey, Jamika, I just wanted to share with you that I have applied to and been accepted into MIT’s program where I get to study more about tech and design experiences, which was really exciting to hear.
And that’s exactly what we want to be able to do, to provide a space for women to know that there are more opportunities out there. And if we can provide an extra rung in that ladder, another step up to help them see and find those opportunities, we want to do that. Now she got into MIT on her own. I’m not going to take credit for that. But I do think there’s value as she’s communicated to us and she’s come back and participated in other programs, is that there’s nothing like community and exposure and connection, especially when you’re one of a very small number of folks in a space. It can feel like you’re the only one ever. But that I think is a mistake for any of us because it can keep us from doing, being, offering more. And that for me is exciting.
We’ve got other experiences and stories like that, but I think that’s one of my favorite. It hearkens back to me, our discussion about being uncertain on our paths because we’ve only been told this is what a path should look like.
Steve: And that I can imagine for this woman that you validated a path or she took from that fellowship experience the confidence to do what she was already capable of.
Jamika: I hope so. You know, the research tells us, especially in broadening participation and in areas where we’re trying to get even more women in the field, role models and cohorts matter. So again, that’s part of that academic training of knowing that there are frameworks that I’ve applied before and other training programs that I’ve created for young people who participated in. This is what helps us to grow community and capacity. And so applying that in this context is again, a nice combination of what do I know in this space and then how can I apply it to create the change that I think we need in the world. And so it’s been a really great opportunity to grow, to fail, to have really great experiences, a celebration, which I think is what entrepreneurship is, right? Like, would you agree? Like that’s part of the gig. And so it’s been fun. It’s been a journey, but I hope to keep learning.
Steve: I hear both, you know, trying to support individuals, but also exploring how you can kind of change the systemic aspects. I have a naive view of that. So my questions are going to be naive, but it’s clear how you help these individuals and your example of the symposium at Capital One seems like that’s about the system. And I guess, is this true, the more Black women go into tech, the system does change based on who’s in it. And that I think naturally would change the opportunities for black women in the future. Is that how this works?
Jamika: Yeah, I think so. I mean, I’m one of those people who, I mean, systemic change is hard and it takes a long time. And so I don’t profess that any one thing I do or even any number of things that I do can change the system at Capital One or otherwise, actually. I actually appreciate that we were able to have a conversation around intersectionality at a tech company that happens to be a bank and is also very highly regulated. That hadn’t happened before. And so in my mind, yeah, it was a way to talk about some real experiences for people who needed to be heard. And it wasn’t just Black women, right? I mean, all of us have experiences and all of us have stories. All of us have identities that make us who we are. Race and gender are the obvious choices, but they don’t stop there. Where did we grow up? What’s our educational background? How many languages do we speak? Are we living in this country, but maybe coming from another country, right? How many degrees do our parents have, right? There are all these dimensions of who we are, including how freely and able we are able to move in the world. Like that’s a level of identity that we often don’t think about. But all of these dimensions and all of the dimensions are very rarely held by one person. There are so many parts of who we are that makes that conversation around intersectionality so meaningful for everybody. It is not an exclusive term. It’s actually an inclusive term.
And I appreciate that we’ve been able to have that conversation at Capital One, which I hope begins to empower others or help others feel empowered to have similar conversations. And I’d be very curious to know if any other organization is having that level of conversation because I haven’t heard it. I’ve been invited to lead those conversations in other organizations, in university as well, but it’s also not an easy conversation. So I think what my approach is in that recognizing that there is an opportunity for systemic change is to know that the system can be fixed, but that isn’t necessarily my job.
I want to make sure that for those of us who are in the system, that we feel supported, that we have what we need to succeed and persist. And that when I have an opportunity as Jamika to lend my own experiences and expertise in an effort like through the National Academies of Science and Engineering, which is a pretty big organization, then I can also share, well, here’s what else we can do. Here’s what we’ve learned from blackcomputeHER, for example, that really helps our community members persist. Or here’s what I’m learning that we’ve discussed in an organization that is not all Black women, but we see the value of hearing our voices. I think finding ways to, again, be myself in both worlds is what I strive to do because again, I’m more than the work that I do. I’m Jamika. I happen to be a lot of different things and do a lot of different things and have a lot of different hats that I wear.
So I would love to be able to, I would love to hear other people talk about themselves in that way too, which I think goes back to the user experience. Who are we? That’s really what I think my role as a user researcher is. How do we help people to not just tell us who they are, but to be comfortable being who they are wherever they are? That for me is pretty cool. There’s context in the middle of all that, but at the end of the day, I want to know the people that I’m serving. There are lots of different ways to support that and to support it meaningfully.
Steve: I guess to say yes and to that, yes, we can do a great job at helping people to tell us who they are, but also sharing that information in a way so that people who are otherwise reduced to artifacts on a wiki somewhere or marketing documents or whatever, trace logs through a tool are more fully presented or richly presented and in a way that looks more like how they’d want to be presented.
Jamika: That’s right. That’s right. More of that, please. Yeah. I think that’s always easy or obvious, right? Sometimes even in recruiting participants or thinking about who might give us the feedback. It’s as simple as ensuring that on the one hand, our participant pool is diverse, but how are we ensuring that even for those who might be considered the mainstream audience, that we’re not pigeonholing them in their experiences either? How are we soliciting the kind of feedback that truly is inclusive, not just through a diverse pool of participants? I think again, it’s a different way of thinking about experience and enabling that extra layer of support for the people that we serve.
Steve: Since we’re talking about Capital One a little bit, could you just maybe give a little context to what you work on and say a little more about Capital One and your work there?
Jamika: Yeah. I am head of research for data and AI, and that really means that our researchers on my team and I are responsible for supporting data rich and data inspired products and experiences. And we’re supporting and beginning to support AI and machine learning based experiences as well for our customers.
Steve: And is that part of a business area or a product area at Capital One?
Jamika: Yeah. In particular, Capital One is a highly regulated banking tech company, has lines of business that it supports, card and bank and such, but it also has enterprise level supports and the enterprise supports really enables all functions of the company, including card and bank. And so my role is in the enterprise area of work where I in particular am really interested in ways to scale our efforts and to apply our learnings in ways that not only help our customers externally, but supports our customers internally, which can be associates, right? Or me or others in the organization. So that’s kind of where I’m situated in the kinds of problems that I’m solving.
Steve: When you think about not just Capital One, but the profession that we’re in writ large, and I know you have lots of exposure to researchers from all sorts of different organizations through your own career. In this world of, I’m going to call it like modern day corporate user research, what do you think doing good looks like in our field right now?
Jamika: Doing good. I acknowledge it probably means different things to different people. For me, as a researcher, and as someone who wants to do good in the world, like I just do, it means understanding people’s needs in context and providing opportunities for them to succeed. That’s what that means for me. Success can mean different things to different people. I can guess what success means from a business perspective. I can even guess what success means from a researcher perspective, but ultimately it’s that end user who tells us whether or not we got it right. I want that person to feel as a user, an end user, free to share with us when we got it wrong, but also when we got it right.
Building the right kinds of relationships and the right kinds of processes that support that are really important. Relationships between whom? Well, I think there are lots in the process, but certainly the relationships with our end user. Again, who is our customer? Who are we serving? That requires a level of engagement and relationship that I think allows us to really deeply understand their needs and to support their successes, whatever that looks like. What are they trying to do? What activities or tasks or jobs are they trying to get done? That’s the context that defines what success looks like. I’m very interested, and I think doing good means supporting those tasks in context in a way that helps them feel that they can determine what success is and attain that success in context.
Steve: This is going to be a slight non sequitur. I just want to go way back to something you said early on. You just were talking about yourself as a child. I think you said you were always a researcher and that you always ask questions, but I wonder is there more or do you have a childhood story about a canonical kind of Jamika thing that was like you as a researcher?
Jamika: I know I always ask why, which isn’t so unusual for kids. I remember being insatiable in my questioning. In a way where I remember, and maybe it’s just my mom getting exhausted, but I remember I’d asked a question. I took dance as a kid. Before you really learn proper form, you do a lot of kicks and twirls. I was a little girl. That’s what I did. I remember once I was asked to be part of a choreography for an organization or church or something. I don’t know what it was. I had asked my mom who was helping to choreograph. She wasn’t a dancer per se, but she supported me and my siblings and was always there. I remember thinking, “Mom, what if we do this and this and this? Can we do this and this? Can we kick like this? Can we kick like this?” I do remember her saying to me, “No, there’s too many kicks.” I remember being really pensive and thinking, “Oh.” Too many kicks is obviously a thing. I didn’t realize that as a seven-year-old. But in real world, as someone who’s much older now, and even after learning more about the craft of dance, you don’t want too many kicks unless you’re a Rockette. That’s context. It matters.
But in that moment, I think it was also my mom’s way of saying, “You know what? No more questions. It’s too many kicks.” But she said it so thoughtfully and so calmly that it’s just what I needed. It was the response that got me thinking more. Like, “Oh, that’s why. Too many. Okay.” I think that’s just how I see the world. I need to be able to grapple on my own terms. Even if the person or the people that I’m asking don’t know the answer, some kind of response to get me noodling and to get me on my own path is helpful. I think that’s what drives my thinking about user research too, and about understanding the big questions. That’s what drives me. I don’t know if that really answers your question, but that’s one of my memories. We gather data, and then we have to make sense of that data. Make sense of it.
Steve: There’s an element of that story too that’s about there are rules about the path that are given to us or not given to us. You’re describing a journey of learning what those rules are and then also choosing to interpret them to follow your own heart at different points along the way.
Jamika: That’s right. The guidance is fine and useful, and for me as a structured thinker too, sometimes necessary, but I also have learned and appreciate the grappling that I have to bring to bear. You’re right. Creating my own destination from what I’ve got and knowing that it’s okay, whatever that looks like, and to be okay with the grappling, especially, which can be hard, is also okay.
Steve: Are there ways that people who are listening can support blackcomputeHER?
Jamika: Oh, sure. We’re a non-profit, 501(c)(3), so donations are always welcome. I’d love for folks to just see what we’re doing and to see that we’re not creating a separate community that competes. We’re actually creating a community that can help inform and transform tech and design in ways that otherwise would be pretty empty and not reach what I think are areas of necessity given where tech is going now, given where we have the opportunity to make the greatest impact, areas like AI and areas like advanced learning through technology, whatever the opportunity is, I think it’s richer by the most diverse problem solvers on the job. I think that’s where the opportunity for understanding more about our organization and, frankly, the way we can all work together to do more good together lies.
Steve: I will put a link in the show notes to the organization so people can check that out. Jamika, I really enjoyed and learned a lot from talking with you today. As I always do, thanks for sharing your story and reflecting on it in such a nice way today.
Jamika: Thank you, Steve. It’s been so much fun. Thanks for having me.
Steve: There you go. What do you know? That’s the show. Find Dollars to Donuts wherever you get podcasts and at portigal.com/podcast for all of the episodes with show notes and transcripts. I would really, really, really, really, really be grateful if you would rate and review Dollars to Donuts on Apple Podcasts. Our theme music is by Bruce Todd.
The post 48. Jamika Burge of Capital One first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Akshay Verma, the head of User Research at Duolingo. We talk being qualitative focused in an experimentation-driven organization, research team structures, team rituals, and sharing knowledge between researchers.
I don’t actually want to bemoan or belabor this concept of a room that we’re invited to or not. At Duolingo, I feel it pretty acutely just because we do have a lot of rituals and traditions at Duolingo around how product gets built. And it’s great. It works. It works really well. But, you know, I could spend a lot of time and energy going crazy, being like, “How do I get invited to these rooms?” and then get upset when it doesn’t happen? I actually don’t. I try my best, but I think our energy is probably spent elsewhere.” – Akshay Verma
Show Links* Akshay on LinkedIn * Duolingo * How Duolingo is using its ‘unhinged content’ with Duo the Owl to make people laugh on TikTok * FigJam * IDEO * Stanford d.school * Calendly * How the Underground Dance Music Scene Makes Me a Better User Researcher * Get Familiar With Detroit Techno: 10 Essential Songs
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal.
The other day I debuted a new client workshop on “Boosting User Research Impact.” We did this remotely, with me being in California and the enthusiastic participants in Europe who stayed late. One person used the excellent simile “It’s like I’m being asked to solve global warming” – in this case they were referring to the kinds of research request they get, but it highlighted for me how the team did such a great job at not spiraling into higher-level, structural, and less-easily solved challenges around having impact on stakeholders, but stayed within the framework I gave them, which was to consider specific, practical steps they could take before research, during research, and after research. There’s that cliché that asks, how do you eat an elephant? And the answer is one bite a time. These kind of frameworks turn an elephantine problem like having impact with user research into something more manageable. For this team, it worked very well to dedicate time and space for collaborating creatively in order to change the approach. We also looked at what’s already going well, and how to extend or expand that. In just a couple of hours they generated a number of new changes that they can prioritize and build from ideas into actual programs and practices.
I’d love to talk with you about the challenges your team is facing and how I can help.
Okay, let’s get to today’s episode with Akshay Verma. He’s the Head of User Experience Research at Duolingo.
Steve: Akshay, thanks so much for coming on Dollars to Donuts.
Akshay Verma: Thank you for having me, Steve.
Steve: I really appreciate getting to speak with you.
Akhay: It’s a total joy to be here.
Steve: So you are the head of user research at Duolingo, is that right?
Akhay: That is correct. Everyone’s favorite green owl and unhinged TikTok presence. Yes, that is true.
Steve: I want to know about those symbols a little bit too, but what is Duolingo?
Akhay: Duolingo is the world’s most popular language learning app.
Akhay: We recently expanded beyond language as well. We’re rebranding ourselves as a learning app. It’s also a bit of a video game. I think we really lean into the fact that it’s a really gamified experience. It’s a really fun, gamified way to learn a language or now you can actually learn math and music on Duolingo as well.
Steve: And the owl. I think I know about the owl, but explain the owl.
Akhay: The owl has sort of had a life of its own in the last couple of years and really made a wave on the internet. But yes, our icon, our mascot is a green owl. And somewhere about two to three years ago, our lovely social media manager, Zaria, started experimenting with TikTok and just had the owl take on a life of its own with a lot of really funny, really unhinged TikToks. And so now the owl is kind of like a celebrity. And actually there’s videos of the owl sometimes. They’ll be recording a TikTok in Madison Square Park or whatever, somewhere in New York. And it’s like a celebrity. Like people will come up to Duolingo and take photos with him. So he’s become a cultural icon, but he is sort of the shepherd. He shepherds language learning and learning through the app.
Steve: And his name is Duo.
Akhay: His name is Duo, that’s correct.
Steve: What does user research look like at Duolingo right now?
Akhay: Actually, we’ve been in a period of flux and change over the last year. I think a really exciting one, but a period of flux nonetheless. So user research right now at Duolingo looks like this. And I’ll tell you a bit about what it used to look like about a year ago to also provide a point of contrast. So we actually, about a year ago, moved from being in our product org to being in our design org. And I’ll actually contextualize this a bit further. Duolingo is a pretty small company. We’re about, I would say, 700, 750 employees total right now, which is fairly small. I used to work at Spotify, which I believe the time I was there was closer to 7,000 to 8,000 for comparison. And our product and design teams are around, let’s say 150 people total. And there’s eight people on the research team right now. So actually fairly good ratio, all things considered. But user research at Duolingo right now looks like we’re part of the design org. We’re still settling in, in a way. Change takes time. And so it’s been a year, but we’re still settling into being in the design org. We’re a fairly qualitative team, slowly moving towards being mixed methods. We do a lot of longitudinal research. We exist in a sea of experimentation. And so we have to think a lot about how do we coexist in a company that has a lot of A/B tests. And yeah, we’re just a really lovely, lovely team of people who are passionate about languages and about learning and around gamification. And we each kind of have our specialty and flavor that we bring to the team.
Steve: You said that your team shifted from reporting to product to reporting to design. What led to that?
Akhay: Yeah, a few things led to that. We had a bit of internal restructuring with certain individuals and people leaving and people joining. And it was a really good moment for us to reflect on where should research sit. And so part of it was just riding the wave of other organizational restructuring that was happening. And then this question around the industry standard alignment that research typically has is with design and the type of impact the team was having at least about a year ago and leading up to that moment was probably closer aligned with design. And so we kind of took that moment to try that out. And so I think it was partly just riding the wave of other moves and reorgs that were happening and then trying to think about industry standard wise, where should we be? I have a lot of thoughts on reporting to product versus design just at large. I’ve done both before. There’s pros and cons to both. I’ve reported to an insights function before that’s multiple insights disciplines, and that’s worked really well too. So I have some thoughts, but I think at Duolingo, it works really well for us to be in the design org.
Steve: You teased us with you have thoughts. I guess I want to hear what those are.
Akhay: Yeah, let me give you some context about the teams that I’ve been a part of over the last eight to 10 years. So I used to be on the lovely research team at LinkedIn where we reported into design and design actually reported into product. And so I’ll come back to that in a second, but I sort of grew up, so to speak, in the heyday of our field in that environment of being part of design and then sort of by extension being part of product. I then moved over to Spotify, which I remember back in like 2016, 2017, it was really pioneering in tech where research could fit in an organization. It was one of the first companies that brought together data science and user research, and then eventually other functions like analytics, engineering, and other related insights disciplines into one insights org. They’ve iterated on that model a little bit, but I was at Spotify in that heyday of my teammates were researchers and data scientists, and I would leave work streams and teams across both disciplines. I then moved over to this company called Gong.io, which is this sales product for, it’s an AI product for salespeople. And there I actually reported to our head of product. And so I was very, very closely intertwined with the product org.
And now at Duolingo, as I mentioned, I joined when we were part of the product org and we have since moved over to design. So I’ve sort of seen a bit of the design thing, the product thing, the insights thing. And yeah, you mentioned, not to generalize, but there’s definitely trade-offs in each one of those. And I think it is, not to give the researcher, it depends answer, but it does. It depends on the context of an org and it depends on a few things. It depends on what is the product culture? Is it a very experimentation driven company like Duolingo or Spotify, or maybe not so much like LinkedIn. Again, LinkedIn had a ton of experimentation as well, but I at least worked in parts of the company that were a lot more like strategic and forward thinking and new product building. So I think that really impacted the role that research could play. It also depends on a lot of organizational rituals, right? Like at Duolingo, we have a lot of rituals, like we have product reviews that certain people attend. We have different permutations of leads meetings. There’s a lot of stuff that’s organizational specific.
And so as a result, I think it just, it depends where you are, but generally if I were to generalize, I love being part of product. I actually love being part of product because I think it gives us a closer through line to actually make product impact. And I always say some of the best researchers are product managers who are secretly researchers. And so I think being part of product really helps with that. I felt that definitely at Gong, like I was in product leadership meetings. I was working really closely with my product managers and those were my key stakeholders. But I also like being part of design because if a company like, and Duolingo is a good example of this, if a company really wants us to make very visible, tangible impact on how a product looks and feels and functions and how features are designed, being part of design really, really helps with that. And so I guess this is my jumbled way of saying it depends and pros and cons to both, but I personally love being part of product. And then I didn’t mention the insights disciplines. I loved that long-term. I don’t think that works as well long-term. I don’t think being your own, own discipline that doesn’t ladder up to product or design is actually that effective.
Steve: Why is that?
Akhay: You know, I found that at Spotify, it was interesting because we had so many insights professionals. We had like 150 researchers, probably 400, 500 data scientists, which is crazy. We just had so much visibility, which was really interesting. But I found that since we didn’t report into product and we didn’t report into design, we didn’t have the same level of influence because our set of stakeholders were each other, which was great. But then I think we got lost in some of those rooms. We got lost in those leadership meetings with the rest of your discipline because the rest of our disciplines was just other insights folks. And so I think that was a big downside. I’m trying now to place myself many years back and think about the things I used to feel, but the lingering, I think impression I left Spotify with was again, love Spotify, a huge place in my heart for Spotify. I just think it was tricky to have impact as research, as user researchers specifically with being so disconnected from the other functions.
Steve: How does reporting structure shape the way you’re able to influence people?
Akhay: You know, part of it is very literal. Like I said, even just things like the meetings you’re a part of week over week, which permutations of team meetings or leads meetings or leadership forums are you invited to? Part of it is literally just that. It’s like if you report into X discipline, you will go to X disciplines set of meetings. And so I think that really shapes influence being part of a company, especially bigger companies that have lots of people where a lot of our job is influence and a lot of our job is fuzzy and it’s the people you meet and it’s the people you can have impact with. It’s truly just like, which rooms are you a part of and which rooms are you not? And frankly, reporting structures as silly as it sounds, like has a huge role in that. I think the other thing that’s a little bit more amorphous is the incentive structure that comes with it, right? The incentive structure for someone who’s in, and I’m no longer explaining to someone who’s three by using phrases like incentive structure, but you know, what are you incentivized for by being in a product org? It is shipping great product, making sure it moves metrics and continuing to do that, right? What are you incentivized for if you’re in a design org? Probably something similar, but from a different lens.
And so I think that really impacts our influences research as well. Like ultimately who is deciding what success is, it is the person who’s leading the org you’re in and whether that’s product or design, that’s going to impact that. So I think those are the two things that come to mind, one being very literal and one being a little bit more around what are you ultimately incentivized for? What is research as a discipline incentivized for or with rather? And yeah, what does our success look like as individuals and as teams as a result? Yeah. I mean, I think it’s possible, but again, it’s just, it’s so dependent on the person and it’s so dependent on the rituals that different teams have. And so I do think that’s possible. And I think that’s the classic conversation around having a seat at the table or being invited to the table. The things we often hear about is almost like tropes in our field, like researchers invited and we have a seat or we’re not invited. How do we get a seat at the table? And sometimes it’s literally that it’s like, which org do you report into? And as a result, are you invited or not? Of course, there’s always room for change. Of course, there’s always room for influence, but you’d be surprised you put people together to accomplish certain goals. And we exist in the structures that we are a part of in a company. And that typically ends up being the function that we ultimately represent or report up to or whatever it may be.
So yes and no, I do think there’s opportunity. And I’ve seen great examples of that at Duolingo, for example, because we have a bit more of a blank slate to work with, to be honest, because the research team is quite nascent. And so I’ve been able to experiment with like inviting us to certain again, rooms or meetings or whatever we want to call them that we haven’t been part of before to varying degrees of success. It’s not always been a slam dunk, but that’s not always the case, especially as you get more complex and get to the sizes of a LinkedIn or Spotify, which are like tens of thousands of people. And there’s so much chaos and complexity as a result.
Steve: In a company at the size of Duolingo, when someone like you invites themselves to different kinds of meetings, like you said, varying degrees of success, what does a slam dunk look like and what does a failure look like?
Akhay: I’ll share a recent example with someone from my team. One of the researchers on my team, it’s an area, you know, at Duolingo, we have three main areas similar to most companies, right? We have a learning area that thinks about all the learning content of Duolingo. So all the actual learning related material that the app offers. We have a growth team, which is pretty structured, which is similar to a growth team that you might imagine, which owns all the gamification mechanics, our DAUs, our daily active users, our top line growth metrics that we look at as a company. And then we have the monetization area, which thinks about our subscription. So the ways that Duolingo makes money. So everything from the purchase flow to what value to users we offer from our subscription, et cetera. So that was just a bit of context, but one of the researchers in my learning area, and this is a slam dunk example of, you know, over the years has built great relationships with leads. We typically haven’t been involved in, again, certain permutations of leads meetings, just because research historically has just never been invited.
And recently he mentioned, you know, he was part of a conversation with that group and they were talking about certain product ideas. And the team was pretty amenable to the ideas until said researcher was like, Hey, actually we have like years of data and research around this. And my perspective and intuition is this maybe isn’t the best idea. And we should maybe think of this in a slightly different way. And the team completely agreed and decided, actually, let’s pause on this new thing that we were exploring. And that to me was a slam dunk of like, Hey, we’re not typically in this room. We kind of are testing the waters and inviting ourselves in. And as a result, we can shut things down before. And I don’t even want to use the phrase shut things down because it’s more like, Hey, we can actually like redirect the team before because we have that knowledge and we have an archive of knowledge we have built over the years as a research team. That to me is a great example of like, right, incentive structures and, you know, how do we play a role in these situations? And then let’s think about like, what does, and I’m blanking a bit on your original question now.
Steve: You know, you said it could go well or it could go not well. There’s degrees of success.
Akhay: Yeah.
Steve: And so I was curious what going well looks like and maybe what going not well looks like.
Akhay: Yeah, I think examples of not going well. I think a lot of researchers will be able to relate to this, but this is for better or for worse, something we experienced quite often. Duolingo is a very experimentation driven company. Like that’s the bread and butter of Duolingo is we do a ton of experiments. And oftentimes we are involved in maybe the early parts of planning and experiment that we might run, which can sometimes be pretty laborious. That could take weeks and months of prep and experiment hygiene and analysis. And there’s been so many times me or my team have been parts of those conversations and said, this is not a great idea. And we do it anyway. And then a couple of months happen and what we kind of expected happens.
And I’m not here to like, I’m going to make sure that never happens because that’s impossible. Sometimes you have to let people do what they’re going to do. But that’s a sheer example of like, we are not involved early enough or we’re not involved period. And as a result, we will catch wind of something that we spent two, three months doing. And then a researcher on my team in a one-on-one will be like, had they shown me this, I would have told them exactly this three months ago. And that’s the downside of the, had they shown me this, right, is things just fall through the cracks if you’re not parts of those conversations. And yeah, so I think that’s an example that comes to mind of something I hear a lot. And I know a lot of researchers will be able to commiserate feeling like we’re not involved enough or early enough or in the right, with the right depth or at the right point until it’s too late. Or we never hear of things. And then we go, yeah, we can point to like four different studies we’ve done that would have probably like shaped how you’re thinking about this had you told us.
Steve: I was reflecting back a couple things that you’re saying. One is being in the room, is having that information…
Akhay: Exactly.
Steve: …is being aware of it. That seems like that’s the base. And then from that, you need to be able to, for lack of a better way of putting it, be listened to. And so in your slam dunk example…
Akhay: Exactly.
Steve: …that was a new sort of room to be in. That was new. But I think you’re also saying that the researcher in question had built relationships. Even if they hadn’t been in that room, they had relationships.
Akhay: That’s exactly right.
Steve: Coming to that room is not brand new. Everybody, they knew each other. And then when they made a recommendation or a suggestion, that was taken up.
Akhay: And you know, it’s interesting, just a meta reflection on this. I don’t actually want to bemoan or belabor this concept of a room that we’re invited to or not. I think all of this is very true. And actually at Duolingo, I feel it pretty acutely just because we do have a lot of rituals and traditions at Duolingo around how product gets built. And it’s great. It works. It works really well. But, you know, I could spend a lot of time and energy like going crazy, being like, how do I get invited to these rooms and then get upset when it doesn’t happen? I actually don’t. I try my best, but I think our energy is probably spent elsewhere.
You know, and again, this is sort of an existential reflection on it, but I don’t know if necessarily this metaphorical room is something that I think is like the biggest issue right now and for my team or anything, but I do think it’s an important one and it connects back to our conversation around reporting structures and all of the like company context that ultimately impacts these metaphorical rooms that we are a part of or not part of. But more to say, it’s funny talking about it because I’ve intentionally reflected to myself about how I don’t want to belabor this, especially as a leader of a team, because that’s a recipe for like a constant downward spiral. If you know, we as a team or me as the leader of this team is thinking a lot about just this idea of these rooms and the rooms we’re not a part of. Of course, I do think about it, but I catch myself being like, okay, no, no, no. How do I make best with what I have and slowly in part change over time?
Steve: And then what does that have to do with the rooms that you are in?
Akhay: Exactly and like over time, how do we focus our energy on that? And part of it, again, going back to a reporting structure, right? If we reported into product, you know, I would be in rooms that I’m not in now. That would be great. I would love that. But the downside is I wouldn’t be in a lot of rooms I’m in now being, you know, a design leader. So, you know, trade-offs there, but it’s a very literal thing that impacts the rooms you’re in or not.
Steve: And I think you’re saying that it’s healthier to focus on what relationships and what rooms. You and I are never going to say rooms again after this conversation. We’re going to be so sick of that word. The teams and the meetings where you are there…
Akhay: Yeah.
Steve: …and you’re trying to add value, it’s healthier to focus on what you do have access now to…
Akhay: Yeah. Yeah.
Steve: …and where you are working versus, gee, where you could be working.
Akhay: Yeah, exactly. And I, you know, I say that to a lot of people on my team or just a lot of other researchers a lot too, because we are a fuzzy discipline. We are no matter how much, you know, I think it’s important to like continually have the impact conversation and to be evolving our baseline of what good impact looks like and be able to measure it, you know, as tangibly as we can, but ultimately we are a fuzzy discipline. And so it’s really easy to get stuck in these spirals about, you know, what rooms we’re going to be a part of and what rooms we’re not. And yeah, you’re right. I’m never going to say rooms again after this conversation.
Steve: Sorry, everybody.
Akhay: But I always tell people, no, we got to, we have to make the best of what we have, but also recognize that like, yeah, we can still push for change. And that’s where I come in and that’s where, you know, we all play a role, but that’s where, you know, I do come in. But my method is slow and steady and seeing over time what works and what doesn’t and not letting myself burn out over the context of the company that I exist in. Right. Yeah. Think about it a lot.
Steve: You said a couple of times that, you know, you’re a qualitative team and an experimentation-oriented company. And even in the last things that you were just talking about, you said, “This is fuzzy work, but we have to measure impact.” How do you think about what impact looks like for the fuzzy work that you do in a company whose culture and thought process is oriented towards experimentation, A/B testing, and so on?
Akhay: It’s probably the number one thing I think about. And it’s true. I think one of the joys of Duolingo is that it is very experimentation driven. And again, I say joy because to me it is a good thing. I could probably be like, oh, my team, it’s so hard to be a researcher because we’re so experiment driven at Duolingo. But I don’t think that’s the case. I actually think there’s a lot of work we don’t have to do because we’re going to do the experiment anyway. And I love that because that frees up my team to work on things that aren’t so focused on experimentation. However, the other side of the coin there is it’s hard to show impact, right? If the impact is like, okay, what experiment did your research specifically lead to and what were the metrics that it moved?
A lot of my job has been actually changing our mindset, starting with my team, then moving beyond that to our design org and ultimately to our product team and ultimately to the company changing our mindset that, yes, that’s one form of research impact, but that is by far and away not the biggest form of impact. And so how I think about impact is really three things right now. I think the first is how can we add some context here? So we recently actually restructured the team a little bit where we have three key horizontal audience level work streams that map onto Duolingo’s three-year company strategies. So I sort of did this intentionally about three, four months ago where I was like, I think we just need something much more horizontal that ties very directly to our company goals where I worked directly with our exec team, shopped it around with other function leaders and made sure everyone was A, on board and B, felt excited about this. So as a result of that, the impact I see there is really long-term. It’s like, how do we actually shape our company’s thinking around these key goals by studying the audiences that they pertain to? And without talking specifically about what these goals are, a lot of them are audience focused. There are specific segments of our audience base and essentially not so much always just around growth, but around really creating value for that segment.
And so that to me was like, that’s what I need to do because guess what my team’s really good at? Studying audiences. So as a result of that, that’s the first and the most amorphous form of impact, the one that’s going to take the most time, but the one I’m being really intentional about quarter over quarter, making sure, Hey, like we have these three-year markers, five-year markers for the company. I want to be part of that conversation over the next three years. And it’s not all that my team is doing, but a good chunk of my researchers are now working on these work streams that align to those goals and the work streams are audience understanding. So that’s the first level of impact. I would say the second level of impact is beyond that. I still have at least one researcher working on each of those areas that I had mentioned, you know, the three big areas. And then we actually have some other areas of Duolingo as well, making sure that we have at least a person supporting the month over month, quarter over quarter needs that are very specific to these sub teams. And so that form of impact is still pretty like strategic and it’s still pretty high level, but it’s about like shaping the features that the team might be working on, iterating on those features.
Like an example for growth might be thinking about our gamification mechanics, short, medium, long-term, what does that journey look like? What else do we experiment with? How do we actually implement it in our monetization team? Thinking about, you know, we are thinking about our subscription offerings and the tiers of our subscription. So making sure that we’re helping the team in those conversations at the area level. So that to me is the second level of impact is like at the area levels, making sure we’re impacting how we’re thinking about feature development. And then frankly, the third, maybe the third and sneaky fourth is that what I mentioned, the experiments and the very tactical things. And that’s intentionally the third thing and not the first thing, because I still want that to be part of our portfolio. I still every quarter want to be able to say, you know, we impacted these X number of experiments. Either because our research directly led to them, or, you know, we played a role in them in some way. And of these, these are some metric wins that we had. But I want that to be like a pretty decently small size of our portfolio, because I don’t want to over-index on that, because that’s also a bit of a losing game I find is if you get too bogged down as a research team on like, which metrics did you move? That’s the most obvious way to show your impact, right? But I think it’s the hardest as well.
And that’s where you get in the game of like working backwards and some shoddy calculations around like, well, technically I was a part of this conversation. And therefore I moved this metric. And that’s where I encourage more of like, hey, this is a team sport, like we should be a part of these conversations, but I don’t want to get like, quote unquote, territorial about like what we impacted and how much like that moved metrics. And then relatedly, very tactical improvements. That’s sort of the bread and butter of like usability research. We don’t do a lot of it as a team, but I have two lovely full-time contractors who basically help keep our lights on. And so every week they have a full slate of features and concepts they’re testing, usually very tactical. The teams find a lot of value in it. I think it’s great. We can point again at the end of every quarter, similar to my experiment report, I can say, you know, we tested 13 features across the company and we made these very specific design changes, you know, and these have been implemented or these were put into an experiment.
And so that’s like my third and maybe fourth category is the tactical research and the experimentation impact. So that’s how I think about impact right now, company and strategy level, that’s the longest term and having specific work streams that align to that feature level. Again, helping teams dream up new features or iterate on existing features and helping unblock them at the feature area level and then experimentation and tactical impact.
Steve: For each of these three plus a sneaky fourth level of having impact, I’m wondering how and who and when, like, the decisions about what you’re going to spend time on doing. Yeah, could we talk through each of those and maybe hear how those decisions get made?
Akhay: So basically how do I staff my team to ensure that we’re covering those different layers of impact?
Steve: Yeah, and I guess the leading question part of it is, I don’t know, I have this thing about proactive versus reactive that I want to impose as a presumption in that. So, yeah, who’s working on it and where does the need for them to work on it emerge from?
Akhay: I’ll start with the first. I think that to me is it’s new terrain for us, but to me it’s really obvious. It’s, you know, we have these company goals that we talk about constantly. Like our three to five year strategy as a company. And so those to me are like, okay, if we have this other North Star as our company, I need to have work streams that align to each one of those. And so the need sort of arises at the top, right? It’s like, these are our North Star metrics. We constantly refer to them. We talk about them in all hands. You know, we have documentation around them. And so when I actually presented this to senior leadership a couple of months ago, that’s how I framed it. I’m like, hey, I’m trying something new. These are these new work streams. And actually they are all very, very neatly mapped onto our specific goals we have as a company over the next three to five years.
And so the need sort of arose in that I identified that and I was like, we need to do that because as I mentioned, a lot of these, actually all of them are audience focused. They’re about a specific subset of our audience. And so as a result, about half my team is staffed on those. And here’s the thing. It’s not like these are in isolation, right? Like sure, these ladder up to big company goals, but actually in the short, I would say in the medium to long term, they still will have impact in, I mean, they have to have impact for specific teams as well, because that’s how we get to these goals, right? As teams have to act against these things. And so it’s sort of this twofer of, you know, I want to help shape our company vision and strategy long term, but in the short, medium, long, in the short and medium term, these will also serve area level goals. And so that’s where the need arose, so to speak. And that’s how I’m thinking about staffing them in that, like about half my team is working on work streams that align to those goals. And then the other, you know, second category that’s, I think everyone on top of what they’re doing is still more or less aligned to an area. That’s the area that they represent. That’s the area that they have some context in.
We’re experimenting with some new staffing models, and I now have a research pod of a few researchers who’ll look across two areas, which we can dig into if you’re interested. But nonetheless, I think for the most part, people are aligned to areas. And so just because, but that doesn’t mean they’ll be working on things that quarter for that area. A great example is I have a researcher on my team who’s leading a work stream that has a lot of buy-in from our CEO. We actually meet with him regularly just to keep him updated on what we’re learning and what the next steps are. As a result, she can’t work on monetization, which is the area that she supports, but that’s okay because I made sure that like anything that’s like absolutely critical, we’ll find a way to resource. But I also make sure to set expectations like, hey, this quarter actually like X person is busy working on this other work stream. So I have to do this game of trade-offs. And as a result, this game of like working with stakeholders around the expectations around resourcing for the quarter or for the half or whatever it might be. Again, we have to work with a finite set of resources.
So I do the best that I can to make everyone happy. Not always possible, but you have to do what you have to do. And then the third is that’s the stuff I’ve tried to offload for my team. That’s where we leverage our contractors who work full time. And then the experimentation is like an added bonus. Like a lot of things can lead to experiment ideas. So we as a team just keep a pulse on that. And so that’s sort of like a byproduct of all the work that we’re doing. If possible and when appropriate, we try to move it to experiment ideas that we can implement.
Steve: I’m thinking back to your use of the term rituals to describe, you know, ways that organizations work together. And so within research, yeah, what rituals or practices do you have? You’re talking about sharing information and keeping, you know, track of the pulse of things. How do you all work together?
Akhay: Actually, very, very top of mind for me. We’ve been doing a bit of a ritual reset that I’ve been really excited about. I think one thing that comes to mind a few months ago, I was sort of observing. So we have a very tight knit team. We work really closely together. We have this is a research team. We have our weekly meeting where we keep in touch and chat about what we’re working on and commiserate over things and get to know each other. But I realized there’s so much up leveling that we were missing because we as a team weren’t looking at each other’s work nearly as often as we should be. And so I kind of had a bit of a mandate and I said, hey, if anything you are doing is going to get, and this is subjective, but more than a few eyeballs on it beyond our team and you’ll be the judge of how you want to define that, I want to make sure that our entire team does a round of crit for it.
So if you’re sending out an email at the end of a project that the whole company reads, which we do at the end of every project, we send out a TLDR email to this email list that has pretty much the whole company, I want to make sure that our entire team has a checkpoint to give feedback on it before we send it out. That’s been amazing. I’ve gotten so much great feedback about the feedback process just because it’s so nice to be able to connect the dots. And we were starting to miss that a little bit up until a few months ago. That’s a ritual I love. And that’s also true for things like absolutely if you’re going to have a share out report, like absolutely multiple checkpoints where the whole team looks at it. And I’ve just seen so much work over the last few months be up leveled, we’re able to connect the dots more, have more of a point of view. I mean, yeah, it’s just the nature of the more people can give, the more number of smart people that can look at what you’re producing and give you feedback on it, the better your work will be. So I think that’s a really lovely ritual that we have.
Another thing I try to do every quarter is these sort of zoom out weeks, typically to start or end a quarter where we actually get together and talk about these bigger provocations that we as a team want to pose for the company or for the product team or with as wide of an audience as possible. And that’s also an exercise in connecting the dots. And we had a really fruitful discussion recently where we tried to dig into the archives of our team’s insights and identified examples of great insights and not so great insights. And then dissected why was something great and why was something not? And then actually came up with principles of what we as a team think a great insight is. And I can’t recite it off the top of my head because I forget, but it was things like, actually, I’m not even going to try because they were lovely and great and I probably won’t do them justice.
But we as a team now have a shared definition of this is what a great insight is. And that can be an anchor and a bit of a rubric almost. It’s not so much a rubric, but just a bit of an anchor as we frame our own insights. And would it stand the test of like, hey, we defined that this is a great insight. Is this true? So I’d love to do those zoom out days just to dig back into the archive of our own team, think ahead. I have one coming up next month that I’m thinking about, and it’s going to be around impact. People think a lot about individual impact, so I think I’ll have everyone write an impact statement for themselves as individual researchers. Then zoom out one level. What is your impact on the team or work stream that you have been a part of? Then we’ll zoom out another level. What is your impact in the broader team that you’re a part of? Then we’ll zoom out another level. What is our impact as a team? Things like that. I love rituals like that because I think it’s so important to take a step back, zoom out, and also think about our past. I keep using this phrase of the context that we exist in, but I think it’s really true. So much happens in any given month at a company. People leave, people join, team. We have reorgs. We move to a new discipline. So I always want to make sure, let’s take stock of that and let’s make sure that we show up the best that we can in all of that. So that’s my intention with those rituals is helping us zoom out. What else?
And then we actually, we’ve been in a bit of a flux with joining our design teams rituals because we just moved orgs, like I said. So that’s been actually really fun for the team, having a new set of stakeholders that we are building relationships and rituals with. But I think that’s the stuff that’s really important and that’s the stuff I love to think about.
Steve: When you do those zoom outs or that example of what’s a good insight, how are you able to synthesize and align on that? I guess I asked that question with some bias that I don’t take it for granted that everyone’s going to talk about what impact is or everyone’s going to talk about what quality is and that you’re going to end up somewhere that’s not a giant mess on the wall. How are you collectively able to get somewhere with all this potentially personal and individual judgment disparate takes on things?
Akhay: You know, frankly, it is a bit of a mess on the wall and that’s the thing we have to be comfortable with, but pretty tactically, you know, just leaning into our skills as workshop leaders and strategists. So I do a lot of, you know, we’ll have a pretty collaborative fig jam board with these open ended questions and provocations. I’ll continuously remind the team, this is not meant to be a judging exercise or no one gets surprised for having the most number of great insights on the insight, great insights category. And so it’s a lot of like intentionality around, and I think this is a great thing about researchers. Our job is to make sense of mess. And so we do make a bit of a mess, but then I give us enough time to like distill that mess into whatever it might be. A principle of what makes a great insight or a set of three really impactful nuggets that we want to share with the whole company. That was a recent brainstorm that we did. And so it’s a lot of like, honestly, facilitation skills and more tactically using really, really intentionally designed FigJam boards.
Steve: Maybe you could take a fictitious or actual example and just maybe walk through some of the touch points in the process. I’m curious with the number of people and the sort of artifacts being produced, how do you work through group feedback? What kind of feedback are people getting and how do they structure that interaction to make it feasible and valuable?
Akhay: You know, right now it’s actually fairly unstructured, which I like. I’m all for structure where structure is needed, but this has been fairly organic and it’s been working really well. But tactically what that looks like is, like I said, I have a fairly loose, but still pretty clear definition of like, if you’re going to get eyeballs on it beyond this team, let’s go through a team crit because I want to make sure that we as a team are showing up in the best way that we can and also learning from each other. Right. So as a result, it’s pretty packed. We basically added an extra calendar slot every week where it’s just crit slots. And so we now have twice a week crit slots for just the research team. And there’s only eight of us and almost every week, all of those get filled up. So we’ll have anywhere from three to five people presenting different types of projects. Like I said, everything from a final report to a discussion guide to a survey set of survey questions that they have.
And we keep it pretty unstructured, but if we’re being really, really granular, what that looks like is we’ll have typically 20 to 30 minutes. The person will start with like, here’s what this is. This is the type of feedback that I want. Sometimes it’s literally like, I’m about to send this email out. Can you just like gut check, spelling, grammar, and like framing for me? Love that. Guess what? With eight smart people in a room, it ends up never being just that because we ended up getting into all sorts of other stuff as well, which is great. And that’s the good byproduct of this. And we will typically just have a live Slack thread. And so the person will be presenting and we will all just have a smattering of Slack notes to the person. And so they’re basically getting feedback and advice from many six, seven people as they’re presenting something, or sometimes it’s more discussion based. And then from there, sometimes it’s like, Hey, actually let’s do a follow up round the next time around if it warrants it. So it’s fairly unstructured actually, and it’s worked really well. And again, what I’ve liked is even just in a few short months of doing this, I see a lot more consistency in how we show up, especially for the high visibility things like a company-wide TLDR email with our insights.
But my favorite part is seeing people connect the dots, right? We’ve had people on the team who’ve been here for many, many years who have a lot of historical and organizational context that someone might not have. And as a result of this, they can then augment their insights or their perspective with that context or someone else might be working on a project that the other person’s not privy to and now they can connect the dots and uplevel their work that way. And so that’s my favorite part of it is also just making sure that the more dots we can connect, the more I think our team shows up as polished and as impactful. And so that’s been our crit ritual and it’s been really, really great to see the team just take it in strides and say that they’re learning a lot just from each other by doing that.
Steve: In terms of that connecting the dots piece, you know, you’re only eight people and it’s taking like lots of face time to get access to each other’s knowledge and put that in other places where it’s going to make the work better. And you’ve worked in research teams of different sizes and organizations of different sizes. How does connecting the dots work in a larger research organization?
Akhay: Frankly, it doesn’t. I, you know, again, loved being at Spotify, but the amount of times I would work on something and then realize months into a project that someone else at the other side of the org, far, far away from me had either done this before or was also working on it at the exact same time. And that was sheer, again, the sheer product of organizational design, the fact that we weren’t centralized as a user researcher, insights function. So it didn’t work to the same extent or you spend a lot of your time gathering context. And so we only have hopefully 40 hours of our work week. If we’re spending, you know, half of it building context, that slows us down for sure. And you miss a lot of context. So, you know, in larger organizations, it’s much tougher. I think it worked really well when I was at LinkedIn. We were a very tight knit research team as well. And frankly, I adopt a lot of the ethos of this like crit practice and making sure that we all stay connected, even as we scale and grow bigger from that experience. And again, it’s much more doable with a much smaller company and with a team of eight, but it’s just tough.
The more people you throw into an equation, the harder it will be. And so I think that’s what, as you scale companies and as you scale research teams, that’s where we introduce a lot of chaos sometimes and end up just like spending a lot of our time connecting those dots to varying degrees of success. So yeah, I don’t know. I think it’s tough. I think it’s just really tough in larger organizations.
Steve: You made mention a moment ago about a research pod. I’d love to hear what that is.
Akhay: Yes, this is hot off the press. But one thing I am experimenting with, with my team is, I mentioned we have the three key areas of our product org or rather of our company. And I only have so many researchers and so we’ve tried the embedded model and it works pretty well in some areas, but maybe not so much in others. And then on top of that, I also just have people matters to consider. I want to give people a fresh start. I want to, I’m a big believer in rotating to new areas after, especially after a few years of working on the same thing. I think fresh energy is really, really important. So I’m at this perfect confluence right now of trying to optimize who on my team works on what so that it’s a good match for them and for the area that they’re in. And then trying to optimize for the management structures I have on my team. And then also trying to optimize for just resourcing the highest priority needs across the entire company. And so I kind of dreamt up this research pod of three researchers who will look across two areas, two of the three key areas, and they will have a manager who’s responsible for building those area level relationships across both areas, fielding, prioritizing, kick-starting the right research requests. And then with the three researchers figuring out probably on a quarterly basis, who’s going to work on what.
And so it’s like a semi-embedded model where I as a researcher in this pod could spend one quarter working on one of the two teams and the other quarter working on the other team. I anticipate, this is again, very much hot off the press. I anticipate some of the concern around that will be, well, what will you lose by not being embedded and not being the go-to person for your product director to come to? I actually don’t think much because they will still have a go-to person as their point person, right? It will be the manager of this pod. And I still think you can build context over multiple areas, but what this actually helps free up is we can work on higher priority things. I think the sheer, if you’re aligned to an area, then you’re kind of beholden to what others that area need. But then this other area might need higher priority, just if we were stack ranking at the company level, what the most important things are. I actually think there’s more to be had by being able to work on higher priority things than being completely embedded in one area.
So it’s not necessarily the most revolutionary staffing model, but it’s a little bit different for Duolingo. And I anticipate there will be a bit of pushback and concern, but I’m really excited for it. I think it’s a really nice way to give people the opportunity to work across multiple areas, gain more context, gain more expertise, and then frankly, just be able to work on higher priority things. And so let’s see how it goes. Duolingo is very much like a, you are defined by the area and the team that you’re on. So this will be interesting. And I’m curious to see, maybe I’m wrong, maybe there won’t be any discomfort, but curious to see if there is any discomfort around like, oh, there’s not one go-to IC, individual contributor researcher supporting the area that someone’s working on. So that’s my research pod experiment.
Steve: Maybe we can switch gears a little bit to learn about how you found the field of user research.
Akhay: I love this question because I think every researcher has a really unique story to how they found research. And mine is a bit of a funny one. So it goes back to me studying Italian, which funny enough, you know, language, now I work at Duolingo, but learning a language was actually a really serendipitous way by which I learned about research, but studying Italian in college, and I did not know what research or design or frankly, even tech was. And my language instructor told me about a summer program in Italy.
And this was in the sort of heyday, early days of like IDEO, the d.school, sort of their like early days of research as a discipline. And I knew nothing about that, but it was in Italy. There was a summer program in Italy and it was about design thinking and prototyping. And I was like, this seems interesting design. I don’t know what design is, but I like the concept of design. And so I went, I did the summer program and it sort of changed my life because I learned about, you know, prototyping and research. And this again was in the like early 2010s. It was in the like early days of like the heyday, I say, of, you know, the d.school and IDEO. And I bookmarked that summer and I was like, I want to do this eventually. But then the more I looked into it, the more I felt like you needed a PhD. Again there wasn’t a lot of clear paths into UX research at the time, especially, I mean, it’s still true, but especially about 10 years ago, that was not the case. And then one thing led to another. I ended up working at a design agency, moved back to Italy and through the luck of it all ended up finding my way to the research team at LinkedIn. And so that was my journey, kind of learning about design and research and realizing, oh, I actually already do this.
A lot of my background prior to tech was in, you know, ethnography, sociology, urban studies, a lot of people-centered disciplines, but I was not approaching it knowing that like there was a field that existed in tech that could relate to it. And so when I, I still remember the moment I was like, oh, this is a job I can have. Like I can learn about people, but in the context of like a business or a product that seems too good to be true. And now here we are, I think it’s changed a lot. I think that ethos of like people-centered research teams has changed a lot for better or for worse. I still think at our core, that’s who we are. But I remember the cultural conversation around our field, especially about 10 years ago was like, that is the unique value we bring. I actually believe pretty wholeheartedly, like that unique value for us has changed a lot. Now we have to be much more like product strategists and designers, and we have to look across multiple methods and we have to be fluent in business metrics and experimentation and a lot of things, which frankly, like many years ago was not true, but I think that’s okay. And I think that’s good, but that’s how I found my way to the field.
Steve: Do you find yourself on the other end of those conversations? Like as you said, it’s still a field that’s hard to find its way into, but are you given your role or just things you do in the community? Are you exposed to people that want to join the field or people that are new to the field?
Akhay: Yeah, it’s funny because now there’s so many sort of feeder quote unquote academic programs or boot camps into this field, and in a way that was not true when I was joining. And I had the great privilege of meeting amazing mentors and people who took a chance on me. And I try to be that as much as I can, but as you can imagine, the LinkedIn inbox definitely fills up. But through hiring, LinkedIn actually hire, not LinkedIn, Duolingo actually hires pretty heavily from a student pool. So a lot of our, we have more people who have never worked anywhere but Duolingo than not. And so I really try to tap into that and make sure that we are getting really good talent that has never worked in research before and trying to be parts of talks and conversations and panels as much as I can. But it’s also a different time. I think the access, again, there’s all these programs now that are like trying to be feeder programs into the field of UX research for better or for worse, and similarly with design. So I think it creates a lot more supply and we’re kind of in this flux as an industry and as a field right now that I’m curious to see how that will shake out.
Steve: What are things you look for when you’re talking to someone as a prospective employee or even as an informational kind of conversation? What are things that stand out to you, people that don’t have research experience but that you think could be successful and do a good job at research? What are you looking for?
Akhay: By far and away, the main thing I look for is I think the like gusto and energy someone brings just to who they are. I think so much of our job is actually not research. It is influence. It is driving change. It is inviting yourself to those rooms that we talked about. It is moving those rooms. It is influencing those rooms. And I really look for that. I think that it’s tough because it’s a very, you know, sometimes difficult thing to measure and feel, but I look for that is like how much conviction can you bring to a conversation or a room? Because so, so much of our job is entirely dependent on that. And frankly, maybe this is the bias I have with my background that’s more in the humanities and the social sciences, but I look for people who are good people people, you know? People who are good at understanding people and which sort of relates to that first point as well. Being able to like tell interesting stories about people.
Again, that’s changing a little bit as I mentioned with just the evolution of our field, but no matter what happens, I think that’s what will set us apart as a field. That’s what also set us apart from other disciplines like data science is like we have to be able to pull from our expertise and knowledge as people experts. And so I look for that as well. And that can come in a lot of different ways. And you know, we hear this term curiosity and empathy thrown around a lot. I think it can be a little bit overused, but it relates to everything I just mentioned. And I think that sometimes just shines through. And so that’s really what I look for is like, can you move a room? Can you, you know, how much conviction and gusto do you bring to the way that you talk about work or you talk about a problem? That’s really important to me.
Steve: People that follow this podcast will have heard some interviews in the last couple of months that were previous guests coming back on and talking about what’s different for them or different in the field. In some cases, five years later, in some cases, nine years later. So hypothetically, if let’s say we were gonna have another episode of this together in nine years, what do you think we would talk about?
Akhay: Well I would love to be back in nine years. I think a lot of what we would talk about would be methodologies of years past that we don’t think about anymore. We would talk about, you know, how we exist in the structures of our teams. And so I’ll be specific about the first example. You know, I don’t think with the new tooling that we’re about to have and the new process changes that we’re about to have, we’re not like we probably teams aren’t going to be doing usability testing. Right? Like we can we’ll probably be able to offload a lot of that. And so I think we’ll talk probably very we would talk very specifically about those, you know, methods that we like don’t use or employ anymore and the things that we used to have to do that we don’t anymore.
A good example is I was reflecting today on how nine years ago I would go back and forth over email scheduling a participant interview and spend like a good chunk of my week literally just back and forth without Calendly. Just me, the researcher, scheduling people very manually and it’s changes incremental. So it’s hard year over year to be like, oh, things used to be so different. But if you look at it nine years later, it’s like, oh, yeah, we didn’t like we just didn’t have tools that could do that for us. And so I would imagine like nine years from now, that will probably be the case. Just a lot of the things and the processes that we have are no longer there.
I think the bigger question is, frankly, like. You know, we can have a lot of conversations as researchers. What is the future of our field? Where is this field going? But frankly, I think every discipline is having that same existential moment. This is not unique to us. Data science, product design, everyone is thinking about how is our job going to change? And so nine years from now, yeah, a lot of the jobs will have changed and a lot of the fields might be, you know, a lot of these jobs will probably. Will have to traverse, I think, a lot of different. I don’t know, areas of expertise to. Do meaningful work, I think. And again, reflecting back nine years, I think that’s also true in the early days of that heyday of, you know, the how might we as I call it, I reflect on that. I think I was told that as a researcher, I was supposed to be a little bit objective. I was supposed to like plant seeds and then let other people run with it. And now, nine years later, I tell my team, no, no, no, you plant seeds and then you water them and you run with it and you bring people along with you. But like you have to have a point of view. You have to run with it.
And so nine years from now, I think it will be probably even more extreme. So I’m curious to see how the like boundaries of what value we bring to a team or a company will be different. What new jobs don’t even exist yet. Like who are we partnering with? And you know, what are the types of things that people are doing at companies? Like I don’t even know in nine years what’s, you know, there’s all these disciplines I work with now or types of roles I work with now that weren’t there nine years ago. So yeah, those are some things that come to mind.
Steve: Maybe just before we wrap up, it’d be great to hear about something about you that’s maybe not about work that, you know, let us learn something about you.
Akhay: I always joke that my life is really lovely and it’s really simple because it’s mostly two things. It is Duolingo and it is techno. And again, that’s more a joke. My life is more than that. But yeah, a really big part of my life over the last many, many years has been being involved with like the techno community and basically, you know, underground dance music. The backstory of that actually goes back to I lived in Amsterdam about 10 years ago. And that’s one of the biggest hubs of, you know, underground dance music. And it sort of also changed my life in a way because over the years I’ve collected so many lovely deep friendships and a perspective on the world that comes with it, being able to travel the world with it. I pretty much spend my weekends and my evenings like digging for music and playing music and being inspired by music, thinking about the music scene here in Brooklyn. And so it’s a really big part of my life. And I remember I wrote, now it’s probably been eight years, but I’m going to regret drawing attention to it.
But there’s an article that I wrote on LinkedIn about how being part of that world actually made me a better researcher. I would probably, you know, cringe if I read it now, but it’s true. I think it was a lot about like shaping your perspective on the world, living a, you know, sort of alternative life in a way and being curious about people and philosophizing about people and problems. And yeah, I don’t know. I think a lot of that has translated very implicitly into my life. And I don’t know, I like being very open about it too. You know, that’s something that shapes me. And so is being a researcher and so is, you know, working at Duolingo. And so, yeah, there was a really funny meld of that where there was a meme of Duolingo last year going to this club in Berlin that I go to a lot. And I had like dozens of people at work and elsewhere being like, oh, surely you did this, right? Because this is the perfect meld of you and, you know, the two things that you love and represent. I did not have anything to do with it, but it was true. It was the perfect meld of that. So yeah, that’s a big part of my life.
And also just seeking inspiration from, you know, I love living in New York. I moved here really accidentally. I didn’t even want to live here. And I did. And it was one of the best decisions of my life. And now being really involved with like, yeah, just like finding inspiration from the city and being able to live in New York musically and artistically is really important to me as well. And I think, yeah, it totally feeds into my work as a researcher.
Steve: Why would you cringe looking at that LinkedIn piece now?
Akhay: It’s probably really lofty and like over philosophizing something, which, hey, that’s fine. That’s part of writing LinkedIn think pieces. But maybe I wouldn’t. Maybe I should give it another read. You’ve inspired me, Steve.
Steve: You’re describing a type of music that you like and that you participate in, but how does sort of being interested in something get to the point where you’re actually part of a community?
Akhay: Yeah. Yeah.
Steve: Like, you could list a genre of a thing that you like, and that doesn’t necessarily mean also that’s a community, but that seems to be a huge part of what you’ve drawn from it.
Akhay: It’s true. It is a very, very, probably the biggest part of my life, like I said, outside of work. You know, I think it’s, I tend to do this. I’m learning about myself over the years. I’m very good at focusing on just a few things that I love. And then I really ledge onto them and go really deep. And I mean, like decades long deep research is probably, you know, research is one of them. But when I got into this music, I remember feeling, I don’t even know what I felt, but I remember there was this universe of like terminology around this music that I knew I wanted to uncover. And then I spent many years living across the world and then finding wherever I lived pockets of people that were interested in that. And that was a beautiful realization for me, actually.
I lived in San Francisco when I worked at LinkedIn, which was actually a pretty, I loved LinkedIn, but it was a pretty low point in my life because I never really took on to living in San Francisco. But I remember like I found this little community of people who were into techno and who were into underground techno and like industrial techno. And that typically means they’re, you know, they’re throwing these parties or they’re throwing cultural events and you realize people are the same everywhere. And then I started traveling and I would go to, you know, Rome or Prague or London or wherever. And like, there’s always that community. And it was this beautiful realization that like people are more similar than we’re not. And then honestly, over the years, it’s the sheer number of hours you put into a thing. And so I’d put in a lot of hours into attending events and building connections.
And then when I moved to New York, it’s such a, there’s such a rich cultural scene here for anything, but certainly for techno and for underground music. And that was sort of what I launched onto. I mean, I moved here right as COVID was kind of starting. So it took me a while to kickstart that. But in the last few years, being able to just, yeah, it’s a sheer number of hours you put into a thing and then you realize that becomes your life. And I feel, yeah, just really lucky to like feel so much inspiration from music. And then actually it’s funny, yesterday I went to a friend’s quartet that was playing. And I felt like the same thing I felt about 10 years ago with techno, where I was like, I have no vocabulary to like discuss this with someone or to feel a certain way about it because it’s a whole universe of like, they were playing a piece from a medieval French, I don’t know, composer. And I was like, that just breaks my brain a little bit because I don’t even know what that entails.
And it’s funny because people probably feel that way when I ever start to talk about the intricacies of techno, but like, it’s similar, but so, so different. And it’s kind of beautiful to be at the start of something, not to say I’m going to go on a decades long journey to dig into that, but who knows? We’ll catch up in nine years and see. So yeah, it’s just frankly that it’s the sheer number of like hours you put into something and then the like feeling you get, I think you, you were always iterating on our lives and like the people or the situations or the things that make us happy. It’s true in work. You know, I’ve loved certain places I’ve worked at more than others or people I’ve worked with more than others. That’s obvious, you know, but it’s true. And over the years I’ve realized that’s been the one constant that like people that I’ve gotten along with or the situations I find myself in are happiest when they’re somewhere in that realm. And so as a result, I’ve now narrowed down my life to be like, Hey, I do the things that make me happy. And that’s a big part of it.
Steve: Is there a particular piece of music that is — that’s the one we would know you by?
Akhay: It’s hard to think of a particular piece of music, but recently I’ve been doing a lot of, I was just in Berlin and I spent like four hours at this record store, just listening to records because it’s such a rich country, you know, it’s such a rich city for techno. And I think where I’ve found a lot of inspiration recently is like old nineties, slower, what’s called like acid techno, which is just, you know, it’s the sound of, I don’t know, it’s hard to, again, I still don’t have the vocabulary for this stuff, but the sound is called acid and it’s this like really uplifting sound that just, you know, I collected a lot of records from that. And then this weekend, actually, I’m going to Detroit because that’s the real birthplace of techno and there’s a big festival that celebrates that. And so I’m really excited to get the flavor of, I love what’s called Detroit techno, which is a much more minimal, darker, more industrial, much more repetitive sound. But to answer your question, no, no particular music comes to mind, but genre wise, that’s where I find most inspiration is these days at least. And then, yeah, I think a lot of the, you know, the scene sort of also, it changes pretty fast. And so right now there’s this, you know, the people who are sort of at the top of the scene, so to speak, the people who play these big institutions are the ones I’ve been loving. And so, yeah, I don’t know, it’s hard to describe it without going on to like a full jargony journey, but yeah, just like back to the roots of like 90s Detroit and Berlin techno. That’s what’s been making me happy lately.
Steve: Well, thank you so much for coming on the podcast and talking about your two things, research and techno. It was great to talk to you, and yeah, we’ll pick it up again in nine years and see where we’re at.
Akhay: Yes, I will see you then. Thanks so much for having me, Steve.
Steve: There ya go, whaddya know, that’s the show! Find Dollars to Donuts wherever you get podcasts, and at Portigal dot com slash podcast for all of the episodes with show notes and transcripts. I would love love love for you to rate and review Dollars to Donuts on Apple Podcasts. Our theme music is by Bruce Todd. Today’s remix is by Trinity.
The post 47. Akshay Verma of Duolingo first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Daniel Escher, Director of UX and Research at Remitly. We talk about more ways for researchers to add value, business questions over research questions, and the things that researchers worry about.
Where I think collective identity can be limiting is when someone thinks of themselves as a researcher and says, “Therefore, that means this is my small box of things that I do and ways that I contribute.” And what I always want to do is push that box to be bigger, right? I’m not at all saying that the box doesn’t exist in any way. But we as researchers can drive far more decision-making, far more strategy, far more hypotheses than I think we realize. I think that we tend to want to hand off work to other people when actually what I encourage my team to do is figure out where are the places where actually a handoff doesn’t make sense, but a handshake makes sense. There’s some contact there. Or where does hand-holding make sense, where there’s really extended involvement? – Daniel Escher
Show Links* Steve and Inzovu – Storytelling workshops * Formats Unpacked * Daniel on LinkedIn * Remitly * Nazir Harb Michel on LinkedIn * Savannah Young on LinkedIn * Angelina Erine Theodorou on LinkedIn * José G. Soto Márquez on LinkedIn * James by Percival Everett * Clay Christensen’s Milkshake Marketing (and Jobs to be Done) * RITE Method
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal.
In collaboration with my friends at Inzovu, we’ve been running storytelling workshops for clients. Storytelling is an essential human skill for any team. It drives connections, influences decisions, and inspires empathy. In innovative and creative practices, the ability to tell a compelling story is just as crucial as doing the work itself. While good teams focus on delivering high-quality work, great teams go beyond and wrap the delivery of their work into stories. A team’s ability to tell well-crafted stories is critical for influence, which is always the key to the outcomes they seek. That’s where our storytelling workshops come in. These workshops empower your team by enhancing their storytelling prowess. It’s valuable for a range of roles within growing product and service organizations, like user experience design, user research, product management, marketing, and innovation. We run this in a few ways. One is a series of short online sessions across several weeks with individual and group assignments between meetings. We also have an in-person full-day version. Check out more info at www.inzovu.co/workshop/storytelling. And I’ll put that link in the show notes. One resource that participants have found useful and inspirational is Formats Unpacked, which really opens up the idea of what a story can be. An opening monologue is a format. A podcast interview, which we’re getting to in a moment, is a format. An unboxing video is a format. The same interview, one year apart, is a format. You might enjoy checking out the Formats Unpacked site, which dives into many different approaches to presenting stories and sorts out what it is that makes them work.
Now let’s get to the episode. Today, I’m speaking with Daniel Escher, Director of UX and Research at Remitly. Daniel, welcome to Dollars to Donuts. It’s really great to have you here on the podcast. Thank you.
Daniel Escher: It’s such a pleasure to be here, Steve. Thanks so much.
Steve: I’d like to ask you to start by giving a little introduction.
Daniel: Yeah, I think the very first thing people are surprised to learn about me is I’m the youngest of six children. I realize that’s pretty uncommon in this day and age. But I come from a big family. I have 14 nieces and nephews across myself and my husband. And I grew up in Hawaii, also kind of surrounded by not just my biological family, but very much a sense of community and broader family. And I think I’ve generally taken that throughout the rest of my life, where I try to surround myself with people who are really, really good people, good-hearted people, people from different walks of life.
So after growing up in Hawaii, I moved with my family to Washington State, where I currently live. But I was gone for about 10 years, living in New Jersey for a few of those years, and then seven years in Indiana, getting my PhD in sociology at University of Notre Dame. Now I lead customer research and UX for Remitly. I should actually correct that. I’m one of the people leading UX at Remitly. My official title is director of UX and customer research. Remitly is a company about a dozen years old. We provide cross-border financial services and have offices located around the world, but our headquarters is here in Seattle.
Steve: Can you say a little more about the company and what it does and who you serve?
Daniel: Yeah, so we serve really anyone who needs financial services across borders. And so the focus for us over the last decade or so has been remittances. So people who have moved from one country to another and then are sending money, often back home, to family members, friends. That’s not all of our customers by any means, but that’s definitely the vast majority. So we operate in what we call pairs of countries or corridors. So let’s say sending money from the US to Philippines is one pair or corridor. Sending money from Canada to Nigeria is another one. So our customers range really widely in terms of demographics, in terms of socioeconomic level, and in terms of the needs that they have for the product. And then over the last couple of years, and especially these days, we’re in the process of developing some new products. Unfortunately, I can’t say more about what those are, but they are very exciting and I’m very curious to see us continue to learn in those spaces and then ultimately achieve product market fit and a great customer experience.
Steve: How long have you been working at Remitly?
Daniel: Yeah, it’s been almost six years now, which is pretty wild to think about.
Steve: And so you mentioned both customer research and UX being areas that you’re responsible for. And just over the time that you’ve been at Remitly, how has the set of things changed that you’ve been leading or focused on?
Daniel: Yeah, Remitly, I have realized, is a constantly shape-shifting organism.
And I think a lot of companies are like that. And I didn’t realize that actually until pretty late or pretty recently. And I actually think that us as researchers, many of us as social scientists, the extent to which we can view the place where we work as an organism that is responding and shifting and adjusting can actually help us to be successful. So that’s a quick digression. But to get back to your question, the last six years has looked different in so many ways. When I started out, I was essentially an individual researcher brought in to begin a research function. I hired and built a team, established practices, and then about a year ago moved from reporting into marketing to reporting into product. And that was a function of the needs of the business shifting.
And when I joined the company, the biggest needs for us were understanding our customer audience and our potential audience. And so we did total market segmentation, and then we needed to develop a brand position. And so I was heavily involved in that. So essentially we went from being what looked like a flag store online, because we had one country’s flag, another country’s flag, and then an exchange rate of the currencies, right? Into a much more human brand. And so we’ve shifted from talking about the function of sending money across borders into more of the outcomes of sending money across borders. And that’s often some sense of connection, some sense of accomplishment, some sense of aspiration. And then a lot of that work in marketing is, it’s not like it’s complete, because obviously customers are always changing. But the bigger opportunity in the last couple of years has been more on the product side, as we think about building new products and trying to expand the offerings that we have.
Steve: And when did user experience be part of your, I wanna say remit, but that might be an annoying term to use for the company that we’re talking about,
Daniel: I love it.
Steve: But I just did it and I’m sorry.
Daniel: No, no, we use the term as well. Definitely. I would say from the beginning, user experience has been part of that. But especially in the last about year, I’ve become much more focused on it. And I would say that that’s been a fantastic transition for me. So I’ll share more about that in a second. But just to describe the nuts and bolts. So currently I have four researchers reporting to me, a couple of content designers, and then a product designer. And as that group, we cover everything from upper funnel marketing and brand building into landing pages and app store optimization into our app and our send flow and checkout experience and then post-submit experience.
And then kind of underlying all of this is a new work stream around voice of customer. But as I look at that whole surface area, what has been so helpful is shifting from being focused exclusively on research to also thinking about content design and product design. That has really forced me to go through the, I’ll call it the operational wall in what I do.
So I think as researchers, especially in my first about four years at Remitly, I was really focused on, I do the research, I do it well. And then someone goes and does something with it. And in the past year, by having designers report directly to me, I now have to care about that handoff. I have to care about, okay, what then happens with our hypotheses or with our recommendations? Do the designs actually reflect that? Does the overall customer experience actually reflect that? And so honestly, this past year has been a real crash course through that wall or against that wall at times, as I’m trying to figure out in some ways, how to be a bit of a product manager, how to be a bit of a program manager.
I remember talking to my team a few months ago. And what I told them is that our default mode is to think about our skillset first and then draw lines around that and use that to determine what I do or don’t do. And instead, what I said to them was, think of yourself as being on a school board or the board of a nonprofit or the board of a for-profit, where when you are at that board meeting, no one cares what your degree is in. No one cares what your skillset is. No one cares what your job title is. All of you want the work to get done. The end. Right?
And so, obviously, me as a trained sociologist, I’m bringing in that experience and those credentials and whatnot into the conversation that we’re having or into the thing that we’re doing. But all of those things don’t actually limit me or determine what I’m able to contribute. And in fact, I know that there’s a sense of collective identity and momentum that comes from I am a researcher or I am a designer or I am a product manager. But in actuality, if I think about a researcher, for so many years, I was under this impression that the way I add value as a researcher is by conducting original research. And actually, what I’ve realized is that very few people are interested in that in itself. Instead, what I can bring is my overall skillset and just life background. So I can add value by asking good questions. I can add value by integrating data sources. I can add value by synthesizing what business management is saying, what marketing is saying, and what I heard from some agents in customer service.
And I can do all of that and drive decision-making without ever conducting a customer interview or running a usability study. And in a similar way, designers can add tremendous value or steer decision-making without ever drawing a wireframe in Figma or presenting a prototype of something. So all of that, in summary, I would say is I think that we forget the power that we have to make outcomes happen. And we don’t need to limit ourselves to what is my job title? What do others expect me to do? Okay.
Steve: It’s so exciting to hear you describe, I don’t know, the superset of ways that we bring value. And that pushing pixels or asking questions is not the scope of what any of these disciplines can do, and there’s this larger set of things. But if you expand the definition or the tent or whatever it is, yeah, research is facilitative. Research is working with stakeholders, is working to understand problem spaces, is synthesizing, integrating.
That narrative is a really lovely and rich characterization of what researchers can do. Even in your presentation of it and how I hear it myself, it’s still, like in your board example where no one cares what your discipline is. I don’t know, I think I need more help bridging that back to this example because you’re still saying researchers have this larger set. But I’ll just speak for myself, I like that identity. I want to be seen as a researcher. I don’t wanna be seen as a designer or a generic person on the board. I think there’s issues of identity that we need.
And so you also brought up what do people expect from you or what do we think people expect from us versus how to bring more value than that. I mean, maybe my question here is, do you think there’s an aspect of identity with the name of our discipline, researcher, designer, that is playing a role here in terms of what we’re comfortable with, how we’re seeking to perform and be valued for that performance?
Daniel: Yeah, absolutely. I think that personal identity and collective identity is really powerful, right, and actually, if I think about what drives activists in social movements to continue to show up day after day, it’s some sense of personal identity or collective identity. And so I don’t at all want to dismiss that. I think that that is really powerful. What I see that as though is a launching pad or a springboard into doing really big things.
And where I think collective identity can be limiting is when someone thinks of themselves as a researcher and says, therefore, that means this is my small box of things that I do and ways that I contribute, and what I always want to do is push that box to be bigger. Right, I’m not at all saying that the box doesn’t exist in any way. But we as researchers can drive far more decision making, far more strategy, far more hypotheses than I think we realize. I think that we tend to want to hand off work to other people when actually what I encourage my team to do is figure out where are the places where actually a handoff doesn’t make sense, but a handshake makes sense. There’s some contact there. Or where does hand holding make sense where there’s really extended involvement?
So for example, I think of someone on my team, Naz. He’s a linguist by training. He has been heavily involved in conducting research on how we price and how we merchandise pricing. And I realized that maybe some folks are immediately yawning as I talked about that, but it’s actually a really complex, fascinating space to figure out, what should our price actually be to send money across borders? And then how do we display that price and communicate that price to people who may not have more than a couple years of formal education all the way to people who have PhDs or JDs from people who are sending $15 to people who are sending $15,000 at a time, right? And so I go into this backstory to say that I’ve seen Naz make this transition from this is what the stakeholder needs. And so here’s the research, the end. Over the last two years, he’s shifted from doing that to now he is actually helping to set the pricing strategy in certain places. He is actively working with designers and actually putting other people and teams to work for him. Saying, hey, I need this so that we can learn this thing in market or out of market, and then we’ll be able to make this decision, right?
So I hope that that helps. Like Naz still thinks of himself very much as a researcher, right? He’s not a PM, not a designer, not a business analyst or manager. But he’s going far outside of that kind of traditional box of doing the research and then handing it off to operational teams.
Steve: If Naz, were here and we were to ask him how he made the transition from these two points of thinking about how he brings value. Do you have an idea what he might say?
Daniel: Yeah. Oh, lots of sweat, lots of effort. I would say that for everyone on my team. Myself included. Right? I think the reason I’m doing research and being a researcher sound too easy. Most of the time. I’ve seen in myself over. You know, the last five and a half, six years. A huge swing. I’ve seen, you know, Nas do it. Other people on my team, Angelina, Jose, Savannah. All of the kind of easy advice about research and, and making an impact is, is actually really challenging.
So for example, right. I hear people give advice about, you know, track your impact. Well. Okay. That, that actually. Implies that, you know, what different kinds of impact are. Right. So, so that you’re seeing impact in, in the number of different ways that that can look. And then it implies that you are in the right conversations in the right meetings. That you are focused on the right things in the first place and not trying to do everything at once, which is a classic, classic error of many researchers out there. And so, you know, just that one piece of advice is, is actually many, many months, if not even years of kind of hard work and effort to, to get right. Similarly, even advice about how to build rapport with people in an interview is actually very challenging. Right? Like it takes dozens of times of practicing across a wide variety of people to be able to do that successfully. So I’m starting to get long winded, but, but again, yeah. It’s a tremendous amount of time and effort that, that goes into this.
Steve: I mean, this idea that you’re describing here, the moving from the handoff to the handshake or the handhold and sort of learning to do that over time. It is something that I’ve heard a critique of researchers or as research discipline is that, yeah, researchers want to sort of write something and have people sort of automatically get what to do with it. And the opportunity for researchers is to do what you’re saying and help the work take shape and help it have life and help the other people that you want to impact work with it. There’s also a story that exists at the same time and I guess multiple stories can be true when you’re talking about over time, lots of people, lots of organizations, which is that there are researchers who are, I think to some extent, begging to have that kind of impact. I’m not just here to do interviews, I’m not just here to get data or to, often it’s like I’m not just here to run usability tests after you’ve done things like I can bring more value, research is a process and a discipline can bring more value. And those folks are often, I think, stymied because there isn’t quote buy in or they won’t let me, you hear that.
And I don’t know what we do with that, but I guess a takeaway for me is that there’s multiple sort of vectors that are happening in a profession. And the one that you’re describing is, again, is not new to me. Researchers have this opportunity and it takes, I think you’re also emphasizing, it takes time to kind of shift your role for yourself. I think that your team, you’re not talking about overcoming internal resistance, you’re talking about them all progressing in, yeah, redrawing the box or not being limited by that small box.
Daniel: Yeah, I think that everyone at any company in the world consistently underestimates how long it takes to build relationships, just period. And the reason that I start there is that as soon as we talk about research or design or involvement in the abstract, we have removed relationships from the equation. And actually, almost all of these questions around making an impact, around getting involved in things, come down to personal relationships that we have. To be clear, the sociologist in me is immediately saying, okay, great, Daniel, you’re gonna emphasize the individual. Hey, we also need to talk about the institutional, right? Obviously, right? But stick with me for a second just on the individual story.
So if I think about being a researcher in a setting that is challenging, a setting that is maybe even skeptical, a setting that feels stymieing, the way to start to make progress against that, in my experience, is over time, slowly chipping away at all of those perceptions, right? So I spent probably my first four, four and a half years, Remitly, listening to what other people wanted me to do. Instead of going after, here’s what I think needs to happen, and I’m going to make it happen. So I got all sorts of external advice about set up these, like, self-directed training modules and do this kind of work, not that kind of work, right?
And in actuality, like, where I need to start is, what is the business question? Not what is the research question? What is the business question? Okay, the business question is, we are losing this kind of customer in this country. Okay, what can I do to address that question? And I realized that that work is going to come at the cost of requests from teams that want me to do this little thing or that little thing. But in actuality, the biggest questions for a researcher to answer almost always rise above that kind of fray, right? So like, if you think, no matter what the company is, it’s how do we acquire new customers? How do we keep the ones we have? And why are ones that we have churning? Why are they leaving?
So the extent to which you can position your work to answer one of those three questions, you are going to be making an impact. And what I see researchers often doing is waiting for permission to get involved, right? Like, they are waiting for someone to say, hey, I want you to be involved in this thing. The problem with that is, one, like, oftentimes the requests that come in are not actually the most important, valuable thing that could happen, right? And then the other problem is that you’re waiting for permission, right? Instead of saying, hey, this is the thing that needs to happen and this is the thing that needs to get done.
And so what I often see researchers do is they have some sense for, here are the very important questions, but then here are the requests that I’m getting. So I’m going to try to do everything, right? And that is a recipe for disaster. In actuality, a human being can do maybe two, maybe three things in a quarter, right? Like, set aside all the answering emails, set aside whatever. Like, you can do two or three things in a quarter. And so it’s not impressive to say that you’ve run, you know, 50 usability studies or 100 user interviews or whatever the thing is. The impressive thing is that you have actually answered one of those big three questions around acquisition, retention, and then, you know, churn or dormancy.
Bit of a soapbox there, but I see researchers, you know what? I’m actually going to speak for myself because I don’t want to project onto other people. I have been such a people pleaser for so much of my career. And I have thought that the way that I will get invited into things or the way that I will stand out is by taking some requests, doing it really well, and then being rewarded and recognized for that. And what I’ve learned is that just does not happen, right? People’s time span is really short. Their attention is short. It’s actually in the act of becoming more independent, at least in the U.S. corporate context, that you actually stand out and you demonstrate that you know what truly matters and that you are committed to delivering on it.
Steve: I don’t know if it’s when someone joins your team or through the relationships you have, your team as a leader, how do you help people make that switch from, and you got me when you said people pleaser and waiting for permission, I think that resonates strongly. How do you help people see, I mean, you’re talking about through the looking glass a little bit, there’s a whole other way of looking at this. It doesn’t feel like it’s on a gradient, it feels like it’s a flip from do what you’re kind of assigned, and that’s how you help people versus say no to things and choose independently where you think the biggest value is. That doesn’t feel linear to me, that feels like letting go of a previous model.
Daniel: Yeah.
Steve: And I hear you on things taking time, so I don’t want to sort of reduce this to like an aha moment or anything…
Daniel: Yeah.
Steve: …but how do you get there, how do you manage other people to get there, how do you?
Daniel: Yeah, it’s a light switch that I think takes many months, if not even years, to flip. So you’re right that it is kind of a polar experience. But I think in actuality, there’s a fair amount of gradient involved. For candidates that I’m interviewing and for people that I hire, it starts with I’m looking for three aspects to them, and then there’s kind of an underlying aspect just because of Remitly. But the three that are constant are I’m looking for the person to be curious. I’m looking for the person to be humble. And I’m looking for the person to know their value and the value of others. So curiosity, humility, value.
So let me unpack each of those. Curiosity, I am looking for people who are constantly wondering, what’s ahead? What’s next? What can I learn more of? Why did this thing happen? What is a new method that I need to learn? What’s a creative way to solve this problem? What is some domain knowledge that I could pick up that would help me in what I need to be doing? So just kind of a tremendous curiosity. Like very few things disappoint me just in general in work. But one of them is when I see someone who has just kind of done the work and then just stopped and has said, like, okay, I checked the box. And research is not about checking the box. Like research is kind of this constantly self-propelling activity forward. And so there’s almost never a sense of termination to it. So that’s curiosity.
Humility, I am looking for people who are confident in what they know and what their skill set is and what they can bring, but then also recognize that they still have much more to learn and might be wrong. So humility does not mean submissiveness or shyness or any of those other traits. But it’s really this sense of confidence with the recognition that I have more to learn. I am not perfect. I can and might be wrong for a given thing.
And then value is, you know, I know what I bring to the table. And then importantly, I know what others bring to the table. So this actually ties back to the discussion we were having about the researcher identity versus being like part of a board, so to speak, right? Where I, as a researcher, I’m not going to try to do everyone else’s job, but also I’m not going to let other people do my job for me as well. So those are three traits. And then especially because of Remitly, we have something like 5,000 pairs of countries where people can send money to and from. We have a wide variety of customers. We’re developing additional products. And so, you know, kind of underlying all of these three is a real sense of positionality, a sense of understanding, like, who am I in the world? Who are other people in the world?
I’m really looking for, you know, diverse backgrounds, diverse experiences, language skills, experience conducting cross-cultural research or even like design research, right? Essentially the recognition that people differ. And the way that I show up in the world is not the way that everyone shows up in the world.
Steve: How do those traits set someone up to operate in a way that is independent and is not waiting for permission?
Daniel: Yeah, so the curiosity aspect is that the person is not afraid to ask what may seem like dumb questions, right? Just that’s one example. So for example, Savannah on my team, she is constantly asking these great questions that make obvious assumptions that a given team has or that forces, you know, a team to connect the dots in a way that they haven’t before. I think humility, what helps with that is that a researcher is not showing up as a know-it-all, right? Which helps the relational aspect to develop.
And so I think about like Angelina on my team has formed these really strong partnerships with people in creative marketing. And the reason that she does that is because she is able to bring the knowledge that she has, but then also recognize that she could be wrong, that there’s ways that she can improve. And then that makes that team want her around more. So then she’s in additional conversations, she’s part of additional Slack threads or emails, and then that spirals out even further into involvement into bigger projects, involvement earlier in a project. The way that value then fits in is people want a researcher to be around because they know specifically what that researcher offers. Synthesis of data, integration of data, bridging of different parts of a company together, original data collection, analysis, right? All of those good kinds of things. So they want that in the project, they’re clamoring after that. And then the researcher also knows like, “Okay, I have a big picture mindset. “I understand what we are trying to accomplish.”
And so then I as the researcher, I’m gonna put other people in the spotlight or I’m gonna make sure to bring someone into a conversation when it’s an appropriate time for them to be involved. And of course, everyone appreciates that, right? Like that feeling of, “Oh, hey, I as a product marketer “or I as a program manager got invited into this thing “because Daniel invited me.” That feels good. There’s then reciprocity afterward.
Steve: I’m going to pick up one piece of all the interesting things that you just said. At the beginning, you talked about, you know, asking maybe air quotes, dumb questions, and not being a know-it-all. And it reminds me of when we’re teaching people how to do interviews, there is this thing about asking a question that you already know the answer to or that’s sort of maybe an obvious thing or that the person might think you know the answer to. And, you know, I’ve spent time sort of training people on that, and it’s very nuanced. It’s like it’s the same words, like, I don’t know, “What days of the week do you work?”
And there’s a way to ask that question that is dumb, and there’s a way to ask that question that’s, to your point, curious. And some of it is about body language, some of it is about inflection. I think some of it is about intent, what’s kind of in your heart when you ask that question, to your point, again, about curiosity. The interview setting is different than the setting with our colleagues. I feel like there’s maybe more pressure on us to be a bit of a know-it-all, to be seen as credible. Is there a best practice around how do you ask those dumb questions? Because I hear you on all the value those questions bring, but there’s a way to do that that brings the success that you’re describing for these folks in your team that are so effective, have these great relationships. There’s a way to do that that’s harmful to you, that’s harmful to your esteem or respect that you receive, and there’s a way to do it that it all falls under the label of asking dumb questions. Can you identify the tactic that makes that a successful dumb question?
Daniel: Yeah, that’s a good question, right? And this is actually the challenge of human interactions is as soon as we try to formalize things into rules, we realize that there are thousands of rules and nuances depending on the audience, the interlocutors, the situation, right? So, just kind of starting at the beginning with what you were saying, the idea of feigned ignorance in an interview setting, absolutely, a very powerful technique.
Another one that I’ve seen in interviews or that I’ve used is a tactic of we’re in this together. So some of those interview questions that are really survey questions, how many hours do you work in a week? What’s your job? You can almost put on this air of, “Ugh, all right, here we go. “Just bear with me, just a couple minutes of questions. “Just gotta get through the form, right? “You know how it is.” And then you get responses, persons like, “Yeah, I’m gonna contribute to this. “I understand how it is. “I have my own job where my boss does the same thing to me. “Okay, great.” And then now after two minutes of those survey type questions, you actually have more rapport than you would have otherwise.
But then to the second part of your question, like thinking about the work situation, yeah, I think there’s a couple different things come to mind. Like one is doing a little bit of self-deprecation as a tactic, the classic preface of, “Hey, sorry, this may be obvious, “but could you explain what is this acronym “or what’s the context for this thing? “I think I’m just missing that.” So that’s kind of the most humble way of doing it. Another one, it’s related, but it actually makes you sound better, which is, “I’ve just been in five back-to-back meetings. “I’ve been switching context from CS to marketing “to new product development today. “Remind me, what is the OKR that this addresses?” So now you’ve just kind of given yourself a high five, and then you’ve launched into it.
And then a third tactic is the back channel. So you’re not asking the question publicly, but you’re going to someone that you trust, that you have a good relationship with, and are like, “Yeah, what’s this acronym? “I’m just totally missing it.” And maybe you’re doing that over Slack in a meeting or you’re doing it afterward. So yeah, you’re absolutely right. There is an element of performativity, even in the “no dumb questions.” But in general, I would say that actually, we worry too much about self-presentation. If I think of most companies that do an annual performance review, let’s say, no one remembers how funny you are on Slack. No one is thinking about the time that you made them feel good on June 26, earlier in the year, whatever.
Really, a performance review, which is what researchers are interested in or should be interested in, comes down to what were the three big things you did this year? What was your impact? What did you accomplish for our customers or for the business? And I actually think, like you were talking earlier about, how do you help newcomers to a company shift their mindset into thinking about impact and extending their skill set into the operational realm sometimes? And I think that’s actually a really critical answer, is like, “Okay, great. You did 20 different projects this year or you were involved in whatever many different things. But if you had to boil it down, what are the two or three things that you could say, ‘This is what I did in the year.'”
And really, if you work backward from that, that really simplifies life and reduces a lot of the noise. It’s like, “You know what? I helped reduce customer churn by X percent.” Or, “I made this decision about a new product happen that otherwise would not have happened.” It’s these very kind of core impacts or core demonstrations of what you did that have 10,000 prior steps that led to that thing. But really, that is the big set of things that you’ve done that you should be aiming for.
Steve: So we started a little bit with sort of the moment of asking questions, and you gave some examples about sort of how to frame that, but then I think you’re making this larger point that when you look back, even like on a quarter or an annual basis, those small moments that maybe we have some anxiety about are not the ones that should be reflected on as these, what are we trying to get to? But how do we, in the moment, keep that in mind? Because it’s like I guess you’re talking a little about being present versus looking at it in retrospect. So how does that perspective that, yeah, when we come back to look at this, here’s what the important things are going to be, how does that give us the best confidence to make the right decisions in the moment when they feel uncertain or risky?
Daniel: Mm-hmm. Yeah. Yeah. I often tell my team some phrase along the lines of, “Let’s be human for a minute.” And I actually, in this conversation, want to be human for a minute. I very much recognize and see in myself and see in others a lot of anxiety in researchers. I see a lot of insecurity. And I use those terms not in any sort of pejorative way or in any way that indicates some kind of failing. I use those terms as descriptions of what it means to be a human. I have all sorts of anxiety and all sorts of questions in my mind of, you know, “Am I doing a good job? Am I doing the right thing? If I had done this, you know, X number of years ago, would I be in this place?” Right? I in no way want to dismiss those. I in no way want to suggest that, “Hey, you know, if I just had my act together, all of that would disappear.” It just won’t, right? I actually have admired so much the leaders and authors who actually say, “You know what?
Even in my role, even in my position, even after 30 years, I still get nervous before I speak in public or I still cringe when someone’s going to give me feedback.” Right? So I think then the question becomes, “Okay, I can’t get rid of the anxiety or the insecurity. What do I do in the face of it? Do I try to just push it down?” No, that’s not going to be a recipe for success. Do I try to reframe the situation? That’s often successful. Do I have a mantra that I repeat over and over to myself? Sure, yeah, I do that. If someone is giving me hard feedback, I have this phrase, “If you are not making mistakes, you’re not learning.” And all credit to Percival Everett, his new novel, “James,” the slave character James or Jim, says that at one point. I wrote it down on a Post-it, and I’ve been saying it in my head ever since, basically. So I do think that anxiety, insecurity is present and is never going to go away. I do think that as researchers, as designers, as PMs, analysts, right, on and on, that we can look back retrospectively at what we’ve done, but we can also look forward to where we are trying to get to. So by the end of this year, I want to have accomplished this thing and this thing, right? And I have that right in front of me. I’m looking at that every day. I’m being tireless in going after those things. That doesn’t mean that there aren’t hard times. That doesn’t mean that there’s not anxiety, that I don’t mess up micro interactions. But I’m constantly going toward one, two, maybe three things to get to by a certain point in time.
Steve: I’d like to yes and some of what you’re saying here.
Daniel: Love it. Yeah.
Steve: And just to go back to our, we’re going big picture…
Daniel: Mm-hmm.
Steve: …and then specific example of big picture, we’re going back and forth.
Daniel: Mm-hmm.
Steve: So the asking dumb questions, not knowing something,
Daniel: Mm-hmm. Mm-hmm.
Steve: …in addition to everything that you’re saying, I guess I’ve had the experience of being around when someone else asks a dumb question, in those ways that you modeled,
Daniel: [laughs]
Steve: and just felt enormous relief and gratitude for that person doing that…
Daniel: Yeah.
Steve: …that it’s helpful for me, and that I can see even what it does for the group.
Daniel: Yeah.
Steve: That it unlocks something and unlocks some assumption. I’ve personally received it and I can see the group benefits from it.
Daniel: Yeah.
Steve: And so then that teaches me a little lesson that maybe it’s not about permission. I appreciate your take on waiting for permission, but I get permission when I see somebody bring value and then I think, “Oh, they model it for me. I could do that too.” Or when I’ve taken a risk and done something like asked a question…
Daniel: Mm-hmm. Yeah, absolutely.
Steve: …that I’m not sure if it’s okay for me to ask, and then gotten some approval from someone else, thank you for asking that. I might get a private message afterwards. And if that happens, in that first example…
Daniel: Yeah.
Steve: …when someone does it and I’m grateful for it, I’ve learned enough that I might try to tell them, either publicly or privately, receiving the reinforcements and modeling the reinforcements,
Daniel: Yeah. Yeah.
Steve: and noticing that a behavior that might be unsafe or risky…
Daniel: Yeah.
Steve: …does bring value and has positive consequences, starts to give me– and maybe that’s the reframe the anxiety a little bit and say…
Daniel: Yeah.
Steve: …”Yeah, if you look at some specific examples…
Daniel: Yeah.
Steve: …it’s played out really, really well.” And then modeling is kind of a follow-on for that.
Daniel: Yeah, I’m so glad you bring this up. This raises two things I’ve learned. One is in creating the right ecosystem. And then the other is not waiting to lead. So let me unpack both of those. Like, a lot of this conversation has focused on more of the personal, more of the individual realities of day-to-day life at a company or an agency conducting research. But let’s actually talk about the institutional for a bit. Institutional can kind of sound like a sterile, abstract word. And so oftentimes what I actually describe is what I call an ecosystem. So in an ecosystem, we need to have the right conditions in place for everyone to thrive. And so that means having norms about how we conduct meetings. Or that means having checks for bias in promotion panels or in performance reviews. That means ensuring representation in this group and that group. On and on.
We can talk about this in a number of different, very tactical ways. And what I see then is, okay, so we want to create an ecosystem where people are thriving. And how we can help with that is that we can start to create that ecosystem for ourselves. Even if it is my coworker and myself or my team of four people or my team of 10 people or the broader design org of 40 people or on and on. I think that especially starting out as researchers earlier in career, we are waiting for someone to give us the affirmation or to send us the message after the meeting. But actually you can start doing that on day one right as you are starting your career. You don’t have to wait for things like that to happen. Obviously there is — in what I’m describing, I’m describing kind of a bottoms-up approach to creating that ecosystem. Obviously there is a complement of top-down, right? Like what the company leadership is saying and modeling and demonstrating in their actions, not just their words. But both things really do have to happen together.
And I think actually what you are describing is such a good reminder for me as a director, as a leader in the company. Know that the rest of the day after we have this conversation, I’m very much going to be thinking, okay, who can I call out into the spotlight and recognize? Or what just happened in that meeting that shouldn’t have happened? What can I do differently in the next meeting to make sure that so-and-so is able to speak and contribute more instead of being sidelined like they were?
Steve: So just as we get closer to wrapping up, maybe one thing we haven’t talked about as much, we talked a little about sort of data collecting as one of the roles of research and you expanded that to pulling multiple sources together. And really, we’ve talked a lot about the relationships that create the context or the work to have impact. One piece we haven’t talked about as much is, I’m going to say like a differentiator in research as opposed to a question asker is, is sensemaking, you know, what we do with this data. You know, how do you think about that or how do you practice that with your team?
Daniel: I love that question. I think that one of the biggest differentiators for researchers is the knowledge that we bring. And by knowledge, I mean domain knowledge about various social scientific theories and user research theories. And then also the know-how of how to put those theories to work, especially when we are doing original data collection. What I often see is that researchers come into a corporate setting, let’s say, or an agency setting, and then they actually end up forgetting or very rarely putting to work all of the social scientific theories that exist in the world and that they have learned. And that is actually a tremendous opportunity for us.
So if I think about sociology, for example, which is my main training, I can bring in theories about, let’s say, collective identity from social movements to generate hypotheses for how we might create collective identity among a customer base, which then instills a sense of community and belonging and then can propel some sort of action.
Or if I think about the corporate setting in which I work, if I think about that in terms of social movements, I can actually apply a theory of, let’s say, political opportunity, where I see that there’s division within leaders or there’s something changing in a market that gives us a chance to accomplish something that would have been impossible to do even just last month. I also think a lot about socioeconomic level and research on social class, race, and gender as I’m conducting research. I think it just it sensitizes me to all sorts of different aspects that otherwise I would miss. And then it helps me to make sense of the data, of course. And I’m given examples from sociology, but certainly psychology, economics, political science, all of that feeds in as well.
Where I see a big opportunity for researchers is we often get in this habit of reporting quotes or showing a video clip. And to be clear, those things are very powerful, right? They’re a tool that we can use. But the problem with that is that is only one kind of data. It’s often just as important what the person did not say, or it is important to pay attention to body language or to the fact that off the recording their cell phone rang, which then indicated this thing that can help us design the product, right? So I think often as researchers, we are trying to work backward from, okay, what is the slide deck? Like what is the cool thing that I can put on the slide for the audience? And in doing that, we actually like truncate a lot of our intellectual ability and prowess because we are ignoring or setting off to the side all sorts of other data that exists in the world.
Steve: Are you making the link between that experiential data and some of the theories from social science or other disciplines?
Daniel: Oh, absolutely. Yeah, definitely. A connection there for sure. And then also there’s a connection in terms of the methods that we use in the first place, right? So interviews are great or usability tests are great for that quote or for that clip where the customer can’t find the back button for 40 painful seconds, right? All of that is good, but that’s only a couple of data forms. And so if we actually expand into, okay, I’m not going to limit myself to easy video clips and easy quotes. I’m going to think much bigger about what are the business questions? What do I need to solve? We can then develop really creative methods, deploy those, and then generate really interesting data. And yes, you’re right. It may not be as succinct or kind of easy to understand as a customer quote, but if it’s answering the question, then it’s absolutely what we should be using.
Steve: We’ve talked throughout this conversation about the collaboration and the relationships with the rest of the organization and sort of permission and confidence and a lot of themes here. And I’m thinking about, yes, as a sociologist you have all these theoretical frameworks to kind of bring in. How much of that would be known to people outside your research group?
Daniel: Very little of it, and that is okay. So let me actually use the example of Jobs to be Done, which is a much more kind of approachable framework that I think many people are familiar with. You can use the Jobs to be Done framework. You can make an observation of customers using a milkshake machine in the morning versus in the afternoon like Christensen does in his book. And you can then generate hypotheses for testing and market. Let’s move the milkshake machine in the afternoon over here. Let’s change the size of the cups. Okay, all of that. Great. You are then presenting that information to stakeholders. Where I in the past have gotten off track and where many researchers get off track is they go through that whole analytical process. They go through that whole framework. They may not actually need to do so. All of those frameworks, all of those theories are tools for us to sharpen our thinking and to generate hypotheses. We don’t necessarily need the audience to understand them fully. We may not even need to communicate them at all. And in fact, communicating them may actually just be a distraction for the audience.
What you actually need is what was the outcome of all of that work? And that may reduce literally hours upon hours of observation and careful thinking may reduce down to just a couple of bullet points or a couple of talking points. And that’s okay, right? I think as part of researchers’ insecurity and anxiety, myself included, we want to show all of the hard work that we’ve done. We want to say like, “Hey, this is difficult. Not anyone can do this.” And so we go into a lot of detail. But in actuality, most of our audience actually trusts that we are experts. They trust that we are doing the hard work. And instead, what they want is us to communicate that thing.
I actually think that we can learn a lot from folks in analytics who are really experienced where, my goodness, I know that dozens of hours of data cleaning and munging and various statistical testing has gone into one or two bullet points or a slide and a slide deck. And I don’t need to see all of that work. I know that they are the expert. I know that they’ve done the hard work. And so I can just go in the conversation from what they’re saying and then we can together figure out what does this mean. Sure. Yeah. Yeah. Yeah. Mm-hmm. Mm-hmm. Yeah. Yeah. Thanks for the pushback.
I think this is where the audience is a major overlay to what I’m saying, where I was more speaking for an executive audience that has relatively little time, relatively little bandwidth. And so the approach that I’ve seen be successful is starting with, here’s what we learned, here’s what the implications are. And then if needed, in the slide deck appendix or in the document appendix, here are all of the details about why we are confident in this, what we learned, what our process was to get here. I do think though, as you are working with more operational teams who are really in the nitty gritty with you, like actually creating the marketing campaign or the landing page to test demand for a product or creating the wireframe, whatever, this is where a lot of that detail becomes important. So this is when you’re in a long conversation. It’s probably not a deck or document at that point. It’s probably you with a designer, maybe an analyst and a PM, whoever. And you’re actually having much more of a working session where you’re saying, this is what I’m seeing, here’s why I’m seeing it. You’re answering questions as they come.
And actually, one way that we’ve been doing that much more recently is we’ve been having much more quick iterative share outs with those executional or operational folks versus maybe the share out that happens once or twice with a more executive audience. So in a recent project, for example, I think we were doing share outs every two or three days. And it wasn’t just a one-sided conversation share out, but it was really a discussion, almost kind of like a workshop of, okay, this is what we learned over the last couple of days. Here’s what we think it means. How are you seeing it? We iterate and then we go from there. So I’m kind of describing the RITE method, right? R-I-T-E method. But I think oftentimes that is used more for design and developing, let’s say, wireframes or a prototype. But I think actually the method can apply more broadly beyond that.
Steve: Anything that we didn’t cover today that I should have asked you about?
Daniel: I love that question. I close most of my interviews that way. I want to say two things that I think are often missing when really any kind of leader gives advice and gives prescriptions. One is that this is all a work in progress. I probably make it sound like things are set, that I’ve had these learnings, I’ve made these adjustments, and this is the way that I operate and the team operates. This is one point in time, honestly. I guarantee if we had this conversation in three months or in six months, that I would have a different lens to put on things or I would emphasize certain things more than others. I might even disagree with some of the things that I’ve just told you. And so I do hope that everyone listening to this understands that this really is how I’m seeing the world now. But going back to one of my core values is humility. I’m very willing to be wrong on this or to change on this.
The second thing I’ll say is that a lot of advice, especially I’ll call it out, especially on LinkedIn about research or blog posts even about what to do, what to not do, really misses a lot of nuance. So I’m very much speaking from what I have learned works at Remitly and how I have been able to succeed here. I do think, though, that at a different company, things may be different. And so I do want to encourage folks to use what is useful, toss the rest. That’s totally fine. And I do just have to give a quick meta reflection on myself. This is such a classic researcher way to end a conversation, isn’t it? Like I have all of these thoughts and then I kind of end with, hey, but I could be wrong. Here’s a bunch of qualifiers. And I just I think that’s such a sign that I truly am a researcher. Going back to the earlier part of our conversation. I’m not a product manager. I’m not a CEO. I’m not an analyst.
Steve: I do have a follow-up question though for…
Daniel: Please.
Steve: Your observation of what you see on LinkedIn…
Daniel: Yeah.
Steve: Why did you agree, we’re not on LinkedIn, we’re podcast, it’s a different format, but why did you agree to do this, to be on this podcast?
Daniel: Yeah, I think the detail, right, there’s so much nuance there. There’s so many things that sound kind of trite or pithy in in written form, especially in a forum like LinkedIn. But, you know, as soon as you start talking about them in a discussion, you realize how how one thing connects to another or how, you know, the thing that that you think is generally applicable is actually only applicable to maybe one kind of setting. And so I figured that this would be a great way to have a more detailed, nuanced conversation about what it means to be doing research in this day and age.
Steve: Well, I’m really grateful for you taking the time and having what to me felt like a very nuanced conversation. We covered a lot, but it’s been really interesting and really enjoyable. So I want to thank you for being on the podcast.
Daniel: Oh, you’re very welcome, Steve. Thank you so much for the invitation and for your time.
Steve: All right, everyone. That’s a wrap for today. You can always find Dollars to Donuts wherever you get your podcasts, as well as at portigal.com/podcast for all of the episodes with show notes and transcripts. Want to do me a solid? Rate and review Dollars to Donuts on Apple Podcasts. Our theme music is by Bruce Todd.
The post 46. Daniel Escher of Remitly first appeared on Portigal Consulting.
This episode of Dollars to Donuts features part 2 of my two-part conversation with Reggie Murphy of Zendesk. We talk about psychological safety at work, Reggie’s career journey, and online career resources for UX researchers.
That helps the team be better researchers when they feel like they have a space where, man, I don’t have to be perfect every time. I’m going to definitely strive really hard to do great work and try to be successful. But I have a leader who’s going to have my back if something goes wrong. It works. I want every people leader who’s listening to this to understand that. That you’re not going to get it right every time. But if you set the environment and the intention of being a leader who understands that people will make mistakes, but it’s not that you made the mistake. It’s, okay, how do you learn from it and not do it again? And how that we can set up parameters within the team to address that particular mistake if it was something like a research protocol or something. – Reggie Murphy
Show Links* Steve’s corporate speaking engagements * Steve on the Rock and Roll Research podcast * Interviewing Users, second edition * Reggie on Dollars to Donuts (part 1) * Reggie on LinkedIn * Zendesk * The Fearless Organization: Creating Psychological Safety in the Workplace for Learning, Innovation, and Growth by Amy C. Edmondson * Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity by Kim Scott * Digital Body Language: How to Build Trust and Connection, No Matter the Distance * Magid * I Wish I Knew podcast * Laura Cochran on LinkedIn * UXR resources
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal:Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal
In this episode, I’ve got part two of my two part conversation with Reggie Murphy of Zendesk.
But before we get to that…hire me to speak at your next in-house event. You may see that I speak at conferences and meetups, but most of my talks and presentations are done for in-house audiences. If you’ve got an all-hands or a retreat or a speaker series, book me to deliver a keynote, participate in an Ask-Me-Anything session, or engage in a fireside chat with your leaders. I regularly speak on topics like research skills, the impact of user research on business, innovation and product development, optimizing research influence, and more. Don’t hesitate to get in touch if you have a specific topic in mind. Go to portigal.com/services and portigal.com/speaking for more information.
I was recently a guest on Matt Valle’s Rock and Roll Research podcast. Matt interviews insights and user research pros about research but he also talks to them about their own creative endeavors that may not be related to user research. Here’s a short clip that is about research, but you can check out the whole thing if you want to hear me talk about not only about research but also about Rolling Stones fan communities and creative writing.
Matt Valle: You’ve got a very interesting perspective because you’re sort of growing up when this discipline was nascent and helping to sort of create it to some extent. At the same time, you’re a consumer and a user like all of us, and you experience good design and bad design. Once upon a time, you know, you were able to fight through all of the difficulties and challenges to get things done with technology. So as you think of where it is today, where user research is today and where it’s headed, you know, what does that look like as you look forward?
Steve: Right. I mean, I think, you know, right now we’re in this inflection point of difficulties depending on your industry or geography. And it’s an interesting one because I feel like companies believe that research is important.
Matt: Right.
Steve: but companies don’t necessarily believe consistently that researchers are important.
Matt: Right.
Steve: It goes back to that people who do research thing that, you know, that we talked about before. But I think a lot of things are on a pendulum. And they go back and forth. And it doesn’t mean that if we’re anxious right now about individual prospects or career prospects, I think that’s very valid. And so I’m not trying to negate any of that. But I think what you see over time is that there’s demand and then there’s the demand softens, then the demand increases and then it goes back and forth.
So I don’t know, I mean, I hope I don’t get proved wrong by this. But I think that the demand for researchers to do research will come back. I think it’s going to come back large. And it’s not that oh people who aren’t researchers do poor quality research. I think it’s different. And I think there’s so much need.
And again, right now, I think companies are kind of saying, oh, we don’t need it right now. And they’re justifying that economically. When have you ever worked in a situation where the amount of questions that there were that were important for decisions outpaced and exceeded the resources available to respond to that. So that hasn’t magically gone away.
So I guess I feel medium-term enthusiastic that there’s just a lot to learn and it doesn’t go away. It’s not like, well, we’ve got the answers. Now we don’t need any more research. Even in more mature product companies, I think they continue to be faced with questions of all kinds, like tactical, operational, future, meaning, story, narrative, culture, all this stuff. There’s all across the board. I think there’s lots to be learned that lots of disciplines care about. Strategists care about research, product people care about research, designers care about research.
They don’t care about research. They care about the answers. They care about insights, right? They care about what we can help them understand and decide. So I don’t think that’s going away. And I think it might keep reshaping itself in terms of what we mean by research and who’s doing research, like what activities and so on.
But I guess I just see more, more, more coming.
And again, I’m sorry for anyone that wants to slap me for saying that because that’s not where we are, like today feels very much like somewhat of a brick wall. But I think that’s going to change eventually. I don’t know when.
That was from the Rock and Roll Research podcast. Check out our whole conversation. And of course please pick up a copy of the second edition of Interviewing Users if you haven’t already. Now, let’s get to my the second part of my conversation with Reggie Murphy, the Senior Director of UX Research at Zendesk.
Steve: So we’re talking a lot about processes of collaboration, relationships internally, but maybe we could switch the focus a little bit to you and your team because you’re leading those folks, you’re leading them through change that companies ready for or not ready for. Do you have an approach or philosophy that you bring as a people leader?
Reggie Murphy: Yeah, one of the first things that I learned way back when I started training as a manager is that you can get to where you want to be by helping others get to where they want to be. And I used that philosophy in my mentoring, but I started out as a people leader early in my career. My second job I was managing folks. It was more of a player-coach role, but having deep empathy for teams and how teams need to be connected, build relationships and work together collaboratively has always been on my mind. Promoting diversity and being inclusive, all of that has been on my mind throughout my career. But one of the things that recently, I would say recently maybe over the past five, 10 years or so that I really try hard as a people leader to install in my teams is the idea of psychological safety and radical candor. It’s like two things. I talk about Amy Edmonson’s book, The Fearless Organization, and Kim Scott’s book, Radical Candor, all the time after I read it several years ago. Because especially in tech, you can work on a lot of different types of teams with a lot of different types of people. And the way to get teams to work better together is to provide an environment and a space where people can feel free to share their opinions and ideas without fear of admonishment or being made fun of or anything like that.
And that’s one thing that’s really important to me, and especially in my team here at Zendesk, because we have a different setup than say a company that is just located in maybe two countries. We’re a global company. The 15 people that I have on my team, we are located in Melbourne, Australia, Singapore, Poland, Estonia, Canada, and various parts of the US. We’re a globally distributed team, just the 16 of us. So we’re not able to connect in person very, very much. Last year, we were able to get together everyone who was on my team then.
So number one, it’s just hard because of that, just not being able to be in person. So it’s important that we develop connections, team connections, and an environment where people feel like they’re building relationships with each other. And so we try to do some things in our team meetings in order to foster that. It is really hard. We’re not perfect at it, but I think we’re getting better at it than we were, say, two years ago when I joined this team. But we’re working towards growing as a team. And I want the team to be a family. I know that sounds very squishy or melodramatic, I don’t know. But I feel like you’re working with your teammates 80% of the week. So it’s important that you try to feel like a family. And within families, you want to foster the environment where people feel like, “Hey, I can do something wrong. I can fail at something, and it’s not going to count against me.”
And that’s what Amy Edmondson talks in her book. She talks about avoidable failure and intelligent failure and the differences. And avoidable failures are things that you can… Hey, if you put in the right protocols and controls, you can avoid having typos in your presentation. Those are avoidable failures. But intelligent failures are like, “Hey, you have the best intentions and you are working towards a goal to be successful on a particular project, but just something goes wrong. Let’s say you were trying to step out and do something different, but it just didn’t work very well. We’re going to applaud you if you fail.” And as a people leader, getting back to your question more directly, it’s important that we establish the conditions and the environment where this is possible. And people feel free to work not harder, but smarter, but be innovative, try new things. And if it doesn’t work out, learn from it and move on and do something else great.
Long answer to your question, but I really feel strongly about this, that teams that do this well, and I’ve seen them, they work better together. They’re more successful. People feel like they can grow their careers because their family is supportive of them. Their people leader is supportive of them. And teams where I’ve seen work like this and operate like this, I’ve seen people’s careers just take off and go. And that’s what I try to do.
So that’s my philosophy. And I’m still working at it. It’s still a work in progress, but I am adamant about it. And I love collecting feedback on how I’m doing so that, “Hey, if I’m off center or off course, I want to know about it.” And that’s another thing that helps establish this environment of psychological safety is that the person who’s leading the team is open to this feedback. And you have to be to model that behavior so that your team sees it and they’re able to operate in the same way.
Steve: What’s something you do so that your team sees that you are open to that feedback?
Reggie: Whenever I make a mistake or if I either say something wrong or maybe something, I post the wrong thing on Slack or something, I immediately own it, apologize. Try to own. If you make a mistake, you just got to own it. You got to face it. And I’m trying to be vulnerable in that way.
And I hope that people see that. I’m not a perfect leader, but I’m certainly going to do my best and try really, really hard to push and to do the things I need to do in order to be a positive leader. But if I do something that doesn’t work very well, I’m going to own it. And I think that when people see that, that say, “Hey, okay. Yeah.” Reg is someone who admits a mistake and he can correct it. The burden is not on me to be perfect all the time.
We had a couple of errors on the team over the past six months. When I say errors, some things just didn’t go well with say research sampling or a protocol. Maybe somebody didn’t follow the right protocol for a research project. And there was a consequence where say a customer was contacted when they weren’t supposed to be contacted, that they had opted out of wanting to be contacted for a research study. It happens. And so when something like that happens, it’s very important for the people leader to address the problem, but to be open and to help the person who made the mistake understand that, hey, it was not one of those avoidable errors. You were doing the right thing, but it was a mistake. And in these couple of instances, we talked about the mistake in a meeting. And it’s kind of surprised people that I brought it up. I went at my team to see that, “Hey, yeah, this was a mistake that was made. Here’s how we learned from it.” So there’s no shame. There’s no disgrace. And I talked to them, of course, one-on-one and said, “Hey, I want to talk about the mistake that you made in the team meeting. Is that okay?”
And so I think that that helps the team be better researchers when they feel like they have a space where, man, I don’t have to be perfect every time. I’m going to definitely strive really hard to do great work and try to be successful. But I have a leader who’s going to have my back if something goes wrong. And that’s how I do it, Steve. It works. I want every people leader who’s listening to this to understand that. That you’re not going to get it right every time. But if you set the environment and the intention of being a leader who understands that people will make mistakes, but it’s not that you made the mistake. It’s, okay, how do you learn from it and not do it again? And how that we can set up parameters within the team to address that particular mistake if it was something like a research protocol or something. How can we get better as a team so that we all avoid doing that thing that was the error before?
.
Steve: The avoidable error and the intelligent error. Is this right: one outcome might be figuring out a process or some approach that turns the intelligent error into a future avoidable error. Am I even using the terms properly?
Reggie: Yeah, I think yes.
Steve: So, I’m going to just play back again. So you talked about the connections, the relationships between people. You talked about modeling yourself, making mistakes, vulnerability, taking responsibility.
Reggie: Right.
Steve: And you talked about reactively when something happens, having a safe, positive individual and group learning and going forward with it, improving the situation. Did I miss anything or are there other?
Reggie: No, that’s exactly right. You’re right.
Steve: Well, I’m glad to hear you unpack it because we use a lot of words like psychological safety and team relationships and trust. But sometimes we just take those as read, like, oh, we know what that means. But as a leader, you’re making specific choices that are tactical in nature in order to get to a more mature level of safety and learning and growth among everyone on the team.
Reggie: Yeah. And I think in this day and age when many companies are virtual or hybrid, it’s hard when you’re not in person. It makes human communication harder. And what I’ve learned in working for a really globally distributed company and team is that there is a human on the other side of that Zoom. And it’s utterly important that you get to know them and you try to provide a space for them to feel at home when you’re having a conversation and when the team is having a conversation. It is incredibly hard to do, Steve. And I think our team has grown in this way.
I think we try to find little things that help us. We’re still working at it. It’s not perfect. It’s still sometimes things get lost in translation on Slack. There’s some miscommunication of expectations about things, but I’m hoping that as we continue to grow together, we own it. We own the things that aren’t working. We identify. We’re not afraid to call it out, raise your hand and say, “Hey, this isn’t working. Can we get together and talk about fixing it?”
Tactically, yeah, I do hear psychological safety being thrown around a lot. And so that’s why it’s super important for me as a people leader to make sure that we’re actioning on what it means every day. And that is in one-on-one conversations, that is in team conversations. This is in how we treat each other. One of our core values at Zendesk is we care for each other. I believe that. And the way we do that is to have empathy and all of… That’s another word that’s sort of thrown around. But true empathy is really seeing the world as the other person sees it.
One of my graduate school professors talked about that when you’re conducting qualitative research, it’s seeing the world as other person see it. Well, I use it in a different context. It’s developing empathy for your team is seeing the world as they see it. And I’m hoping that we’re doing well in this area. And I just feel very strongly about it as a people leader that we have to set the tone. We have to set the stage for our teams. And hopefully our teams see that and they’re better off because of it.
Steve: You mentioned up front from this part of our conversation that you’ve got people in all these different parts of the world. So presumably coming from different international cultures themselves, is that a compounding factor besides time zones and lack of being in person? Does culture, in terms of where people are from in the world, impact how you approach creating psychological safety?
Reggie: I think at Zendesk, Zendesk is such a global company. Everybody is in a lot of different places. So from a cultural standpoint, when you step into Zendesk, you know you’re working for a global company. So a variety of geographies. So I don’t know if there are a lot of cultural differences that we have to work through, maybe a little bit. But I think when you establish the environment that no matter who you are, where you are, your background, your life experience, what you look like, you can join Zendesk. You can join the UX research team, the product design organization, and you can fit right on in. We’ll accept you open arms.
Yeah, maybe there are some norms or social norms that’s different from folks who grew up in Europe versus Asia versus Indonesia or South America or in the States. But I think that when you join Zendesk, you’re joining a global company. And I think the people who we attract to our company come in with that thinking in mind, I’m going to work for a global company. And so you understand what that means. But I do think, and this is something interesting, we can kind of go off on a little tangent here, but you do have to install some helpful rules, especially if you’re working virtually, to maybe limit the thing that a cultural difference might be concerning about, if that makes sense.
I think what helps, I remember sometime last year, we did a workshop on something called digital body language. And in the workshop, they talked about what it means to have virtual communication most of the time with the people that you work with. And so in Slack messages, when somebody posts something, we flood it with emoji reactions. We respond with a thumbs up or happy face or how we feel. We let that person know digitally that we heard them. That’s one. Second, I have one of my senior managers is based in Singapore. 12 hour difference right now from me to Singapore. There’s no great time to meet. Somebody has to take the hit. So either it’s 8 p.m. in the evening for me, 8 a.m. for my senior manager or vice versa. So we accept that and we make it work.
The other thing is when we’re working async and I know my colleagues in APAC, in Asia Pacific area, Melbourne, Singapore, when they’re Slacking me and I receive that message, whenever it is that they slack me, I wake up, I get that message, I respond immediately or I try to respond so that when they wake up, they get it. It’s that kind of thing. It’s very minor, but that means you’re heard, you’re responsive, you’re prompt. That’s having good digital body language.
I think those kinds of things, when you do them well, it fosters the type of culture that helps minimize what your question was before, any cultural differences that might sort of stand in the way of great communication. You have to be intentional about it though. So I try to foster that. It’s not perfect for everyone at the company, but I think we try. That’s the good thing about working at Zendesk.
It sounds like I’m doing a commercial for Zendesk, but I love what I do. I love where I do it. So I talk about the environment and the culture. I think we do really, really try hard to get it right.
Steve: You’ve mentioned just throughout our conversation, graduate school, some previous employers, some of your training. It’d be great to hear how you found user research, a little bit of how you got to where you are today.
Reggie: I went to undergraduate school thinking that I would be a news reporter, live on the scene, Reggie Murphy, Channel Nine News. I worked at the campus radio station. I majored in history, minor than communication, thinking that I was going to be a news reporter at some point. Didn’t work out that way. I didn’t know what I wanted to do when I graduated. So I went immediately to a master’s in communication program at the University of Tennessee. And there I learned media. I learned research though. I mean, in a graduate program, you’re going to learn research methods. And I leaned into that. And so some of my first job was working as a salesperson at a radio station. And I didn’t really care about the sales process too much. So that didn’t make me a really good salesperson, but I loved showing the client or potential customer the research that would help them make a decision about whether or not they wanted to buy advertising on our radio station. That’s what I love the most. So after a couple of years of being unsuccessful as a salesperson, I decided to go back to school and to the doctoral program and learn more research methods. At that time, I still didn’t know. UX wasn’t even a term then. It was human computer. I don’t even think HCI was a coined term at that point. It was mostly human factors at that point, showing my age. My seniority, I should say.
Steve: They’re really old term, if you want to just feel less bad, when I came in, there were still people talking about the MMI, which was the Man Machine Interface.
Reggie: Oh my. Yes, that is back in the day. So when I left graduate school in the PhD program, I had a couple of options. I interviewed for several academic jobs. I thought maybe I’d want to teach, but I found a company called Frank Magid Associates. Right now, I think they’re now just Magid Advisors, I believe. And this company hired PhD researchers to work for their company. Folks with PhDs to work as research analysts. Love that job. That’s when the first research trip I took in that role was in Singapore. Conducting was in Taiwan and Singapore. So three months in, after getting a PhD, I find myself in Taipei doing focus groups with a translator, Mandarin translator. It was an amazing experience to just be dropped into that environment. And so I was hooked on just, at that time, I was a market researcher.
But then at that moment, the internet was becoming the internet. So a lot of our clients were television, radio stations, startup internet companies developing websites. And so we began a practice of doing usability testing for these websites. I remember the first one was excite.com. Now I’m really showing my seniority. And I should say the rest is history. I was hooked on this idea of understanding how people were interacting with technology. So then my next job was the company that I referred to earlier in the people leadership role, the Gannett USA Today company. And that’s where I was for 12 years.
And during those 12 years, I moved from being a market researcher to this kind of, when UX became UX, sort of this amalgamation of market research, UX researcher, and the iPhone came out and we were developing USA Today to be on, well, we already developed the website. So now we were building iPhone apps and iPad apps at that time. And so I was bitten by the tech industry bug at that moment. But then 2012 happened and I was laid off. So that was a pivotal moment in my career where I had been working in the media industry primarily for the first 10, 12 years of my career.
And so after I was laid off, I joined a company which was a boutique enterprise research and design consultancy. And so we basically, our customers would hire us to come into their company. And so let’s say they installed a very big enterprise system like SAP or Oracle or one of these big platforms so that their company could do financial management or recruiting. SAP is a big platform. It may not work very well in certain places depending on how you implement it. And so companies would hire us to come in and understand why their employees weren’t adopting it. And then we’d go in and do contextual inquiry, ethnography research to help figure out what the problems were to help the company redesign these interfaces so that their employees could use it. I was hooked.
When that moment happened, I was hooked because I began to… I mean, I was doing so much research with people who were working on business tools and business platforms. And I would sit there and watch them painfully work through a system that was not helpful in helping them get their work done. And I just develop a lot of empathy around that. And I just remember… That’s why it’s really interesting I’m working for Zendesk now because at the time I had a customer who was a utility company and we were conducting research and I was sitting there watching them. And if you think about a customer service representative, this is why I now have a lot of empathy when I have a customer service issue and I’m calling or I’m messaging on a platform. There is somebody behind there, there’s a human being trying to navigate four or five conversations, find data and information to help you with your concerns. And I watched it with my own eyes in several projects with that company. And it was just amazing to me that software isn’t developed very well for certain industries.
Anyway, I was there for about three and a half years. The company was acquired and then I moved on to Facebook now, Meta. And that’s when I think my career really took a turn for the better. I mean, it went up because I was able to now to work for a growing company. That was at the time when it was hockey stick growth. We were hiring many people. This is maybe eight years ago and it was amazing. I worked on groups, the first iteration of groups, Workplace, which was their enterprise tool. I think their competitor was Microsoft Teams at the time. I worked on the camera effects platform. So this is when Facebook was trying to be Snapchat and they built the Me Too features to do the frames and bunny ears and all that stuff. Amazing work.
But I was drawn back into enterprise work when I decided to switch to an IC role. So that was an interesting thing. I went from a people manager and about halfway in my career at Meta, I transitioned to an IC role and worked on the internal recruiting platform that they built internally. And I was helping the platform that recruiters use to recruit folks for jobs at Facebook. Spent a couple of years there and then moved on to Vanguard, a financial services company, very popular, 40 at that time, 40 year old company. Grew a team there, had a lot of success there. It was like a left turn going from social media to financial services industry.
But it was great. I’m glad I know now a whole lot more about 401ks and change of ownership transfers. So I increased my finance acumen by working at Vanguard. But then the pandemic hit and I wanted a remote job, a fully remote job. And so Twitter came along, a company formerly known as, had an amazing time there until, well, we all know what happened. But I left before all of that happened. But I was a director of a creation and conversations team. And our main focus was primarily the tweeting experience. And that was really fun to work on. We worked on now, I think it’s still called Twitter Spaces, but the audio feature where you can talk to anyone in the world on Twitter. We worked on that. And that was a really cool experience. But I left, we had some changes in the company and I was looking for a new opportunity and that’s how I found Zendesk. And it’s where I am now. Yeah. Yeah.
Steve: You mentioned that approach called “I Wish I Knew” that came from Twitter. People might also know that as the brand of the public-facing podcast that Twitter ran.
Reggie: Right. Yeah. So if you go to that podcast, I am in the first episode of that podcast. It was,
Steve: Did you have an involvement in that?
Reggie: Yeah, we named that podcast that, it was a great experience. I think we did maybe five or six episodes. Yeah, it was great. We were basically telling the story of how research informs decision-making at Twitter. And so I can’t remember all the episodes, but then this was three years ago when we produced those episodes. I still go back and listen to them every now and then when I want to relive some of the glory days of Twitter, I would say when I was there.
Steve: I’m not sure if I know of another example of a corporate research team podcast about research. There are obviously other research podcasts, but I haven’t seen a company share what they were doing that way.
Reggie: We loved it. It was galvanizing for our team. Everybody wanted to, I wouldn’t say everybody, but we had a lot of hands raised to, “Hey, when can I do an episode? When can I do an episode?” And it was an intentional effort. We really were doing some great things on our research team to influence product strategy to grow and develop the understanding of how people were navigating the Twitter platform at the time. And I loved being able to tell that story and tell how we were doing it. And so those episodes were in service to doing that.
Steve: So just hearing you reflect on your own research career. You and I had a conversation before we had this one. And you used this great phrase, I’m going to read back to you. I don’t know if I tripped or fell into user research.
Reggie: Yeah, I think it’s — I feel like that’s what happened. I did not know in graduate school that I’d be doing this job that I’m in now. I didn’t — you know, you don’t know what you don’t know when you’re that young. And many people who are in PhD programs, they may work for 10 years and go back. And that’s what it was like in the cohort I was with at the University of Tennessee. Many of my colleagues, they had already had pretty successful careers doing what they were doing. But I was one of the only ones who sort of was pretty junior in my life at that point in time. So I didn’t know what I didn’t know. And I think just working in the market research industry and being in this business during the time of the development of the internet was really helpful in developing my career as a researcher, but a researcher in tech. And once the discipline of UX was established at some point in the early 2000s, I sort of just, yeah, tripped and fell into it. I was already there.
But there was a period of time, Steve, in my early on in my career where I was trying to learn a new methodology every year. I was challenging myself to learn, you know, something like predictive analytics and data science. And at one point, I went to a conference and saw an eye tracking machine. I was introduced to that. I learned that. And then ethnography came into play. So as a researcher, I was — every year, I was trying to learn something new. And then once UX became a thing at some point in the mid to early 2000s, you know, I was there. Yeah, fell into it, loved it, and began just nurturing my career in it.
Steve: What are ways in which you are helping, whether it’s Zendesk researchers on your team or the community of researchers that we’re all part of in general, what are ways that you are supporting them?
Reggie: When I was at Twitter in 2021, we had a program that we installed on Black History Month that year where we had opened our expertise to folks from underrepresented groups across the world to help nurture their careers or interest in research and design. And I had a couple of people who I began to mentor during that month. So we had basically four weeks of connecting with our mentees. And during those conversations, I began to start compiling. I would give advice, and I would find links to articles or tutorials or, you know, how-to articles and tutorials, and I would put it in a Word doc. And my mentee and I would — each time we met, we’d go through it, and I’d offer some advice. So that doc started filling up. I had already had a Google Sheet that had a bunch of links to design thinking materials from my days of being trained at the IDEO company. And so I kept those links. One of my colleagues and I, who I worked with at the time, had first developed it, shared it with me, and we sort of kept it as a doc.
After that mentorship program was over, I said, “I have all these links that were helpful in these conversations. Why not just put them in a doc?” And because it was helpful, because after that period of time when I was doing a lot of mentoring, more people were reaching out that year because that was the year we were really in the middle of the pandemic. There are a lot of people transitioning from one career to another. They were looking at UX research as an opportunity. We were on Twitter Spaces a lot doing something called Becoming a UX Researcher. So that was something that we started, and each month we’d have an episode where we invite people to come and talk about their career and their role as a UX researcher. It was so much fun. I think we did five episodes. We archived those episodes and put them as a link in this Google Sheet.
And so, Steve, by the end of that year, that Google Sheet became this mammoth link farm. It was like, I called it my Craigslist sheet of UX research. That’s basically what it looked like. It wasn’t sexy at all. It wasn’t pretty. And now I had like seven or eight tabs of different things that was linked. Well, let’s fast forward. I began talking to my colleague, Laura Cochran, who’s my former colleague and dear friend. We’ve done a lot of work together over the years. And I said, Laura, can you help me build a website where we can merge all these links together and offer it as a UX research career platform, a place where you can go and get information, career resources, tools, information to help you grow your career.
And so over the past few months, we’ve been building out this website, and it’s now in soft launch mode. We don’t have a name for it yet, which I want to ask you a question about that in a little bit. But it’s just called UXR. And the purpose or the goal of this site is really to help you grow your research career. That’s number one. Just understand what it takes to be a great researcher. Maybe you’re a junior in your career. Maybe some of these materials will help you get to the next level. The second thing we hope this website will do is it just inspire maybe folks who are thinking about entering the field of UX research. And so we’re trying to collect stories from others who have made the successful leap from, say, finance to UX research or marketing to UX research or something else to UX research. And we want to highlight those stories, what were the lessons learned, how tos, so that folks who are interested can learn from others who have done it. And then we want to connect people who have shared this information on other platforms. We want to connect them here on this platform so that it’s all together.
So if you’re someone who’s sort of junior in your career, we’re hoping that this website sort of helps you grow, develop your skills. If you are trying to go from one career field to UX research, it’ll help you sort of inspire you with stories that you can learn from others. And we’re super excited by this. And so to your original questions, what am I, you know, mentoring is one thing. And this is sort of the next level of mentoring is that, you know, all the advice I’ve been giving people who were interested in becoming a UX researcher or growing their career, I’m trying to put it in this website. But here’s the interesting thing that we want to try to do on this website is provide curated information from others. So like, obviously, there are companies all across the world who conduct research. And we have articles on the site now from, say, user interviews, dScout, some of the platforms. The Nielsen Norman Group distributes a lot of information. So we have a few articles there. But we want curated articles from UX researchers who have written a blog post or have logged, have a video that they’ve shared. Like I have some of my friends and colleagues who have written articles about their experience going from one industry to UX research, or maybe there’s a methodology that they’re an expert in that they have shared information about on Medium or Substack, some of those places. We want to curate those articles.
So eventually, hopefully over time, those are the majority of articles on the site from individual researchers who are sharing about their experience or talking about a methodology that they’re expert in. Because I think people learn better from other people. And I’m excited about this, Steve. It’s very new. We literally just the day before we’re recording this podcast, we soft launched it. And we’re now in the middle of sort of sharing it with our friends and colleagues, getting some early feedback so that we can, you know, do what you do when you launch a product. You soft launch it, get feedback, make some iterations. And hopefully over the next few weeks or so, we’ll find a moment to really launch it in a bigger way. If you can add it to the show notes so that folks can take a look and we have links there where they can give us direct feedback about the site or if you have a story of you that you posted about your journey from one career to UX research. Or if you have posted about something else that you’ve done or something that you’re expert in in the UX research field, you know, we’d love to highlight and showcase your story on the site. So there’s a place where you can share your information with us and we’ll speak with you and put it on our site.
Steve: Is there anything else you think we should cover in our conversation today?
Reggie: I think we’ve covered quite a bit today, Steve. I have a question for you. And this is probably a selfish question, but I love the title of your podcast, Dollars to Donuts, and I don’t know. I haven’t — I don’t know if anyone has asked you this before, but I imagine they probably have because we are searching for a name of the website that I was just describing and so I’m curious how you came up with the name Dollars to Donuts or does it have a hidden meaning that people don’t know about?
Steve: Well, the phrase “dollars to donuts” is sort of about hunches. It’s an archaic phrase, I guess, but dollars to donuts, if I don’t miss my bet, this is what I think is going to happen. So I think that’s how it was used. I’ve never really, again, it’s an archaic term. So my brute force method was to go to the Wikipedia page of, I think it’s aphorisms, or there’s some category of short phrases. And I think the aphorisms are much longer. And I just sat there and paged through it until it was, I guess it was at the D, so it was not too far. But I was looking for something to spark, because I didn’t want to call it, I don’t know, User Research Leadership Podcast or Steve Portigal’s, like I didn’t want that sort of obvious name, I wanted a name, right? Like, you know, I wish I knew, like, that’s an amazing name. And I didn’t know you had that source for it. But I had to kind of make it up. But when I hit it, there’s that moment of discovery where you find something. So the phrase, I think, speaks to the delight of discovery of research. People that have followed me on different platforms might have known that I am quite the donut enthusiast. So it had sort of a personal meaning to me. I think about the old trope of focus groups with the M&M’s in the observation room. And that idea of turning like junk food into profit, right? You kind of going from M&M’s to insights to new products. And I mean, that that one’s a bit of a reach, but that like, Oh, yeah, you go from from donuts to I guess it’s the other way around, right? It’s the dynamic between the food and the money at the end.
So that’s sort of my rationalization for it. But I think when I saw it, I was like, Oh, yeah, I mean, I just started scrolling through pages to try to find a term that I could kind of make a pun out of or co-opt or kind of reuse.
Reggie: I am looking forward to going through that exercise to create a name for this website. So I’ll let you know how it goes. When we rename it, I’ll send it to you.
Steve: Right, we’re gonna share it and it’s pre, it’s pre named, it’s pre named version, but the next.
Reggie: Yes. It’s a pre-named version.
Steve: Well, I just appreciate so much your generosity of time and sharing and all the great stuff that you’ve done that I think there’s a lot for me here to learn from. It’s just really great to speak with you, Reggie. Thank you.
Reggie: It was a pleasure, Steve. Thank you for inviting me on your podcast. I had a great time.
Steve: That’s it for today, and as always, I hoped you learned something. And of course you can always find Dollars to Donuts on all the places where you find podcasts, as well as at Portigal dot com slash podcast for all of the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 45. Reggie Murphy of Zendesk (part 2) first appeared on Portigal Consulting.
This episode of Dollars to Donuts features part 1 of my two-part conversation with Reggie Murphy of Zendesk. We talk about aligning the work of the research team with stakeholder OKRs and empowering non-researchers to do user research.
The researcher would go into these meetings and say we’re going to do a “I Wish I Knew” exercise, where we start thinking about what we’re building for our customers, what are the questions outstanding that we still don’t have an answer to. We’d go through that exercise, and then we’d prioritize that list. I can’t tell you how valuable those exercises were and how our stakeholders looked at us and said, “Wow, I did not know that research could add this kind of value to our conversation,” because it really helped them see. You know, that question that we’ve been battling around in these meetings isn’t really the one that’s most important. It’s this one. And to see it all together was a revelation for some of our stakeholders. I can’t tell you how important that was. – Reggie Murphy
Show Links* Interviewing Users, second edition * Portigal Consulting services, including training * The Product Manager Podcast: How To Master User Interviews To Build More Lovable Products * Reggie on LinkedIn * Zendesk * The Maze podcast: Scaling research through democratization with Reggie Murphy * OKRs * Zendesk Relate
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with people who lead user research in their organization. I’m Steve Portigal.
In this episode, I’ve got part one of my two-part conversation with Reggie Murphy, who is the Senior Director of UX Research at Zendesk.
But before we get to that, when I went to revise Interviewing Users, I started with all that I had learned about teaching user research over the past 10 years. In fact, I’ve been leading user research training workshops for a lot longer, and that was a key source for the first edition way back in 2013. Across all this time, helping folks get better at user research has been a big part of my consulting practice. Sometimes I’ll teach a workshop as part of a conference, but mostly I run workshops for a company or a team within a company. Often the teams are interested in up-leveling skills, developing a shared language around user research, or just being more effective at customer interviews. Regardless of the experience and skill mix in your organization, I can lead a session that will help you learn and grow together. The people that get value from training with me include those with almost no experience with research, people who have responsibility but not a lot of grounding, and folks with a great deal of expertise. I typically find a mix in any group, and learning together in real time with me and with each other, whether it’s in person or remote, is an excellent way to improve your organization’s research practice. Please get in touch to find out more about how I can help you do research better and do better research.
I recently spoke about user research with Hannah Clark on the Product Manager podcast. I’ll link to the whole episode, but right now, here’s an excerpt.
Hannah Clark: Something I really wanted to make sure that we end off on before I let you go is bias. Because we always hear that interviewers should be as unbiased as possible. So how should we think about bias in a user interview context?
Steve: Right, and that word in our society, outside the domain we’re talking about today, it’s a really bad word, right? It talks about discrimination, racism, sort of everything that is screwing up our society is bias. So the word is a bad word, but I think when we talk about bias in interviews, we’re talking about cognitive bias.
And one thing is I’d encourage people to be a little more forgiving of themselves. It’s how our brains work, right? There’s reasons why human beings have these biases. Confirmation bias is where you hear what you expected to hear. You already had an idea, and so when you hear somebody say that, you’re like, “Yeah, see, I was right.” So that’s not good, right? You want to sort of do better at that. And there are tactics for this, like, “Hey, before we start doing research, let’s all talk about what our assumptions and expectations are, not as hypotheses to test, but just to say them out loud or write them down so that they’re not kind of clenched within our chest, but they’re just things that we can look at and like, ‘Oh yeah, this is a thing that might happen.'” And that sets you up a little bit better to see those biases when they come up and to let go of them, to have things confirmed, but also see something different.
But there is a compassion for ourselves that’s necessary. I want to offer just like a short story about my own encounter with my own bias. It’s kind of a story about me doing something not great, or at least feeling something not great and kind of overcoming it.
And it’s a story about going to interview small businesses and going into this agency, and the agency had the name of the founder on the wall, and I could go in. It’s like this creative environment, lots of fairly young, hip people kind of riding around. And I’m there to meet the founder, which I don’t know how I would get access to this person. And they come out, and they’re older than I am, and I was younger than I am now. And we go into this conference room, and we start talking, and I’m asking about goals and expectations and planning, all the stuff we want to understand about small businesses. And I realize at some point, this guy’s talking about his short-term goals and his long-term goals. And I realize that I am surprised by his articulation of long-term goals. And I realize that the reason I am surprised by it is that I had judged him. I mean, it’s my own ageism. I had decided this guy is sort of this figurehead founder who doesn’t really, is not really involved, and is like there to interview, not do the work. And it’s a terrible story that I had come up with about him based on what I brought into the room, my own biases, my own ageism. And so when I realized that, this is all happening in my head. Like, I’m asking questions, he’s giving me information. And I realize, oh, my questions are sort of based on my mental model, which is completely wrong.
And it doesn’t always have to be a horrible ism like this. We all have our mental models about people. And if we can hear them being wrong, then we can redirect. We can like, oh, tell me more about your long-term goals. And so as a researcher, yes, I wish I was a person who didn’t participate in isms and was not ageist myself. But we all are some amount to some extent. And I think, you know, there’s a very unkind way to have that manifest itself. And there’s just a more normal human level of that. And I don’t know that I can be the arbiter of what those levels are. But I’m not saying this to be proud of my ageism, but just to be kind of full disclosure with everybody.
When I had that moment, it was like really, really awesome. Like it was just such a great feeling of like it was even joyful, like, oh, I was wrong. I understand this person in a like a much deeper way. And actually to get over myself to be able to do that. My goal is to understand this person in a rich way. That’s what’s really exciting about research and what makes me be able to go get this stuff and kind of bring it back. And so again, they’re not always this extreme, but you often have to get over yourself a certain amount.
So I mean, what was happening for me was I could hear myself. I could hear where I was almost clenching, like trying to steer his story a certain way. And he was steering in another way. And just to feel that tension between what he wanted to tell me and what I wanted to talk about, which happens in every interview to a certain extent. And some of that’s just topic based. But here it was sort of identity based.
And so there was insight for me in that about myself, about the topic, about this guy. And I say that at the risk of being judged for my own bias, but with the hope that we can all get better at hearing our own bias and in the moment and kind of grappling with it and being intentional about the choices we make in the interview to get to what we’re trying to get to.
That’s from the Product Manager podcast. I hope you’ll check out the updated edition of Interviewing Users if you haven’t already. So let’s get to the first half of my conversation with Reggie Murphy, the Senior Director of UX Research at Zendesk.
Reggie, welcome to Dollars to Donuts. It’s so excellent to have you here.
Reggie Murphy: It’s great to be here, Steve. Thanks for the invite.
Steve: I like how we’re talking about here as if we’re in a physical place. We are just on a website talking to each other.
Reggie: We are in a virtual place.
Steve: Yes, I’m virtual here. First of all, what’s your title, your job role?
Reggie: I am Senior Director of UX Research at Zendesk. But for the past couple of months, I have been in an interim role as Head of Design for Zendesk also. So I basically have two jobs right now.
Steve: Just two.
Reggie: So pretty busy.
Steve: For people that don’t know, what is Zendesk?
Reggie: Zendesk is a complete customer service solution. So we help companies develop really positive and connected conversations with their customers through a system that allows them to get feedback from their customers to hear issues and concerns. And our software allows our customers to triage and prioritize in tickets. So our main software is a ticketing system. And so our customers and users can share what their concerns are, issues are, and the system will enable the agents, the customer service agents at our customers to resolve those issues and problems in an efficient way.
Steve: So the users, I guess we have users and we have customers.
Reggie: Right.
Steve: Just thinking from a user research perspective, who is it that you are learning about, what categories of individuals do you learn about in order to improve the product?
Reggie: So our primary customer, if we think about cohorts that we conduct research on, there are two. First, it’s the administrators. And that is the people who set up our platforms. And they could be so they’re like the director of CX, customer experience at an organization. They could be director of customer support. They could also be the chief information officer. So there is a variety of roles of folks who are responsible for purchasing our product and installing it within their organization. And then there is someone that we call admin or an administrator who sets it up so that their organization can use it. So that’s one cohort that we conduct research on to learn how that we best improve the product for them.
And then you have the customer service support agent. And that’s the title in some companies. And that is the person who is on the front lines talking with the end user. So their customer. So for example, Grubhub is a client is a customer. So their end user would be someone who say just ordered some food and maybe has an issue with their order and they go online and they use say the messaging system to alert Grubhub that, hey, I have an issue with my order. I didn’t get the right thing. So that’s the end user. And so the customer service agent at Grubhub is using our system to talk with that end user to help resolve that issue. And so we spend a lot of time researching the admins, how they are setting up the system and the agents and the interface that they’re using in order to talk with the customers that are using their product.
Steve: Do you ever deal with the end user of your clients’ customer support systems?
Reggie: Not a lot. It’s, you know, we have to make strategic decisions about the work that we do because of the size of our team.
We’d love to do more work with the end user. And we’ve done a few projects. But by and large, we spend the vast majority of our time speaking with admins and agents because we feel like that’s where we get the most bang for our buck and who we’re really building for. And I believe if we do the right thing and we build a thing right for the admins and the agents, then the end user will have a delightful experience also, hopefully. Yeah.
Steve: So you’re talking a little about, I guess, where you have to prioritize and what are the areas that you’re thinking about the most. Can you maybe say a little bit about that in terms of establishing research or building research? Where have you decided to look and where have you decided to focus?
Reggie: So when I first joined, and I talk about this in a podcast I recorded for Maze recently, where we were a team of 11 researchers. I have two research operations folks and I have two research managers. So a team of 16, including myself, relatively small, considering we partner with 200 plus scrum teams in the organization, in the broader product organization, which is about 1900 people. So relatively small team.
So when I first joined Zendesk, we were spread out throughout the organization. We were getting work done, but most of the majority of the work was primarily tactical. And we were working towards the end of the product development life cycle, sort of in the evaluation phase. Okay, we’ve already built the thing. Now let’s go and talk to customers and see if it works. And great. We needed to do that type of research, but I believe just philosophically that UX research teams, especially one as small as the one I had inherited when I joined, we needed to be working more towards the beginning of the product development life cycle. Because I thought that that’s where, I think that that’s where the most value can be extracted from a user research team.
Over the past couple of years, we’ve been shifting our work to achieve that goal. And in doing so, the way we’ve done it is we have aligned the work that we do to the company’s OKRs. So each year, a company has about six or seven big OKRs. And then those translate into other OKRs at the organization level. And we align to those big OKRs. And so I’ve spent the last couple of years orienting our team around that structure. And it’s made our ability to conduct more strategic research at the beginning of the product development life cycle better, more efficient, and we’re adding more value.
We still, of course, do tactical research. We still do usability testing and concept testing. Should we go with concept A or B? We still do that. But we’ve seen better success now that we are oriented in this way. And I believe that we’re going to continue as our team, hopefully we get bigger, but let’s say, assume that we stay the same size for the next year or two. I believe this helps us work on the most important work at the company. Now when we do that, obviously there’s some research that still needs to be done that maybe our team can’t do themselves. And that’s what I talked about in the Maze podcast.
And so over the past couple of years, we’ve undergone a pretty extensive effort to democratize some of our research methods. Obviously you can’t, you know, someone who’s not a professional researcher, there is certain types of research that we believe that we can enable them to do. And then there’s other types of research, the more extensive, deep research that we believe that, hey, let us handle it. And so we’ve developed a pretty extensive plan that I’ll talk about in that podcast to help designers or product managers who we may not be able to partner with very closely on their particular area. We’ve enabled them to do very basic user testing using the usertesting.com platform or just basic interviews just to get some feedback or insight for a particular customer problem that they’re trying to understand.
So I think we’ve been pretty successful developing that system over the past couple of years so that we have some balance now. Our researchers are doing really valuable strategic work, foundational research, some tactical research, and we’ve enabled other functions, not researchers, to do research that we aren’t able to help them with on a day-to-day basis.
Steve: I will put a link to that interview in the show notes.This idea of giving other folks tools to do certain kinds of research, democratization, I’ve heard people refer to it as the D-word. It feels like a controversial area. Just having worked on this for a period of time and had success, do you have a hot take on the hot take, I guess, about democratization?
Reggie: I’ve read a lot of hot takes on this lately. I do. You and I have been in this business for quite some time. And those of us who have many years of working in different companies and different settings, I’m a little old school, so I believe that, hey, at one point, let me just say at one point of my career, only researchers should do research. I took a hard line on that.
But when you work with other functions in the way that I’ve worked across my time at Facebook, well now Meta and X, formerly Twitter, and some other companies, you learn that there are other functions like some designers and PMs who also have research experience that if you were able to enable them a little bit and provide them some guidance and the right tools and set the right boundaries and parameters for how they structure a research program, that they can do valid research that they can be confident in.
At one point, yeah, I admit, Steve, it made me cringe that I would allow someone to come in and do a usability test without me or without the team. But I’ve backed away from that in that because we’re now in a world where we have limited resources, headcount, we have constraints on time, we need to build and ship. And so we have to be flexible and adapt. And I think over the years, I have begun to really understand what that means in terms of delegating research. And now I don’t cringe as much because I am working with very capable, very intelligent people who are open and understanding that, “Hey, look, I’m not a researcher, so please help us. We really need assistance here.” And so at Zendesk, we’ve set up a really solid model where we can be confident in the other research work that other people are doing.
So I don’t know if that’s a hot take, that’s a meandering answer, but I no longer have that firm hardline stance that I did back in the day because now I have worked long enough with cross-functional stakeholders that just want to do the right thing. They’re not going, you know, maybe there’s some rogues here and there, but I’m working with people who really want to learn and understand customers and they’re willing to listen and learn and be educated by a research team in order to do it and to do it well so that they can be confident in the outcome.
Steve: And just to affirm the deep roots of that cringe,the fear of what other people will do, way way back, I remember working at an agency doing in context research. And our clients would occasionally ask to show up and sit in. And so we’re not even talking about leading or have any ownership of or any responsibility for, but just be present during. And that was a horrifying concept, that just the mere presence of somebody, they would disrupt this elegant, finely nuanced dynamic. And I think we kind of started to compromise and we would stage this. We talked about re-interviewing somebody or finding a friend of a friend, recruiting something so that we knew it wouldn’t screw up. In a consulting situation, again, early days of the work, so it wasn’t maybe well understood.
Reggie: Right, exactly.
Steve: Just the sense that this was fragile and could be ruined, and that would hurt the work and hurt us and hurt the relationship. I think the roots of what you’re talking about go back and back and back. And when we say them now, it’s ridiculous. What I’m saying sounds ridiculous. How could you ever prevent people from coming or not want them to come? Now we want the opposite, right?
Reggie: I remember some of those days, but I think in the second company I worked for, which was a media company, the Gannett company that owned USA Today, maybe eight or nine years into my career there, we began to do more ethnographic research. This is when I was doing a lot of training with the IDEO company and design thinking was the thing at that moment.
And this is why my stance on this softened because we spent a lot of time in the field with other non-researchers. We would intentionally invite them so that they could see for themselves what customers were saying. We were tired of going out and conducting a piece of research, bringing it back, and people not believing us. You know, because those are questions that come up, “Well, what did they say?” And there was skepticism.
And as much as we tried to convey the value of the work and do all the things to make people feel confident, you know, there was always that bit of skepticism. So we began to empower and engage our stakeholders and invite them to infield assignments. And that’s when I believe that I saw it from a different perspective. I saw some genuine, sincere desire to sit and listen. And we would tell them, we had a sheet and I still have some of these administrative protocol sheets where it’s stakeholder, basically how to be a good stakeholder 101. Here’s what you do. Here’s what you don’t do. Just sit and listen. Don’t say anything. So after a while though, we would allow them to probe, ask questions because we would educate them on how to do it.
Now for those who are still on that hard line today, who would cringe at me even suggesting that we do that, I would say that, yeah, there are probably some cross-functional partners who may not be the type of person who would do very well with this responsibility if you give it to them to do it solo. So then what do you do? Well, at Zendesk, we have sort of a consultancy model. So if you’re a PM, product manager, and you have something that you want to, you have a question, you don’t have a research partner. We have a research request process that we triage these requests. And then we set up time, office hours, time with a researcher, and we’ll consult with you to understand what is the problem that you’re trying to explore and understand. And given the level of urgency or what we believe to be the level of understanding that you need, the precision that you need to have to have a really successful outcome, we’ll decide I think you can do this on your own. Here are some tools. You can go into our learning management platform. You can do our certification course and here are the tools, go for it. Let us know if you have any problems.
That’s one route. The other route is after that consultation, maybe we don’t feel confident and maybe the person who’s asking for help may not feel as confident. And there we can make an informed decision about maybe we go into the direction of, “Hey, we’ll consult with you along the way. It’s not like you’re flying blind.” That gives us, the research team, confidence and comfort that we’re doing the right thing and we’re not sacrificing integrity or quality of the work and the outcomes are what we want or what we envision them or expect them to be from the research.
Steve: I’m having a small, if not big, aha from that. Some of the discussion around democratization is, yeah, look at the nature of the work, the urgency, the complexity. Sometimes it’s around the method. But I think if I’m hearing properly, you’re also saying, look at the team, the requester, and their capability. Educatability is maybe not the right word, but you want the work to be successful and who’s gonna execute it is a factor there. So you choose how to respond. That’s something that you factor in in choosing what path to recommend.
Reggie: Absolutely. I have another example to give you. Recently our company had its biggest customer event. It’s called Relate. It was in Las Vegas this year. The product team took a lot of people to the event this year, which hadn’t really been done previously at an event like this where you have 1,500 customers. And the research team, the design team, we were all there and we set up an area where customers could come and talk with us about various product ideas that we were launching at the event, but also to help us look into the future. We called it co-designing the future of Zendesk. And it was just three researchers, well, four researchers, including myself at the event. We had designers and the remainder were product managers.
So in order for us to be successful at these customer conversations, we developed sort of some tools, some guidance that we shared with the organization. Say, “Hey, if you are speaking with a customer, here are some questions that you can ask them and here’s how you can ask them.” And we enabled them to, I believe, have some really fruitful, enriching conversations. And I felt good about that. I felt good about what we learned. And I think as you grow and mature in your career as a people leader, which I am, but even as a UX researcher, you begin to develop sort of this level of, okay, it can’t be all bad. Like there are ways that you can adapt and get the type of outcome that you expect. Is every piece of research that a non-researcher will do will be perfect? No. Will they make mistakes? Yes. But on balance, I believe that if you set up the right parameters, provide the right guidance and consultation, then you can really get the outcome that you want. I think that we did that at this conference this year.
Steve: did you see some of your non-researcher colleagues being successful in their interactions with customers?
Reggie: It was amazing to see. They were engaging, they were listening, probing in different places appropriately. Because we were sharing some concepts on a big corkboard wall. And to be a good researcher, and I think I made the comment in the Mace podcast that everybody can be a researcher. I’ve read lots of articles where I say, “Everybody can be a researcher.” I truly believe that any time that you are in front of a customer, no matter who you are at the company at Zendesk. So let’s just say Zendesk, I’m just talking about Zendesk right now. Any employee, no matter your title, anytime you are in front of a customer, you can be a researcher. You have to be a researcher, really. Because that’s how you learn what that customer is trying to convey to you. And I was so proud of us and the guidance that we gave them. But in watching how we interacted with customers, you would have thought it was a team full of researchers. And I’m not going to take all the credit for it, but because I think we were just working with, I have an incredible team, teammates, designers and product managers at the company. In order to have a really solid conversation with a user or a customer, in our case, by and large, all you have to do is really listen and ask why and how. And I think, you know what I’m talking about, Steve. You ask those questions that can really help move the conversation along so that you can get to the understanding of what that customer’s need may be. Right.
Steve: It’s simple and easy and really hard all at the same time. Maybe that’s why as a field we have kind of fraught conversations about this. Everyone is a researcher is, I mean, it’s a good hot take because I think it’s a distressing comment and what do we mean by research? What do we mean by researcher? I think you’ve provided the context, but that little pull quote, and I promise not to just pull that out of context on you. But that whole quote, it distresses people.
Reggie: Yeah.
Steve: But I think you’re framing it in a really positive way. And you’re right, it makes me wonder, and I am certainly guilty of this, collecting stories of when we see things go wrong. I mean, I like those because they’re good learning moments, but
Reggie: Absolutely.
Steve: I think there is a tendency or risk of focusing on this narrative that this is hard and people can’t do it. But when you just look on the other side of the mirror like I think you’re doing and saying like, this is hard and people can do it. It unlocks a lot more potential for the outcomes that we all want, the kind of information that we’re looking for.
Reggie: If I can say one more thing about this. What I’ve enjoyed about this process of enabling our cross-functional partners to do research, empowering them to do it, is that it makes them better product managers. It makes them better designers because now they have the customer in mind. And I can’t tell you how many organizations or teams I’ve been on where that wasn’t the case. And I’ve had to kind of sort of jump in the middle of conversation and say, “Hey, wait, has anybody asked the customer what they think about what we’re doing here?” And so I love that the organization that I’m in are thirsty for these opportunities to, number one, talk to customers, but number two, to learn from a team of expert researchers on how to do it. Because in the end, it’s going to make them better.
And I feel very strongly about that, that, hey, we’re not where we want to be, but I think our entire organization is on a really nice journey up the maturity curve on becoming more customer first and understanding the customer better. And these opportunities like the Relate event and just other projects that we’re conducting internally, I think are really good ways for our teammates across functions to learn, grow in their own careers and to be better at understanding how to gather information to make more informed decisions about what we are creating, building and shipping for our customers.
Steve: Maybe I can use that as a segue to go back to something you said quite early on in our conversation. One of the areas where you’ve been pushing to build that maturity, and you talked about shifting where research is focused from kind of evaluative late in the product cycle to more strategic and other things that are taking place earlier in the cycle. And you talked about kind of realigning. And you said a little bit at a high level about how OKRs were kind of the unlock, I guess, to kind of get in there. And because I think you’re describing a very common challenge that maybe less mature organizations face or research leaders face or research teams of one.
I’m only allowed to do this work. No one wants this other work for me. I keep recommending it. And it’s sort of that I think some people are kind of stuck in these little traps where they can’t change the perception of what value they can bring. And therefore, they can’t get the opportunities to bring that value. So could you unpack a little bit more about how you changed what the opportunity was for research at Zendesk?
Reggie: There were a couple of things that I did. First of all, the first six months, I just was assessing what our relationships were like with our key stakeholders. And I came to find out that, you know, while we had some syncs with them, they weren’t necessarily on a cadence. When I say syncs, one-on-one meetings with our design leaders and product leaders and engineering leaders. So, number one, we need to set up a consistent cadence where we are connecting with the leadership of these particular work streams on a consistent basis.
Then we would put ourselves, make sure that we are positioned appropriately in the important conversations that are happening earlier in the product development lifecycle. When the customer problem is being talked about, or what we think are the customer problems, researchers who are in that moment, and they’re in those meetings early on, are able to help that conversation understand, well, how do we know that this is a problem, number one? And I think it was very important at the very beginning to establish a clear and consistent cadence of syncing with other cross-functional stakeholders. So that was the very first thing, and it was developing that relationship. So I think the relationship that we had, and I might be exaggerating, was we have an idea, it’s pretty baked, research, come on in. Let’s tell you about this idea. Now let’s go set up a research program to test it. I’m not saying all the work was like that, but that’s what I observed. I said, we got to fix that. So that was the first, I think, thing that I really tried to do was develop a closer relationship.
Steve: So a very sort of tactical thing, you’re setting up a cadence of one-on-one relationships with the people in these other parts of the organization. And one of the outcomes there is to get into those meetings where a researcher can identify assumptions and challenges and so on. Are those meetings where those conversations are happening, are those literally the one-on-ones you’re setting up?
Reggie: Well, they were the workstream leads meetings. So there’s scrum teams all over, and they’re small, medium, and large sizes. But there were meetings where the design leadership, the product manager, the product marketing manager, maybe there was a data science leader. That core team that is working on that thing, I wanted to make sure that not only were we in those meetings at a consistent cadence, but that we were developing one-on-one relationships with everybody in that group. So the relationship is one thing where you’re doing a one-on-one, and you’re just learning about that person, and you’re really understanding what that person needs and wants.
But from a team level, now you are in the thick of it, and you are in those conversations, like you said, where the customer problem is talked about from the very start. And in addition to that, we installed some protocols or exercises or activities where the researcher will facilitate a conversation around what are the potential customer problems. We call it I Wish I Knew. And this was something that I learned about when I was working at the company formerly known as Twitter. And it was an exercise. It was essentially a brainstorming exercise.
And when I first joined Zendesk, it was right around May and June. And then I guess right around September, October, we started annual planning for the next year. And that was a perfect time to install this activity. So we piloted it in a couple of work streams. This was a moment where the researcher could go into these meetings and say, “This meeting, we’re going to do an IWIK exercise, I Wish I Knew exercise, where as we start thinking about annual planning and the products that we were thinking about building for our customers, what are the questions outstanding that we still don’t have an answer to or areas that we still need to explore?” And so we’d go through that exercise, and then we’d prioritize that list that they would come up with.
I can’t tell you how valuable those exercises were and how our stakeholders looked at us and said, “Wow, I did not know that research could add this kind of value to our conversation,” because it really helped them see. Because especially once you get into the prioritization work, you look and you go, “That question that we’ve been battling around in these meetings isn’t really the one that’s most important. It’s this one.” And to see it all together was sort of a revelation for some of our stakeholders. So when we installed that exercise, it changed the game a bit in terms of our relationship and our position or brand inside the minds of our cross-functional partners, because now we’re providing a value at the very beginning of the product development lifecycle that is helping establish, “Okay, so these are the problems that we need to be worrying about as we think about planning for the next year.” Can’t tell you how important that was.
And then, so if I think about the next part of this was as the company started looking at their annual plan and defining these big buckets of areas that we wanted to work in. So for example, intelligent conversations is sort of one of them, just throwing that out there. And this is all of our sort of artificial intelligence work. And we think about within that big bucket, there’s like maybe, I don’t know, I’m just picking this up, there’s 15 projects or 15 streams of work that need to happen. And we strategically aligned a researcher to that category of work to work with those teams to figure out, well, where can research lean into the most to have the most impact in that work stream that laddered up to that OKR. And when a researcher is in that position and they’re reporting out research, and at the very first couple of slides, this is the research project that is designed to learn 1, 2, 3, A, B, C that will help inform OKR XYZ. That is how you operate. That is how you become a valuable research team. It’s not perfect yet. I think we still have room to grow and get better at it. But over the first couple of years, I think we’re doing a really good job at setting up a structure where we are able to add that kind of value to our product development organization.
Steve: Can I play back some of what I heard?
Reggie: Yes.
Steve: Because I think this is a universal complaint and concern, right? Not getting access, not getting permission. You talked about the individual relationships with people and as part of that, being in those meetings, being in those team meetings. I think it’s on a case-by-case basis, right? How do you get invited to that? But there’s an audacity a little bit of showing up and saying, “Hey, we’re here to be part of this meeting,” then making a contribution. But I think maybe biding your time a little bit. I’m now putting words in your mouth. But building the relationships, being in the meetings, hearing what’s happening, and then choosing a point at which to bring value. And the value that you’re bringing there is facilitative of others. It’s not, “Here’s what we know. Oh, we have the answers.” But just, “Let’s help you talk about what you have to decide and learn.”
Reggie: You’re onto something. So yes, we’re doing that. But in addition, we are bringing some answers to that meeting because one of the other things I did in the first year is we established a research and insights library, a research repository. So we aggregated all the UX research that had been done to date at the company, and we put it in our confluence site, and we set up a nice searching function. So literally every research project that we’ve done that we could find is there. And so in those meetings, when we are having conversations about different customer problems that may not have been answered, we may be able to answer some of them because we now can point to this library and we say, “Hey, wait a second. A couple of years ago, we explored this problem at this angle. So the question that you’re asking, we’ve already covered that piece of it, but maybe we can explore it in this way, in a new way.” And so I don’t know if I’m countering what your assertion was earlier, but I’m basically saying yes, we are able to bring some answers to that meeting for the purpose of helping triage the priority or prioritize the list of questions that come out in the brainstorming of the I Wish I Knew brainstorm session.
Steve: So you are countering it, which was the outcome of me doing, like, “Here’s what I think I heard” question for you. And then I guess the thing that I took away from you sort of describing that presentation where a researcher is part of a work stream and says, “We learned these things. It affects these projects.” I think you said, like, in the first couple of slides, we talk about things like oh, speaking the language of business. Sometimes people talk about you got to know, like, the business model of the company. But I think you’re giving a very actionable definition of what it means to speak the language of business, which is to know what initiatives or what projects or what OKRs to be able to frame the information you’re bringing in terms of what things it ties to that are already what people are concerned about. So doing that translation, so, like, why this is important.
Reggie: Absolutely, because it’s going to ladder up and inform this OKR. Now what I want to mention is this is how we set up our structure to help the researchers align directly with valuable work and strategic work that they can do. But in addition to that, we are also pushing ourselves to even look beyond that specific work and look horizontally. And that has really galvanized our team.
For example, I mean, adoption. If you’re at a B2B company, adoption is a huge priority. So not only are we doing these discrete projects that are laddering up to the OKRs, we’re also stepping back and looking across the organization and saying, okay, well, out of all the projects that we’re working on, what are we learning about customer adoption in all of them? And how can we now inform how the company is thinking about how we help our customers adopt our product and extend it and upgrade, add more? Like what do they need? And so I’m so excited by this work that we’ve recently done around adoption. And we’re seeing a lot of our cross-functional partners leaning in on it and asking us for it. And we’re trying to do a little bit more. But the reason why I’m mentioning this, I just want to sort of set the structure of how we’re maturing as a team is that, yes, we’re aligned to these OKRs.
And it’s actually freeing us up to do the strategic work, but also to pause, step back a little bit and even look higher than that. And I just can’t wait to see what we do for the rest of this year and next.
Steve: What’s the mechanism for different researchers working on different projects? They have different whatever, it’s documents, it’s in their heads. How do you collectively find those overarching issues and kind of pull them together?
Reggie: You know, with the adoption work, it kind of organically happened. We do have check-ins every quarter where our research team, we get together and we do what we call research readouts for that quarter. And this is where a researcher will share all the projects that they’re working on, what they intend to learn. Maybe some of the projects are already in flight, so they’ll give a status update. And in that moment, the purpose of those meetings is to find opportunities where we can collaborate and work together on different projects that are similar. And organically, that’s when these opportunities surface. And I think that’s what happened with the adoption work. There were several projects around adoption. And oftentimes, even with a team as small as ours, that you may not know everything that your research colleague is working on. And that’s why we do these quarterly syncs where they read out what they’re working on. It’s super-duper helpful for us being able to do this.
And that’s what happened. It just sort of organically happened. But I think we’re trying to be more intentional on finding opportunities like that, where out of all the streams of research that we’re conducting, what are we learning that is sort of a global insight that is meaty enough that we can highlight it as a thing that the company needs to lean in on and care about? That to me is where value is from a UX research team, even as small, medium, and large, no matter what size you– if you’re doing that on a consistent basis, senior leadership, executive leadership, they pay attention. And I’ve seen it. And I think what insight has done to us, we’re starting to gain this traction and awareness that we’re doing this kind of work. And I love it. And I encourage anyone who’s listening to this podcast to start thinking in this way, because you can get so buried into the day-to-day research. And you really want to add value to that work stream. And you really want to do well and be successful. But you also have to look ahead. And you also have to look beyond it a little bit. And when you do so, you can find opportunities like that to tackle issues that you may not see in just one particular, one discrete project.
Steve: And that’s the way it is, at least for part one. Stay tuned to this donut channel for part two with Reggie coming soon. A reminder that you can always find Dollars to Donuts at all the podcasty places. Plus, you can visit portigal.com/podcast for all of the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 44. Reggie Murphy of Zendesk (part 1) first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I catch up with Leanne Waldal, five years after she first appeared on the podcast. She’s now a Principal in User Experience at ADP.
A couple of years ago, I realized I know things. We all know things, but sometimes we go through life thinking there’s always something more for us to know, or we don’t know as much as others. A couple of years ago I was like, oh, I know some stuff. I could share it. If I think of myself at 23, 24 years old, I had people who were my age now who were telling me things that I listened to and got advice from. I’m that person now. I can be the person who like gives people advice or says, I don’t actually know everything, but here’s some things I learned over the years that might help you. It makes me feel good to do that. It boosts my confidence. It helps me feel like I can actually do something that’s not just my craft or not just my job for a paycheck or not just this, but I actually have something to offer. And that’s a great feeling. – Leanne Waldal
Show Links* Interviewing Users, second edition * Beyond The Surface: Navigating The Depths Of User Research With Steve Portigal (Greenbook Podcast) * Leanne Waldal on Dollars to Donuts, 2019 * Leanne on LinkedIn * ADP * Badass: Making Users Awesome by Kathy Sierra * Dropbox * Orbiting the Giant Hairball: A Corporate Fool’s Guide to Surviving with Grace by Gordon MacKenzie * Organic Online * Howard Rheingold * NASDAQ crash, 2000 * San Francisco 2004 same-sex weddings
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal. In this episode I catch up with Leanne Waldal, 5 years after she was first on Dollars to Donuts.
But before we get to that, I updated my classic book, Interviewing Users, to incorporate what I’ve learned in 10 more years of being a researcher and in teaching other people. After the book came out, I spoke about user research with Karen Lynch on the Greenbook podcast. Here’s a short clip.
Karen Lynch: I mean, there’s even some frameworks for creating a knowledge management platform for yourself, you know, how to have a database of your own research. So excellent applications for a smaller shop that might not have access to platforms and tools. But here’s a way you can kind of create your own hub, knowledge hub. You did a very—a good job, solid job, an important job of also providing different—you know, here are some forms. You gave structure to the field is what you did. You know, here’s some forms that you can look at for, you know, debriefing your interviews after you conduct them. Here are some kind of, like—I don’t want to call them templates. But here’s the framework that you can do for creating your discussion guide. Here’s some tools you can lose—use to synthesize your data. So you were—you’ve given some very tangible tools in this book for anybody who is really trying soup to nuts to go off on their own for the first time or just get to know the field that maybe they’ve been hired in, really practical, tangible things that researchers can borrow from. I mean, again, having been in the field for a long time, some of this was—you know, some of this was a part of my practice. But I’m like, “You know what? That’s a great debrief form. That one really stood out to me.” For example, Interview Debrief Form, where it’s not just take notes on what you just experienced, but it’s prompting your brain to think through what it means. So kudos to you on that. Is that just a practice of yours? Or did something kind of stimulate that thought that you’d might want to include those?
Steve: Well, you know, there’s this interesting part of research where it’s collaborative and facilitative. I mean, it’s not just what, you know, I, as the researcher, or me, as part of the research team, learns. It’s, you know, the people that we’re working with. And so I have an obligation to them. Boy, that sounds like—it sounds very—more moralistic than I mean. Like, I can do a better job if I can help them learn something and take something away. But, if I also—if I hear what they’re taking away, especially—I’m, you know, I’m not the domain expert. I work as a consultant, so I come into an area that somebody else inhabits. And so they’re going to always see things in the research that I won’t see. It’s really helpful for me to understand what didn’t they hear that person say. Like, if there’s a gap in what they took away, then I now know I need to kind of emphasis that because it’s—there’s a takeaway that’s obvious to me that isn’t to them. So that needs to be surfaced as I share back. So I can get that out a debrief. And, when I hear what they heard and what surprises them, I understand, yeah, how they’re framing the world, what’s relevant information. Like, I’m getting this indirect feedback. So, you know, I—like, it’s not my natural way of being to have a template for an activity. I’d rather just chat. And sometimes that suits me well, and sometimes I need to put a little more structure in it. So, I think, you know, writing up a debriefing guide—well, I think there’s—having something formal like a template or a tool you can use sort of reminds me that this is an important part of the process. I need to make time for it and mental space for it, and I need to tell the people I’m collaborating with that they should leave time for it. And guess what? This is a serious activity. I don’t—I’m not just trying to, like, get coffee with you and, “Hey… what did you think?” I’ve got a document here. So there’s a little bit of a theater. And I don’t mean that in an unkind way. But there’s a little bit of a, you know, a formality to it that reminds me to take it seriously and that shows my collaborators that I value what they have to say and that I, you know, I’ve got some format for that.
That’s from the Greenbook podcast. I’ll put the whole episode in the show notes. And I hope you’ll check out this updated edition of Interviewing Users, and share it with all of the people in your network. You can help me and help the book by writing even a tiny review on Amazon.
So, let’s get to my conversation with Leanne. She’s now a Principal in user experience at ADP.
Well, Leanne, thank you for coming back to Dollars to Donuts after, I guess, five years
Leanne Waldal: Good to be here.
Steve: or so. It’s nice to talk to you again. It’s nice to talk to you again for this podcast. That introduction presumes that you and I have not spoken in the intervening five years,
Leanne: We have.
Steve: but we have.
Leanne: Yes, indeed.
Steve: We have. The secret is out. Let’s talk about some of the things that you’ve been up to professionally in the last five or so years. We can kind of start anywhere and go anywhere, but I think that’s one of the things we’d like to cover and just hear what some of your experiences have been and what you’re thinking about nowadays with user research.
Leanne: Yeah, and the wonderful thing about the last five years is it was about four years ago that the pandemic started. And when we spoke about five years ago, it was probably six months before this pandemic started and it changed research. So I went and worked for a company and helped them understand using ethnographic research in Germany and France, what was going on with the markets that they were trying to sell into their help teams really understand what they needed to do to better sell into those markets.
And then I joined an agency. And at the beginning of 2020, I was selling a project to a company with my teammates in the San Francisco office. And we had planned out like many of us in early 2020, all the things that we were going to do in this project over the course of 2020. And then of course, the pandemic happened and we all went home. And so we had to shift the type of research we were doing from going out and talking to people in person and seeing them in person to doing everything over video and then also running some surveys.
One interesting thing that happened in the summer of 2020 is humanity got tired of answering surveys. So I was doing mixed method research in the summer of 2020 with interviews with people about their experiences. And then I also wanted from those experiences to measure them at scale with a survey to find out, well, if I talk to 20 people and then I take these things and find out, you know, how do a thousand people respond to these experiences? It was a really easy target. It was millennials in California. It took like 35 days to get a thousand responses. And that was fascinating to me because it was really easy for us at that time to get people to talk to us on video. But I talked to the panel provider we were using at that time and they said, people are tired of answering surveys. This pandemic is wearing people out. We have all sorts of personal things we’re dealing with and we’re also terrified the world’s about to end.
And so then in 2021, before the vaccines came out, we were doing some work with a startup at the agency I was at. And I realized that some of their early like Kickstarter supporters were in the San Francisco area. And I said, this is about motorcycles. Motorcycles are used outside. We could actually meet people in person. Like that’s the thing I miss the most. I’m an extrovert. For all the people who are introverts, the pandemic was just wonderful for them to a certain extent. Like lots of people I know who are introverts were happy to work from home and happy to be at home. Me, I miss people.
And so I started up ethnographic research again in 2021 by meeting motorcyclists in a park. Got some great pictures of all of us in masks standing about eight feet apart from each other handing out water bottles and shouting across the grass to each other. When I left the agency, I went to work for a startup and introduced again ethnographic research to them because they had a mobile app but hadn’t really watched people use it out in the real world. And so I started just in the San Francisco Bay area to just run a pilot, like let’s go get some people using this and see how they use it. And then got the funding to go visit some customers and visit some consumers and some other people.
So I was tromping around in the snow in Minnesota in like negative degrees following somebody who was using our app also tromping through the snow, which was super fun. And when that startup laid me off, I did some consulting. I did some consulting for a large company, also doing some ethnographic research, going into some offices of the people who use their products to show the pain points. And also then started what I started doing, which I hadn’t done in a while, which was really fun was looking qualitatively at the pain points that were happening and then going to the people who have numbers from revenue and numbers from usage analytics and figure out how much money is being lost because of the user experience or how much money could be gained from cleaning up some of the user experience.
And then I joined ADP in June of last year as a principal in user experience. And I’m leading all sorts of research, including recently some ethnographic research again with one of our clients, which is super fun, teaching people, teaching people and teaching people.
Steve: I think we like to forget things that happened during the pandemic because, as you said, people were worried that everything was going to end. So that slow survey response or slow and low survey response, do you know, has that rebounded in the time since then?
Leanne: It definitely has. So by the time I was at a startup that I was at, I think we did some ethnographic research, some interview research, and then we had like, you know, six key things that we wanted to understand from a certain consumer population. I think we got all the response we needed in like 11 days. So from what I’ve seen in that sort of consumer research and mapping survey research to qualitative research, we’ve gone back to getting people to respond to it more quickly. But if you remember the summer of 2020, we were terrified. Why would you even if you’re offered a gift card to answer a 10 question survey, like spend time on that when you’re trying to take care of your kid, dogs and yourself?
Steve: We were terrified and we also felt, I think, again, it’s hard to remember with any clarity, but there was this sense of trying to preserve some aspects of normalcy, like family and work and so on. But you could see why people would want to run a survey because that’s part of their job and they want to continue feeling normal, but it’s a really, it’s an interesting observation that people didn’t want to respond to surveys because of what everything that was going on.
Leanne: Yeah. And as researchers, we lost some of our ability to understand people’s experiences because of the pandemic and being locked down. So we were limited to diary studies that are about things like surveys and interviews.
Steve: I remember having a lot of, we had a lot of complicated feelings about a lot of things, but watching that period where research was starting to pivot to remote and I don’t know, I felt like there was a certain gleefulness in being able to guide that. I don’t want to make it like a Coke versus Pepsi thing, but it seems like there is a remote versus in-person belief system or, you know, adherence and that when in-person became impossible, the folks that could give guidance and best practices on remote, it was hard to not see that guidance being offered because it was everywhere.
Leanne: Oh, of course. It was amazing at that time for people who didn’t know how to do remote. Like the agency I was at prided itself on doing contextual inquiry all the time. So teaching people how to do things more remote, I always like to mix things up and tell people like you can’t do all remote or all contextual or all quant. You need to mix it up to understand a human experience.
Steve: But that doesn’t work with our desire to make a single declarative, this is the best way to do it, Leanne, what do you mean, what do you mean mix it up?
Leanne: Oh, we all combine complexities in our experiences and how we do things. I find it really hard to stick just in one camp.
Steve: And I think it’s interesting, you talk about being an extrovert kind of driving you to, if I understood, yes, extroverts suffered from lockdown, but that as a researcher who’s also extroverted, you really were, I think, creative in trying to find ways to do some portion of context and kind of get yourself out there and get to where people were.
Leanne: Yeah, I felt really lucky to be working for this agency in San Francisco where we had an office where the windows open. So we were coming back to fully masked with all the windows open, standing apart from each other just to see each other. Do something on a whiteboard together, sit in a room and talk together just to, because you can’t just like one thing about the difference between desk research remotely and seeing someone in person is you can’t see the Post-it notes around the monitor. We all got Post-it notes around our monitor. We all got things, you know, I’ve got stuff here today to remind us of what to do.
And if you have a workplace, you’ve also decorated your space. So, you know, we’re decorator crabs, basically. We bring in pictures of our family and someone brings us back, you know, a token from some trip and we put it on our desk. And you know, the team all goes out and does something fun together and we put a picture on our bulletin board and you can’t see that. And recently something interesting I noticed was that, which I hadn’t thought of before is that with this remote, like you and I just look at each other, you know, on video, if I shared my screen with you, you can only see one screen at a time. Well, a lot of people work with two or three screens now. So how do you in a remote world see how they’re comparing things across two screens? Like they could, I could share one screen with you and then another, but you couldn’t actually see my experience of, you know, comparing this Word doc with this Word doc. So that fascinated me. I was like, Oh, amazing. The world has also changed that monitors are cheap and we can have multiple now.
Steve: I would have to ask whether or not you had multiple monitors and ask to see them.
Leanne: Remotely, you might just assume your screen is your screen. Like think about if you’re doing research with engineers, you know, you and I are both of a certain age. So you remember when we only had one monitor and it was like a big thing, the two of them on a desk, there’s no way you could have two monitors. Well, now if you go look at any software developer who’s working, they’ve got a couple of monitors or three monitors. They’ve got curved monitors. You know, we just sort of surround ourselves with all this stuff to look at. And you can only see that if you go see people in person, we don’t yet have a way with any of zoom or any other video technology, be able to see across all of those monitors all at once. So I think that’s an interesting point of innovation for these video companies.
Steve: And we’re still talking about research where the thing we would be looking at would be something that takes place on a screen.
Leanne: So I want to be able to see what you’re doing. Also, you know, motorcyclists can’t see their motorcycle. If I talk to you on video, maybe you can bring like your phone or something out to show me your motorcycle, but I can’t get that giddy experience you have about talking about it person.
Steve: Sorry, did you say the giddy experience?
Leanne: Yeah!
Steve: Who’s giddy in that example?
Leanne: The motorcyclists. Motorcyclists feel very, very strongly about their motorcycle. And I also noticed, because I’d been doing in the consulting gig I did before where I’m working now, I was doing some remote interviews. And then where I’m working now, I do remote interviews and go out and talk with people. And I’m noticing this distinct difference between how vulnerable someone will be with me and how much they’ll share with me in person versus on a screen. And that, if you think about it, just makes human sense. We can sense each other better and share things better when we’re in person than when we’re on a screen.
Steve: When you and I did the previous episode of this podcast, I came to your office and we sat in a room holding microphones to do it.
Leanne: Yeah, exactly. Yeah.
Steve: Technically, I guess we could have done that again, but I think we rely on this technology. We’re using remote screen-based recording, blah, blah, blah, that has become the default, even when it’s not the only option.
Leanne: Yeah, and I think it’s fine. This gives us opportunities to talk to people we can never talk to. But I think we have to make sure we remember the importance of doing things in person. Humans are people who sense things off of each other, and it’s important to make sure that’s there in the mix with everything we’re doing.
Steve: And you set up off the top to teaching and you talked about going with your colleagues
Leanne: Mm-hmm.
Steve: and teammates and clients and so on into the field. We have a number of different parties that have a different experience.
Leanne: Mm-hmm. Mm-hmm.
Steve: So the motorcyclist who’s giddy, us the researcher, and I mean, I’ll just say like,
Leanne: Mm-hmm.
Steve: I need to be in the room with that giddy person. So I need something that I’m missing from the remote. But then there’s also the people that you’re taking out that you want to give them a sense
Leanne: Mm-hmm.
Steve: of their customers and their users. What’s been your experience over these few years in trying to do that for your clients and stakeholders?
Leanne: Well, for that sort of work, bringing marketers, salespeople, engineers, product managers, designers, et cetera, out into the field for consumers or customers, B2B or B2C or B2B2C, it never fails to surprise them. I’ve been doing this for a long time, and it’s the same thing over and over. And I love that sort of teaching moment of bringing someone out to an experience and just saying, I’m going to run the camera. If you don’t know what to say, I have no problem keeping the conversation going and saying, tell me more, tell me more, show me more. What about that? You get to just observe, or you can participate. It’s all up to you. There’s no rules, really. And I’ve never had someone say that they didn’t or weren’t surprised by something that they saw or they heard or observed. And usually, the things people are most surprised by are the environments that people are in. Because if you only see me in this room, like right now, I’m in a phone booth in a WeWork. I probably look like I’m in a sauna. You have no idea what’s going on around me. And so to see the vast experience of someone’s house or of someone’s business or someone’s office or following them along while they do something out in the world, usually people I’m with are as surprised by that as they are by the things in the product or the service or whatever it is that trying to understand the experience of. Well, for B2B, when I’ve worked in B2B,
Steve: What is it that these folks are learning from these experiences?
Leanne: I’m trying to help people understand the humanity of the people who are using the product. That it’s not just someone using a credit card to pay for a subscription, or it’s not just the buyer and their team who are at work and using your product in their work. It’s not just a job. There’s a human behind all that who’s got needs and has got friends at work or has colleagues or people that influence or don’t, or they have other products they’re using around your products that you can’t see from your usage analytics. So I’m trying to get them to see the giddiness of the motorcyclist, not just like, “I am this demographic and I’ve got this much money to buy this kind of product,” but look at how they feel about it. That’s what brand campaigns and marketing campaigns do. They get into our feelings. And so I try to get the product side of a company to understand that piece, that you can get more engagement in your product.
You can get people to use it more if you get them to feel good about it. It’s like Kathy Sierra’s book, “Make Your Users Badass,” or something. I probably just mangled that. Something like that. You want your users and your customers to talk about you at a cocktail party after two drinks. You want them to remember you so well. And then that Maya Angelou quote about, “I don’t remember what you said or did. I remember how I felt.” That’s the human. And so in companies, marketers know that. Brand people know that. I try to help product and design people remember that because they know that too. We just don’t remember that when we’re focused so much on usage analytics. It’s like, yeah, usage analytics and revenue analytics are important, but they aren’t important in a vacuum. You have to also make people feel good and feel smart when they’re using your product. Yeah.
Steve: Have you seen any shift in the appetite for the kind of, you know, understanding that you’re enabling people to gain?
Leanne: Research and design appetite has gone up and down in the tech industry over the past 20, 30 years that I’ve been in it. Right now, there’s a lot of people out there saying nobody has appetite for us. I’m like, well, maybe they have an appetite for a certain type of work that we do. We don’t have an appetite for all the work that we do. And that appetite ebbs and flows, just like the appetite for anything else. It’s just humans and trends and businesses and business decisions. And my advice to everybody has always been like, you can create an appetite for what you provide. It’s a sales technique. Understand what somebody needs, meet them where they are. Instead of being like, here are the five things I do. Here’s the checklist, sort of like McDonald’s, like here are the things you can order from me. Oh, you don’t want any of these? And instead say, what are you looking for? What are you trying to achieve? What are you doing? How can I help?
Steve: I want to clarify, or I’m trying to think about this as a question and not a statement, but I might just go with a statement. There’s a difference between, you know, when you say, find out what people need and then help them achieve it, that doesn’t mean if they don’t ask for contextual research, you don’t do contextual research. And maybe this is where the McDonald’s thing breaks down for me, or needs like another metaphor layer on top of the metaphor. Because if you ask what they need, no one needs research, they need the information or the decision or whatever they’re going to do about it. And so you have a lot of ways to get to that outcome. But if you say to them, hey, I do research, to your point, right, if you say, hey, I do research, do you want research? No, I don’t.
Leanne: Yeah, exactly. Or also like waiting for someone to ask you for a certain kind of research. When someone who doesn’t know the breadth and the depth of what you can do with research, asks you for something, they might not know what they need or want. They’re only asking based on their own knowledge. So I always find it’s better to figure out what kind of research to do or how to prioritize it or where to be by hanging out with people and finding out, say, what are you working on? What’s coming up? What do we know? What do we not know? Here’s these analytics. Do we know why these analytics look this way? Well, we could find that out this way, or we could do this this way, and see which one of those things that I start proposing to them based on what I heard them say they’re working on or what they’re trying to achieve, sort of like sparks a conversation. And then research is desired and wanted and valued and invited to the table. But I see a lot of researchers, and I coach researchers on this, not going in with like, I’m going to run a survey on this, or we’re going to do some interviews, or I’m going to do this unmoderated. Nobody knows what unmoderated research is outside of the research community. So stay away from the methodology. Stay away from like, I’m going to do this and stick with like, I’m your partner and I want to help you out
Steve: The labels that are always anchored in my mind is the difference between proactive and reactive. And you’re talking about being proactive. And even I get a picture as you kind of talk that there’s a relationship and there’s something over time. And I kind of hang out with them and see what they’re thinking about and what they’re talking about. That is not a — this is sales, it’s not a sales call, it’s a relationship-based sale.
Leanne: Yeah. And that’s why, as you and I have talked before, I don’t want to
go back to run a consulting firm. I don’t want to be a consultant unless I have to, for, you know, to get revenue in my life. I enjoy being a part of a team to like, nurture relationships and get collaborations going and work with people so we can do something together, because I don’t like being a lone wolf. I don’t like being a single point of failure. And, you know, I want to do things together.
Steve: In some of the conversations we’ve had where you’ve said — made these comments before about, you know, the relationships, I get the sense, and you can correct me here, those relationships aren’t handed to you. You’re seeking them out.
Leanne: The startup I last worked for, after I got laid off, customers of that startup were still texting me up to like, three or four months after I left the company, because they didn’t know I’d left the company. Every time they text me, I’d have to let them know. But I’d gotten such a good relationship with them that was beyond just like, I want to know how you use this product. It was like, I want to know how to use this product. And how did you start this business? And, and tell me about your family and this little town you live in. And where’s the best place to get lunch? And, you know, I would get invited over for dinner with their families. And you want that sort of relationship, which is very much like a sales tactic, because that’s what people who sell want. They want to get under your skin a little bit to um, understand you better, because that helps them sell to something that you need. And that’s all about relationships and collaboration.
Steve: What things, if any, are different in the sales relationships and collaboration that you want to nurture with users, customers, versus the colleagues, the people that you work with that you’re also wanting to do collaboration with?
Leanne: Well, you have a different goal when you’re a salesperson. You’re trying to hit a number. You’re trying to get someone to buy something. My goal with anybody is, I want to help you out. I want to see if there’s something you need that I can help you with. I want to see if, you know, you know, the types of skills that I have and the types of things I can do and the craft I have can help you. And if it can’t, I’m not going to, I’m not going to waste my time, you know, doing something if it’s not going to help you. I don’t know that all researchers or research teams see themselves as a service organization, but we basically are a service organization. We’re providing a service to and with people who hopefully will use and are interested in using the insights and the learnings they can get with us, from us, to build product, to design things, to market something, to sell something.
Steve: Hearing you talk is really refreshing for me because I think it’s easy for us, or for me at least personally, to get sucked into all of this as adversarial. I don’t know, we hear a lot of stories from each other, you know, in this work, we tell a lot of adversarial stories, persuading, getting permission, convincing.
Leanne: Do you pay attention to the subreddits on research? Because it’s full of that. Yeah. I just watch it and read it.
Steve: And sales, I mean, being a consultant, you know, I have lots of peers where we talk about sales, and the more — even though I’ve been doing this for a long time, I still keep learning and relearning that sales is not a persuasion adversarial kind of work. It is a — it’s all the things that you’re talking about, it’s relationships and how can I be of value to you? And that way of being is — I think you live that way, and so you work that way.
Leanne: Well, I learned it from running the consulting company. And then when I was at Dropbox and they were creating a sales team, I sort of like did a swap with the sales team. I was like, you teach me how to sell things because I actually want to know this more. Like I see, I see value in knowing these techniques. And, you know, you teach me that and give me access to people you’re selling to, and I’ll teach you about them. And that was a really valuable collaboration then. Because Dropbox was just getting into the enterprise. We were just starting to sell to larger customers. And I was working with all of the new enterprise account managers to basically learn what they did, but then also learn who they were selling to, and then feed that back to them to say, you know, I think you could sell this customer this, or this customer needs this.
Steve: What kind of things did you learn for yourself about sales from that?
Leanne: Well, that was when sort of like a chandelier went off. And I realized that like techniques we use in research to understand someone’s experience are the exact same thing sales people do. So a lot of researchers will be like, nobody does research like we do. Product managers don’t do it. They ask leading questions. Sales people are just trying to sell. And I’m like, hmm, I’m starting to see that. And this was back then. So about 10 years ago when I was like, oh, salespeople are actually doing a lot of research and product managers are actually doing a lot of research. Why does it matter how you ask the question? If what you’re trying to do is discover something or understand something, as long as you get to the end goal, the path you got there doesn’t really matter.
Steve: And did you learn anything about — again, you’re using sales in this very elevated way to describe how you work with colleagues, for example. Is this the point at which you started to develop those skills further?
Leanne: Yeah, that was the job where I grew up. I had run my own consulting firm for 17 years. So from when I was in my mid to late 20s until I was in my early 40s. And when I took my first job after that long consulting period at Dropbox, I was like, oh, I’m going to learn how to be an adult inside of a corporation now. It doesn’t matter what age you are when you learn how to do that. So it went through that learning curve there.
Steve: The idea of growing up is really — that’s a big one.
Leanne: Well, I don’t think we all ever grow up. We just grow up in certain ways. It’s like, oh, now I know how to behave professionally. I didn’t know how to behave professionally before.
Steve: I mean, you ran a consultancy for 17 years, so you at least were able to survive in that time.
Leanne: But you know this. It’s a different sort of professional to service clients and manage clients and sell things than to learn how to behave and how to manage the politics and the relationships inside of a corporation. There’s this really great book called Orbiting the Giant Hairball, which I go back to every once in a while, because it’s just all about the humanity in these large corporations or in these mid-sized startups and how do people get along and work together and make decisions and collaborate and get things done. It’s very different from running your own thing.
Steve: You mentioned just offhand, I think, that you talk to researchers and people are reaching out to you for advice, and I’d just love to hear kind of what that’s like, and I don’t know what’s coming up for people right now that you’re talking to them about.
Leanne: So what’s coming up is questions about how to, particularly for researchers early in their career who maybe started doing research in 2020, they’ve never done contextual inquiry. They’ve never done ethnographic research. They’re working at a company where they don’t have a manager or a leader who’s encouraged them to do that. They started remote. A lot of these people went to college remote, graduated in 2020 or 2021, got a job as an early career researcher, and they’ve just never done this. And so I basically start with the basics, like here’s how you observe someone. The other thing I noticed, and I also talked with some people recently about what they’re seeing of people who are earlier in their career who went to school during the pandemic and now are coming out, that it’s the same thing that those of us who are older went through in the pandemic.
Leanne: We lost a certain amount of social ability. And most of us, or a lot of us, got that back in the
Leanne: last year or two. We started meeting people in person again. We started going to dinner parties. We started going out to bars, went to concerts. Some of us went back to offices and figured out how to be in an office again. But there’s a lot of fear of, “I don’t know how to do this, and I’m scared to do this, and I’ve never done this before,” that a lot of it I attribute to the pandemic. And I think those of us who do mentoring and coaching need to be aware of that and teach people how to do that. I know someone who is teaching people how to dress to go to an office, how to wash your hair, how to have conversations with people. And that’s something that’s really specific to this point in time that didn’t exist before. So say, like, 2016, what was I mentoring and coaching people about? It was more relationship coaching. Like, how do you get along with a product manager who disagrees with your research results? Or how do you have influence? Or how do you learn a new skill? Or you’re a junior researcher, you want to become a senior researcher, what do you need to do to show that so that you can get that promotion? I’m not hearing that so much anymore. I’m hearing more around, well, one, how do I get a job? But then also, sort of like, how do I do these things that nobody ever taught me to do and I never had a chance to do before, and I want to try to do them now? Some of these people I’m coaching are the only researcher in a company. So they don’t even have someone to sort of like manage them or say, this is how we did it in the past, 2019 and earlier. So yeah, I just I like to help people out that way. I sort of feel like the things I can do right now are improve relationships among different teams. And I can also help out people who are trying to grow up in their career.
Steve: What else is coming up in these mentorship conversations you’re having?
Leanne: Presentation skills. So a lot of people come to me and say, I’ve been giving presentations, you know, over Zoom for years, and now I’m being asked to give a presentation in person. And I thought, how do you not know how to give a presentation in person? I’m like, oh, you’ve never worked in an office where you had to stand up in a room in front of a table of eight people with a pointer, you know, remember how we used to plug like a USB thing into our laptops, and then that we’d have a clicker and then presentation would show up on screen? Well, people are being asked to do that again. But that’s another soft skill that nobody’s taught them how to do. And one thing that is an advantage of presenting over video is you can have notes. You know, nobody can see that you’ve got your PowerPoint and like presenter view with your notes, or nobody can see that you’ve got like a notebook, you know, with like, oh, this is what I say on this slide. When you’re in a room, you’re on stage. And I think a lot, a lot, a lot of us forgot how to be on stage, who used to be on stage. But some people have never been on stage. And now they’re being asked to be on stage. And they didn’t get that practice at college, because you always presented your papers and everything over video or in a small classroom. I’m really glad that my daughter started college in person and is in college in person. So she’s getting that like those that like social growth that you need before you start to turn into a young adult trying to get yourself into the workforce.
Steve: It’s a fascinating observation that maybe is obvious to everyone, but I had never really thought of this. There’s a significant cohort of people in the workforce who have never, who don’t have a pre-pandemic norm to return to around research, presenting, business travel, any of those kinds of things.
Leanne: Yeah. Just like professional skills, how to get along with people. Yeah.
Steve: It’s interesting that at least the people that you’re in contact with have some awareness that they have a gap. Seems like that would be the first step to addressing it is knowing that it’s missing, that there’s a thing that is expected of you.
Leanne: Yeah. Yeah, there are a couple people I mentor who come to me in panics, like, I have to do this, and I’ve never done it before. And I’m like, oh, it’ll be okay.
Steve: Yeah, what do you tell someone who hasn’t done a business trip before, who hasn’t worked in an office? What’s the granularity of advice here?
Leanne: That even me in my early 50s, I still have new things I have to do every once in a while. And we can all do new things and hard things, and you will actually be okay. And then we can get down to brass tacks and go through the tactics. What, how do you pack? Or how do you like memorize things or practice? A lot of people who give presentations over video don’t practice first, because you’ve got all these supports around you to do it. You’re wearing your sweatpants and you put on a button down shirt, but you’ve got sweatpants on. You’ve got all your Post-it notes nearby of what to say. And so without all of those supports, what are you going to do? Well, you have to practice more. You have to think ahead. You have to plan, make lists if that’s your thing. And I think it’s really surprised people that they need to do that.
Steve: What’s the guidance, this is decontextualized, I guess, but what’s the guidance for a person who’s new to in-person research?
Leanne:Well, I say, you know, go find someone to bring
with you who’s done it before. So you aren’t, you and everybody else isn’t brand new to it all. It’s this, you know, it’s something like, you don’t all want to be new to doing in-person research on the group that’s going out to do it. See if you can find someone who’s done it before. Or find someone from marketing sales, who’s done something similar. Who’s at least like gone to visit customers or has gone to conferences and meet with customers. Your first pancake of the first time you do anything will always be a little rough. So it’s always good to have someone there who, you know, can advise you a little, or, you know, steer you in a slightly different direction when they see you going astray. And if you don’t have that, then, you know, just be gentle with yourself. Whatever you’re doing is the best you could possibly do.
Steve: That’s the lesson for everything, right? Let’s just be gentle for yourself. What do you get out of mentorship?
Leanne: Well, a couple of years ago, I realized I know things. And, you know, we all know things, but sometimes we go through life thinking there’s always something more for us to know, or we don’t know as much as others. And it was a couple of years ago when I was like, oh, I know some stuff. I could share it. You know, maybe it would provide some value to people who, you know, like if I think of myself at 23, 24 years old, I had people who were my age now who were telling me things that I listened to and got advice from. And it just sort of popped into my head. I was like, oh, I’m that person now. I can be the person who like gives people advice or says like, you know, I don’t actually know everything, but here’s some things I learned over the years that might help you. So, and it, it makes me feel good to do that. It boosts my confidence. It helps me feel like, oh, I can actually do something that’s not just my craft or not just my job for a paycheck or not just this, but like I actually have something to offer. And that’s a great feeling. Yeah. And then you’re sort of surprised, like, oh, I actually know how to do this.
Steve: I don’t know if this happens for you this way, but sometimes I don’t know what I know until I’m in a situation where I’m asked to help somebody out.
Leanne: Yeah. And other people will say to me, like, well, Leanne, we see you as an expert and everything. I’m like, yeah, but I don’t see myself that way. Yeah. Yeah. And for me, it meant I was taking it on in a way that felt comfortable for me. Like I don’t need to be or want to be the expert who, you know, gets on stage everywhere or has the big title or anything like that.
Steve: But to choose to be a mentor is to partially take on the role of the expert. The theme of some of your earlier points about creating good collaborations around research are understanding what somebody needs and how you can be helpful to them, and that you like to be helpful. And that seems to manifest in mentorship as well.
Leanne: But I do like to help people. Yeah. And that’s that being a helpful person is something that’s often attributed more to women than to men. And I used to when I was younger, sort of like sort of like push away things that were like, oh, that’s stereotypically female. And I’m like, OK, so what if I’m doing something that’s stereotypically female? I like it. It makes me feel good. Makes me feel strong. People appreciate it. And so I’m going to own it. But there are those of us who are Gen X feminists who grew up in a time when you had to sort of reject things that were stereotypically female. So I’m starting to embrace more of it now.
Steve: Is that Gen X at 25 and Gen X at 50 are approaching life differently?
Leanne: Yes, I was bald at 25 with an attitude.
Steve: There’s so many good follow-up questions to the statement, I was bald at 25.
Leanne: I’d come out of the closet probably like, I don’t know, three years before, two years before, something like that, and declared myself to the world as a dyke. So I shaved my head and I wore everything rainbows. And yeah, it was fun. I look at that part of myself and I’m like, oh, that was awesome. I don’t need to do that now.
Steve: The part of you that we’re talking about that likes to help, how did that part of you manifest when you shaved your head and wore rainbows?
Leanne: Oh, it didn’t manifest itself at all then. I was angry at the world. Yeah. Yeah. It was not an easy world to come out in 25, 30 years ago. I was, let’s see, 25.
Steve: When did you start your consultancy?
Leanne: I was working for AT&T at that time. AT&T Wireless, because I’d worked for a startup cell phone company. I was in Seattle. I moved to San Francisco in 1996, and that improved my life immensely. So I worked for a little startup called Organic Online in 1996. It’s now a huge company, but at that time it was like 30 people. Left there and worked for a startup that was started by Howard Rheingold, if anybody remembers him. And they ran out of money, so it got laid off. And then I was doing, people were asking me to do contract work for them. And at this time I wasn’t doing research at all. I was doing QA and server performance load testing. And so I was doing contract work and someone said, oh, you’re getting so much contract work, you could start a company. So I started a consulting company in the fall of 1997 at the age of 26, where I was very brazen and thought I could do anything. And yeah, and sold, just sold projects to all sorts of startups and tech companies. 1997, ’98, ’99, the money was falling off trees and started selling research when a company had asked if we could do it and I didn’t know what it was. And so I went and asked some friends and someone told me and I was like, oh, well, let’s figure this out. In the late ’90s in the web and tech industry, if you just basically said you knew how to do something, someone would pay you to do it and then you just found other people and figured out how to do it. Sort of fake it till you make it.
Steve: Is there a point professionally where the seeds of what you’re talking about now, because I think you’re describing a way of being, that’s about finding out how to help people and doing that. Can you identify some of the seeds of that? When you were angry and wearing rainbows, that was not present there, but where did it start to emerge and how you worked?
Leanne: Oh, yeah. When the NASDAQ crashed. So do you remember the NASDAQ? NASDAQ crashed in the spring of 2000, which was actually a bigger deal for those of us in the San Francisco Bay Area than the terrorist attack in 2001, because it had a bigger effect on our life here on the West Coast. I had to lay off a considerable number of people at my startup company, at my consulting company. And I had never done that before. I didn’t know how to do that. I was a terrible manager. I was like running. People who worked for me then thought I was great, but I look back at that and I’m like, oh, compared to me now and having gone through management training and everything, I didn’t know what I was doing. And I wished at that time that I could be able to do more for people who had relied on me for a source of income to pay their rent and pay their bills to support them better when we just lost tons of clients and revenue streams all at once after the NASDAQ crashed. Yeah, probably started then because then the company turned mostly into a research company. We still did QA probably until about 2010 or so, but we were primarily doing research projects and I was primarily hiring people who were in research.
And I also had got married and had a kid and it changes your life. So in 2004 was when Gavin Newsom made marriage legal for 45 days in San Francisco and my wife was pregnant. And so we got married when she was pregnant, told her dad he should bring a shotgun to city hall because it was a shotgun wedding, bought a house, got a letter from the California Supreme Court saying our marriage wasn’t valid so we had to run down and get domestically partnered before the kid was born. And for all of us, it’s usually things like that that happened during our life that help us gain a little more empathy for experiences in other people. It’s just the act of being human and growing older and having experiences that makes you understand other people’s experiences. And oh, everybody holds pain and everybody holds things that they won’t tell you about and we’re all here to help each other.
Steve: And just hearing you talk about it, even thinking like the layoffs, how we handle endings, I’m reminded of something that you and I talked about, not on this podcast, but just years ago. And I was catching up with you during a period of time where, I guess you were leaving a job and you were thinking about how you wanted to leave everything. Would you mind kind of describing some of what you did and how you thought about it?
Leanne: Sure. I had been laid off because a company was reorganizing and the organization that I was overseeing no longer existed. So there was actually no place for my role anymore. And I was given time. So it wasn’t like the startup that laid me off a little over a year ago where it was just like, you’re done today, turn your laptop in. It was like, here’s the package and you’ve got this many days and you can still come into the office but you don’t have to do work anymore. Being told that you don’t have to do anything anymore, I was like, lots of people depend on me here. I have lots of relationships I need to pass on to someone else. So the initial sort of angry or hurt response to being laid off, I was like, okay, I can have this angry or hurt response because I love this job. But I also understand the business decision behind it. And I also understand that there are people who will need things from me before I leave. So one of my colleagues actually said when I met with them on the last day and said, here’s the budget for the team and here’s the things, I documented all the things that we were doing. Here’s all the people that we had relationships with. I already sent out email intros and made sure everyone had them. What they said to me was like, they’d never seen someone leave that way before. And I said, well, why would you want to burn bridges?
The tech community is really small. And just like when I’ve been harassed or mistreated at a company, I haven’t been the person to want to sue the company because this tech community is so small and I don’t want to be that person. So I want to take other ways to file a complaint or make something known or make sure that things are documented. And so it’s the same thing with leaving, whether it’s my choice to leave, when I’ve made a choice to leave a company, I’ve done the same thing. I make sure everything’s documented. Everybody has a relationship pass off and everyone’s going to be taken care of. So like when I’ve left companies, you know, and given like 60 days notice, I’ve made sure that whoever is reporting to me, they knew who they’re going to report to next. We had like closed off our relationship. You know, I told them how to get in touch with me outside of the company. You know, if they want any coaching or mentoring in the future, because as long as I’m working, I want to make sure I maintain good relationships with everybody. I think that’s the most important part of work, is relationships and colleagues.
Steve: And I think when we talked about it, you said, these people, you may manage them in another job, or they may manage you in another job, or they may interview you for another job, or you may interview them for another job.
Leanne: Yeah, exactly. Yeah. I mean, I have had people who worked for me in the past who will just say misbehaved and then applied for a job on my team, you know, at another company I was at. But just like that thing, like, I don’t remember what you said or what you did. I remember how you made me feel. I look at their resume in a stack and I’m like, no, like you did something that really harmed my agency or you did something that was really professionally inappropriate. You know, I can’t have you working for me anymore. That company that laid me off, the next company I was at, I was working with the sales team and they wanted to sell to that company that had laid me off. And I said, sure, I know people there. I can, I can help set up something. And we, so we set up this whole lunch. We were going to lunch at a restaurant in San Francisco and waiting in line to be seated was the person who had like, laid me off. And, you know, we gave each other a hug and I was like, you know, we’re okay. Like there doesn’t need to be an adversarial relationship when there’s business decisions made and when we all live in the same town and we’re all in the same tech community.
But people do set things up that way sometimes. And I think the harm it does is to yourself. It’s sort of like, I tell people, you have a choice of what you’re going to do with the feelings you have right now. You’re feeling frustrated. You’re feeling overwhelmed. You’re feeling disrespected, undervalued, whatever. Okay. Those are feelings. What are you going to do about it? Like you have control over your actions and your next steps and what you say and what you do. This is no longer about research. This is about being a human in a workplace.
Steve: Yeah, which is the core of research.
Leanne: Yeah. Well, and it’s also how to show value in research is to be that human who’s professional and can manage situations and keep a certain sort of like emotional regularity.
Steve: You know, the successful researcher or leader or manager who does a great transition when they’re laid off is also the person that is good at understanding what people’s needs are and proposing ways to help them accomplish it so that they collaborate and research together. It is the same set of values and life skills that you’re talking about. I mean, I think we went someplace really interesting and I like what you said, you know, these are human skills and I do think this is about research even though it’s about being a human.
Leanne: Well, that’s what I coach people who are looking for a job is to focus more on like, what is the hiring, like use your research skills. What is the hiring manager looking for? Does that fit you? Because there’s all sorts of researchers out there right now who are hurting because they can’t find work. I think the way that we find value again and find ourselves jobs again is to use those skills we have as researchers to understand what people want.
Steve: Is there anything in today’s conversation that we didn’t get to you bring up?
Leanne: No, that’s been great, Steve. I like the way it just sort of wandered. This has been fun.
Steve: Thanks Leanne for being just so wide reaching in what you have to share and kind of digging into a lot of related aspects. It’s really very interesting and inspirational for me personally to have this conversation with you. Thank you.
Leanne: Oh, you’re welcome. And thank you, Steve. Always a joy to talk with you.
Steve: Okay donut friends, thanks a whole heap for listening to this episode! Don’t forget, you can always find Dollars to Donuts where all the podcasts are, or visit Portigal dot com slash podcast for all of the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 43. Leanne Waldal returns first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I catch up with Leanne Waldal, five years after she first appeared on the podcast. She’s now a Principal in User Experience at ADP.
A couple of years ago, I realized I know things. We all know things, but sometimes we go through life thinking there’s always something more for us to know, or we don’t know as much as others. A couple of years ago I was like, oh, I know some stuff. I could share it. If I think of myself at 23, 24 years old, I had people who were my age now who were telling me things that I listened to and got advice from. I’m that person now. I can be the person who like gives people advice or says, I don’t actually know everything, but here’s some things I learned over the years that might help you. It makes me feel good to do that. It boosts my confidence. It helps me feel like I can actually do something that’s not just my craft or not just my job for a paycheck or not just this, but I actually have something to offer. And that’s a great feeling. – Leanne Waldal
Show Links* Interviewing Users, second edition * Beyond The Surface: Navigating The Depths Of User Research With Steve Portigal (Greenbook Podcast) * Leanne Waldal on Dollars to Donuts, 2019 * Leanne on LinkedIn * ADP * Badass: Making Users Awesome by Kathy Sierra * Dropbox * Orbiting the Giant Hairball: A Corporate Fool’s Guide to Surviving with Grace by Gordon MacKenzie * Organic Online * Howard Rheingold * NASDAQ crash, 2000 * San Francisco 2004 same-sex weddings
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal. In this episode I catch up with Leanne Waldal, 5 years after she was first on Dollars to Donuts.
But before we get to that, I updated my classic book, Interviewing Users, to incorporate what I’ve learned in 10 more years of being a researcher and in teaching other people. After the book came out, I spoke about user research with Karen Lynch on the Greenbook podcast. Here’s a short clip.
Karen Lynch: I mean, there’s even some frameworks for creating a knowledge management platform for yourself, you know, how to have a database of your own research. So excellent applications for a smaller shop that might not have access to platforms and tools. But here’s a way you can kind of create your own hub, knowledge hub. You did a very—a good job, solid job, an important job of also providing different—you know, here are some forms. You gave structure to the field is what you did. You know, here’s some forms that you can look at for, you know, debriefing your interviews after you conduct them. Here are some kind of, like—I don’t want to call them templates. But here’s the framework that you can do for creating your discussion guide. Here’s some tools you can lose—use to synthesize your data. So you were—you’ve given some very tangible tools in this book for anybody who is really trying soup to nuts to go off on their own for the first time or just get to know the field that maybe they’ve been hired in, really practical, tangible things that researchers can borrow from. I mean, again, having been in the field for a long time, some of this was—you know, some of this was a part of my practice. But I’m like, “You know what? That’s a great debrief form. That one really stood out to me.” For example, Interview Debrief Form, where it’s not just take notes on what you just experienced, but it’s prompting your brain to think through what it means. So kudos to you on that. Is that just a practice of yours? Or did something kind of stimulate that thought that you’d might want to include those?
Steve: Well, you know, there’s this interesting part of research where it’s collaborative and facilitative. I mean, it’s not just what, you know, I, as the researcher, or me, as part of the research team, learns. It’s, you know, the people that we’re working with. And so I have an obligation to them. Boy, that sounds like—it sounds very—more moralistic than I mean. Like, I can do a better job if I can help them learn something and take something away. But, if I also—if I hear what they’re taking away, especially—I’m, you know, I’m not the domain expert. I work as a consultant, so I come into an area that somebody else inhabits. And so they’re going to always see things in the research that I won’t see. It’s really helpful for me to understand what didn’t they hear that person say. Like, if there’s a gap in what they took away, then I now know I need to kind of emphasis that because it’s—there’s a takeaway that’s obvious to me that isn’t to them. So that needs to be surfaced as I share back. So I can get that out a debrief. And, when I hear what they heard and what surprises them, I understand, yeah, how they’re framing the world, what’s relevant information. Like, I’m getting this indirect feedback. So, you know, I—like, it’s not my natural way of being to have a template for an activity. I’d rather just chat. And sometimes that suits me well, and sometimes I need to put a little more structure in it. So, I think, you know, writing up a debriefing guide—well, I think there’s—having something formal like a template or a tool you can use sort of reminds me that this is an important part of the process. I need to make time for it and mental space for it, and I need to tell the people I’m collaborating with that they should leave time for it. And guess what? This is a serious activity. I don’t—I’m not just trying to, like, get coffee with you and, “Hey… what did you think?” I’ve got a document here. So there’s a little bit of a theater. And I don’t mean that in an unkind way. But there’s a little bit of a, you know, a formality to it that reminds me to take it seriously and that shows my collaborators that I value what they have to say and that I, you know, I’ve got some format for that.
That’s from the Greenbook podcast. I’ll put the whole episode in the show notes. And I hope you’ll check out this updated edition of Interviewing Users, and share it with all of the people in your network. You can help me and help the book by writing even a tiny review on Amazon.
So, let’s get to my conversation with Leanne. She’s now a Principal in user experience at ADP.
Well, Leanne, thank you for coming back to Dollars to Donuts after, I guess, five years
Leanne Waldal: Good to be here.
Steve: or so. It’s nice to talk to you again. It’s nice to talk to you again for this podcast. That introduction presumes that you and I have not spoken in the intervening five years,
Leanne: We have.
Steve: but we have.
Leanne: Yes, indeed.
Steve: We have. The secret is out. Let’s talk about some of the things that you’ve been up to professionally in the last five or so years. We can kind of start anywhere and go anywhere, but I think that’s one of the things we’d like to cover and just hear what some of your experiences have been and what you’re thinking about nowadays with user research.
Leanne: Yeah, and the wonderful thing about the last five years is it was about four years ago that the pandemic started. And when we spoke about five years ago, it was probably six months before this pandemic started and it changed research. So I went and worked for a company and helped them understand using ethnographic research in Germany and France, what was going on with the markets that they were trying to sell into their help teams really understand what they needed to do to better sell into those markets.
And then I joined an agency. And at the beginning of 2020, I was selling a project to a company with my teammates in the San Francisco office. And we had planned out like many of us in early 2020, all the things that we were going to do in this project over the course of 2020. And then of course, the pandemic happened and we all went home. And so we had to shift the type of research we were doing from going out and talking to people in person and seeing them in person to doing everything over video and then also running some surveys.
One interesting thing that happened in the summer of 2020 is humanity got tired of answering surveys. So I was doing mixed method research in the summer of 2020 with interviews with people about their experiences. And then I also wanted from those experiences to measure them at scale with a survey to find out, well, if I talk to 20 people and then I take these things and find out, you know, how do a thousand people respond to these experiences? It was a really easy target. It was millennials in California. It took like 35 days to get a thousand responses. And that was fascinating to me because it was really easy for us at that time to get people to talk to us on video. But I talked to the panel provider we were using at that time and they said, people are tired of answering surveys. This pandemic is wearing people out. We have all sorts of personal things we’re dealing with and we’re also terrified the world’s about to end.
And so then in 2021, before the vaccines came out, we were doing some work with a startup at the agency I was at. And I realized that some of their early like Kickstarter supporters were in the San Francisco area. And I said, this is about motorcycles. Motorcycles are used outside. We could actually meet people in person. Like that’s the thing I miss the most. I’m an extrovert. For all the people who are introverts, the pandemic was just wonderful for them to a certain extent. Like lots of people I know who are introverts were happy to work from home and happy to be at home. Me, I miss people.
And so I started up ethnographic research again in 2021 by meeting motorcyclists in a park. Got some great pictures of all of us in masks standing about eight feet apart from each other handing out water bottles and shouting across the grass to each other. When I left the agency, I went to work for a startup and introduced again ethnographic research to them because they had a mobile app but hadn’t really watched people use it out in the real world. And so I started just in the San Francisco Bay area to just run a pilot, like let’s go get some people using this and see how they use it. And then got the funding to go visit some customers and visit some consumers and some other people.
So I was tromping around in the snow in Minnesota in like negative degrees following somebody who was using our app also tromping through the snow, which was super fun. And when that startup laid me off, I did some consulting. I did some consulting for a large company, also doing some ethnographic research, going into some offices of the people who use their products to show the pain points. And also then started what I started doing, which I hadn’t done in a while, which was really fun was looking qualitatively at the pain points that were happening and then going to the people who have numbers from revenue and numbers from usage analytics and figure out how much money is being lost because of the user experience or how much money could be gained from cleaning up some of the user experience.
And then I joined ADP in June of last year as a principal in user experience. And I’m leading all sorts of research, including recently some ethnographic research again with one of our clients, which is super fun, teaching people, teaching people and teaching people.
Steve: I think we like to forget things that happened during the pandemic because, as you said, people were worried that everything was going to end. So that slow survey response or slow and low survey response, do you know, has that rebounded in the time since then?
Leanne: It definitely has. So by the time I was at a startup that I was at, I think we did some ethnographic research, some interview research, and then we had like, you know, six key things that we wanted to understand from a certain consumer population. I think we got all the response we needed in like 11 days. So from what I’ve seen in that sort of consumer research and mapping survey research to qualitative research, we’ve gone back to getting people to respond to it more quickly. But if you remember the summer of 2020, we were terrified. Why would you even if you’re offered a gift card to answer a 10 question survey, like spend time on that when you’re trying to take care of your kid, dogs and yourself?
Steve: We were terrified and we also felt, I think, again, it’s hard to remember with any clarity, but there was this sense of trying to preserve some aspects of normalcy, like family and work and so on. But you could see why people would want to run a survey because that’s part of their job and they want to continue feeling normal, but it’s a really, it’s an interesting observation that people didn’t want to respond to surveys because of what everything that was going on.
Leanne: Yeah. And as researchers, we lost some of our ability to understand people’s experiences because of the pandemic and being locked down. So we were limited to diary studies that are about things like surveys and interviews.
Steve: I remember having a lot of, we had a lot of complicated feelings about a lot of things, but watching that period where research was starting to pivot to remote and I don’t know, I felt like there was a certain gleefulness in being able to guide that. I don’t want to make it like a Coke versus Pepsi thing, but it seems like there is a remote versus in-person belief system or, you know, adherence and that when in-person became impossible, the folks that could give guidance and best practices on remote, it was hard to not see that guidance being offered because it was everywhere.
Leanne: Oh, of course. It was amazing at that time for people who didn’t know how to do remote. Like the agency I was at prided itself on doing contextual inquiry all the time. So teaching people how to do things more remote, I always like to mix things up and tell people like you can’t do all remote or all contextual or all quant. You need to mix it up to understand a human experience.
Steve: But that doesn’t work with our desire to make a single declarative, this is the best way to do it, Leanne, what do you mean, what do you mean mix it up?
Leanne: Oh, we all combine complexities in our experiences and how we do things. I find it really hard to stick just in one camp.
Steve: And I think it’s interesting, you talk about being an extrovert kind of driving you to, if I understood, yes, extroverts suffered from lockdown, but that as a researcher who’s also extroverted, you really were, I think, creative in trying to find ways to do some portion of context and kind of get yourself out there and get to where people were.
Leanne: Yeah, I felt really lucky to be working for this agency in San Francisco where we had an office where the windows open. So we were coming back to fully masked with all the windows open, standing apart from each other just to see each other. Do something on a whiteboard together, sit in a room and talk together just to, because you can’t just like one thing about the difference between desk research remotely and seeing someone in person is you can’t see the Post-it notes around the monitor. We all got Post-it notes around our monitor. We all got things, you know, I’ve got stuff here today to remind us of what to do.
And if you have a workplace, you’ve also decorated your space. So, you know, we’re decorator crabs, basically. We bring in pictures of our family and someone brings us back, you know, a token from some trip and we put it on our desk. And you know, the team all goes out and does something fun together and we put a picture on our bulletin board and you can’t see that. And recently something interesting I noticed was that, which I hadn’t thought of before is that with this remote, like you and I just look at each other, you know, on video, if I shared my screen with you, you can only see one screen at a time. Well, a lot of people work with two or three screens now. So how do you in a remote world see how they’re comparing things across two screens? Like they could, I could share one screen with you and then another, but you couldn’t actually see my experience of, you know, comparing this Word doc with this Word doc. So that fascinated me. I was like, Oh, amazing. The world has also changed that monitors are cheap and we can have multiple now.
Steve: I would have to ask whether or not you had multiple monitors and ask to see them.
Leanne: Remotely, you might just assume your screen is your screen. Like think about if you’re doing research with engineers, you know, you and I are both of a certain age. So you remember when we only had one monitor and it was like a big thing, the two of them on a desk, there’s no way you could have two monitors. Well, now if you go look at any software developer who’s working, they’ve got a couple of monitors or three monitors. They’ve got curved monitors. You know, we just sort of surround ourselves with all this stuff to look at. And you can only see that if you go see people in person, we don’t yet have a way with any of zoom or any other video technology, be able to see across all of those monitors all at once. So I think that’s an interesting point of innovation for these video companies.
Steve: And we’re still talking about research where the thing we would be looking at would be something that takes place on a screen.
Leanne: So I want to be able to see what you’re doing. Also, you know, motorcyclists can’t see their motorcycle. If I talk to you on video, maybe you can bring like your phone or something out to show me your motorcycle, but I can’t get that giddy experience you have about talking about it person.
Steve: Sorry, did you say the giddy experience?
Leanne: Yeah!
Steve: Who’s giddy in that example?
Leanne: The motorcyclists. Motorcyclists feel very, very strongly about their motorcycle. And I also noticed, because I’d been doing in the consulting gig I did before where I’m working now, I was doing some remote interviews. And then where I’m working now, I do remote interviews and go out and talk with people. And I’m noticing this distinct difference between how vulnerable someone will be with me and how much they’ll share with me in person versus on a screen. And that, if you think about it, just makes human sense. We can sense each other better and share things better when we’re in person than when we’re on a screen.
Steve: When you and I did the previous episode of this podcast, I came to your office and we sat in a room holding microphones to do it.
Leanne: Yeah, exactly. Yeah.
Steve: Technically, I guess we could have done that again, but I think we rely on this technology. We’re using remote screen-based recording, blah, blah, blah, that has become the default, even when it’s not the only option.
Leanne: Yeah, and I think it’s fine. This gives us opportunities to talk to people we can never talk to. But I think we have to make sure we remember the importance of doing things in person. Humans are people who sense things off of each other, and it’s important to make sure that’s there in the mix with everything we’re doing.
Steve: And you set up off the top to teaching and you talked about going with your colleagues
Leanne: Mm-hmm.
Steve: and teammates and clients and so on into the field. We have a number of different parties that have a different experience.
Leanne: Mm-hmm. Mm-hmm.
Steve: So the motorcyclist who’s giddy, us the researcher, and I mean, I’ll just say like,
Leanne: Mm-hmm.
Steve: I need to be in the room with that giddy person. So I need something that I’m missing from the remote. But then there’s also the people that you’re taking out that you want to give them a sense
Leanne: Mm-hmm.
Steve: of their customers and their users. What’s been your experience over these few years in trying to do that for your clients and stakeholders?
Leanne: Well, for that sort of work, bringing marketers, salespeople, engineers, product managers, designers, et cetera, out into the field for consumers or customers, B2B or B2C or B2B2C, it never fails to surprise them. I’ve been doing this for a long time, and it’s the same thing over and over. And I love that sort of teaching moment of bringing someone out to an experience and just saying, I’m going to run the camera. If you don’t know what to say, I have no problem keeping the conversation going and saying, tell me more, tell me more, show me more. What about that? You get to just observe, or you can participate. It’s all up to you. There’s no rules, really. And I’ve never had someone say that they didn’t or weren’t surprised by something that they saw or they heard or observed. And usually, the things people are most surprised by are the environments that people are in. Because if you only see me in this room, like right now, I’m in a phone booth in a WeWork. I probably look like I’m in a sauna. You have no idea what’s going on around me. And so to see the vast experience of someone’s house or of someone’s business or someone’s office or following them along while they do something out in the world, usually people I’m with are as surprised by that as they are by the things in the product or the service or whatever it is that trying to understand the experience of. Well, for B2B, when I’ve worked in B2B,
Steve: What is it that these folks are learning from these experiences?
Leanne: I’m trying to help people understand the humanity of the people who are using the product. That it’s not just someone using a credit card to pay for a subscription, or it’s not just the buyer and their team who are at work and using your product in their work. It’s not just a job. There’s a human behind all that who’s got needs and has got friends at work or has colleagues or people that influence or don’t, or they have other products they’re using around your products that you can’t see from your usage analytics. So I’m trying to get them to see the giddiness of the motorcyclist, not just like, “I am this demographic and I’ve got this much money to buy this kind of product,” but look at how they feel about it. That’s what brand campaigns and marketing campaigns do. They get into our feelings. And so I try to get the product side of a company to understand that piece, that you can get more engagement in your product.
You can get people to use it more if you get them to feel good about it. It’s like Kathy Sierra’s book, “Make Your Users Badass,” or something. I probably just mangled that. Something like that. You want your users and your customers to talk about you at a cocktail party after two drinks. You want them to remember you so well. And then that Maya Angelou quote about, “I don’t remember what you said or did. I remember how I felt.” That’s the human. And so in companies, marketers know that. Brand people know that. I try to help product and design people remember that because they know that too. We just don’t remember that when we’re focused so much on usage analytics. It’s like, yeah, usage analytics and revenue analytics are important, but they aren’t important in a vacuum. You have to also make people feel good and feel smart when they’re using your product. Yeah.
Steve: Have you seen any shift in the appetite for the kind of, you know, understanding that you’re enabling people to gain?
Leanne: Research and design appetite has gone up and down in the tech industry over the past 20, 30 years that I’ve been in it. Right now, there’s a lot of people out there saying nobody has appetite for us. I’m like, well, maybe they have an appetite for a certain type of work that we do. We don’t have an appetite for all the work that we do. And that appetite ebbs and flows, just like the appetite for anything else. It’s just humans and trends and businesses and business decisions. And my advice to everybody has always been like, you can create an appetite for what you provide. It’s a sales technique. Understand what somebody needs, meet them where they are. Instead of being like, here are the five things I do. Here’s the checklist, sort of like McDonald’s, like here are the things you can order from me. Oh, you don’t want any of these? And instead say, what are you looking for? What are you trying to achieve? What are you doing? How can I help?
Steve: I want to clarify, or I’m trying to think about this as a question and not a statement, but I might just go with a statement. There’s a difference between, you know, when you say, find out what people need and then help them achieve it, that doesn’t mean if they don’t ask for contextual research, you don’t do contextual research. And maybe this is where the McDonald’s thing breaks down for me, or needs like another metaphor layer on top of the metaphor. Because if you ask what they need, no one needs research, they need the information or the decision or whatever they’re going to do about it. And so you have a lot of ways to get to that outcome. But if you say to them, hey, I do research, to your point, right, if you say, hey, I do research, do you want research? No, I don’t.
Leanne: Yeah, exactly. Or also like waiting for someone to ask you for a certain kind of research. When someone who doesn’t know the breadth and the depth of what you can do with research, asks you for something, they might not know what they need or want. They’re only asking based on their own knowledge. So I always find it’s better to figure out what kind of research to do or how to prioritize it or where to be by hanging out with people and finding out, say, what are you working on? What’s coming up? What do we know? What do we not know? Here’s these analytics. Do we know why these analytics look this way? Well, we could find that out this way, or we could do this this way, and see which one of those things that I start proposing to them based on what I heard them say they’re working on or what they’re trying to achieve, sort of like sparks a conversation. And then research is desired and wanted and valued and invited to the table. But I see a lot of researchers, and I coach researchers on this, not going in with like, I’m going to run a survey on this, or we’re going to do some interviews, or I’m going to do this unmoderated. Nobody knows what unmoderated research is outside of the research community. So stay away from the methodology. Stay away from like, I’m going to do this and stick with like, I’m your partner and I want to help you out
Steve: The labels that are always anchored in my mind is the difference between proactive and reactive. And you’re talking about being proactive. And even I get a picture as you kind of talk that there’s a relationship and there’s something over time. And I kind of hang out with them and see what they’re thinking about and what they’re talking about. That is not a — this is sales, it’s not a sales call, it’s a relationship-based sale.
Leanne: Yeah. And that’s why, as you and I have talked before, I don’t want to
go back to run a consulting firm. I don’t want to be a consultant unless I have to, for, you know, to get revenue in my life. I enjoy being a part of a team to like, nurture relationships and get collaborations going and work with people so we can do something together, because I don’t like being a lone wolf. I don’t like being a single point of failure. And, you know, I want to do things together.
Steve: In some of the conversations we’ve had where you’ve said — made these comments before about, you know, the relationships, I get the sense, and you can correct me here, those relationships aren’t handed to you. You’re seeking them out.
Leanne: The startup I last worked for, after I got laid off, customers of that startup were still texting me up to like, three or four months after I left the company, because they didn’t know I’d left the company. Every time they text me, I’d have to let them know. But I’d gotten such a good relationship with them that was beyond just like, I want to know how you use this product. It was like, I want to know how to use this product. And how did you start this business? And, and tell me about your family and this little town you live in. And where’s the best place to get lunch? And, you know, I would get invited over for dinner with their families. And you want that sort of relationship, which is very much like a sales tactic, because that’s what people who sell want. They want to get under your skin a little bit to um, understand you better, because that helps them sell to something that you need. And that’s all about relationships and collaboration.
Steve: What things, if any, are different in the sales relationships and collaboration that you want to nurture with users, customers, versus the colleagues, the people that you work with that you’re also wanting to do collaboration with?
Leanne: Well, you have a different goal when you’re a salesperson. You’re trying to hit a number. You’re trying to get someone to buy something. My goal with anybody is, I want to help you out. I want to see if there’s something you need that I can help you with. I want to see if, you know, you know, the types of skills that I have and the types of things I can do and the craft I have can help you. And if it can’t, I’m not going to, I’m not going to waste my time, you know, doing something if it’s not going to help you. I don’t know that all researchers or research teams see themselves as a service organization, but we basically are a service organization. We’re providing a service to and with people who hopefully will use and are interested in using the insights and the learnings they can get with us, from us, to build product, to design things, to market something, to sell something.
Steve: Hearing you talk is really refreshing for me because I think it’s easy for us, or for me at least personally, to get sucked into all of this as adversarial. I don’t know, we hear a lot of stories from each other, you know, in this work, we tell a lot of adversarial stories, persuading, getting permission, convincing.
Leanne: Do you pay attention to the subreddits on research? Because it’s full of that. Yeah. I just watch it and read it.
Steve: And sales, I mean, being a consultant, you know, I have lots of peers where we talk about sales, and the more — even though I’ve been doing this for a long time, I still keep learning and relearning that sales is not a persuasion adversarial kind of work. It is a — it’s all the things that you’re talking about, it’s relationships and how can I be of value to you? And that way of being is — I think you live that way, and so you work that way.
Leanne: Well, I learned it from running the consulting company. And then when I was at Dropbox and they were creating a sales team, I sort of like did a swap with the sales team. I was like, you teach me how to sell things because I actually want to know this more. Like I see, I see value in knowing these techniques. And, you know, you teach me that and give me access to people you’re selling to, and I’ll teach you about them. And that was a really valuable collaboration then. Because Dropbox was just getting into the enterprise. We were just starting to sell to larger customers. And I was working with all of the new enterprise account managers to basically learn what they did, but then also learn who they were selling to, and then feed that back to them to say, you know, I think you could sell this customer this, or this customer needs this.
Steve: What kind of things did you learn for yourself about sales from that?
Leanne: Well, that was when sort of like a chandelier went off. And I realized that like techniques we use in research to understand someone’s experience are the exact same thing sales people do. So a lot of researchers will be like, nobody does research like we do. Product managers don’t do it. They ask leading questions. Sales people are just trying to sell. And I’m like, hmm, I’m starting to see that. And this was back then. So about 10 years ago when I was like, oh, salespeople are actually doing a lot of research and product managers are actually doing a lot of research. Why does it matter how you ask the question? If what you’re trying to do is discover something or understand something, as long as you get to the end goal, the path you got there doesn’t really matter.
Steve: And did you learn anything about — again, you’re using sales in this very elevated way to describe how you work with colleagues, for example. Is this the point at which you started to develop those skills further?
Leanne: Yeah, that was the job where I grew up. I had run my own consulting firm for 17 years. So from when I was in my mid to late 20s until I was in my early 40s. And when I took my first job after that long consulting period at Dropbox, I was like, oh, I’m going to learn how to be an adult inside of a corporation now. It doesn’t matter what age you are when you learn how to do that. So it went through that learning curve there.
Steve: The idea of growing up is really — that’s a big one.
Leanne: Well, I don’t think we all ever grow up. We just grow up in certain ways. It’s like, oh, now I know how to behave professionally. I didn’t know how to behave professionally before.
Steve: I mean, you ran a consultancy for 17 years, so you at least were able to survive in that time.
Leanne: But you know this. It’s a different sort of professional to service clients and manage clients and sell things than to learn how to behave and how to manage the politics and the relationships inside of a corporation. There’s this really great book called Orbiting the Giant Hairball, which I go back to every once in a while, because it’s just all about the humanity in these large corporations or in these mid-sized startups and how do people get along and work together and make decisions and collaborate and get things done. It’s very different from running your own thing.
Steve: You mentioned just offhand, I think, that you talk to researchers and people are reaching out to you for advice, and I’d just love to hear kind of what that’s like, and I don’t know what’s coming up for people right now that you’re talking to them about.
Leanne: So what’s coming up is questions about how to, particularly for researchers early in their career who maybe started doing research in 2020, they’ve never done contextual inquiry. They’ve never done ethnographic research. They’re working at a company where they don’t have a manager or a leader who’s encouraged them to do that. They started remote. A lot of these people went to college remote, graduated in 2020 or 2021, got a job as an early career researcher, and they’ve just never done this. And so I basically start with the basics, like here’s how you observe someone. The other thing I noticed, and I also talked with some people recently about what they’re seeing of people who are earlier in their career who went to school during the pandemic and now are coming out, that it’s the same thing that those of us who are older went through in the pandemic.
Leanne: We lost a certain amount of social ability. And most of us, or a lot of us, got that back in the
Leanne: last year or two. We started meeting people in person again. We started going to dinner parties. We started going out to bars, went to concerts. Some of us went back to offices and figured out how to be in an office again. But there’s a lot of fear of, “I don’t know how to do this, and I’m scared to do this, and I’ve never done this before,” that a lot of it I attribute to the pandemic. And I think those of us who do mentoring and coaching need to be aware of that and teach people how to do that. I know someone who is teaching people how to dress to go to an office, how to wash your hair, how to have conversations with people. And that’s something that’s really specific to this point in time that didn’t exist before. So say, like, 2016, what was I mentoring and coaching people about? It was more relationship coaching. Like, how do you get along with a product manager who disagrees with your research results? Or how do you have influence? Or how do you learn a new skill? Or you’re a junior researcher, you want to become a senior researcher, what do you need to do to show that so that you can get that promotion? I’m not hearing that so much anymore. I’m hearing more around, well, one, how do I get a job? But then also, sort of like, how do I do these things that nobody ever taught me to do and I never had a chance to do before, and I want to try to do them now? Some of these people I’m coaching are the only researcher in a company. So they don’t even have someone to sort of like manage them or say, this is how we did it in the past, 2019 and earlier. So yeah, I just I like to help people out that way. I sort of feel like the things I can do right now are improve relationships among different teams. And I can also help out people who are trying to grow up in their career.
Steve: What else is coming up in these mentorship conversations you’re having?
Leanne: Presentation skills. So a lot of people come to me and say, I’ve been giving presentations, you know, over Zoom for years, and now I’m being asked to give a presentation in person. And I thought, how do you not know how to give a presentation in person? I’m like, oh, you’ve never worked in an office where you had to stand up in a room in front of a table of eight people with a pointer, you know, remember how we used to plug like a USB thing into our laptops, and then that we’d have a clicker and then presentation would show up on screen? Well, people are being asked to do that again. But that’s another soft skill that nobody’s taught them how to do. And one thing that is an advantage of presenting over video is you can have notes. You know, nobody can see that you’ve got your PowerPoint and like presenter view with your notes, or nobody can see that you’ve got like a notebook, you know, with like, oh, this is what I say on this slide. When you’re in a room, you’re on stage. And I think a lot, a lot, a lot of us forgot how to be on stage, who used to be on stage. But some people have never been on stage. And now they’re being asked to be on stage. And they didn’t get that practice at college, because you always presented your papers and everything over video or in a small classroom. I’m really glad that my daughter started college in person and is in college in person. So she’s getting that like those that like social growth that you need before you start to turn into a young adult trying to get yourself into the workforce.
Steve: It’s a fascinating observation that maybe is obvious to everyone, but I had never really thought of this. There’s a significant cohort of people in the workforce who have never, who don’t have a pre-pandemic norm to return to around research, presenting, business travel, any of those kinds of things.
Leanne: Yeah. Just like professional skills, how to get along with people. Yeah.
Steve: It’s interesting that at least the people that you’re in contact with have some awareness that they have a gap. Seems like that would be the first step to addressing it is knowing that it’s missing, that there’s a thing that is expected of you.
Leanne: Yeah. Yeah, there are a couple people I mentor who come to me in panics, like, I have to do this, and I’ve never done it before. And I’m like, oh, it’ll be okay.
Steve: Yeah, what do you tell someone who hasn’t done a business trip before, who hasn’t worked in an office? What’s the granularity of advice here?
Leanne: That even me in my early 50s, I still have new things I have to do every once in a while. And we can all do new things and hard things, and you will actually be okay. And then we can get down to brass tacks and go through the tactics. What, how do you pack? Or how do you like memorize things or practice? A lot of people who give presentations over video don’t practice first, because you’ve got all these supports around you to do it. You’re wearing your sweatpants and you put on a button down shirt, but you’ve got sweatpants on. You’ve got all your Post-it notes nearby of what to say. And so without all of those supports, what are you going to do? Well, you have to practice more. You have to think ahead. You have to plan, make lists if that’s your thing. And I think it’s really surprised people that they need to do that.
Steve: What’s the guidance, this is decontextualized, I guess, but what’s the guidance for a person who’s new to in-person research?
Leanne:Well, I say, you know, go find someone to bring
with you who’s done it before. So you aren’t, you and everybody else isn’t brand new to it all. It’s this, you know, it’s something like, you don’t all want to be new to doing in-person research on the group that’s going out to do it. See if you can find someone who’s done it before. Or find someone from marketing sales, who’s done something similar. Who’s at least like gone to visit customers or has gone to conferences and meet with customers. Your first pancake of the first time you do anything will always be a little rough. So it’s always good to have someone there who, you know, can advise you a little, or, you know, steer you in a slightly different direction when they see you going astray. And if you don’t have that, then, you know, just be gentle with yourself. Whatever you’re doing is the best you could possibly do.
Steve: That’s the lesson for everything, right? Let’s just be gentle for yourself. What do you get out of mentorship?
Leanne: Well, a couple of years ago, I realized I know things. And, you know, we all know things, but sometimes we go through life thinking there’s always something more for us to know, or we don’t know as much as others. And it was a couple of years ago when I was like, oh, I know some stuff. I could share it. You know, maybe it would provide some value to people who, you know, like if I think of myself at 23, 24 years old, I had people who were my age now who were telling me things that I listened to and got advice from. And it just sort of popped into my head. I was like, oh, I’m that person now. I can be the person who like gives people advice or says like, you know, I don’t actually know everything, but here’s some things I learned over the years that might help you. So, and it, it makes me feel good to do that. It boosts my confidence. It helps me feel like, oh, I can actually do something that’s not just my craft or not just my job for a paycheck or not just this, but like I actually have something to offer. And that’s a great feeling. Yeah. And then you’re sort of surprised, like, oh, I actually know how to do this.
Steve: I don’t know if this happens for you this way, but sometimes I don’t know what I know until I’m in a situation where I’m asked to help somebody out.
Leanne: Yeah. And other people will say to me, like, well, Leanne, we see you as an expert and everything. I’m like, yeah, but I don’t see myself that way. Yeah. Yeah. And for me, it meant I was taking it on in a way that felt comfortable for me. Like I don’t need to be or want to be the expert who, you know, gets on stage everywhere or has the big title or anything like that.
Steve: But to choose to be a mentor is to partially take on the role of the expert. The theme of some of your earlier points about creating good collaborations around research are understanding what somebody needs and how you can be helpful to them, and that you like to be helpful. And that seems to manifest in mentorship as well.
Leanne: But I do like to help people. Yeah. And that’s that being a helpful person is something that’s often attributed more to women than to men. And I used to when I was younger, sort of like sort of like push away things that were like, oh, that’s stereotypically female. And I’m like, OK, so what if I’m doing something that’s stereotypically female? I like it. It makes me feel good. Makes me feel strong. People appreciate it. And so I’m going to own it. But there are those of us who are Gen X feminists who grew up in a time when you had to sort of reject things that were stereotypically female. So I’m starting to embrace more of it now.
Steve: Is that Gen X at 25 and Gen X at 50 are approaching life differently?
Leanne: Yes, I was bald at 25 with an attitude.
Steve: There’s so many good follow-up questions to the statement, I was bald at 25.
Leanne: I’d come out of the closet probably like, I don’t know, three years before, two years before, something like that, and declared myself to the world as a dyke. So I shaved my head and I wore everything rainbows. And yeah, it was fun. I look at that part of myself and I’m like, oh, that was awesome. I don’t need to do that now.
Steve: The part of you that we’re talking about that likes to help, how did that part of you manifest when you shaved your head and wore rainbows?
Leanne: Oh, it didn’t manifest itself at all then. I was angry at the world. Yeah. Yeah. It was not an easy world to come out in 25, 30 years ago. I was, let’s see, 25.
Steve: When did you start your consultancy?
Leanne: I was working for AT&T at that time. AT&T Wireless, because I’d worked for a startup cell phone company. I was in Seattle. I moved to San Francisco in 1996, and that improved my life immensely. So I worked for a little startup called Organic Online in 1996. It’s now a huge company, but at that time it was like 30 people. Left there and worked for a startup that was started by Howard Rheingold, if anybody remembers him. And they ran out of money, so it got laid off. And then I was doing, people were asking me to do contract work for them. And at this time I wasn’t doing research at all. I was doing QA and server performance load testing. And so I was doing contract work and someone said, oh, you’re getting so much contract work, you could start a company. So I started a consulting company in the fall of 1997 at the age of 26, where I was very brazen and thought I could do anything. And yeah, and sold, just sold projects to all sorts of startups and tech companies. 1997, ’98, ’99, the money was falling off trees and started selling research when a company had asked if we could do it and I didn’t know what it was. And so I went and asked some friends and someone told me and I was like, oh, well, let’s figure this out. In the late ’90s in the web and tech industry, if you just basically said you knew how to do something, someone would pay you to do it and then you just found other people and figured out how to do it. Sort of fake it till you make it.
Steve: Is there a point professionally where the seeds of what you’re talking about now, because I think you’re describing a way of being, that’s about finding out how to help people and doing that. Can you identify some of the seeds of that? When you were angry and wearing rainbows, that was not present there, but where did it start to emerge and how you worked?
Leanne: Oh, yeah. When the NASDAQ crashed. So do you remember the NASDAQ? NASDAQ crashed in the spring of 2000, which was actually a bigger deal for those of us in the San Francisco Bay Area than the terrorist attack in 2001, because it had a bigger effect on our life here on the West Coast. I had to lay off a considerable number of people at my startup company, at my consulting company. And I had never done that before. I didn’t know how to do that. I was a terrible manager. I was like running. People who worked for me then thought I was great, but I look back at that and I’m like, oh, compared to me now and having gone through management training and everything, I didn’t know what I was doing. And I wished at that time that I could be able to do more for people who had relied on me for a source of income to pay their rent and pay their bills to support them better when we just lost tons of clients and revenue streams all at once after the NASDAQ crashed. Yeah, probably started then because then the company turned mostly into a research company. We still did QA probably until about 2010 or so, but we were primarily doing research projects and I was primarily hiring people who were in research.
And I also had got married and had a kid and it changes your life. So in 2004 was when Gavin Newsom made marriage legal for 45 days in San Francisco and my wife was pregnant. And so we got married when she was pregnant, told her dad he should bring a shotgun to city hall because it was a shotgun wedding, bought a house, got a letter from the California Supreme Court saying our marriage wasn’t valid so we had to run down and get domestically partnered before the kid was born. And for all of us, it’s usually things like that that happened during our life that help us gain a little more empathy for experiences in other people. It’s just the act of being human and growing older and having experiences that makes you understand other people’s experiences. And oh, everybody holds pain and everybody holds things that they won’t tell you about and we’re all here to help each other.
Steve: And just hearing you talk about it, even thinking like the layoffs, how we handle endings, I’m reminded of something that you and I talked about, not on this podcast, but just years ago. And I was catching up with you during a period of time where, I guess you were leaving a job and you were thinking about how you wanted to leave everything. Would you mind kind of describing some of what you did and how you thought about it?
Leanne: Sure. I had been laid off because a company was reorganizing and the organization that I was overseeing no longer existed. So there was actually no place for my role anymore. And I was given time. So it wasn’t like the startup that laid me off a little over a year ago where it was just like, you’re done today, turn your laptop in. It was like, here’s the package and you’ve got this many days and you can still come into the office but you don’t have to do work anymore. Being told that you don’t have to do anything anymore, I was like, lots of people depend on me here. I have lots of relationships I need to pass on to someone else. So the initial sort of angry or hurt response to being laid off, I was like, okay, I can have this angry or hurt response because I love this job. But I also understand the business decision behind it. And I also understand that there are people who will need things from me before I leave. So one of my colleagues actually said when I met with them on the last day and said, here’s the budget for the team and here’s the things, I documented all the things that we were doing. Here’s all the people that we had relationships with. I already sent out email intros and made sure everyone had them. What they said to me was like, they’d never seen someone leave that way before. And I said, well, why would you want to burn bridges?
The tech community is really small. And just like when I’ve been harassed or mistreated at a company, I haven’t been the person to want to sue the company because this tech community is so small and I don’t want to be that person. So I want to take other ways to file a complaint or make something known or make sure that things are documented. And so it’s the same thing with leaving, whether it’s my choice to leave, when I’ve made a choice to leave a company, I’ve done the same thing. I make sure everything’s documented. Everybody has a relationship pass off and everyone’s going to be taken care of. So like when I’ve left companies, you know, and given like 60 days notice, I’ve made sure that whoever is reporting to me, they knew who they’re going to report to next. We had like closed off our relationship. You know, I told them how to get in touch with me outside of the company. You know, if they want any coaching or mentoring in the future, because as long as I’m working, I want to make sure I maintain good relationships with everybody. I think that’s the most important part of work, is relationships and colleagues.
Steve: And I think when we talked about it, you said, these people, you may manage them in another job, or they may manage you in another job, or they may interview you for another job, or you may interview them for another job.
Leanne: Yeah, exactly. Yeah. I mean, I have had people who worked for me in the past who will just say misbehaved and then applied for a job on my team, you know, at another company I was at. But just like that thing, like, I don’t remember what you said or what you did. I remember how you made me feel. I look at their resume in a stack and I’m like, no, like you did something that really harmed my agency or you did something that was really professionally inappropriate. You know, I can’t have you working for me anymore. That company that laid me off, the next company I was at, I was working with the sales team and they wanted to sell to that company that had laid me off. And I said, sure, I know people there. I can, I can help set up something. And we, so we set up this whole lunch. We were going to lunch at a restaurant in San Francisco and waiting in line to be seated was the person who had like, laid me off. And, you know, we gave each other a hug and I was like, you know, we’re okay. Like there doesn’t need to be an adversarial relationship when there’s business decisions made and when we all live in the same town and we’re all in the same tech community.
But people do set things up that way sometimes. And I think the harm it does is to yourself. It’s sort of like, I tell people, you have a choice of what you’re going to do with the feelings you have right now. You’re feeling frustrated. You’re feeling overwhelmed. You’re feeling disrespected, undervalued, whatever. Okay. Those are feelings. What are you going to do about it? Like you have control over your actions and your next steps and what you say and what you do. This is no longer about research. This is about being a human in a workplace.
Steve: Yeah, which is the core of research.
Leanne: Yeah. Well, and it’s also how to show value in research is to be that human who’s professional and can manage situations and keep a certain sort of like emotional regularity.
Steve: You know, the successful researcher or leader or manager who does a great transition when they’re laid off is also the person that is good at understanding what people’s needs are and proposing ways to help them accomplish it so that they collaborate and research together. It is the same set of values and life skills that you’re talking about. I mean, I think we went someplace really interesting and I like what you said, you know, these are human skills and I do think this is about research even though it’s about being a human.
Leanne: Well, that’s what I coach people who are looking for a job is to focus more on like, what is the hiring, like use your research skills. What is the hiring manager looking for? Does that fit you? Because there’s all sorts of researchers out there right now who are hurting because they can’t find work. I think the way that we find value again and find ourselves jobs again is to use those skills we have as researchers to understand what people want.
Steve: Is there anything in today’s conversation that we didn’t get to you bring up?
Leanne: No, that’s been great, Steve. I like the way it just sort of wandered. This has been fun.
Steve: Thanks Leanne for being just so wide reaching in what you have to share and kind of digging into a lot of related aspects. It’s really very interesting and inspirational for me personally to have this conversation with you. Thank you.
Leanne: Oh, you’re welcome. And thank you, Steve. Always a joy to talk with you.
Steve: Okay donut friends, thanks a whole heap for listening to this episode! Don’t forget, you can always find Dollars to Donuts where all the podcasts are, or visit Portigal dot com slash podcast for all of the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 43. Leanne Waldal returns first appeared on Portigal Consulting.
For this episode of Dollars to Donuts I had a wonderful conversation with Celeste Ridlen, the Head of Research at Robinhood
This is a fundamental leadership-y thing where no two people are going to do that same leadership role the same way. You’re never going to do them the same way as somebody else. And that’s actually a really good thing because the situation may call for exactly what you can offer. But because of that, if you’re looking to other people to decide whether or not you’re going to be suited to doing that role, it’s kind of like thinking about whether or not you should be a writer based on whether or not you can write exactly like Mary Shelley. – Celeste Ridlen
Show Links* Interviewing Users, second edition * Steve on the CX Chronicles * Celeste on LinkedIn * Robinhood * Cognitive Psychology, Florida State University * Dr. Michael Kaschak * Dr. Roy Baumeister * Dr. Dianne Tice * San Jose State University Human Factors and Ergonomics Master’s Degree Program * Oracle * Symantec Corporation * Time by Pink Floyd * Airbnb * Mary Shelley * Grateful Dead
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal.
Did you know there’s a new, significantly updated second edition of Interviewing Users? Of course you did. I hope you’ve checked it out and are recommending it to everyone you know. Shortly after the book came out, I had a conversation with Adrian Brady-Cesana for his CX Chronicles podcast. The link to the whole episode is in the show notes, but here’s a quick excerpt where we talk about domain expertise in user research.
Adrian Brady-Cesana: I’d love for you to share a couple examples or a couple stories of some of the things that you’ve seen working with your clients and working in your business around sort of how you’ve seen some of the companies that had really incredible teams or some of the commonalities or some of the things that you saw again and again and again with your clients that really had a solid handle on how they sort of built up their team, built out the different roles and really kind of stratified how their team was going to be taking care of their customers.
Steve: I think there’s a lot of pressure on people doing research right now to carry it yourself all the way through. And I think this is such collaborative work. And I just, I think I’ve seen more success when there is some collaboration and that’s a big, collaboration is a big, big term.
But one thing that, you know, your question makes me think of is complexity. Like I think as user research as a practice has grown, it’s finding its way into many more complex domains like installing and maintaining and configuring servers and network devices, not even just servers, but the whole, the whole infrastructure. I worked years ago on a credit default swap trading, and you might or might not have heard that phrase, but boy, just dig and dig and dig. And it’s like, it doesn’t make any sense until you’ve really been involved.
And, you know, so for me as a consultant, but even my clients who are on teams, like they’re, they’re not necessarily domain experts. And so I think this really interesting challenge comes up for whether you’re a researcher, whether you’re someone else in the organization that’s out talking to customers is, is trying to navigate that balance between like, how much do I need to understand about this?
And so for me, I think one thing I’ve seen that to be really successful, it goes back to the collaboration thing is when you pair up someone who’s great at research, which is, okay, I don’t know about this. I want you to explain it to me. And someone who is great at the domain, whose job isn’t to ask questions. Their job is to hear what doesn’t make sense about the technology or about the deployment or about the process. And that collaboration is really, really sharp, I think, and has a great effect on, you know, when you’re talking to customers and users, I think sometimes we’re nervous because while we want to be seen as credible, especially if it’s an actual customer, right? We ask for their time. We want to go talk to them. You’re going to send some idiot that doesn’t know what they’re talking about. That isn’t necessarily the reaction you’ll get, but I think it’s sometimes the reaction that we fear.
And so it can be really a really great triangle between a user or a customer who has, who’s a practitioner of something very complex, you know, and a person from the producer or, you know, maker side of it, the company side, who’s knows the domain and someone who knows how to listen and ask questions and follow up and sort of facilitate this.
When I see researchers kind of getting immersed into a domain, they do build up some competency, but some of these things are decades of specificity and really kind of elusive stuff. So I think just to go back to your question, I think teams where there’s bandwidth for collaboration and you can bring in people with different perspectives, different domain and process expertise to create a great interview for the customer that you’re talking to.
Like it’s a good experience to talk to a researcher and a domain expert because you just, you can watch who they make eye contact with as they kind of see like, oh, you’re the, I’ve had people even tell me, oh, okay, you’re the question asker and you’re the person that knows that you’re the engineer. Like people can figure that out. And it’s a really, nobody’s pretending to be anything that they aren’t.
And it really, I think can be very harmonious, but you have to create the bandwidth that kind of support that collaboration on the team. So everybody can work together to get the insights that we want to get from the people we’re building for.
Again, that was from the CX Chronicles podcast. Now let’s get to my conversation with Celeste Ridlen. She’s the head of research at Robinhood.
Celeste, thank you so much for being on Dollars to Donuts. It’s really great to have you here.
Celeste Ridlen: It’s awesome to be here. Thanks for inviting me.
Steve: Can we start with an introduction from you, build the rest of the conversation off from that?
Celeste: Yeah. What would you like to know in my introduction? I exist.
Steve: You exist.
Celeste: My name is Celeste. I’m the head of research at Robinhood. I’ve been doing research for 15 years now. My background is in human factors and ergonomics. I live in San Francisco, long walks on the beach, that kind of stuff.
Steve: How did you discover the field of human factors?
Celeste: I was in a cognitive psych lab, like working in a cognitive psych lab at Florida State University, and I was trying to think about my next step. I liked to joke that with a psychology degree and an English degree, you basically are qualified to be a mall security guard. I was looking at grad school and I decided that I was going to talk to Dr. Kashak who was running the lab at the time about my interests, because I had tried on neuroscience and I didn’t want to hurt animals and place electrodes on rats’ brains, so I cast aside neuroscience and then social psychology. I worked with Baumeister and Tice and that was interesting, but also so vague and not applicable, and much love to social psychology, but just so vague. I was in Dr. Kashak’s lab and I was asking him about what his advice would be for what I should study if I was most interested in cognitive psych. He asked me if I liked technology, if I liked industrial engineering, like making things. We had this great conversation about human factors, which is essentially cognitive ergonomics. As a field that started blooming right around the time we started industrializing weapons, which is a weird historical fact that I got excited about. That’s how I got into that.
Steve: How did you find a program for yourself? This was grad school. You did choose grad school.
Celeste: I did. Well, no, there wasn’t any, I mean, now there’s HCI and stuff like that, but at the time there weren’t programs dedicated to that yet. I looked around, there were a few programs. One of them was at Georgia Tech. One of them, I don’t even remember where they all were, but they were approximations of what I was looking for. Some of them were called human factors, some of them weren’t. The one I ended up going with was at San Jose State, and yes, right up there in terms of name and stature with Georgia Tech. But I chose San Jose State specifically because they had an applied, terminal master’s degree program, there was an applied emphasis. Where they had relationships with tech companies in the Bay Area and NASA, and they were working directly with these companies to get students into exactly the jobs that I was excited about. That’s why I chose it. I knew I didn’t want to be an academic. I had an end goal. I didn’t want things to be vague. I wanted to see the fruits of my labor immediately. Then I, sight unseen, came out to California, have never looked back.
Steve: Was there a first applied job or project or something that came to you through that program?
Celeste: Yeah, a lot of people had stuff come directly from school. I happened to sit next to somebody in one of my classes. I mean, I don’t know if it was like a stats class or I don’t remember which one it was. But there was somebody who was already working at Oracle, and she let me know about a job opening. And I wasn’t even done with grad school, and they took a chance on me. I was a contract physician. In retrospect, it was probably like a low lift, but it was my first sort of foray into things. And so, no, it wasn’t like a direct, it wasn’t directly because of like this, the program itself, but it was the people inside of it. Like you literally never know who you’re sitting next to. So it was lucky for me.
Steve: What kind of things were you doing in that first Oracle job?
Celeste: Oh, God. One time I transcribed an offsite, like a field visit that I did not attend. So I had headphones on, and I had to hand transcribe 20 plus hours of people spewing acronyms that I didn’t know or understand. So that was fun. I did a lot of like participant recruiting. This was a joy, but definitely a labor of love there. I did a lot of like synthesis and conducting studies, but I did a lot of the stuff that people either have automated now or would not consider in like an entry level research job now, which is character building, let’s call it. Without sort of going through your resume step by step, but what’s,
Steve: I don’t know, you can assess what the next marker is like, what’s another role that came after that, that was significant for you?
Celeste: Yeah. So I hopped from that to a full time position at Symantec about a year later. I don’t even think Symantec is a company anymore, which is at the time it was like 25, 30 years old. And so there’s sort of like two camps in tech, or at least there were at the time. These old, like slow moving companies with long, long product cycles, like Oracle, Symantec, it’s like there’s a lot to an implementation. And so you’ve got like a five year release cycle, like it’s really, really long. And then you had new companies that were very young, right, like on the edge of being what I would call startupy, like where it’s, everybody’s doing everything. And so I jumped from Symantec, which was like, it was a great job. They also took a chance on me. And I shifted into working in sort of like the newer building phase of tech where like there were almost no researchers. And we had to sort of build a process and perspective and relationships as the company’s like trying to grow like crazy. So Twitter was a really, really pivotal point in my career where I got really, really excited. It was like a dream job. I always wanted to work on Twitter. Like it was, you know, this 2013 Twitter, that means something different now, for sure. But I was so excited. And it was amazing to be surrounded by people that had that much passion and energy. And I, you know, I was part of that bricklaying process, right? Like I had a boss. And I loved it so much that I got invited to do another bricklaying at Airbnb a few years later. I was at Airbnb for a very long time. We went from 20 researchers to like, I don’t know, over 100 at our peak. So that was a lot of bricks.
Steve: Can you explain the bricklaying metaphor?
Celeste: Yeah, what? That doesn’t make sense to you?
Okay, so what I mean by that is you have this sort of like nascent or non-existent research team, like research function, let’s call it like a discipline. Maybe there’s one person, maybe there’s five people, but everyone’s just kind of like still being very reactive because the company doesn’t know how to work with research yet and doesn’t really know what part of the culture it fits in, where it fits, so on and so forth, like how to engage, that kind of stuff. So what I mean by bricklaying is like there’s process, right? So how do you recruit participants? What are sort of like the safety issues with your research participant agreement or your NDA? What are the safety issues with the way that you reach out to your participants? So I was like building a lot of programmatic structure on top of then hiring people, trying to identify and prioritize research questions, all the things, like all the how-to’s, all the — so I was interviewing like a million people every week. I was participating in like a crap ton of interviews for the company and then also of the users. So it was just nonstop assessing things, basically, but figuring out like do we have a crit, as an example, as a research team? Do we do like a weekly critique? What does that culture look like? Is it required? Are we forcing you to do that? Those kinds of things are the bricklaying. Like they’re all part of the bricklaying. It’s not just like hiring and setting up process, but it’s also like what kind of culture do we want to have and what’s mandatory, what’s optional, and like what needs to be grassroots versus top-down, that sort of thing. It’s fun. It’s really, really fun.
Steve: I don’t know if this is part of the bricklaying metaphor, but what about things like how much we know or don’t know about people that we’re building for?
Celeste: Yeah, that’s a big one. I mean, when you first start — let’s keep going with the bricklaying metaphor — when you first start, like you could put a brick anywhere, so to speak, and you would have impact, right? Because you didn’t know anything and now you know something. But that gets like more complicated and more nuances is necessary when you start, when you’re sort of saturating on a particular topic. There is a moment to either move on or rethink it. And so, yeah, that’s definitely a part of it.
Steve: Right. Now you got me thinking of the Tetris metaphor where the more bricks you put down, the more difficulty placing those future bricks. There’s more scrutiny and impact on that choice.
Celeste: Yeah. Yeah. I’m very good at Tetris. I don’t know if I’m good at bricklaying, but this is the third company I’ve moved to where I felt like that was a part of what my role entailed. Because it’s not just in research, too. It’s the company at large. You’re defining process and practice as a group of people who’s maybe under a thousand to then like three thousand, five thousand people in a very fast — like the only way to address any of that is to just figure out how to scale yourself and figure out what’s important and what isn’t, because you have to make some decisions very quickly there.
Steve: I do want to move on to Robinhod in a moment, but I have a question that I want to go back. So what’s your title at Robinhood?
Celeste: I am the head of research at Robinhood.
Steve: And hearing you talk, I’m inferring that bricklaying is not just the purview of people who have leadership titles. You’re describing Twitter and Airbnb as roles where you didn’t have a leadership title, I think, but your job bricklaying was part of what you were doing. It was the context for your work.
Celeste: Yeah, you and I could go on forever on a much belabored subject about the difference between management and leadership. I was a manager at Airbnb, but you’re right that the bricklaying metaphor is not specific to somebody with a leadership title or somebody who is a people manager. It’s something everybody has to build together. And it depends a lot on the chemistry of both the research team and the broader company you’re at. So it never looks the same way twice.
Steve: Well, I’d love to hear if it’s possible, you know, different companies, different roles, but being an individual contributor, being a people manager, being a leader, again, the companies are different. So maybe at the question, the comparison isn’t right. But would you be able to characterize either ways that one could or ways that you have been involved in bricklaying? I love the metaphor. Just these three different contexts for bricklaying, you’re coming into it with what seem like different responsibilities or titles.
Celeste: Totally.
Steve: What is it like? How do you compare and contrast across the three?
Celeste: I think in all three cases, the similarities were around, no one’s going to tell you that you absolutely need to be doing that unless you’re like, you know, the pet of something that is kind of objectively, everything is your job, right, to some degree. So I think that one, it’s a little bit easier to believe that everything is within your purview. But for other roles, I came in and there are opportunities, there are problems everywhere, and you can either decide that they’re not your problem, which is an approach, or you decide that you’d like to solve it. Nobody’s going to probably yell at you for solving it. So you take it on.
So things like rewriting interview questions, like that’s something small that I did. Creating processes and how-to docs was something I did like every five minutes at Twitter. And there was like a running joke that no one was going to know how to do anything after I quit, because there was no one to write how-to docs anymore. I’m sure they wrote them plenty.
But I think it’s just about deciding if you have a perspective or a skill that can be lent to that particular problem or like opportunity, and then just doing it. I mean, there’s nuance there, right? You have to ask questions and make sure you’re not hurting anybody’s feelings, someone else isn’t working on it, that kind of thing.
But a lot of times in those environments, the best thing you can do is just say, “Hey, no one has a problem. I’m going to do this. Good? Good? Yes? Anyone want to weigh in? No? Awesome. I’m going to take this on.” And then everyone is pretty thankful if they remember that you did it at all, which is, you shouldn’t be doing it for credit anyway.
Steve: Over the course of your history, have you worked in environments that weren’t in the bricklaying mode? It seems like the first two you described were that.
Celeste: Yes. Yeah. So both Symantec and Oracle had very well-funded and staffed research disciplines. They weren’t massive. I think Symantec, there were still only like six of us, but it’s not enormous for a 20,000-person company at the time. But there was already a way of doing things within the broader UX team, within the company. We had a process, we had practices, we had tooling. No one was starving for those sorts of things. And so everything was kind of plug and play. There were great jobs, but the environments themselves are different. You learn different. You get reps for doing the work there a lot more than doing the work and the stuff around the work, if that makes sense.
Steve: If I’m applying for a job in user research, let’s just imagine that’s the thing I’m doing. It’s like, ask your doctor if Flonase is right for you. Ask what kind of environment is right for whom? Having seen both and participated in both these, if there are only two, and I’m sure there’s nuances here.
Celeste: There’s a lot.
Steve: Yeah. If places where it’s more plug and play versus places where there’s a lot of problems to be solved and opportunity to solve them, what’s your advice for people into, I guess to me, I hear two parts. How do I assess what the situation is? And then how do I self-assess about what works for me?
Celeste: I think it seems to me like there are a lot of ways to do all of those things, but I’m going to tell you a couple of things. One is how to assess or how to self-assess. But then the second or third, depending on how you just numbered those in your head, would be people that I’ve seen succeed in either of those situations, of which there are probably shades, it’s a spectrum or something like that, because they might be different too. So when I think about assessing, when you’re interviewing for a job, I think the thing that I would look for is who’s funding the research team and why do they think that a research team should exist? Those are things that you would think you get the same answer every time, but you extremely do not. Sometimes people want a research team or have a research team because it’s a box to check. This is what we do to make products. This seems like a thing past a certain point of company maturity that we would definitely want to have. So I’m not entirely sure, but I’m going to, here you go. I’m going to just check that box and it’s going to be great. And you’re going to be my researcher. So there’s stuff like that that can kind of give you a glimpse into the research maturity of the organization or whatever you want to call it.
Another question I like to ask, so around funding, do you have the research? Do you have the tooling you feel like you need to do your job? If you needed a new tool, what would it take to get it? Is it a procurement question? Do you have a budget? Do you have to go talk to somebody else about budget? Those are other things.
And then asking questions even about headcount. So like, how did this role open? Why does someone think it’s important? Why did someone decide to do this over hiring another engineer? Sometimes those yield interesting answers too. And you can kind of tell where in a research maturity a company is based on that. What I see in terms of success for one sort of a person over another, I hate putting it like that because I think you can be a different sort of person at different places in your career too. But if you’re interested in sort of the bricklaying, let’s call it, I’ve seen people say they want to do that. And then they jump in, they realize like, oh shit, nobody writes anything down here. There is leadership lacking and cross-functional ways. And I can either take the reins or just get really upset. And you never know how you’re going to react in that situation until you find yourself in it.
So I’ve seen people who were like, yes, I want that. I want that. I’m excited about that. And then they get there and they’re just very uncomfortable, very frustrated. And what they actually wanted was the excitement of moving quickly, but they didn’t understand all of the things that surround that, that you need to take along with the moving quickly and the excitement.
Steve: What led you to Robinhood and the role that you have?
Celeste: Yeah. So this is a complicated question. So Twitter, Airbnb, and Robinhood, while they are pretty disparate in terms of topic, like your customers, things like that, they’re not in the same vertical. The similarities they have besides the company phase and stage is that they’re all very, very mission driven, like very strongly mission driven. And once you’ve worked at a company like that, I should just speak for myself here because maybe this is not exciting to everyone else, but once I worked at Twitter, I was like, I can never have anything but this ever again. Because when people really believe in it, when people really believe in the work you’re doing and believe it’s for a purpose that isn’t just like contributing to the capitalist abyss, it’s more motivating and more exciting to me to get up every day and like, and focus on a mission. It’s what I return to when I’m trying to prioritize things, when I’m feeling like I don’t know what to do. It’s a nice way to hold the center.
So on top of that, when I was leaving my last job, I interviewed, you know, there are many mission driven companies. And so like I interviewed for some heads of research roles actually, and I was even surprised to be offered some. I was leading the largest team at Airbnb when I left. It was like, it’s called hosting, which was Homes, which is the biggest part of the business, Homes, Community. We had an Olympics team and then experiences was also a part of it. I just felt like I was collecting Pokemon at that point. Like they just kept giving me stuff.
But so I basically decided that I didn’t want to be a head of research. This is why this is a funny story. I opted for Robinhood for the mission. And also because I was excited to lead research and like a lateral move across a few teams reporting to the head of research at the time. It was okay with me because I had all these like ideas about what being a head of research would mean, which is why I didn’t want to be one. So like I won’t be as close to the work anymore, or I’m going to spend all my days like writing and rewriting career frameworks, or I have to be super front and center. That’s another one that like comes up for me all the time. I’m not, I’m chatty, but I’m like fairly introverted. And I don’t personally love being like the star of the show.
So I was also kind of looking at people that I had reported to over the years and or known that like then went on to be heads of research. And I just couldn’t see myself doing what they were doing, or at least what I thought they were doing. And anyway, this is all about me joining Robinhood. But basically, through a series of twists and turns in my first few months, I found myself in the role I was avoiding, which is fun.
But looking back, I think my biggest gap in thinking at the time was that I forgot or I didn’t know that leadership roles are what you make them. So this isn’t very researchy, or even like UX-y. I think this is just like a fundamental leadership-y thing where no two people are going to do that same leadership role, whether it’s the head of research, the CEO, the COO, whatever, because those are all equivalent roles, right? The same way. Like you’re never going to do them the same way as somebody else. And that’s actually a really good thing because the situation may call for exactly what you can offer. But because of that, if you’re looking to other people to decide like whether or not you’re going to be suited to doing that role, it’s kind of like thinking about whether or not you should be a writer based on whether or not you can write exactly like, I don’t know, Mary Shelley. I love that that’s the first one I thought of. But she did it her way and she wrote Frankenstein, right? And then you’re going to do it your way and maybe not write Frankenstein. And just because you can’t write Frankenstein doesn’t immediately invalidate whatever it is you are going to write.
So that was a really long answer for why are you doing this job? But it was like a personal growth moment where the very thing I was avoiding, I had to confront. And I learned a lot about it.
Steve: Maybe there’s some blurring between what we think we can’t do and what we think or we know that we don’t want to do.
Celeste: That’s beautiful. Thank you. Oh, yeah. I agree.
Steve: Because I think you’re talking about like you’re looking at other, right, we all compare ourselves to people, whether we compare ourselves to Mary Shelley or someone who’s been ahead of research, but you’re calling out just to reflect back and kind of make sure I understand. You’re kind of calling out, well, you didn’t want to do the job because you didn’t want to sort of spend your energy and time and focus doing the things you saw other people doing. But the aha is that there’s a way for you to be being a leadership role that isn’t those things.
Celeste: That’s exactly it. The things that you’re good at are the things you should be doing as a head of research or as literally any other leadership role. And you should be surrounding yourself with people that are good at the things that you’re bad at, because together you’re going to make beautiful music or write Frankenstein, you know, whatever. Right. Whatever you want to extend there. I really like that that’s the first person I thought.
Steve: That’s a good improv moment. Don’t think about it. Just say it. I want to ask about “mission driven.” I think you’re saying, you know, once you sort of had a taste of that at Twitter, that became important to you. It was. And you wanted to. I guess I want to ask, is there a distinction between mission driven as just kind of a cultural quality and like the specific mission? Which of the bits of it are the ones that are calling to you so strongly?
Celeste: No shade to the broader corporate lifestyle or maybe much shade. I don’t know. But I think when you’re every company has like a mission and a vision and like values and they there’s lots of pomp and circumstance around those things. And I think the distinction is in the discussions that they’re integrating into every day. So at Oracle and Symantec, as much as I loved the people there and I respect and admire the work that they were doing, we weren’t talking about the broader like, what are all of us collectively across all these products trying to do together? What is our ultimate goal beyond making money? Like what’s important to us? What is the legacy we want to leave as a company? And yeah, it exists somewhere written down in some wiki, corporate wiki. But like it just it wasn’t a part of the day to day conversation.
Whereas like at Twitter, we were obsessed with being the global town square. We were obsessed with making it with enabling people to discuss and connect and like communicate. And it was really exciting and invigorating to be working side by side with people that were like, I don’t know how we’re going to do this, but we’re going to do it and it’s going to be amazing.
And then at Airbnb, belong anywhere isn’t just like an advertising tagline. Like everybody is talking about the ways that we’re going to make people feel more or less like they belong based on design choices, strategic directions. Like it’s infused in everything.
And it’s the same with Robinhood, ours is democratized finance for all. We talk about it literally every week. It is a constant in every meeting. It emerges, the language emerges, we’re weighing trade-offs and thinking about it.
And it sounds a little culty at its worst, but I like to think that at its best it’s a force for good. I think you are what you measure. And so when you’re running a lot of experimentation and you’re looking at all these metrics that maybe build up to something that you didn’t actually want to aim towards, but just made sense in the individual examples, you can return to the mission as like, okay, but are these metrics democratizing finance or are they doing this other thing over here? Have we lost sight of it? Whereas at Oracle, at Symantec, I felt like that was not necessarily the lighthouse.
Steve: Let me throw a different metaphor in here because we’re doing so well with them.
Celeste: We’re killing it with the metaphors.
Steve: I wouldn’t go to a, well, I guess we’d have to turn on a time machine, but I wouldn’t go to a Grateful Dead concert for absolutely anything. But if I was going to go to one, going with a friend of mine that loves music and loves the dead would be the way to do it, to be in that experience with someone who is into it. I’m using that as an analogy for the mission-driven thing. I don’t care about if the dead and whatever the dead is about is the mission. I don’t actually care about it. It’s not my mission. Belong anywhere might or might not be my mission.
But if you’re going to go to a concert or if you’re going to work in an environment, one where that kind of passion and commitment and attention to detail and thoroughly building out every aspect of decisions being made based on that, you’re kind of highlighting how powerful that is and how rewarding that is. I guess I’m asking, is that still true if the mission is one that you are ambivalent about, say, versus 100% bought into? That’s what I’m probing on here.
Celeste: Yeah. First of all, I love the metaphor because you’re right. You could go to a show, and if you went to the show with your best friend and they were crazy about the dead, you would have a completely different experience because energy is a very human thing. Energy is infectious that way. The passion, the enthusiasm, you can’t help but kind of like, “Oh man, I was going to say ‘Catch a whiff.’ That’s a little too on the nose with the Grateful Dead reference.” But we’re going to do this all day.
Yes, I actually do think that if you are fairly ambivalent about the mission. I don’t think I woke up in 2021 and was like, “You know, what really needs to happen in the world is finance needs to be democratized for everyone, for all.” I don’t think that I woke up feeling that way. I definitely don’t think that I looked at Twitter’s mission and was like, “Man, I have never been more like — it is my life’s calling to be a part of the global town square.” I think the difference is that the mission has to resonate at least a little bit. You have to be like, “Yeah, of course. I mean, yes. Do I agree with the idea of democratizing finance for all? Absolutely. I love that.” I didn’t know I wanted to do that until I started looking into it. But it doesn’t make it any less important to me, especially if I’m surrounded by people that are all agreeing and believing in the same thing. So it helps, but I don’t think — it’s not like you have to be born with that passion in mind in order for it to be infectious. I was looking for a less disgusting word, but that’s what we’re going with.
Steve: When we’re talking today, start of spring 2024, how long have you been at Robinhood?
Celeste: Oh boy. How long have I been at Robinhood? I’ve been at Robinhood for two years, and I’m sighing deeply because every year is getting shorter in my life. Just like Pink Floyd promised. I’ve been here for two years. I just celebrated my two-year anniversary in the beginning of December, so a lot has happened during that time. I started in December of 2021. A few months after that, the head of research left.
My job changed every two to three months for probably over a year, which isn’t inherently weird, except that 2022 happened, which meant that the crypto markets and the stock markets, everything was going down in 2022. There was a huge wave of layoffs across tech. We also had to lay people off. All of that is super difficult as an employee, but also as a leader of a team. It’s really, really tough, especially when you spend some time thinking that this is not what you wanted to do. So when I look back at the two years, it feels like more than two years, but it also feels like I just blinked and two years passed. So there’s a lot of cognitive dissonance in that length of time.
Steve: What’s the cognitive dissonance?
Celeste: It feels like a lot and like a little at the same time.
Steve: Yeah. So over those two years, what are some bricks that you’ve laid?
Celeste: So the team predates me by a lot. It’s not like I built the team and built all these processes. The team predates me by a lot. It was originally put together by my predecessors. I deeply appreciate everybody who did that for what and who they left behind. Those have been gifts. But the bricklaying in this case was that Robinhood had just IPO’d over the summer and was sort of nestling into everything changes when you IPO.
Airbnb IPO’d while I was there, Twitter IPO’d while I was there. Everything changes. There’s a lot more you need to do. Because your funding looks different, there’s a lot more that you need to be held accountable to. And so a lot of process changes when that happens. The research team had also been through a lot. There were some big, dramatic leadership and team changes during the entire year of 2021, and I showed up in December. So a lot of my bricklaying, I wouldn’t say it was like building a team from the ground up, but it was sort of like there was a lot of healing that had to happen as a group of people.
When I started, I was hearing a lot about people not trusting each other or feeling like if so-and-so got promoted, why didn’t I? I don’t even think their work is good. There wasn’t really very much teaminess. So a lot of the bricklaying was rebuilding cohesion and trust, thinking about things differently, re-evaluating tooling because everybody, when you’re laying people off, are also reassessing the budget and the tools that you need or don’t need. So a lot of really tricky stuff like that happening at the same time. And then kind of re-evaluating research’s relationship with the company, which was a pretty tall order, but it’s been fun.
Steve: What are things that you can do to build teaminess?
Celeste: Not sure I have a good answer to this because I’m going to give you that delicious research answer, it depends.
Steve: Yes!
Celeste: You’re welcome.
Steve: You’re going to ring a bell right now.
Celeste: Yeah. Someone somewhere is furious at me for this, but it depends on the situation. We had layoffs at Airbnb when I was still there and it was devastating and we had to sort of rebuild our emotional baseline also. And it was tough to find all the loose ends and figure out what work still needed to be done, what work had fallen off and that was okay, who we were without the people that we had lost and things like that. It’s really, it’s hard to be laid off and it’s very hard to lay off. It’s just like a no one wins situation.
So in that, in like the Airbnb case, the teaminess came from just, actually it might be the same as Robinhood, being consistent. Everybody showing up and being exposed to each other and just talking about what was hard, what wasn’t working, do people have ideas, here’s what I’m doing, what are you doing, but being really active and pushing for contact and like pretty regular consistent contact and trying to foster moments of recognition when something was going well. There was a lot of stuff like that.
I don’t know if I have like a silver bullet answer though, because it depends so much on like the way that things are bad or that need healing. We definitely went from people not being willing to help each other because they were worried about not getting credit to people helping each other without thinking twice about getting credit and those were signals to me that we were like nature was healing. We’re on the right track.
Steve: I know you’re apologizing a little bit for the answer not being like a full list of three things like you didn’t say, oh, we had an offsite or we had a cake, so it’s like, but I really am struck by the fact that you’re talking about like intentional ways of being that are maybe smallish, but that are sustained over time. But that sounds much harder and much sort of less obvious to come up with or to execute. And that seems like, well, if you want to make change, it does depend, but there’s a set of tools that you’re drawing from and a set of principles is, I don’t know, subtler than a fix it kind of approach.
Celeste: Fixing it, especially with something as fragile as like people’s chemistry and sentiments feels like a fool’s errand. Like coming in and being like, let me show you how to do things. I’m going to fix these feelings. That’s not really, I don’t think you’re going to get anywhere if you approach it that way. Someone has probably done that, but I don’t think that’s possible in the toolkit that I have. Maybe Mary Shelley’s done it.
Steve: She was a great management and leadership consultant in her coaching business.
Celeste: Yeah, good call. What comes to mind is that there were two things that were similar about both circumstances. One was sort of the community that I’m talking about, which like you can’t inauthentically build community. You have to push it through connection. You have to get people to connect with each other. And sometimes it’s awkward and it’s not mandatory fun. That doesn’t work. But if you can give people real reasons to show up and be there for each other and be honest with each other, you’re going to, it would be hard for that not to move in a positive direction. I think human beings just crave connection and community. Even the introverted ones, it turns out.
But the second thing is like, this is a businessy answer, but just hear me out. Transparency. So here’s what I know. Here’s what I don’t know. Here’s what I’m doing. Things like that are during, especially things like layoffs or moments where the team has really lost a lot of trust in either each other, the situation around them, the circumstances. They know that you’re doing everything you can to contribute to a shared understanding. It’s again, it’s hard not to move in a positive direction if people are like, you’re being as honest as you can with me. And that opens the opportunities for, I’m having a hard time with this and I didn’t want to say anything until I felt like you were also showing up and spilling your guts about what’s happening. So yeah.
Steve: You also mentioned re-evaluating researchers’ relationship with the company. What is that about?
Celeste: The company loves research. I don’t think the company will ever stop loving research. Some of that, again, love dearly my predecessors and their approach to it. I got really lucky, but, and it was always going to have a place because our co-founders started doing research themselves. When they started the business, they focused a lot on listening to people, on observing people do things, like on very researchy things. So the research team was always going to be a core function at Robinhood. I didn’t invent that. I’m not going to create that. But when I started, there was this, they had grown a ton very quickly, like the entire company did. And every kind of PM, GM, executive marketer up and down the chain was and still is asking about research. We need to do it on every last thing.
We have an unholy amount of opportunities to be user-centered in what we’re doing. And the tone for that is consistently being set at the top, which is delightful. And I don’t feel like I’ll ever see that quite in the same way in my career. So it just feels like I’ve struck gold. They’re always asking about our customers and their perspective. They know that our co-founders are doing the same. Everybody’s asking about this.
But when no one ever questions if research should be involved, and if everyone is insisting that research study everything all the time, you’re faced with a different set of problems. Your relationship to the organization is different because researchers, as I’m sure you’ve seen this too, we’re so happy to be included and consulted because we’re so unused to that that our relationship ends up being, yes, I’m going to say yes to everything because I’m so excited to be in this position and be consulted and be listened to.
And so that’s going to take anybody who starts at Robinhood by complete surprise. Everyone’s always like, “You said we had a seat at the table, but holy smokes, we really have a seat at the table.” Not prioritizing, not saying no. You end up spreading yourself really thin. You end up studying things you really don’t need to study because the leverage isn’t there. It’s not some sort of force multiplier all the time. Not everything needs research.
Actually there’s a general manager that I work with that loves to use this metaphor. Speaking of our metaphors, he likes to say, “If I’m opening up an ice cream shop, do I really need to do research that I should offer vanilla? I just know that vanilla should be on the menu. So do we really need to do research on that and why?” My response to that, of course, is like, in the vanilla answer, sure, but do we know that where you’re opening the ice cream shop is actually a place where people are interested in vanilla? Because you’re assuming a really narrow, maybe it’s Americans only, but whatever. There’s a group of people that like vanilla generally universally, but very generally. Is that target market where you’re opening up your shop? On top of that, is it a fancy vanilla? Are we using elaborate beans from some rare island or whatever? Or is it this just straight up vanilla, no frills? What is resonating? What’s needed based on the context of the situation?
But also I kind of agree with him that maybe we don’t need to do a study about vanilla at all. Maybe we need to understand everything around the vanilla. And that to me is I think the relationship with research that is ongoing and needs to change. If you can’t, as a product manager or some other leader, have a perspective on something without research, I feel a little uncomfortable with that. You should have a point of view. I want information to inform it. But if I don’t, because it’s a fairly inconsequential thing, I think you should still be able to make that choice anyway so that I can work on the stuff, my team can work on the stuff that’s the most important based on the things we’re good at. Not every question is going to be able to be answered by research. So that’s what I mean by the relationship. And it’s nobody’s fault that this is the state of affairs. It’s just what happens when the pendulum swings really hard in the other direction.
Steve: Right. You talked a little bit about saying no, but in the vanilla example, it’s almost like, hey, here’s seven more questions that are harder to parse out, riskier if your assumptions are wrong, and that create more context to the vanilla question begs a bunch of larger questions. And so that’s not saying no, though. That’s like, no, we’re not going to do the vanilla thing. But hey, how’s about…
Celeste: There you go. That’s what I mean. It’s not that I think we should be disengaging from every part of the conversation. It’s just that we need to be asking the right questions. And sometimes the questions that we’re being asked that we say yes to are still vanilla ice cream questions, like just the baseline yes or no vanilla without the nuance, the subtext, all the stuff around the things. And again, it’s just because research is such a part of the product building and marketing culture at Robinhood that people come to you with very strong requests about doing vanilla ice cream research. And we just have to keep making sure that we’re working on the things that are the most important from a business UX timing perspective to make sure that we’re not just answering lower leverage questions. What’s the approach to changing that relationship in the way that you’re articulating?
I spent a lot of time with product leaders and GMs and things like that talking about what the most important decisions they need to make, questions they have are, and then kind of going over… I’ve been made fun of before because I’m like, that’s an experimental question. Like straight up, just run an experiment. It’s faster and more reliable. You’ll have more confidence if you run an experiment. People laugh at me because they’re like, well, wait, don’t you collect data a different way? Yeah. I mean, just because I’m a hammer doesn’t mean everything looks like a nail to me. So I spend a lot of time doing that because it does need to come from the top. But then making sure that people have the tools to be able to feel like they can say no because a lot of researchers feel like it’s going to damage the relationship between them and their product teams if they’re not able to do everything that’s being asked of them, which is disheartening. So we work a lot on that.
We talk a lot about what a higher leverage question looks like or how to meet the short-term moment of what the team needs while making it into a longer-term study also. So I don’t mean in length of time, but I mean, I’m sure you’ve heard this a bunch, Steve. You know how there’s this debate within research about whether strategic or tactical is a better use of our time? I find that interesting because I don’t think they are separate. I think that you can make sort of like what people would refer to as very highly tactical research like, “Do people understand this? Whoa!” You can make it strategic by asking questions inside of it and putting together a broader narrative. If you’re telling me that like, “I’m too senior to do usability testing,” or whatever it is, I’ve heard that multiple times, which is unholy. But it tells me more about how you’re thinking about structuring your research in a way that you’re thinking about insights and what value you bring to the team. It tells me a lot more about that than it does about your seniority or anything else. And so it’s not about like doing less usability research, but it’s about doing the research that really needs to be done. And so we talk about how to prioritize, we talk a lot about like this over that, and we’re just pretty open about what’s important and why. Back to your point about transparency, I think. I’ve been wrong before.
Steve: Yeah? Not in this conversation.
Celeste: Probably several times in this conversation, but never about Mary Shelley. You’re welcome for that.
Steve: I’m thinking about this relationship between researchers on your team and any particular product team and so on, where, yeah, what you said, there might be a near-term question, but there’s a way to do that research that also… \Either you refactor that question into a higher value question, or you do, and you didn’t say this, but I am going to say yes and, you do kind of a yes and of the short-term implication and the longer-term implication. But just this whole exploration of like, what are research being asked for? What does research provide? Just makes me think about questions that come up, like, do we give recommendations? Like what are the kind of outputs of research? And I know it depends. Of course it depends. But does this, I don’t know, does this area provoke anything for you, a point of view, or something you’re thinking about?
Celeste: On the topic of recommendations, I’ve seen at some companies the idea that if people aren’t taking your recommendations, that you have not had impact, which I find charming. Like I personally do not want to work somewhere. I’d be really concerned if someone was taking 100% of my team’s advice all the time. We’re looking through a really specific lens. There’s all these other inputs to consider. It’s really great when the research is informing a decision, but sometimes the decision is not going to go with what the research is recommending. And I don’t think that’s a failure on the researcher, as long as everybody’s aware of the trade-offs and everyone understands the insights.
And by insights, what I mean is like what the data itself translates to for us as a product, as a business, whatever the case may be. It’s okay to have wrong recommendations, but it would be weird if you were taking my advice all the time because it would mean that you’re really only considering one input or weighing it heavier than others. So that feels weird, but I do think, to your point about it being a bit of a controversial topic, I do think we should have recommendations, like be giving recommendations, not because we’re all geniuses, although maybe that’s true. But because we are the most informed about what the data means, what we can say and what we can’t say based on what we did, thinking about things like intellectual honesty and rigor and all of that, I don’t know why we would ever think that we were not suited to making some sort of recommendation.
Because other people are going to take the information that you give and then just make their own recommendation, but that’s filtered through an entirely different lens with way less of the broader context that you’ve collected. It’s okay if your recommendation is bad, and you should probably shape it in a way that makes it shelf-stable, right? If you say, “Move the button to the right,” that’s not shelf-stable, because at some point someone will probably look at your research years down the line and go, “What in the hell were they talking about when they made that recommendation?” But people preferring design A over design B is just sharing data. People preferring design A over design B because the information they were looking for was a lot easier to find, that’s more shelf-stable, right? You understand why, you understand the context of the insight itself, and even if I don’t do exactly what you say, maybe I end up going with design B anyway, maybe I can make design B better based on the way that you frame that. So there’s a lot in there.
Steve: I want to pick out what a recommendation is, because people prefer design A over design B, that’s data. People prefer design A over design B because they can find the information. Okay, that’s an insight. So then the recommendation would be, go with design A.
Celeste: Right. Yeah, but you have to have that other piece in there for it to make any sense at all.
Steve: We should go with design A because it helps people to find the information they’re looking for.
Celeste: Words are hard, Steve.
Steve: I don’t know, just as a counter-argument a little bit, and I think you’re on board with this, the risk of being wrong there is you only know what you know about that. So design B makes us more money, design B we can implement faster, design B is compliant with something, design B is consistent with what we’re doing in three other platforms.
Celeste: Absolutely.
Steve: So I think that’s an example of them not going with design B and that being okay because you’ve given them the information that you have to make that recommendation.
Celeste: Yes. So I would argue that if you’re not incorporating things like which design would make us more money or if you’re not incorporating those pieces of information into your process, you’re leaving value on the table. You should be helping the person who needs to make the decision make it with all that context in there. Anything you can help them with. You have an amazing sense of synthesis and distillation and not everybody’s fantastic at that. So if you can pull that in, you probably should do that. But yes, you’re right that even if your recommendation is wrong, it’s not always that you’re completely incorrect. It’s like not right now because we would prefer to make more money with design B or whatever it is. But you’ve planted this seed and I understand that what people need is not what this is providing and so we can get successive approximations closer to it.
Steve: Right. I think the recommendation discussion sometimes leaves me cold because it loses all the context that you’re providing. It’s the recommendation and the insight in the larger context of the set of things that might be factors for this team. And I think you’re saying you should try to know all of them.
Celeste: Why not?
Steve: I’ll just add there’s always going to be something that you won’t know.
Celeste: Of course.
Steve: I don’t know, it reminds me of like trying to write like Mary Shelley. I think if you do, if you do like creative writing and you do workshops like you get feedback that says, here’s what the problem is and here’s what the solution is. And I think it can be really helpful when you’re the writer to choose not to do that, but to do it, ignore the advice in an informed way. Well, that’s not what my objective is. That’s not my strategy. We need short term wins here, whatever. And I think we’re agreeing strongly here. And it’s just some of the language gets oversimplified when it gets, you know, boiled down to like a social media posts.
Celeste: Yeah, I think this particular topic is one that is not great for like a LinkedIn rant. And yet, it’s all we see.
Steve: But great for a podcast episode, right? Because we can dig in.
Celeste: Sure.
Steve: You brought up my other another one that I take umbrage at which is the, you know, not we’re not having impact if it doesn’t get taken up. And I liked what you said. I’ve also heard people say my research doesn’t have value. If no one takes action, it’s even more binary, right? If that my research impact is sort of higher level than value. My work is worthless. If someone doesn’t do the thing that I told them to do. And I feel so sad because that’s researchers saying that, like as if we don’t have enough people telling us we’re without value. We’ve sort of taken on as a profession taken on some of that, you know, willingly devaluing proactively. I’m going to devalue it for you. It’s depressing. And that makes me sad.
Celeste: Yeah, I agree. It also not to I’m sure that the folks saying this are perfectly smart people. But to me, when I hear that kind of stuff, it feels like it lacks curiosity. So you’re saying that unless there’s like something I can see in front of me, or unless there’s like a direct action being taken, that I have failed when I think if you had a little more curiosity about it, you could notice little hints like, well, but these people are changing their language. And they’re using the language of the people that we’re studying instead of whatever business bullshit we’ve come up with. And that’s impact.
I think there’s we’ve changed the roadmap where we’ve just decided not to go this route. We’ve saved three months of engineering. So it’s almost that inaction is action. There is there’s lots of stuff like that. And I think if you are open to seeing it, it makes itself known to you. But if you are not curious how your research is sort of creeping through an organization, you’re going to end up with it flying right over your head. And then you assume you have no value or whatever it is you said, it’s depressing.
Steve: You know, you’re bringing your the same way that you talked about bringing teaminess is it seems very analogous to the way you’re talking about both creating and sort of realizing the value that it’s smaller signals over a longer period of time with some I’m not sure if the constancy thing applies here or not.
Celeste: Maybe.
Steve: But it’s a binary outcome kind of thing that yeah, taking a slower and more looking for looking for those subtle more subtle signals.
Celeste: Yeah, you’re reminding me of I’ve had this conversation a few times after becoming a manager where people are like just shouldn’t do research on this area because they’re not valuing it the way they should, which often the subtext is they’re not taking my advice or they’re doing things the way that I fundamentally disagree with. You probably know this better than anybody, but this is not a field for the faint of heart. This is like a full on eternal marathon. It took me three years at Airbnb and many different approaches to convince people that non drip pricing was the right call. It cost us a lot of money and we had to find ways to offset that. And every time I got knocked down about it, I could have chosen to never bring it up again.
But it’s chipping away. It’s finding advocates. It’s finding other people that agree or believe it. If I can convince one person each time of something that I thought that I believe in, I’m doing great. That’s impact. And so you’re right that there’s like little subtleties, but it’s also just a long game. Like all of this is a long game. You’re lucky if you have a short term win. You’re super lucky.
Steve: Anything else to add today?
Celeste: No, I mean, thanks for inviting me to do this. It’s very flattering to talk about my thoughts and feelings on things we mutually like for however long this just was. It was very nice. Thank you.
Steve: Yeah, thank you. You shared a lot of interesting perspectives and good history. So yeah, thanks a lot for taking the time.
Celeste: This was awesome.
Steve: Over and out, good buddies. That’s all for this episode. Thanks for listening. Recommend Dollars to Donuts to your peers. You can find this podcast in all of the usual places. A review on Apple Podcasts helps others find it. Go to portugal.com/podcast to find all the episodes, including show notes and transcripts. Our theme music is by Bruce Todd.
Celeste: Did you say a favorite puppet?
Steve: Favorite puppet.
Celeste: What came up for you just now?
Steve: You know so many things about so many things and I was just trying to open up the possibility space.
Celeste: Favorite puppet. I’m going to be thinking about that. What is my favorite puppet? Now I know exactly. It’s the, I can’t remember the other guy’s name, but it’s the two angry balcony dudes in the Muppets. One of them is named Astoria or something like that, or Waldorf, and the other is named after another hotel and it’s just not coming to me. But those are.
Steve: His name is, his name is Statler. Statler. I just Muppet, Muppetsplained you here.
Celeste: You win. You win the Muppets.
Steve: Yeah.
Celeste: I also just like that there’s no, nothing is sacred to them. They just go after whatever, whatever is in front of them. And they do a lot of like lame dad joke moments too.
Steve: So. That’s excellent. I’m not going to get any actual puppet content.
Celeste: Puppet content
Steve: So thank you for, thank you for taking that literally. I didn’t know what you were going to share, but…
The post 42. Celeste Ridlen of Robinhood first appeared on Portigal Consulting.
For this episode of Dollars to Donuts I had a wonderful conversation with Celeste Ridlen, the Head of Research at Robinhood
This is a fundamental leadership-y thing where no two people are going to do that same leadership role the same way. You’re never going to do them the same way as somebody else. And that’s actually a really good thing because the situation may call for exactly what you can offer. But because of that, if you’re looking to other people to decide whether or not you’re going to be suited to doing that role, it’s kind of like thinking about whether or not you should be a writer based on whether or not you can write exactly like Mary Shelley. – Celeste Ridlen
Show Links* Interviewing Users, second edition * Steve on the CX Chronicles * Celeste on LinkedIn * Robinhood * Cognitive Psychology, Florida State University * Dr. Michael Kaschak * Dr. Roy Baumeister * Dr. Dianne Tice * San Jose State University Human Factors and Ergonomics Master’s Degree Program * Oracle * Symantec Corporation * Time by Pink Floyd * Airbnb * Mary Shelley * Grateful Dead
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal.
Did you know there’s a new, significantly updated second edition of Interviewing Users? Of course you did. I hope you’ve checked it out and are recommending it to everyone you know. Shortly after the book came out, I had a conversation with Adrian Brady-Cesana for his CX Chronicles podcast. The link to the whole episode is in the show notes, but here’s a quick excerpt where we talk about domain expertise in user research.
Adrian Brady-Cesana: I’d love for you to share a couple examples or a couple stories of some of the things that you’ve seen working with your clients and working in your business around sort of how you’ve seen some of the companies that had really incredible teams or some of the commonalities or some of the things that you saw again and again and again with your clients that really had a solid handle on how they sort of built up their team, built out the different roles and really kind of stratified how their team was going to be taking care of their customers.
Steve: I think there’s a lot of pressure on people doing research right now to carry it yourself all the way through. And I think this is such collaborative work. And I just, I think I’ve seen more success when there is some collaboration and that’s a big, collaboration is a big, big term.
But one thing that, you know, your question makes me think of is complexity. Like I think as user research as a practice has grown, it’s finding its way into many more complex domains like installing and maintaining and configuring servers and network devices, not even just servers, but the whole, the whole infrastructure. I worked years ago on a credit default swap trading, and you might or might not have heard that phrase, but boy, just dig and dig and dig. And it’s like, it doesn’t make any sense until you’ve really been involved.
And, you know, so for me as a consultant, but even my clients who are on teams, like they’re, they’re not necessarily domain experts. And so I think this really interesting challenge comes up for whether you’re a researcher, whether you’re someone else in the organization that’s out talking to customers is, is trying to navigate that balance between like, how much do I need to understand about this?
And so for me, I think one thing I’ve seen that to be really successful, it goes back to the collaboration thing is when you pair up someone who’s great at research, which is, okay, I don’t know about this. I want you to explain it to me. And someone who is great at the domain, whose job isn’t to ask questions. Their job is to hear what doesn’t make sense about the technology or about the deployment or about the process. And that collaboration is really, really sharp, I think, and has a great effect on, you know, when you’re talking to customers and users, I think sometimes we’re nervous because while we want to be seen as credible, especially if it’s an actual customer, right? We ask for their time. We want to go talk to them. You’re going to send some idiot that doesn’t know what they’re talking about. That isn’t necessarily the reaction you’ll get, but I think it’s sometimes the reaction that we fear.
And so it can be really a really great triangle between a user or a customer who has, who’s a practitioner of something very complex, you know, and a person from the producer or, you know, maker side of it, the company side, who’s knows the domain and someone who knows how to listen and ask questions and follow up and sort of facilitate this.
When I see researchers kind of getting immersed into a domain, they do build up some competency, but some of these things are decades of specificity and really kind of elusive stuff. So I think just to go back to your question, I think teams where there’s bandwidth for collaboration and you can bring in people with different perspectives, different domain and process expertise to create a great interview for the customer that you’re talking to.
Like it’s a good experience to talk to a researcher and a domain expert because you just, you can watch who they make eye contact with as they kind of see like, oh, you’re the, I’ve had people even tell me, oh, okay, you’re the question asker and you’re the person that knows that you’re the engineer. Like people can figure that out. And it’s a really, nobody’s pretending to be anything that they aren’t.
And it really, I think can be very harmonious, but you have to create the bandwidth that kind of support that collaboration on the team. So everybody can work together to get the insights that we want to get from the people we’re building for.
Again, that was from the CX Chronicles podcast. Now let’s get to my conversation with Celeste Ridlen. She’s the head of research at Robinhood.
Celeste, thank you so much for being on Dollars to Donuts. It’s really great to have you here.
Celeste Ridlen: It’s awesome to be here. Thanks for inviting me.
Steve: Can we start with an introduction from you, build the rest of the conversation off from that?
Celeste: Yeah. What would you like to know in my introduction? I exist.
Steve: You exist.
Celeste: My name is Celeste. I’m the head of research at Robinhood. I’ve been doing research for 15 years now. My background is in human factors and ergonomics. I live in San Francisco, long walks on the beach, that kind of stuff.
Steve: How did you discover the field of human factors?
Celeste: I was in a cognitive psych lab, like working in a cognitive psych lab at Florida State University, and I was trying to think about my next step. I liked to joke that with a psychology degree and an English degree, you basically are qualified to be a mall security guard. I was looking at grad school and I decided that I was going to talk to Dr. Kashak who was running the lab at the time about my interests, because I had tried on neuroscience and I didn’t want to hurt animals and place electrodes on rats’ brains, so I cast aside neuroscience and then social psychology. I worked with Baumeister and Tice and that was interesting, but also so vague and not applicable, and much love to social psychology, but just so vague. I was in Dr. Kashak’s lab and I was asking him about what his advice would be for what I should study if I was most interested in cognitive psych. He asked me if I liked technology, if I liked industrial engineering, like making things. We had this great conversation about human factors, which is essentially cognitive ergonomics. As a field that started blooming right around the time we started industrializing weapons, which is a weird historical fact that I got excited about. That’s how I got into that.
Steve: How did you find a program for yourself? This was grad school. You did choose grad school.
Celeste: I did. Well, no, there wasn’t any, I mean, now there’s HCI and stuff like that, but at the time there weren’t programs dedicated to that yet. I looked around, there were a few programs. One of them was at Georgia Tech. One of them, I don’t even remember where they all were, but they were approximations of what I was looking for. Some of them were called human factors, some of them weren’t. The one I ended up going with was at San Jose State, and yes, right up there in terms of name and stature with Georgia Tech. But I chose San Jose State specifically because they had an applied, terminal master’s degree program, there was an applied emphasis. Where they had relationships with tech companies in the Bay Area and NASA, and they were working directly with these companies to get students into exactly the jobs that I was excited about. That’s why I chose it. I knew I didn’t want to be an academic. I had an end goal. I didn’t want things to be vague. I wanted to see the fruits of my labor immediately. Then I, sight unseen, came out to California, have never looked back.
Steve: Was there a first applied job or project or something that came to you through that program?
Celeste: Yeah, a lot of people had stuff come directly from school. I happened to sit next to somebody in one of my classes. I mean, I don’t know if it was like a stats class or I don’t remember which one it was. But there was somebody who was already working at Oracle, and she let me know about a job opening. And I wasn’t even done with grad school, and they took a chance on me. I was a contract physician. In retrospect, it was probably like a low lift, but it was my first sort of foray into things. And so, no, it wasn’t like a direct, it wasn’t directly because of like this, the program itself, but it was the people inside of it. Like you literally never know who you’re sitting next to. So it was lucky for me.
Steve: What kind of things were you doing in that first Oracle job?
Celeste: Oh, God. One time I transcribed an offsite, like a field visit that I did not attend. So I had headphones on, and I had to hand transcribe 20 plus hours of people spewing acronyms that I didn’t know or understand. So that was fun. I did a lot of like participant recruiting. This was a joy, but definitely a labor of love there. I did a lot of like synthesis and conducting studies, but I did a lot of the stuff that people either have automated now or would not consider in like an entry level research job now, which is character building, let’s call it. Without sort of going through your resume step by step, but what’s,
Steve: I don’t know, you can assess what the next marker is like, what’s another role that came after that, that was significant for you?
Celeste: Yeah. So I hopped from that to a full time position at Symantec about a year later. I don’t even think Symantec is a company anymore, which is at the time it was like 25, 30 years old. And so there’s sort of like two camps in tech, or at least there were at the time. These old, like slow moving companies with long, long product cycles, like Oracle, Symantec, it’s like there’s a lot to an implementation. And so you’ve got like a five year release cycle, like it’s really, really long. And then you had new companies that were very young, right, like on the edge of being what I would call startupy, like where it’s, everybody’s doing everything. And so I jumped from Symantec, which was like, it was a great job. They also took a chance on me. And I shifted into working in sort of like the newer building phase of tech where like there were almost no researchers. And we had to sort of build a process and perspective and relationships as the company’s like trying to grow like crazy. So Twitter was a really, really pivotal point in my career where I got really, really excited. It was like a dream job. I always wanted to work on Twitter. Like it was, you know, this 2013 Twitter, that means something different now, for sure. But I was so excited. And it was amazing to be surrounded by people that had that much passion and energy. And I, you know, I was part of that bricklaying process, right? Like I had a boss. And I loved it so much that I got invited to do another bricklaying at Airbnb a few years later. I was at Airbnb for a very long time. We went from 20 researchers to like, I don’t know, over 100 at our peak. So that was a lot of bricks.
Steve: Can you explain the bricklaying metaphor?
Celeste: Yeah, what? That doesn’t make sense to you?
Okay, so what I mean by that is you have this sort of like nascent or non-existent research team, like research function, let’s call it like a discipline. Maybe there’s one person, maybe there’s five people, but everyone’s just kind of like still being very reactive because the company doesn’t know how to work with research yet and doesn’t really know what part of the culture it fits in, where it fits, so on and so forth, like how to engage, that kind of stuff. So what I mean by bricklaying is like there’s process, right? So how do you recruit participants? What are sort of like the safety issues with your research participant agreement or your NDA? What are the safety issues with the way that you reach out to your participants? So I was like building a lot of programmatic structure on top of then hiring people, trying to identify and prioritize research questions, all the things, like all the how-to’s, all the — so I was interviewing like a million people every week. I was participating in like a crap ton of interviews for the company and then also of the users. So it was just nonstop assessing things, basically, but figuring out like do we have a crit, as an example, as a research team? Do we do like a weekly critique? What does that culture look like? Is it required? Are we forcing you to do that? Those kinds of things are the bricklaying. Like they’re all part of the bricklaying. It’s not just like hiring and setting up process, but it’s also like what kind of culture do we want to have and what’s mandatory, what’s optional, and like what needs to be grassroots versus top-down, that sort of thing. It’s fun. It’s really, really fun.
Steve: I don’t know if this is part of the bricklaying metaphor, but what about things like how much we know or don’t know about people that we’re building for?
Celeste: Yeah, that’s a big one. I mean, when you first start — let’s keep going with the bricklaying metaphor — when you first start, like you could put a brick anywhere, so to speak, and you would have impact, right? Because you didn’t know anything and now you know something. But that gets like more complicated and more nuances is necessary when you start, when you’re sort of saturating on a particular topic. There is a moment to either move on or rethink it. And so, yeah, that’s definitely a part of it.
Steve: Right. Now you got me thinking of the Tetris metaphor where the more bricks you put down, the more difficulty placing those future bricks. There’s more scrutiny and impact on that choice.
Celeste: Yeah. Yeah. I’m very good at Tetris. I don’t know if I’m good at bricklaying, but this is the third company I’ve moved to where I felt like that was a part of what my role entailed. Because it’s not just in research, too. It’s the company at large. You’re defining process and practice as a group of people who’s maybe under a thousand to then like three thousand, five thousand people in a very fast — like the only way to address any of that is to just figure out how to scale yourself and figure out what’s important and what isn’t, because you have to make some decisions very quickly there.
Steve: I do want to move on to Robinhod in a moment, but I have a question that I want to go back. So what’s your title at Robinhood?
Celeste: I am the head of research at Robinhood.
Steve: And hearing you talk, I’m inferring that bricklaying is not just the purview of people who have leadership titles. You’re describing Twitter and Airbnb as roles where you didn’t have a leadership title, I think, but your job bricklaying was part of what you were doing. It was the context for your work.
Celeste: Yeah, you and I could go on forever on a much belabored subject about the difference between management and leadership. I was a manager at Airbnb, but you’re right that the bricklaying metaphor is not specific to somebody with a leadership title or somebody who is a people manager. It’s something everybody has to build together. And it depends a lot on the chemistry of both the research team and the broader company you’re at. So it never looks the same way twice.
Steve: Well, I’d love to hear if it’s possible, you know, different companies, different roles, but being an individual contributor, being a people manager, being a leader, again, the companies are different. So maybe at the question, the comparison isn’t right. But would you be able to characterize either ways that one could or ways that you have been involved in bricklaying? I love the metaphor. Just these three different contexts for bricklaying, you’re coming into it with what seem like different responsibilities or titles.
Celeste: Totally.
Steve: What is it like? How do you compare and contrast across the three?
Celeste: I think in all three cases, the similarities were around, no one’s going to tell you that you absolutely need to be doing that unless you’re like, you know, the pet of something that is kind of objectively, everything is your job, right, to some degree. So I think that one, it’s a little bit easier to believe that everything is within your purview. But for other roles, I came in and there are opportunities, there are problems everywhere, and you can either decide that they’re not your problem, which is an approach, or you decide that you’d like to solve it. Nobody’s going to probably yell at you for solving it. So you take it on.
So things like rewriting interview questions, like that’s something small that I did. Creating processes and how-to docs was something I did like every five minutes at Twitter. And there was like a running joke that no one was going to know how to do anything after I quit, because there was no one to write how-to docs anymore. I’m sure they wrote them plenty.
But I think it’s just about deciding if you have a perspective or a skill that can be lent to that particular problem or like opportunity, and then just doing it. I mean, there’s nuance there, right? You have to ask questions and make sure you’re not hurting anybody’s feelings, someone else isn’t working on it, that kind of thing.
But a lot of times in those environments, the best thing you can do is just say, “Hey, no one has a problem. I’m going to do this. Good? Good? Yes? Anyone want to weigh in? No? Awesome. I’m going to take this on.” And then everyone is pretty thankful if they remember that you did it at all, which is, you shouldn’t be doing it for credit anyway.
Steve: Over the course of your history, have you worked in environments that weren’t in the bricklaying mode? It seems like the first two you described were that.
Celeste: Yes. Yeah. So both Symantec and Oracle had very well-funded and staffed research disciplines. They weren’t massive. I think Symantec, there were still only like six of us, but it’s not enormous for a 20,000-person company at the time. But there was already a way of doing things within the broader UX team, within the company. We had a process, we had practices, we had tooling. No one was starving for those sorts of things. And so everything was kind of plug and play. There were great jobs, but the environments themselves are different. You learn different. You get reps for doing the work there a lot more than doing the work and the stuff around the work, if that makes sense.
Steve: If I’m applying for a job in user research, let’s just imagine that’s the thing I’m doing. It’s like, ask your doctor if Flonase is right for you. Ask what kind of environment is right for whom? Having seen both and participated in both these, if there are only two, and I’m sure there’s nuances here.
Celeste: There’s a lot.
Steve: Yeah. If places where it’s more plug and play versus places where there’s a lot of problems to be solved and opportunity to solve them, what’s your advice for people into, I guess to me, I hear two parts. How do I assess what the situation is? And then how do I self-assess about what works for me?
Celeste: I think it seems to me like there are a lot of ways to do all of those things, but I’m going to tell you a couple of things. One is how to assess or how to self-assess. But then the second or third, depending on how you just numbered those in your head, would be people that I’ve seen succeed in either of those situations, of which there are probably shades, it’s a spectrum or something like that, because they might be different too. So when I think about assessing, when you’re interviewing for a job, I think the thing that I would look for is who’s funding the research team and why do they think that a research team should exist? Those are things that you would think you get the same answer every time, but you extremely do not. Sometimes people want a research team or have a research team because it’s a box to check. This is what we do to make products. This seems like a thing past a certain point of company maturity that we would definitely want to have. So I’m not entirely sure, but I’m going to, here you go. I’m going to just check that box and it’s going to be great. And you’re going to be my researcher. So there’s stuff like that that can kind of give you a glimpse into the research maturity of the organization or whatever you want to call it.
Another question I like to ask, so around funding, do you have the research? Do you have the tooling you feel like you need to do your job? If you needed a new tool, what would it take to get it? Is it a procurement question? Do you have a budget? Do you have to go talk to somebody else about budget? Those are other things.
And then asking questions even about headcount. So like, how did this role open? Why does someone think it’s important? Why did someone decide to do this over hiring another engineer? Sometimes those yield interesting answers too. And you can kind of tell where in a research maturity a company is based on that. What I see in terms of success for one sort of a person over another, I hate putting it like that because I think you can be a different sort of person at different places in your career too. But if you’re interested in sort of the bricklaying, let’s call it, I’ve seen people say they want to do that. And then they jump in, they realize like, oh shit, nobody writes anything down here. There is leadership lacking and cross-functional ways. And I can either take the reins or just get really upset. And you never know how you’re going to react in that situation until you find yourself in it.
So I’ve seen people who were like, yes, I want that. I want that. I’m excited about that. And then they get there and they’re just very uncomfortable, very frustrated. And what they actually wanted was the excitement of moving quickly, but they didn’t understand all of the things that surround that, that you need to take along with the moving quickly and the excitement.
Steve: What led you to Robinhood and the role that you have?
Celeste: Yeah. So this is a complicated question. So Twitter, Airbnb, and Robinhood, while they are pretty disparate in terms of topic, like your customers, things like that, they’re not in the same vertical. The similarities they have besides the company phase and stage is that they’re all very, very mission driven, like very strongly mission driven. And once you’ve worked at a company like that, I should just speak for myself here because maybe this is not exciting to everyone else, but once I worked at Twitter, I was like, I can never have anything but this ever again. Because when people really believe in it, when people really believe in the work you’re doing and believe it’s for a purpose that isn’t just like contributing to the capitalist abyss, it’s more motivating and more exciting to me to get up every day and like, and focus on a mission. It’s what I return to when I’m trying to prioritize things, when I’m feeling like I don’t know what to do. It’s a nice way to hold the center.
So on top of that, when I was leaving my last job, I interviewed, you know, there are many mission driven companies. And so like I interviewed for some heads of research roles actually, and I was even surprised to be offered some. I was leading the largest team at Airbnb when I left. It was like, it’s called hosting, which was Homes, which is the biggest part of the business, Homes, Community. We had an Olympics team and then experiences was also a part of it. I just felt like I was collecting Pokemon at that point. Like they just kept giving me stuff.
But so I basically decided that I didn’t want to be a head of research. This is why this is a funny story. I opted for Robinhood for the mission. And also because I was excited to lead research and like a lateral move across a few teams reporting to the head of research at the time. It was okay with me because I had all these like ideas about what being a head of research would mean, which is why I didn’t want to be one. So like I won’t be as close to the work anymore, or I’m going to spend all my days like writing and rewriting career frameworks, or I have to be super front and center. That’s another one that like comes up for me all the time. I’m not, I’m chatty, but I’m like fairly introverted. And I don’t personally love being like the star of the show.
So I was also kind of looking at people that I had reported to over the years and or known that like then went on to be heads of research. And I just couldn’t see myself doing what they were doing, or at least what I thought they were doing. And anyway, this is all about me joining Robinhood. But basically, through a series of twists and turns in my first few months, I found myself in the role I was avoiding, which is fun.
But looking back, I think my biggest gap in thinking at the time was that I forgot or I didn’t know that leadership roles are what you make them. So this isn’t very researchy, or even like UX-y. I think this is just like a fundamental leadership-y thing where no two people are going to do that same leadership role, whether it’s the head of research, the CEO, the COO, whatever, because those are all equivalent roles, right? The same way. Like you’re never going to do them the same way as somebody else. And that’s actually a really good thing because the situation may call for exactly what you can offer. But because of that, if you’re looking to other people to decide like whether or not you’re going to be suited to doing that role, it’s kind of like thinking about whether or not you should be a writer based on whether or not you can write exactly like, I don’t know, Mary Shelley. I love that that’s the first one I thought of. But she did it her way and she wrote Frankenstein, right? And then you’re going to do it your way and maybe not write Frankenstein. And just because you can’t write Frankenstein doesn’t immediately invalidate whatever it is you are going to write.
So that was a really long answer for why are you doing this job? But it was like a personal growth moment where the very thing I was avoiding, I had to confront. And I learned a lot about it.
Steve: Maybe there’s some blurring between what we think we can’t do and what we think or we know that we don’t want to do.
Celeste: That’s beautiful. Thank you. Oh, yeah. I agree.
Steve: Because I think you’re talking about like you’re looking at other, right, we all compare ourselves to people, whether we compare ourselves to Mary Shelley or someone who’s been ahead of research, but you’re calling out just to reflect back and kind of make sure I understand. You’re kind of calling out, well, you didn’t want to do the job because you didn’t want to sort of spend your energy and time and focus doing the things you saw other people doing. But the aha is that there’s a way for you to be being a leadership role that isn’t those things.
Celeste: That’s exactly it. The things that you’re good at are the things you should be doing as a head of research or as literally any other leadership role. And you should be surrounding yourself with people that are good at the things that you’re bad at, because together you’re going to make beautiful music or write Frankenstein, you know, whatever. Right. Whatever you want to extend there. I really like that that’s the first person I thought.
Steve: That’s a good improv moment. Don’t think about it. Just say it. I want to ask about “mission driven.” I think you’re saying, you know, once you sort of had a taste of that at Twitter, that became important to you. It was. And you wanted to. I guess I want to ask, is there a distinction between mission driven as just kind of a cultural quality and like the specific mission? Which of the bits of it are the ones that are calling to you so strongly?
Celeste: No shade to the broader corporate lifestyle or maybe much shade. I don’t know. But I think when you’re every company has like a mission and a vision and like values and they there’s lots of pomp and circumstance around those things. And I think the distinction is in the discussions that they’re integrating into every day. So at Oracle and Symantec, as much as I loved the people there and I respect and admire the work that they were doing, we weren’t talking about the broader like, what are all of us collectively across all these products trying to do together? What is our ultimate goal beyond making money? Like what’s important to us? What is the legacy we want to leave as a company? And yeah, it exists somewhere written down in some wiki, corporate wiki. But like it just it wasn’t a part of the day to day conversation.
Whereas like at Twitter, we were obsessed with being the global town square. We were obsessed with making it with enabling people to discuss and connect and like communicate. And it was really exciting and invigorating to be working side by side with people that were like, I don’t know how we’re going to do this, but we’re going to do it and it’s going to be amazing.
And then at Airbnb, belong anywhere isn’t just like an advertising tagline. Like everybody is talking about the ways that we’re going to make people feel more or less like they belong based on design choices, strategic directions. Like it’s infused in everything.
And it’s the same with Robinhood, ours is democratized finance for all. We talk about it literally every week. It is a constant in every meeting. It emerges, the language emerges, we’re weighing trade-offs and thinking about it.
And it sounds a little culty at its worst, but I like to think that at its best it’s a force for good. I think you are what you measure. And so when you’re running a lot of experimentation and you’re looking at all these metrics that maybe build up to something that you didn’t actually want to aim towards, but just made sense in the individual examples, you can return to the mission as like, okay, but are these metrics democratizing finance or are they doing this other thing over here? Have we lost sight of it? Whereas at Oracle, at Symantec, I felt like that was not necessarily the lighthouse.
Steve: Let me throw a different metaphor in here because we’re doing so well with them.
Celeste: We’re killing it with the metaphors.
Steve: I wouldn’t go to a, well, I guess we’d have to turn on a time machine, but I wouldn’t go to a Grateful Dead concert for absolutely anything. But if I was going to go to one, going with a friend of mine that loves music and loves the dead would be the way to do it, to be in that experience with someone who is into it. I’m using that as an analogy for the mission-driven thing. I don’t care about if the dead and whatever the dead is about is the mission. I don’t actually care about it. It’s not my mission. Belong anywhere might or might not be my mission.
But if you’re going to go to a concert or if you’re going to work in an environment, one where that kind of passion and commitment and attention to detail and thoroughly building out every aspect of decisions being made based on that, you’re kind of highlighting how powerful that is and how rewarding that is. I guess I’m asking, is that still true if the mission is one that you are ambivalent about, say, versus 100% bought into? That’s what I’m probing on here.
Celeste: Yeah. First of all, I love the metaphor because you’re right. You could go to a show, and if you went to the show with your best friend and they were crazy about the dead, you would have a completely different experience because energy is a very human thing. Energy is infectious that way. The passion, the enthusiasm, you can’t help but kind of like, “Oh man, I was going to say ‘Catch a whiff.’ That’s a little too on the nose with the Grateful Dead reference.” But we’re going to do this all day.
Yes, I actually do think that if you are fairly ambivalent about the mission. I don’t think I woke up in 2021 and was like, “You know, what really needs to happen in the world is finance needs to be democratized for everyone, for all.” I don’t think that I woke up feeling that way. I definitely don’t think that I looked at Twitter’s mission and was like, “Man, I have never been more like — it is my life’s calling to be a part of the global town square.” I think the difference is that the mission has to resonate at least a little bit. You have to be like, “Yeah, of course. I mean, yes. Do I agree with the idea of democratizing finance for all? Absolutely. I love that.” I didn’t know I wanted to do that until I started looking into it. But it doesn’t make it any less important to me, especially if I’m surrounded by people that are all agreeing and believing in the same thing. So it helps, but I don’t think — it’s not like you have to be born with that passion in mind in order for it to be infectious. I was looking for a less disgusting word, but that’s what we’re going with.
Steve: When we’re talking today, start of spring 2024, how long have you been at Robinhood?
Celeste: Oh boy. How long have I been at Robinhood? I’ve been at Robinhood for two years, and I’m sighing deeply because every year is getting shorter in my life. Just like Pink Floyd promised. I’ve been here for two years. I just celebrated my two-year anniversary in the beginning of December, so a lot has happened during that time. I started in December of 2021. A few months after that, the head of research left.
My job changed every two to three months for probably over a year, which isn’t inherently weird, except that 2022 happened, which meant that the crypto markets and the stock markets, everything was going down in 2022. There was a huge wave of layoffs across tech. We also had to lay people off. All of that is super difficult as an employee, but also as a leader of a team. It’s really, really tough, especially when you spend some time thinking that this is not what you wanted to do. So when I look back at the two years, it feels like more than two years, but it also feels like I just blinked and two years passed. So there’s a lot of cognitive dissonance in that length of time.
Steve: What’s the cognitive dissonance?
Celeste: It feels like a lot and like a little at the same time.
Steve: Yeah. So over those two years, what are some bricks that you’ve laid?
Celeste: So the team predates me by a lot. It’s not like I built the team and built all these processes. The team predates me by a lot. It was originally put together by my predecessors. I deeply appreciate everybody who did that for what and who they left behind. Those have been gifts. But the bricklaying in this case was that Robinhood had just IPO’d over the summer and was sort of nestling into everything changes when you IPO.
Airbnb IPO’d while I was there, Twitter IPO’d while I was there. Everything changes. There’s a lot more you need to do. Because your funding looks different, there’s a lot more that you need to be held accountable to. And so a lot of process changes when that happens. The research team had also been through a lot. There were some big, dramatic leadership and team changes during the entire year of 2021, and I showed up in December. So a lot of my bricklaying, I wouldn’t say it was like building a team from the ground up, but it was sort of like there was a lot of healing that had to happen as a group of people.
When I started, I was hearing a lot about people not trusting each other or feeling like if so-and-so got promoted, why didn’t I? I don’t even think their work is good. There wasn’t really very much teaminess. So a lot of the bricklaying was rebuilding cohesion and trust, thinking about things differently, re-evaluating tooling because everybody, when you’re laying people off, are also reassessing the budget and the tools that you need or don’t need. So a lot of really tricky stuff like that happening at the same time. And then kind of re-evaluating research’s relationship with the company, which was a pretty tall order, but it’s been fun.
Steve: What are things that you can do to build teaminess?
Celeste: Not sure I have a good answer to this because I’m going to give you that delicious research answer, it depends.
Steve: Yes!
Celeste: You’re welcome.
Steve: You’re going to ring a bell right now.
Celeste: Yeah. Someone somewhere is furious at me for this, but it depends on the situation. We had layoffs at Airbnb when I was still there and it was devastating and we had to sort of rebuild our emotional baseline also. And it was tough to find all the loose ends and figure out what work still needed to be done, what work had fallen off and that was okay, who we were without the people that we had lost and things like that. It’s really, it’s hard to be laid off and it’s very hard to lay off. It’s just like a no one wins situation.
So in that, in like the Airbnb case, the teaminess came from just, actually it might be the same as Robinhood, being consistent. Everybody showing up and being exposed to each other and just talking about what was hard, what wasn’t working, do people have ideas, here’s what I’m doing, what are you doing, but being really active and pushing for contact and like pretty regular consistent contact and trying to foster moments of recognition when something was going well. There was a lot of stuff like that.
I don’t know if I have like a silver bullet answer though, because it depends so much on like the way that things are bad or that need healing. We definitely went from people not being willing to help each other because they were worried about not getting credit to people helping each other without thinking twice about getting credit and those were signals to me that we were like nature was healing. We’re on the right track.
Steve: I know you’re apologizing a little bit for the answer not being like a full list of three things like you didn’t say, oh, we had an offsite or we had a cake, so it’s like, but I really am struck by the fact that you’re talking about like intentional ways of being that are maybe smallish, but that are sustained over time. But that sounds much harder and much sort of less obvious to come up with or to execute. And that seems like, well, if you want to make change, it does depend, but there’s a set of tools that you’re drawing from and a set of principles is, I don’t know, subtler than a fix it kind of approach.
Celeste: Fixing it, especially with something as fragile as like people’s chemistry and sentiments feels like a fool’s errand. Like coming in and being like, let me show you how to do things. I’m going to fix these feelings. That’s not really, I don’t think you’re going to get anywhere if you approach it that way. Someone has probably done that, but I don’t think that’s possible in the toolkit that I have. Maybe Mary Shelley’s done it.
Steve: She was a great management and leadership consultant in her coaching business.
Celeste: Yeah, good call. What comes to mind is that there were two things that were similar about both circumstances. One was sort of the community that I’m talking about, which like you can’t inauthentically build community. You have to push it through connection. You have to get people to connect with each other. And sometimes it’s awkward and it’s not mandatory fun. That doesn’t work. But if you can give people real reasons to show up and be there for each other and be honest with each other, you’re going to, it would be hard for that not to move in a positive direction. I think human beings just crave connection and community. Even the introverted ones, it turns out.
But the second thing is like, this is a businessy answer, but just hear me out. Transparency. So here’s what I know. Here’s what I don’t know. Here’s what I’m doing. Things like that are during, especially things like layoffs or moments where the team has really lost a lot of trust in either each other, the situation around them, the circumstances. They know that you’re doing everything you can to contribute to a shared understanding. It’s again, it’s hard not to move in a positive direction if people are like, you’re being as honest as you can with me. And that opens the opportunities for, I’m having a hard time with this and I didn’t want to say anything until I felt like you were also showing up and spilling your guts about what’s happening. So yeah.
Steve: You also mentioned re-evaluating researchers’ relationship with the company. What is that about?
Celeste: The company loves research. I don’t think the company will ever stop loving research. Some of that, again, love dearly my predecessors and their approach to it. I got really lucky, but, and it was always going to have a place because our co-founders started doing research themselves. When they started the business, they focused a lot on listening to people, on observing people do things, like on very researchy things. So the research team was always going to be a core function at Robinhood. I didn’t invent that. I’m not going to create that. But when I started, there was this, they had grown a ton very quickly, like the entire company did. And every kind of PM, GM, executive marketer up and down the chain was and still is asking about research. We need to do it on every last thing.
We have an unholy amount of opportunities to be user-centered in what we’re doing. And the tone for that is consistently being set at the top, which is delightful. And I don’t feel like I’ll ever see that quite in the same way in my career. So it just feels like I’ve struck gold. They’re always asking about our customers and their perspective. They know that our co-founders are doing the same. Everybody’s asking about this.
But when no one ever questions if research should be involved, and if everyone is insisting that research study everything all the time, you’re faced with a different set of problems. Your relationship to the organization is different because researchers, as I’m sure you’ve seen this too, we’re so happy to be included and consulted because we’re so unused to that that our relationship ends up being, yes, I’m going to say yes to everything because I’m so excited to be in this position and be consulted and be listened to.
And so that’s going to take anybody who starts at Robinhood by complete surprise. Everyone’s always like, “You said we had a seat at the table, but holy smokes, we really have a seat at the table.” Not prioritizing, not saying no. You end up spreading yourself really thin. You end up studying things you really don’t need to study because the leverage isn’t there. It’s not some sort of force multiplier all the time. Not everything needs research.
Actually there’s a general manager that I work with that loves to use this metaphor. Speaking of our metaphors, he likes to say, “If I’m opening up an ice cream shop, do I really need to do research that I should offer vanilla? I just know that vanilla should be on the menu. So do we really need to do research on that and why?” My response to that, of course, is like, in the vanilla answer, sure, but do we know that where you’re opening the ice cream shop is actually a place where people are interested in vanilla? Because you’re assuming a really narrow, maybe it’s Americans only, but whatever. There’s a group of people that like vanilla generally universally, but very generally. Is that target market where you’re opening up your shop? On top of that, is it a fancy vanilla? Are we using elaborate beans from some rare island or whatever? Or is it this just straight up vanilla, no frills? What is resonating? What’s needed based on the context of the situation?
But also I kind of agree with him that maybe we don’t need to do a study about vanilla at all. Maybe we need to understand everything around the vanilla. And that to me is I think the relationship with research that is ongoing and needs to change. If you can’t, as a product manager or some other leader, have a perspective on something without research, I feel a little uncomfortable with that. You should have a point of view. I want information to inform it. But if I don’t, because it’s a fairly inconsequential thing, I think you should still be able to make that choice anyway so that I can work on the stuff, my team can work on the stuff that’s the most important based on the things we’re good at. Not every question is going to be able to be answered by research. So that’s what I mean by the relationship. And it’s nobody’s fault that this is the state of affairs. It’s just what happens when the pendulum swings really hard in the other direction.
Steve: Right. You talked a little bit about saying no, but in the vanilla example, it’s almost like, hey, here’s seven more questions that are harder to parse out, riskier if your assumptions are wrong, and that create more context to the vanilla question begs a bunch of larger questions. And so that’s not saying no, though. That’s like, no, we’re not going to do the vanilla thing. But hey, how’s about…
Celeste: There you go. That’s what I mean. It’s not that I think we should be disengaging from every part of the conversation. It’s just that we need to be asking the right questions. And sometimes the questions that we’re being asked that we say yes to are still vanilla ice cream questions, like just the baseline yes or no vanilla without the nuance, the subtext, all the stuff around the things. And again, it’s just because research is such a part of the product building and marketing culture at Robinhood that people come to you with very strong requests about doing vanilla ice cream research. And we just have to keep making sure that we’re working on the things that are the most important from a business UX timing perspective to make sure that we’re not just answering lower leverage questions. What’s the approach to changing that relationship in the way that you’re articulating?
I spent a lot of time with product leaders and GMs and things like that talking about what the most important decisions they need to make, questions they have are, and then kind of going over… I’ve been made fun of before because I’m like, that’s an experimental question. Like straight up, just run an experiment. It’s faster and more reliable. You’ll have more confidence if you run an experiment. People laugh at me because they’re like, well, wait, don’t you collect data a different way? Yeah. I mean, just because I’m a hammer doesn’t mean everything looks like a nail to me. So I spend a lot of time doing that because it does need to come from the top. But then making sure that people have the tools to be able to feel like they can say no because a lot of researchers feel like it’s going to damage the relationship between them and their product teams if they’re not able to do everything that’s being asked of them, which is disheartening. So we work a lot on that.
We talk a lot about what a higher leverage question looks like or how to meet the short-term moment of what the team needs while making it into a longer-term study also. So I don’t mean in length of time, but I mean, I’m sure you’ve heard this a bunch, Steve. You know how there’s this debate within research about whether strategic or tactical is a better use of our time? I find that interesting because I don’t think they are separate. I think that you can make sort of like what people would refer to as very highly tactical research like, “Do people understand this? Whoa!” You can make it strategic by asking questions inside of it and putting together a broader narrative. If you’re telling me that like, “I’m too senior to do usability testing,” or whatever it is, I’ve heard that multiple times, which is unholy. But it tells me more about how you’re thinking about structuring your research in a way that you’re thinking about insights and what value you bring to the team. It tells me a lot more about that than it does about your seniority or anything else. And so it’s not about like doing less usability research, but it’s about doing the research that really needs to be done. And so we talk about how to prioritize, we talk a lot about like this over that, and we’re just pretty open about what’s important and why. Back to your point about transparency, I think. I’ve been wrong before.
Steve: Yeah? Not in this conversation.
Celeste: Probably several times in this conversation, but never about Mary Shelley. You’re welcome for that.
Steve: I’m thinking about this relationship between researchers on your team and any particular product team and so on, where, yeah, what you said, there might be a near-term question, but there’s a way to do that research that also… \Either you refactor that question into a higher value question, or you do, and you didn’t say this, but I am going to say yes and, you do kind of a yes and of the short-term implication and the longer-term implication. But just this whole exploration of like, what are research being asked for? What does research provide? Just makes me think about questions that come up, like, do we give recommendations? Like what are the kind of outputs of research? And I know it depends. Of course it depends. But does this, I don’t know, does this area provoke anything for you, a point of view, or something you’re thinking about?
Celeste: On the topic of recommendations, I’ve seen at some companies the idea that if people aren’t taking your recommendations, that you have not had impact, which I find charming. Like I personally do not want to work somewhere. I’d be really concerned if someone was taking 100% of my team’s advice all the time. We’re looking through a really specific lens. There’s all these other inputs to consider. It’s really great when the research is informing a decision, but sometimes the decision is not going to go with what the research is recommending. And I don’t think that’s a failure on the researcher, as long as everybody’s aware of the trade-offs and everyone understands the insights.
And by insights, what I mean is like what the data itself translates to for us as a product, as a business, whatever the case may be. It’s okay to have wrong recommendations, but it would be weird if you were taking my advice all the time because it would mean that you’re really only considering one input or weighing it heavier than others. So that feels weird, but I do think, to your point about it being a bit of a controversial topic, I do think we should have recommendations, like be giving recommendations, not because we’re all geniuses, although maybe that’s true. But because we are the most informed about what the data means, what we can say and what we can’t say based on what we did, thinking about things like intellectual honesty and rigor and all of that, I don’t know why we would ever think that we were not suited to making some sort of recommendation.
Because other people are going to take the information that you give and then just make their own recommendation, but that’s filtered through an entirely different lens with way less of the broader context that you’ve collected. It’s okay if your recommendation is bad, and you should probably shape it in a way that makes it shelf-stable, right? If you say, “Move the button to the right,” that’s not shelf-stable, because at some point someone will probably look at your research years down the line and go, “What in the hell were they talking about when they made that recommendation?” But people preferring design A over design B is just sharing data. People preferring design A over design B because the information they were looking for was a lot easier to find, that’s more shelf-stable, right? You understand why, you understand the context of the insight itself, and even if I don’t do exactly what you say, maybe I end up going with design B anyway, maybe I can make design B better based on the way that you frame that. So there’s a lot in there.
Steve: I want to pick out what a recommendation is, because people prefer design A over design B, that’s data. People prefer design A over design B because they can find the information. Okay, that’s an insight. So then the recommendation would be, go with design A.
Celeste: Right. Yeah, but you have to have that other piece in there for it to make any sense at all.
Steve: We should go with design A because it helps people to find the information they’re looking for.
Celeste: Words are hard, Steve.
Steve: I don’t know, just as a counter-argument a little bit, and I think you’re on board with this, the risk of being wrong there is you only know what you know about that. So design B makes us more money, design B we can implement faster, design B is compliant with something, design B is consistent with what we’re doing in three other platforms.
Celeste: Absolutely.
Steve: So I think that’s an example of them not going with design B and that being okay because you’ve given them the information that you have to make that recommendation.
Celeste: Yes. So I would argue that if you’re not incorporating things like which design would make us more money or if you’re not incorporating those pieces of information into your process, you’re leaving value on the table. You should be helping the person who needs to make the decision make it with all that context in there. Anything you can help them with. You have an amazing sense of synthesis and distillation and not everybody’s fantastic at that. So if you can pull that in, you probably should do that. But yes, you’re right that even if your recommendation is wrong, it’s not always that you’re completely incorrect. It’s like not right now because we would prefer to make more money with design B or whatever it is. But you’ve planted this seed and I understand that what people need is not what this is providing and so we can get successive approximations closer to it.
Steve: Right. I think the recommendation discussion sometimes leaves me cold because it loses all the context that you’re providing. It’s the recommendation and the insight in the larger context of the set of things that might be factors for this team. And I think you’re saying you should try to know all of them.
Celeste: Why not?
Steve: I’ll just add there’s always going to be something that you won’t know.
Celeste: Of course.
Steve: I don’t know, it reminds me of like trying to write like Mary Shelley. I think if you do, if you do like creative writing and you do workshops like you get feedback that says, here’s what the problem is and here’s what the solution is. And I think it can be really helpful when you’re the writer to choose not to do that, but to do it, ignore the advice in an informed way. Well, that’s not what my objective is. That’s not my strategy. We need short term wins here, whatever. And I think we’re agreeing strongly here. And it’s just some of the language gets oversimplified when it gets, you know, boiled down to like a social media posts.
Celeste: Yeah, I think this particular topic is one that is not great for like a LinkedIn rant. And yet, it’s all we see.
Steve: But great for a podcast episode, right? Because we can dig in.
Celeste: Sure.
Steve: You brought up my other another one that I take umbrage at which is the, you know, not we’re not having impact if it doesn’t get taken up. And I liked what you said. I’ve also heard people say my research doesn’t have value. If no one takes action, it’s even more binary, right? If that my research impact is sort of higher level than value. My work is worthless. If someone doesn’t do the thing that I told them to do. And I feel so sad because that’s researchers saying that, like as if we don’t have enough people telling us we’re without value. We’ve sort of taken on as a profession taken on some of that, you know, willingly devaluing proactively. I’m going to devalue it for you. It’s depressing. And that makes me sad.
Celeste: Yeah, I agree. It also not to I’m sure that the folks saying this are perfectly smart people. But to me, when I hear that kind of stuff, it feels like it lacks curiosity. So you’re saying that unless there’s like something I can see in front of me, or unless there’s like a direct action being taken, that I have failed when I think if you had a little more curiosity about it, you could notice little hints like, well, but these people are changing their language. And they’re using the language of the people that we’re studying instead of whatever business bullshit we’ve come up with. And that’s impact.
I think there’s we’ve changed the roadmap where we’ve just decided not to go this route. We’ve saved three months of engineering. So it’s almost that inaction is action. There is there’s lots of stuff like that. And I think if you are open to seeing it, it makes itself known to you. But if you are not curious how your research is sort of creeping through an organization, you’re going to end up with it flying right over your head. And then you assume you have no value or whatever it is you said, it’s depressing.
Steve: You know, you’re bringing your the same way that you talked about bringing teaminess is it seems very analogous to the way you’re talking about both creating and sort of realizing the value that it’s smaller signals over a longer period of time with some I’m not sure if the constancy thing applies here or not.
Celeste: Maybe.
Steve: But it’s a binary outcome kind of thing that yeah, taking a slower and more looking for looking for those subtle more subtle signals.
Celeste: Yeah, you’re reminding me of I’ve had this conversation a few times after becoming a manager where people are like just shouldn’t do research on this area because they’re not valuing it the way they should, which often the subtext is they’re not taking my advice or they’re doing things the way that I fundamentally disagree with. You probably know this better than anybody, but this is not a field for the faint of heart. This is like a full on eternal marathon. It took me three years at Airbnb and many different approaches to convince people that non drip pricing was the right call. It cost us a lot of money and we had to find ways to offset that. And every time I got knocked down about it, I could have chosen to never bring it up again.
But it’s chipping away. It’s finding advocates. It’s finding other people that agree or believe it. If I can convince one person each time of something that I thought that I believe in, I’m doing great. That’s impact. And so you’re right that there’s like little subtleties, but it’s also just a long game. Like all of this is a long game. You’re lucky if you have a short term win. You’re super lucky.
Steve: Anything else to add today?
Celeste: No, I mean, thanks for inviting me to do this. It’s very flattering to talk about my thoughts and feelings on things we mutually like for however long this just was. It was very nice. Thank you.
Steve: Yeah, thank you. You shared a lot of interesting perspectives and good history. So yeah, thanks a lot for taking the time.
Celeste: This was awesome.
Steve: Over and out, good buddies. That’s all for this episode. Thanks for listening. Recommend Dollars to Donuts to your peers. You can find this podcast in all of the usual places. A review on Apple Podcasts helps others find it. Go to portugal.com/podcast to find all the episodes, including show notes and transcripts. Our theme music is by Bruce Todd.
Celeste: Did you say a favorite puppet?
Steve: Favorite puppet.
Celeste: What came up for you just now?
Steve: You know so many things about so many things and I was just trying to open up the possibility space.
Celeste: Favorite puppet. I’m going to be thinking about that. What is my favorite puppet? Now I know exactly. It’s the, I can’t remember the other guy’s name, but it’s the two angry balcony dudes in the Muppets. One of them is named Astoria or something like that, or Waldorf, and the other is named after another hotel and it’s just not coming to me. But those are.
Steve: His name is, his name is Statler. Statler. I just Muppet, Muppetsplained you here.
Celeste: You win. You win the Muppets.
Steve: Yeah.
Celeste: I also just like that there’s no, nothing is sacred to them. They just go after whatever, whatever is in front of them. And they do a lot of like lame dad joke moments too.
Steve: So. That’s excellent. I’m not going to get any actual puppet content.
Celeste: Puppet content
Steve: So thank you for, thank you for taking that literally. I didn’t know what you were going to share, but…
The post 42. Celeste Ridlen of Robinhood first appeared on Portigal Consulting.
In this episode of Dollars to Donuts Carol Rossi returns to update us on the last 9 years. She’s now a consultant who focuses on user research leadership.
I’m happy making the contributions that I’m making, even though it’s hard to directly measure impact. I’m hearing from people that they’re finding value from the work that we’re doing together. I want to leave people with this idea that the work that we’ve done together is valuable to them, whether it’s tomorrow, or two years from now they see value in it in some way that they couldn’t have anticipated. I’m trying to be as clear as I can about focusing in the areas where I think I can make the best contribution and have the most impact. And keep reexamining, how do I feel about the work that I’m doing? And what am I getting back from people? – Carol Rossi
Show Links* Interviewing Users, second edition * Tent Talks Featuring: Steve Portigal * Amazon Reviews for Interviewing Users * Elevate Your Team’s Impact with Storytelling * Carol Rossi on Dollars to Donuts, 2015 * Carol on LinkedIn * Carol’s website * Edmunds * NerdWallet * Prioritizing Research for Impact on Maven * GeoCities * When the Healthiest Person You Know Gets Lung Cancer
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal. In this episode, I catch up with Carol Rossi, nine years after she was first on Dollars to Donuts.
There’s a bigger and better new edition of my classic book, Interviewing Users. As part of the launch of the book, I spoke with Russ Unger for his Tent Talk speaker series. Here’s a little clip.
Russ Unger: What’s your approach to ensuring that the feedback gathered from user interviews is effectively communicated and incorporated into the design process?
Steve: The first part of that I think is that you have to do something. You have to make sense of what you gather. Some of this kind of goes to maturity of any individual practice. I think the less experienced folks are, the more they want to just take what they remember about what was said and type it up. And that verb is, that’s stenography maybe, or collation as far as you get. You put these pieces together. And then you’re just taking requests or gathering complaints. You might as well use a survey for that. I think it’s the iceberg model, right? Some of it is above the surface, but a lot of it is below the surface. Below the surface means going back to what was said and looking at it and making inferences. What wasn’t said? How was it said? What said at the beginning and what said at the end? And that’s just within one interview. What did person A say? What did person B say?
And there’s a whole new chapter about this. It’s the analysis and synthesis process. And some folks say that the ratio should be two to one. For every hour of the feedback that you gather, you should spend two hours analyzing and synthesizing. And I think in a less evolved practice, it’s the inverse. You might spend half an hour for every hour or even less. The caveat here is not every research question merits this. If we are looking for, I don’t know, choice preference between something and something else, we might be really clear about what that is. We come back and say, do this.
But for anything where we want to understand why or understand opportunities or understand motivation, a new space you want to go into, characterize a customer that we haven’t worked with before, it really is worthwhile to go and do this analysis and synthesis.
How do we have impact? We have to have something impactful to say. I just want to say that. Some other factors that I think can make or break it is working collaboratively with stakeholders, the folks that you want to inform, influence, take action before you do the research. And so having an understanding of what business challenges are or business goals, like what are we trying to do as a company? And then formulating really good research questions. What are we going to learn in order to inform that? And then choosing methods and approaches that can support that. And not doing that in a vacuum. And then this has the effect of switching your role from being proactive to reactive.
I think it’s hard to have an impact with reactive work. Those requests that come are often late. They’re often based on a shallow assumption about what kind of value research can provide. And so you are going to give a thumbs up, thumbs down in some direction. So your sort of role as a provider of these kinds of insights is diminished. If you can be proactive, which means maybe understanding a roadmap or what decisions are being made or who else is going to do what and proposing research on your own roadmap that is intentional and is ahead of time, you leave space, of course, for things that come up, fire drills and so on.
But trying to work in a proactive, collaborative way, aligning on goals and then putting the effort in to make sense changes the whole conversation about what you’ve learned. You get to that point of sharing with somebody.
That’s part of a larger Tent Talk. You can check out the whole show and definitely buy your postal carrier and barista their very own copy of the second edition of Interviewing Users. If you want to help me out, write a very short review of Interviewing Users on Amazon.
Over the last couple of years, I’ve been partnering with Inzovu to run training workshops about storytelling. Storytelling is an essential human skill that powers how teams work together with each other and with their colleagues. I’ll put a link in the show notes with more info about what I’ve been up to with Inzovu. And if storytelling is something you’d like to build up in your organization, reach out to Inzovu or to me.
Okay, let’s go to my conversation with Carol. She’s a consultant with a focus on user research leadership. Carol, welcome back to Dollar a Donuts after nine years since we last talked. It’s great to talk to you again.
Carol Rossi: Yeah, thanks, Steve. I can’t believe it’s been nine years.
Steve: Time does fly. Let’s talk about those nine years. You know, what’s been the shift in your evolution in your professional world since then?
Carol: When we last talked on the show, I was at Edmunds and I was leading the UX research team there. And I had been there at that point, I guess, four years. I had started the team there and then ended up staying at Edmunds until 2017. And then I took a moment, because I’d been there for quite a long time, and took a moment to kind of ask myself what I wanted to do next. I call it my gap year. So I did some consulting, some really contract work as well as like consulting, helping people think about how to set up a team.
And then in 2018, I went to NerdWallet and that involved a move. So I was in LA for the bulk of my career. 2018, I moved to San Francisco for the job at NerdWallet. And that was an established team that I led for about four years. And I mean, we can go into detail about any of this stuff, but basically left NerdWallet in 2022 and started a consultancy where I’m now focused on helping companies, helping leaders know how to get the most impact from research
Steve: Can we talk about NerdWallet a little bit and then talk about your consulting work now?
Carol: Yeah, Sure.
Steve: So it was an established team. Is that right?
Carol: Yeah, it was. So there were three people on the team. There was actually an open headcount when I joined. We ended up doubling the size of that team. So we still remained a relatively small team, but we did get some additional people. We actually, I think some of the work that I’m really proud of there is that we went from having these researchers doing sort of very siloed work, or even though they were all researchers, they were hardly really working with each other even. And then developed that team to the point where we had a lot more strategic impact. We started a voice of customer program. Two of the people on the team became managers during the time that I was there. So they saw a fair amount of professional growth. And when I left, again, there was this voice of customer program established, as well as a program to train designers and PMs and content managers to do some of their own research. We had, well, on the market research side, they were doing some brand work. We were doing some kind of explorations about how that played out in product. So there were more things that are more sort of horizontal activities we were doing, and also empowering some people to collect their own insights, as well as deepening the impact of our team.
Steve: When you talk about coming in and the researchers that were there were siloed, my mind starts to go to that embedded word and what that means. But I think you’re talking about siloed in a grander scheme of things. But I don’t know, what does siloed look like then?
Carol: I think it’s a really good distinction. The difference between siloed and embedded to me is that embedded can be and is a very valuable way to participate in a product development team.
So it’s like, and we ended up with this sort of hybrid model, I would call it. Because at the time that I left, the team was reporting to ultimately me, but they were dedicated to specific focus areas. So we had one person working on the logged in experience and that involved maybe three pods. We were calling them pods, but squads, whatever, product trio, whatever language we use to talk about the combination of the PM, the designer, the content strategist, and some number of engineers. So we’d have one researcher per, let’s say, three of those pods, but they were all within a focus area.
So one was dedicated to the logged in experience. We had, for example, a couple people working on what we call the guest experience or shopping. So if you’re looking for, so I should say NerdWallet is a company that provides advice and products to consumers who might be looking for financial products. Consumers might be looking for a credit card or a mortgage or a personal loan or whatever. So you can either go and read some articles and then get linked to some potential credit cards for you based on what you’re interested in and your credit score and those kinds of things. Or you can download the app, log in, and get tailored advice based on your specific situation. So those are, at the time, were separate areas of the company in terms of the way the development was divided up.
So I think embedded to me is there’s a very healthy relationship with those pods where the researcher is either dedicated to one or maybe crosses over a couple of those areas, of those pods. But siloed to me is people are working on something so exclusively that maybe there isn’t a lot of conversation across. And I think what you lose in that kind of model is opportunity to take advantage of research that might be going on in an adjacent area or even a very different area but has relevance to what you’re doing.
And so you can have a lot more efficiency across the research function if you’re not re-doing work, you know. Or people are learning techniques from each other, you know. Or people are partnering so that there’s some broader impact across these different focus areas. So there might be — because to the consumer, to the ultimate user, the customer, they’re not seeing it, right, as these sort of separate areas. They’re seeing it as one experience. And sometimes in order to do product development, you have to divide things up.
So how do we keep the flow and the things that need to be similar across the experience and have it make sense by looking for those areas of, you know, similarity or continuity or whatever the word is that you want to use there. Some of the things that we did that worked really well were have just — so first of all, I should just be really clear. Because it was a manageable team, I mean, small enough team, we could do things like have team time every week where researchers felt like they were able to have a dedicated, you know, I think it was an hour or something, but a dedicated time where they could talk about some of the stuff they were doing, present problems to each other, learn from each other, like have time to be able to say, I’m doing this thing, I think there might be some relationship to what you did last year or what so-and-so did who’s not even here anymore and what can we talk about there.
So I think there’s — with a small enough team, you can definitely have people, you know, embedded or partially embedded within specific areas so they’re having maximum impact in those areas, but still conversation across. So I think that’s one thing that we did. Another thing we did was have kind of a loose repository. We weren’t using a really fancy tool. We just literally had, you know, a wiki where all of the research that was done was available. So people could go in and see what had been done and see if there was something relevant to them. And that could be like product managers, designers, anybody could go in and look at that and see. And then they’d usually come back and ask us questions. Hey, I saw this thing, you know, I wonder how that can be relevant to our team. So I think there are a few things that you can do.
Steve: You mentioned that you put in programs to teach other folks who are not career researchers to do research. What did that look like? How did that work?
Carol: I think the way that I’ve seen that work well is to create — when I’ve created a three-part series, workshop series. And so we start with these three workshops and then we do ongoing coaching. So it’s not just a matter of taking a, you know, a training session. And the first workshop is really setting up the research for success. And so that’s really about planning and study. So then there we talk about starting with the business objective, you know, like people will often start with a research question. Like we need to know X. Okay, well, why do you need to know X? Like there’s some business reason why you need to know it. So what’s the thing you need to know? Why do you need to know it? What decisions will be made as a result of that? And then what’s the best way to get that answer? Obviously, you know, in what timeframe do you need to know it and those things as well.
But starting with that framework to give people an appreciation for the fact that we don’t just run a study because we have a question. We kind of put context around it. Even if it’s a lean and I’m using the language of run a study, but this is like the language that some people are using is having conversations with customers or collecting insights, whatever language people are using. It’s the same thinking. And in that first workshop, we talk a lot about reducing bias, making sure we’re not asking leading questions or, you know, the way that we’re writing a task or something that we’re going to put up on a prompt that we’re going to put up on an unmoderated tool for a participant to engage with whatever. We talk a lot about how to do that in a way that those are going to be effective. And by the end of the first workshop, everybody has a lightweight research plan. I give a template. So there’s a template that has all those elements in it. And there are a lot of tips and tools and sample. Questions and sample tasks. So it’s pretty plug and play, but the foundational understanding is there in terms of, you know, not introducing bias and some of those other elements.
The second workshop is literally run a study. So when I was at Edmunds and we were doing in-person research, we would recruit a bunch of participants to come in and we’d have designers, PMs, engineers running their own interviews and, you know, we’d sit and give feedback often. Now what we do is all unmoderated. These workshops are all online now remote. So, you know, it’s an unmoderated tool and they set it up. They set up their study and the tool, and then we, you know, wait for the results to come in the videos or whatever.
And then the third workshop is in researcher language synthesis. And that’s the, like, how do you go from all this data that you just got to actionable insights? So we look at, we talk about the data. We talk about findings that come from there. We talk about insights that are really most important. And then we talk about prioritizing those insights according to the business objective, back to the business objective, back to the decisions that need to be made. What are the most important of all of those insights? Cause you might get a lot of stuff, you know, out of even a lean study. What are the things you need to take action on? And then we talk about taking action. And I have a, again, there’s a template for summarizing their findings of their study, but there’s a table that shows like, what was the insight? Okay. It was high priority. So we’re going to take action. We’re going to do this thing. Who is going to do this thing? It’s assigned to a team. Who’s the point person. Maybe it’s the PM on the team. Maybe it’s the designer. By what date is this thing going to be done? So everybody on the team now has agreed beyond the person that ran the study. They go back to the team, have the conversation. Everybody has agreed. Here’s what we’re going to do as a result. And then that goes into that table goes into their summary. And then there’s a way to go back. If the person that’s running this study. Is not one of the people on that team in this case, they probably are because they’re the designer or the PM or whatever. But you can go back and see what was actually done. Was it, you know, when was it done? What impact was gained by that study. And you can then add the impact t your impact tracker.
Then there’s the coaching that happens after the training. And that’s really vital to help people sear in the knowledge from the training and get feedback as they go along and ask questions. So sometimes the designer will go to the researcher who led the training and ask, will you take a look at my plan ’cause I’m going off the template a bit and I wanna make sure this makes sense. Or they have a question about synthesis because they get something that they didn’t anticipate and they wanna talk through how to do it. Or they need help figuring out how to message something to somebody that wasn’t on the team but needs to get these insights. So there are things that come up in life and sometimes it’s feedback on something that they’re doing. Like when I was at Edmonds and we were doing live interviews, we’d actually have a conversation after each set of interviews with the people that were running them and how did you think that went? And if we saw something that maybe they could benefit from, we would share that with them. So I think that’s a really important part of it and something I incorporate into the workshop that I do to train, it’s not just the training, it’s the follow-up coaching as well.
Steve: I think there’s a lot of hand-wringing off and on over the years about the risks and the consequences of, what did you call them? Non-career researchers, that’s a great term.
Carol: People who do research, I think is what people are saying now.
Steve: You know, we talk about the consequences of these kinds of programs that allow non-career researchers or people who do research. If we empower them as we’re kind of sort of the gatekeepers of the skills and the knowledge to do research, which may not even be an accurate framing anyway, ’cause people are doing research anyway.
Carol: Yeah.
Steve: here’s sometimes some hand-wringing about unintended consequences or intended consequences. I don’t know, with these programs of these different organizations, were there longer term kinds of changes that you noticed?
Carol: Yeah, I think it’s a good question. And I’ll just say, I don’t have that argument anymore with people. I have stopped trying to defend how this can work and how I’ve seen it work well, because the fact is, I’m just really realistic. First of all, I’ve seen it work well in this way that we talked about where there’s this sort of training set and then this sort of coaching activity, and there’s a conversation. It’s an ongoing conversation. And so what I’ve seen work well, one of the things that I’ve seen come out of that that’s been really beneficial is that people who have gone through this program tend to have a better sense, when we’ve been in-house, tend to have a better sense of how to work with research and have a better appreciation for the research that the researchers are doing, the career researchers are doing. And that partnership is richer. I have seen it go awry. I’ve seen people go through a few workshops, refuse the coaching, and then do things like put an app in front of consumers and say, “Do you like it?” So it’s not without risk. I totally get that.
At the same time, I’ve stopped having that discussion with the researchers that are worried about the field being diluted, or I’ve stopped using the word democratize, ’cause we’re not democratizing. We’re helping people do stuff that, frankly, they’re already doing. So why wouldn’t we help them do it better? So I think what, and I see it now in the consultancy, I’m really focused, if I say that I’m focused on helping leaders in companies that maybe don’t have a research leader, or maybe they’ve got one or two researchers, or no researchers, and they’ve got all of these other people out having conversations with customers, why wouldn’t I want to help them do that in a way that it’s gonna be more effective, where they’ll get good data? Because we all know that if you just go out and put an app in front of somebody and say, “Do you like it?” You’re not gonna get, it used to be garbage in, garbage out, right? That language still applies decades later.
So yes, there are risks. I know what the risks are. I think I named one of them anyway. People just go, “Well, why can’t I do persona research?” Or whatever, probably not the best example, but just helping them realize there are things that you need to know, and I get that you need to know those things, and you’re probably not gonna get what you’re looking for with this method. And so having those conversations, it doesn’t mean that once I leave, they’re not gonna try to do that anyway. I can’t control that. Even if I’m in the company, I can’t control that.
So I think the risk of people who do research or non-career researchers doing this just without any guidance is greater than the risk of them thinking they can do something that they really need a career researcher for. And I think it’s not, this is not unrelated to, I mean, it’s a bit of a tangent, but it’s not unrelated to the thing that we see where companies think they want research and they hire someone. I’m seeing, I was seeing more of this like before the big sort of layoffs happened starting at the end of 2022, I guess. I was seeing more first researcher roles that were a player coach, kind of lead manager, which I think is great. I think that’s what I would advise clients to do if you’re gonna get one person, make sure they’re at that level.
But I do still see companies hiring more junior people. And I know what they’re thinking. They’re thinking we need someone to do some research. So they’ll get someone who’s very smart and very well-trained in their research chops, but there may be a senior researcher or maybe more junior than that. And then they’re overwhelmed with, they don’t have a sense of the landscape or how to manage in that kind of an environment. They aren’t getting mentorship in their research work. And then there’s sort of a, like there can be at the company, it kind of a, well, that didn’t really work out. So we don’t need research. You know, there’s sort of, instead of the concept of research being seen, instead of research being seen as like a concept or a practice that kind of associated with a person. And then they go, we don’t need any researchers. We just need to do this ourselves. And so I feel like that has, I’ve seen that a bit.
And I’ve seen, I mean, I’ve also been, some of the people that come to me for coaching are people in that situation because they’re researchers that are not getting mentorship and they’ve kind of been thrown into this situation where they just don’t have the experience to be able to manage all the pieces that go with it because it’s not just about running studies. And I think I totally get the excitement about being the first researcher, you know, and when someone wants you to play that role. And I mean, it’s, you know, there’s a lot of trust that goes into that. I also know people that are sort of senior researcher level, I’m just throwing these terms out. I mean, it’s all, you know, it just depends on the person, but who would say, and you know, they’re in a career search and we’re talking about their career search and they’re like, I don’t want to be the first person ’cause I know what’s involved in that. So, you know, I think it’s like, yeah, I get why someone would take that job, even if they maybe have like a couple of years of experience ’cause it’s exciting. And I also hear people are probably qualified, you know, who have been working for six or seven years. And again, those numbers are just, who knows, you know, it just depends on the person. And they’re like, I don’t want to do that ’cause I know how hard it is.
Steve: We sort of shifted in this conversation a little bit to talking about your consultancy. What did you start and why?
Carol: I had been, towards the end of my time at NerdWallet, I had been getting calls from coworkers asking for help to set up a research program. Like, how do I get started if I want to set up research? And, you know, I was just having these conversations and realizing that I was really excited about this topic and that it’s your beginning. The beginning point is really exciting to me, right? So when I left NerdWallet, I started looking at open roles at the time. And they were this, like I was saying, player/coach kind of role, right? And so it’s like you’re doing some of the bigger research while you’re setting up operations, while you’re setting up a roadmap, while you’re setting up, you know, all the infrastructure and everything. And I had already done that. I had done it a couple times. So I realized I wasn’t excited about doing that again.
And what I was excited about was the leadership components of that. And so the coaching or advising, and we can talk about what I think the differences are there, but, you know, the sort of training, helping people become more self-sufficient, either leaders feel like they’re stronger at supporting a research practice, whether they have researchers or not. Again, like we were saying earlier, helping designers, PMs, you know, et cetera, feeling confident that they can collect insights. If they’re going to do it anyway, we may as well help them do it well. So those are the pieces that I realized I was more interested in. And also just having conversations with people about the importance of operations and thinking about research ops from the beginning or the middle, wherever you are, and how that can be such a force multiplier, you know, such a way to move forward more quickly by spending some time on infrastructure, tools, templates, like having some kind of process, knowing, you know, having some way for people to capture the insights that they’re collecting and share it in whatever way that, however that looks like things that are going to help you do things better and faster later.
So those were the pieces that I was really interested in. And I decided to just go out on my own. I have, you know, I was out on my own for a while through, let’s see, like through most of the 2000s, that looked more like contract research work at that point. And I was doing that in parallel with other work that I was doing that was not tech. But at this time I was like, I’m going to go all in on this consulting model and see what happens. And that was like towards the end of 2022.
Steve: Since you teased us with coaching versus advising, I’m going to ask you to take the bait. What do you think the difference is?
Carol: I mean, I think, and this isn’t like, you know, genius. I think this is the way that a lot of people distinguish those. But to me, coaching is more, let me start with advising. Coaching is more like I’m working with the head of design or I’m working with somebody, you know, head of product or someone in that team that’s in a leadership role to help them see, you know, for themselves, like how that can, how research can be, have more impact or, you know, again, whether they have researchers or not. And so advising, I think has much more of a, like we’re in a conversation and I’m giving them ideas or tips.
Coaching is more of a, I’m working with, I don’t do the big sort of life coaching or big picture career coaching. Like, should I do this anymore necessarily? Because I’m not like trained as a coach where I would do life coaching kind of thing. It’s more like, you know, somebody is in an ops role and wants to shift to a research role and they have all the training to do that, but people aren’t seeing them as a researcher. What do they need to do with their portfolio, their resume? How do they need to talk about the work? Somebody gets laid off, you know, it’s a surprise. They’re trying to prepare for their next role. Somebody is, like I said, in a role where they’re like the only researcher and they’re not getting the mentorship. They got feedback on a specific thing and they don’t really know how to work on it. And their manager isn’t really kind of maybe helping them figure it out. Like it’s a very specific engagement around a topic that we can say, here’s the end goal and here are the steps that you can go through to get to that end goal. And what are the milestones that we can look at along the way, even if it’s just like four weeks or six weeks.
It’s a very specific set of things that we’re doing to get somebody to a particular place. Whereas advising is also there’s a set sort of, you know, sort of a set like arrangement, a number of sessions or whatever. But it’s more like me tossing out advice or ideas, maybe more than I would in a coaching model.
Steve: I’m going to use a word you haven’t used, but when you talk about coaching, I think a little about facilitation. Whereas in the advising, you have a best practice or an idea or suggestion. In the coaching, you’re kind of working along the path to get this person to articulate specific goals, that kind of thing.
Carol: It’s kind of like they are going to do the work to get to a certain place. And I am helping facilitate that. And it’s the way that I work with people in coaching, it’s like there’s actually a worksheet that we use. And the worksheet kind of starts with like, what, again, I sort of should distinguish I’m not doing this sort of big picture, like what is my life about, but I do start with like, what’s your mission statement as a researcher? And what is your broader goal over the next few years? And then what are you trying to get to in the next few months, whatever that timeframe is? And that’s a worksheet where it’s like, literally, what steps are you going to take to get there? What, you know, how are you going to know that you’ve achieved that step? So what milestones are we looking for? What does success look like? When are we going to say you’re done with that step and, you know, maybe addressing a different step?
And so it’s not super linear like that, but it really is. It’s like a, you know, a template. And I found that that worked really well. Actually developed the template when I was at NerdWallet, because I found it worked really well for the team to help them think through either the broader, like, I want to get to be a manager. How do I do that kind of thing? Or the very specific, they got feedback on a performance review about something and over the next few months they want to work on it. And so it’s a really simple template and approach, but that’s how I keep the coaching engagements to like a particular goal that people are going for.
Steve: So coaching engagements, advising engagements, what are the other ways in which you’re working for whomever?
Carol: So I do workshops and I have one workshop that’s really targeted to researchers or, I mean, it could be anybody, but mostly the people who come are like lead researcher or managers or senior researchers or designers. It could be PMs as well, but that’s Prioritizing Research for impact. And you know, there’s a lot of conversation about impact. It’s really the thing that we have had to make sure that we’re measuring, right? It’s not about, and what is impact? We can talk about that in a minute, but the workshop is about how to think through how you’re going to get to impact. It’s not just run the studies that you want. It’s not just run the studies that somebody’s telling you they want. It’s like, what’s the business objective that we’re trying to achieve? What decisions are going to be made if we have this information for a particular study? What do we already know about this? And then we sort of go through this framework based on clarity, risk and cost. So what do we already know that’s clarity? What do we still need to know? What’s the risk of going forward without more research, any research? And what’s the cost of doing research? What’s the cost of developing this?
And there’s a worksheet. It’s really a spreadsheet that we toss all of this information into and have the conversation about each of these possible research projects. And then at the end, you can see what’s high priority, what’s medium, what’s low priority. And then we also talk about how to involve, who do you involve in this prioritization process? How do you involve partners? And then when we get to the end, like who’s the ultimate decision maker for research? That may not be the person that, I mean, sometimes people come into the workshop and they’re like, well, the person who’s making the ultimate decisions, the person who should be really. So that’s a conversation to have. And then after the decisions have been made, what are some best practices to convey prioritization decisions? Transparency, you know, share the work, show people how you got to that decision. Hopefully they were either involved in the conversation up front or someone on their team was who has helped them understand the process. And so nobody is super surprised at the end, ideally.
And then sharing out the results, like literally share the worksheet with everybody that needs to have it so they can see what decisions were made, which projects were prioritized against what other projects. And then for each of the, you know, if it’s sort of low priority and you’re not going to move forward, how do you communicate that? If it’s high priority, how do you communicate that? And then we end up with a lot of things that are sort of medium, like we need to do something, but we don’t need to do a fresh study. And so maybe that’s a researcher is going to go sit through what we already know, and that will save the team time by not doing fresh research because we already know a lot about it. So we have high clarity, but it is high risk to move forward without doing anything else, you know, and the cost to do this research, meaning like go through this stuff is pretty low relative to the cost of going through development and getting it wrong, which is pretty high. So pulling those levers in, you know, in the workshop, we go through this for like three research projects so people can actually do it by the end of the workshop, they’ve prioritized three projects, then they can take that back to their organization and use that tool, the worksheet.
Yeah, it’s on Maven, which is a platform that has, it’s actually a really good platform for all kinds of workshops and leadership. There are workshops on AI now, there’s all kinds of stuff in there. So that’s the one that’s about prioritizing research for impact. I also have one that I literally call it see maximum impact from customer conversations. And that’s a it’s creating a game plan for the, it could you can call it your research program, you can call it your, you know, customer insights practice, you can, however you describe the thing you’re trying to do by having customer conversations. But the idea is that we do that one’s really tailored to like, leadership.
So the people that come are usually like head of product, head of design, product ops, you know, UX leaders, whatever, it’s leadership role. It could be somebody who’s starting a research team who’s a researcher, and they haven’t done this before the kind of player coach person we’re talking about. But the idea is at the end of that workshop, we have a game plan. We do a gap analysis, what’s the current state of research, I’m just going to call it research shorthand, you know, what’s the ultimate desired state. And then let’s make a three month, very specific three month plan to get there. And we look at infrastructure, meaning tools, processes, training, whatever’s going on there, the operational pieces, we look at staff, that could mean you have a researcher, it could mean people doing research, it could mean there’s some operations person on another team that’s helping you recruit, could be anything. And then we organically in the conversation, we start to talk about the research roadmap, because people will come in and they’ll go, well, the most important thing we need to know is X. And so it’s not a workshop to lay out your whole research roadmap.
But those pieces come in, the thing we ultimately need to know is this. Right now we need to know this other piece. So yeah, that’s also on Maven. I’ve run it internally within the company for, you know, a handful of leaders. And I’ve also started running it on Maven. The third workshop that I have right now is this training, you know, designers, PMs, content strategists, whoever, to do their own research. And that’s the thing that we talked about earlier, three parts, planning a study, executing a study, synthesizing to get to actionable insights, and then some coaching. And that one I’ve been running within companies, and I’m going to put it on Maven soon. It’s in the process of moving. I can still run it within a company, but it’s in the process of also becoming available on Maven.
Steve: The one for leaders, the title has customer conversations, not research in it.
Carol: What I’m finding, and I’m not the only one, I’ve been in conversation with a lot of people that are finding this. I mean, in this conversation, we’re talking about research, I’m using that language. But you know, my target audience really is like head of product, head of design. And so there can be, and I think in the last year and a half, become even more of a challenge with the word research in that audience that sometimes people think it means it’s going to be big, it’s going to be expensive, it’s going to take a lot of time. And yeah, sometimes it might be big, expensive and take a lot of time if what you need to know is foundationally something really important to your business, right? That you don’t know that’s going to, you know, make it or break it kind of thing, right?
But I think a lot of what people need is not necessarily that. And I don’t want people, I don’t want those leaders to think that having conversations with customers needs to be big, expensive and take a lot of time. Of course, they’re doing research, you know. But like if you look on my website right now, the word “research” does not appear in until you scroll below the fold. And so I’m experimenting with the way to talk about the offerings that go beyond the word research, because I, unfortunately, I used to be much more of a purist, like many, many years ago earlier in my career. Well, people need to know that research can be lean. Yeah, people are going to figure out that research can be read because we’re going to do it. You know that I don’t need to be preaching about it and I don’t need to be stuck on using that language. I think one of the things that’s held us up in the past as a field is that we’ve been too attached to language process ideas that aren’t necessarily current anymore. And so I’m like, call it whatever you want, you know, like we’re going to do this thing and I think it’s going to help your business and I’m not attached to the word.
Steve: When you started talking about developing this business for yourself, you kind of hinged on like what was exciting to you. And I’m wondering, you know, now that you’re kind of up and going, like do you find in a different experience for yourself when you are doing this, say, through Maven and it’s for the public, for lack of a better term, versus working with an organization and kind of going into that organization? Is there any differences for you when what you’re doing in those different kinds of venues?
Carol: You know, I have this really deep background in teaching. And so for me, leading workshops is really fun and it’s really exciting and working within the organization can also be fun and exciting. It just, they are different and I enjoy both. Yeah, I mean, I like the kind of bringing people together from different organizations and seeing the kinds of experiences they bring in to the workshop and they get a lot of benefit out of that conversation. I mean, this is the feedback that I get, like not only was it valuable to get the, you know, the material and the worksheets and whatever insights I’m bringing and facilitation, but the experience that other people are bringing in from, you know, if we do this publicly is really valuable. And frankly, sometimes I have to really rein it in because they can just start going on and trying to solve each other’s stuff, you know, help each other solve things and, which is great and I love it when they, at the end, you know, people say, let’s connect on LinkedIn and keep the conversation going. I’m actually about to set up a way for people to keep the conversation going across cohorts. So that’s something that I’m going to be doing later this year as well, because there’s so much benefit that people find from checking in, you know? So yeah, it’s different and they’re both interesting to me for very different reasons.
Steve: You had offered to give a little more definition about what impact meant. So I want to loop back to that.
Carol: I’ve been looking at and following what other research leaders are saying about this too. And I think that one thing that we seem to all agree on is that impact goes beyond what I call product impact. So, you know, pretty obvious that impact means we do some research, we come up with some insights, you know, we take the most important of those and we do something to change the existing product or we move into an area that’s new and we see some kind of impact that we can measure in terms of, you know, lift in engagement or revenue or customer satisfaction or whatever the thing is that we’re measuring, right, from a business perspective. That’s one kind of impact, but there are other types. And so I think there are three things.
One, product impact, like I just described, and organizational impact. And that’s stuff like what we were talking about earlier, seeing teams understand better how to work with career researchers by going through the process of learning how to do some research for themselves. I would call that organizational impact. Organizational impact is, you know, content strategist understands, you know, let’s cut that one. I’m going to stick with the first one. Operational impact is stuff like efficiency. So and this, again, relates back to something I said earlier, but we look at this, you know, we try to prioritize the most important and most impactful research. We look at something where we already have a lot of information. We have a lot of clarity about this problem, but maybe this team doesn’t know it.
So for example, we sort of real example, there was a new team spun up around a very important initiative. So the product manager, the designer, and the content strategist were all new, but there was a researcher that had been doing research in that area. And so the product designer, content strategist, designer thought they needed to do a six-week sprint to uncover, you know, where they needed to go with this very important thing. And researcher knew that there was a lot of information already. Researchers spent, you know, something like half a day going through all the information that they had, sat down with this trio, shared the information with them in an hour. Researchers spent like four hours. We can calculate the cost of that time, saved this trio the first three weeks of the sprint. We can calculate the cost of the time that they would have spent and do math and say, we spent X dollars. We saved X dollars here. And they were able to go straight to concept testing because there was all this foundational work that had already been done. So that’s an example of operational efficiency. And I don’t know that we, there are people talking about this, some people talking about this, but I don’t know that we’ve spent as much time on those calculations as a field as I think we could.
Steve: Are there impacts that are not measurable or not easily measurable but still kind of make your list?
Carol: I’m sure there are. I think I’ve pulled my list down to three. I mean, if you look at some of the things that people have been writing about, there are like these much more detailed models. I think it goes back to what are you going to do with this impact? Like if we want to be able to go back to leadership team, or we want to be able to put, you know, at the end of a quarter on an OKR spreadsheet, like what our impact was, we need to make it digestible by other teams and leaders. And so I think we can, I feel having studied this for a while, that everything kind of rolls up to one of those three areas. So I haven’t found something that doesn’t roll up to those three areas, let me put it that way. And I think that if we keep it simple like that, we’re much more likely to be able to say we can see, you know, like we saved X dollars by not doing a bunch of extra research on this project. And that’s something that we can talk about very clearly. I think the related to this is that it can be hard to measure impact period. And we know that, you know, so if you’re a researcher who’s a shared resource across multiple teams, you work on one thing, you go off to work with another team, how are you going to know what team A did a month later, unless someone, you know, comes back and tells you, you may have to go back and ask, hey, what happened from that study? So you know how to describe the impact that you’re having.
But we need to be making the effort to try and find out. I mean, it’s hard. As a consultant, it’s hard for me to know what the ultimate impact is of these workshops and the coaching and the advising unless people tell me. And I also know quite well from my teaching experience, sometimes people learn a thing and then it’s not until, you know, a while later that it actually kicks in for them. So I think that when we’re talking about training that doesn’t have a direct sort of relationship to work that’s happening right now, yeah, it’s hard for me to even know what impact I’m having. But I think it’s really, really important for us to continually try to make sure we can get as much as we can about that. Thank you. So, obviously, none of us knows the future, and we can’t talk about the future unless we talk about how we got where we are and where we are now, right? So I think I actually want to back up to a bit of like the difference between nine years ago and now, because I think it’s relevant to this.
So when you first invited me to do this sort of redo, have this redo conversation, one of the prompts was what’s changed and my first thought was everything. And then I went back and listened to the original conversation from nine years ago, and I realized, oh, more than everything has changed. And nine years is a long time. So we would expect that there would be shifts. But aside from the obvious, like the pandemic, remote work, that kind of stuff, just listening back and thinking about the way I talked about the work then, the way we all were talking about the work then, and the way we talk about the work now, we’ve been talking about impact. We have not, I haven’t used the word qualitative research or design thinking. And the last conversation was all about that, because that’s where we were at that point in the industry. And so that was what was making the work successful then. But we were, if we look even further back, the internet, the history of the internet, right? I was at GeoCities in 1998. We were making it up as we went along. And I remember reading the IPO paperwork and it said, we have no idea how we’re going to make money from this thing. And so that was normal. And then we had the boom and we had the bust.
And then, you know, so through like 2000s, everybody was talking about design thinking into maybe late 2010s. Now it’s all about impact. So the way that we characterize the work has really shifted. I think for me also, when I think future, being at this point in my career, I start asking myself, what is my legacy? Which sounds really fancy. It’s not like I think I’m capital L legacy, like I’m a celebrity or something, but I think we all kind of go, I’ve been doing this for a long time. Like, what am I going to leave this field? What am I contributing and what impact do I want to have now as I go along? And then what am I going to be leaving whenever I decide to stop this? So I kind of look at all of that and I go, where are we now? What does the future look like? Obviously AI. I mean, we don’t need to say much more about that. We need to figure out how that’s going to, how do we use AI tools and that’s changing every single day. How do we use those tools to help the work that we’re doing now?
I mean, when people ask me, what do I need to do? I actually had a call like this yesterday, person got laid off. What do I need to be thinking about and what do I need to do to position myself for my next role? It’s like, you need to be studying AI tools. And like, if you haven’t already done that, like get jumped in. Right. So that’s one kind of really obvious thing. I think another thing that we’re seeing now that’s not going to go away, that’s going to be in the future is this idea of people who do research, right? Non-career researchers collecting some of their own insights. We have to just, we can’t stick our heads in the sand and say, make it go away. It’s not going away. It’s here. It’s been here for a while and we need to figure out how to jump on that. We need to be mixed methods researchers.
You know, it’s funny because when I started, I came out of human factors school and that was very quantitatively focused. And then when I started working, I just started at an era when the work was very qualitatively focused. And so now we’re shifting back towards generalists. So I think everybody needs to be some kind of mixed methods researcher. And I think most people are going to end up being sort of T-shaped, like you’re very strong in some areas more than others, but I don’t think we can go out anymore and say, I only do ethnographic, deep qualitative research and I don’t know anything about writing a survey. Like I just don’t know that that’s going to be possible moving forward.
And another area that I think is really important for us is to, for people who haven’t already been doing this, because some of us have been doing this for a while, but triangulating insights across different sources. So knowing how to dive a bit into analytics data, you know, understanding something about behavioral science, if you don’t already, you know, making friends with the people who run customer support. So you know what they’re hearing, like, do you have a market research function, you know, like all of these other insights functions that I personally think and have thought and have seen work really well, where we’re like, totally working together in a very collaborative way. You know, I think at a minimum, like knowing what they’re doing, if you’re not in an environment where their culture is that collaborative, but having some way to look at things across multiple types of insights functions.
So this is a bit of a personal aside, but it’s very relevant to this question. So I went public in January with the fact that I had lung cancer late last year, and I decided to go public with it, because I thought it might be valuable to people. And I’ve gotten, I mean, you know, in terms of personally what that did for me, it’s, you know, it’s just sort of I could have gone through a full examination of my whole life and career. Oh, my God, do I want to keep doing this? And what I realized is, I’m happy making the contributions that I’m making, even though it’s hard to directly measure impact. I’m hearing from people that they’re finding value from the work that we’re doing together. And so that’s what I want to leave the world with. I want to leave people with this idea that the work that we’ve done together is valuable to them, whether it’s tomorrow, they’re taking the prioritization worksheet back to their company, or we have this coaching conversation and two years from now, they see value in it in some way that they couldn’t have anticipated. So I think that’s really vague and broad. But, you know, I’m trying to be as clear as I can about focusing in the areas where I think I can make the best contribution and have the most impact. And that what zaps my energy, what gives me energy, like you were talking about earlier, I really like teaching these public workshops, as well as doing the work internally. So I’m going to keep doing the public workshops. Yeah, just keep reexamining what, how do I feel about the work that I’m doing? And what am I getting back from people?
Steve: I think you’re saying that talking about or going public with your medical situation prompted people to reach out to you in a way that highlighted the importance to you of the impact of the work that you’re doing. Is that correct?
Carol: Yeah, it was one of the things that did that. I mean, and also just literally in terms of impact of that article. Many people have told me, oh, I went and got a checkup, because I realized I hadn’t been taking care of my health. Oh, I smoked like many years ago, I should go check that out. Or I hugged my child more closely, I called my parents, the human elements of it, as well as the physical health elements were that was really rewarding. And I don’t know what I expected. But I don’t know, for some reason, I didn’t. I don’t know why I didn’t necessarily expect all of that.
Steve: Well, yeah, you have no template for, no prior in what the response to that is going to be.
Carol: No template. I mean, just to throw this out there. And just as another, like, I didn’t write this in the article, but I didn’t even know how to tell the clients that I was working with. And I and that’s where I said, I’m going on sabbatical. And then people thought I was taking a fancy vacation. And then I said, Well, I’m taking a medical leave. And then they worried a lot and started slacking me. Are you okay? What’s going on? How are you? What do you even say when you need to take two months off or whatever it was, if you don’t want to disclose because I wasn’t ready to disclose that. So I don’t even have a template for that is what I’m saying.
Steve: Yeah, now you’ve had that experience, so you’ve learned from that experience.
Carol: Hopefully helped other people.
Steve: We’ve been talking in and around impact at various levels and yet this article, the examples you just gave from your writing of maybe it’s outcomes, not impact. I don’t know. I don’t want to jargonize it, but the kinds of things that happened as a result of your action that you found meaningful and that people reported back that they found meaningful. I don’t want to take a personal experience and try to force map it into something professional, but I guess I’m just seeing echoes throughout our conversation.
Someone saying I hugged my kid is very interesting. That was the action they took. That was something they shared with you, and that was something that had meaning for you as a result of it. When we started off talking about founding your consultancy and determining what you wanted to do for that, what kind of offerings you had, I was just struck by the fact that you used the filter of what excited you.
Now we’ve been talking about changes and even looking ahead, present moment to “future.” I guess just maybe try to tie those things together. Are there things about the near future, the distant future, whatever time horizon we have for future, are there things about that with the work that you’re doing that excite you?
Carol: The thing that I’m excited about for this year is to actually do more of the public workshops. And so I think I mentioned I’m going to roll out the research, you know, lean research for designers and PMs to be public. I’ve got some other ideas that I’m working on, like, you know, some of the pain points that I hear from customers are finding the right people finding the right participants for research, which is a lot easier and B2C than it is in B2B. But there are some things that we can talk about. That’s going to be a workshop being having more conversation around knowing when do you do this yourself? And when do you hire a career researcher? What are the operations that you need to put in place to have your conversations with customers be effective? Like, there are topics like that, that I’m exploring for either short workshops or longer ones because those are things that I’m hearing about. And I like that public forum. So I’m excited to be rolling those out later this year.
Steve: Carol, it’s really great to have this chance nine years later and talk about what’s changed more than everything and the work that you have done and that are continuing to do.
Carol: Yeah, thanks so much for including me.
Steve: Thank you for taking the time. It’s great to chat with you.
Carol: It’s been really fun.
Steve: That’s it for today. I really appreciate you listening. Find Dollars to Donuts where podcasts are podcasted, or visit portigal.com/podcast for all of the episodes, complete with show notes and transcripts. Our theme music is by Bruce Todd.
The post 41. Carol Rossi returns first appeared on Portigal Consulting.
In this episode of Dollars to Donuts Carol Rossi returns to update us on the last 9 years. She’s now a consultant who focuses on user research leadership.
I’m happy making the contributions that I’m making, even though it’s hard to directly measure impact. I’m hearing from people that they’re finding value from the work that we’re doing together. I want to leave people with this idea that the work that we’ve done together is valuable to them, whether it’s tomorrow, or two years from now they see value in it in some way that they couldn’t have anticipated. I’m trying to be as clear as I can about focusing in the areas where I think I can make the best contribution and have the most impact. And keep reexamining, how do I feel about the work that I’m doing? And what am I getting back from people? – Carol Rossi
Show Links* Interviewing Users, second edition * Tent Talks Featuring: Steve Portigal * Amazon Reviews for Interviewing Users * Elevate Your Team’s Impact with Storytelling * Carol Rossi on Dollars to Donuts, 2015 * Carol on LinkedIn * Carol’s website * Edmunds * NerdWallet * Prioritizing Research for Impact on Maven * GeoCities * When the Healthiest Person You Know Gets Lung Cancer
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. I’m Steve Portigal. In this episode, I catch up with Carol Rossi, nine years after she was first on Dollars to Donuts.
There’s a bigger and better new edition of my classic book, Interviewing Users. As part of the launch of the book, I spoke with Russ Unger for his Tent Talk speaker series. Here’s a little clip.
Russ Unger: What’s your approach to ensuring that the feedback gathered from user interviews is effectively communicated and incorporated into the design process?
Steve: The first part of that I think is that you have to do something. You have to make sense of what you gather. Some of this kind of goes to maturity of any individual practice. I think the less experienced folks are, the more they want to just take what they remember about what was said and type it up. And that verb is, that’s stenography maybe, or collation as far as you get. You put these pieces together. And then you’re just taking requests or gathering complaints. You might as well use a survey for that. I think it’s the iceberg model, right? Some of it is above the surface, but a lot of it is below the surface. Below the surface means going back to what was said and looking at it and making inferences. What wasn’t said? How was it said? What said at the beginning and what said at the end? And that’s just within one interview. What did person A say? What did person B say?
And there’s a whole new chapter about this. It’s the analysis and synthesis process. And some folks say that the ratio should be two to one. For every hour of the feedback that you gather, you should spend two hours analyzing and synthesizing. And I think in a less evolved practice, it’s the inverse. You might spend half an hour for every hour or even less. The caveat here is not every research question merits this. If we are looking for, I don’t know, choice preference between something and something else, we might be really clear about what that is. We come back and say, do this.
But for anything where we want to understand why or understand opportunities or understand motivation, a new space you want to go into, characterize a customer that we haven’t worked with before, it really is worthwhile to go and do this analysis and synthesis.
How do we have impact? We have to have something impactful to say. I just want to say that. Some other factors that I think can make or break it is working collaboratively with stakeholders, the folks that you want to inform, influence, take action before you do the research. And so having an understanding of what business challenges are or business goals, like what are we trying to do as a company? And then formulating really good research questions. What are we going to learn in order to inform that? And then choosing methods and approaches that can support that. And not doing that in a vacuum. And then this has the effect of switching your role from being proactive to reactive.
I think it’s hard to have an impact with reactive work. Those requests that come are often late. They’re often based on a shallow assumption about what kind of value research can provide. And so you are going to give a thumbs up, thumbs down in some direction. So your sort of role as a provider of these kinds of insights is diminished. If you can be proactive, which means maybe understanding a roadmap or what decisions are being made or who else is going to do what and proposing research on your own roadmap that is intentional and is ahead of time, you leave space, of course, for things that come up, fire drills and so on.
But trying to work in a proactive, collaborative way, aligning on goals and then putting the effort in to make sense changes the whole conversation about what you’ve learned. You get to that point of sharing with somebody.
That’s part of a larger Tent Talk. You can check out the whole show and definitely buy your postal carrier and barista their very own copy of the second edition of Interviewing Users. If you want to help me out, write a very short review of Interviewing Users on Amazon.
Over the last couple of years, I’ve been partnering with Inzovu to run training workshops about storytelling. Storytelling is an essential human skill that powers how teams work together with each other and with their colleagues. I’ll put a link in the show notes with more info about what I’ve been up to with Inzovu. And if storytelling is something you’d like to build up in your organization, reach out to Inzovu or to me.
Okay, let’s go to my conversation with Carol. She’s a consultant with a focus on user research leadership. Carol, welcome back to Dollar a Donuts after nine years since we last talked. It’s great to talk to you again.
Carol Rossi: Yeah, thanks, Steve. I can’t believe it’s been nine years.
Steve: Time does fly. Let’s talk about those nine years. You know, what’s been the shift in your evolution in your professional world since then?
Carol: When we last talked on the show, I was at Edmunds and I was leading the UX research team there. And I had been there at that point, I guess, four years. I had started the team there and then ended up staying at Edmunds until 2017. And then I took a moment, because I’d been there for quite a long time, and took a moment to kind of ask myself what I wanted to do next. I call it my gap year. So I did some consulting, some really contract work as well as like consulting, helping people think about how to set up a team.
And then in 2018, I went to NerdWallet and that involved a move. So I was in LA for the bulk of my career. 2018, I moved to San Francisco for the job at NerdWallet. And that was an established team that I led for about four years. And I mean, we can go into detail about any of this stuff, but basically left NerdWallet in 2022 and started a consultancy where I’m now focused on helping companies, helping leaders know how to get the most impact from research
Steve: Can we talk about NerdWallet a little bit and then talk about your consulting work now?
Carol: Yeah, Sure.
Steve: So it was an established team. Is that right?
Carol: Yeah, it was. So there were three people on the team. There was actually an open headcount when I joined. We ended up doubling the size of that team. So we still remained a relatively small team, but we did get some additional people. We actually, I think some of the work that I’m really proud of there is that we went from having these researchers doing sort of very siloed work, or even though they were all researchers, they were hardly really working with each other even. And then developed that team to the point where we had a lot more strategic impact. We started a voice of customer program. Two of the people on the team became managers during the time that I was there. So they saw a fair amount of professional growth. And when I left, again, there was this voice of customer program established, as well as a program to train designers and PMs and content managers to do some of their own research. We had, well, on the market research side, they were doing some brand work. We were doing some kind of explorations about how that played out in product. So there were more things that are more sort of horizontal activities we were doing, and also empowering some people to collect their own insights, as well as deepening the impact of our team.
Steve: When you talk about coming in and the researchers that were there were siloed, my mind starts to go to that embedded word and what that means. But I think you’re talking about siloed in a grander scheme of things. But I don’t know, what does siloed look like then?
Carol: I think it’s a really good distinction. The difference between siloed and embedded to me is that embedded can be and is a very valuable way to participate in a product development team.
So it’s like, and we ended up with this sort of hybrid model, I would call it. Because at the time that I left, the team was reporting to ultimately me, but they were dedicated to specific focus areas. So we had one person working on the logged in experience and that involved maybe three pods. We were calling them pods, but squads, whatever, product trio, whatever language we use to talk about the combination of the PM, the designer, the content strategist, and some number of engineers. So we’d have one researcher per, let’s say, three of those pods, but they were all within a focus area.
So one was dedicated to the logged in experience. We had, for example, a couple people working on what we call the guest experience or shopping. So if you’re looking for, so I should say NerdWallet is a company that provides advice and products to consumers who might be looking for financial products. Consumers might be looking for a credit card or a mortgage or a personal loan or whatever. So you can either go and read some articles and then get linked to some potential credit cards for you based on what you’re interested in and your credit score and those kinds of things. Or you can download the app, log in, and get tailored advice based on your specific situation. So those are, at the time, were separate areas of the company in terms of the way the development was divided up.
So I think embedded to me is there’s a very healthy relationship with those pods where the researcher is either dedicated to one or maybe crosses over a couple of those areas, of those pods. But siloed to me is people are working on something so exclusively that maybe there isn’t a lot of conversation across. And I think what you lose in that kind of model is opportunity to take advantage of research that might be going on in an adjacent area or even a very different area but has relevance to what you’re doing.
And so you can have a lot more efficiency across the research function if you’re not re-doing work, you know. Or people are learning techniques from each other, you know. Or people are partnering so that there’s some broader impact across these different focus areas. So there might be — because to the consumer, to the ultimate user, the customer, they’re not seeing it, right, as these sort of separate areas. They’re seeing it as one experience. And sometimes in order to do product development, you have to divide things up.
So how do we keep the flow and the things that need to be similar across the experience and have it make sense by looking for those areas of, you know, similarity or continuity or whatever the word is that you want to use there. Some of the things that we did that worked really well were have just — so first of all, I should just be really clear. Because it was a manageable team, I mean, small enough team, we could do things like have team time every week where researchers felt like they were able to have a dedicated, you know, I think it was an hour or something, but a dedicated time where they could talk about some of the stuff they were doing, present problems to each other, learn from each other, like have time to be able to say, I’m doing this thing, I think there might be some relationship to what you did last year or what so-and-so did who’s not even here anymore and what can we talk about there.
So I think there’s — with a small enough team, you can definitely have people, you know, embedded or partially embedded within specific areas so they’re having maximum impact in those areas, but still conversation across. So I think that’s one thing that we did. Another thing we did was have kind of a loose repository. We weren’t using a really fancy tool. We just literally had, you know, a wiki where all of the research that was done was available. So people could go in and see what had been done and see if there was something relevant to them. And that could be like product managers, designers, anybody could go in and look at that and see. And then they’d usually come back and ask us questions. Hey, I saw this thing, you know, I wonder how that can be relevant to our team. So I think there are a few things that you can do.
Steve: You mentioned that you put in programs to teach other folks who are not career researchers to do research. What did that look like? How did that work?
Carol: I think the way that I’ve seen that work well is to create — when I’ve created a three-part series, workshop series. And so we start with these three workshops and then we do ongoing coaching. So it’s not just a matter of taking a, you know, a training session. And the first workshop is really setting up the research for success. And so that’s really about planning and study. So then there we talk about starting with the business objective, you know, like people will often start with a research question. Like we need to know X. Okay, well, why do you need to know X? Like there’s some business reason why you need to know it. So what’s the thing you need to know? Why do you need to know it? What decisions will be made as a result of that? And then what’s the best way to get that answer? Obviously, you know, in what timeframe do you need to know it and those things as well.
But starting with that framework to give people an appreciation for the fact that we don’t just run a study because we have a question. We kind of put context around it. Even if it’s a lean and I’m using the language of run a study, but this is like the language that some people are using is having conversations with customers or collecting insights, whatever language people are using. It’s the same thinking. And in that first workshop, we talk a lot about reducing bias, making sure we’re not asking leading questions or, you know, the way that we’re writing a task or something that we’re going to put up on a prompt that we’re going to put up on an unmoderated tool for a participant to engage with whatever. We talk a lot about how to do that in a way that those are going to be effective. And by the end of the first workshop, everybody has a lightweight research plan. I give a template. So there’s a template that has all those elements in it. And there are a lot of tips and tools and sample. Questions and sample tasks. So it’s pretty plug and play, but the foundational understanding is there in terms of, you know, not introducing bias and some of those other elements.
The second workshop is literally run a study. So when I was at Edmunds and we were doing in-person research, we would recruit a bunch of participants to come in and we’d have designers, PMs, engineers running their own interviews and, you know, we’d sit and give feedback often. Now what we do is all unmoderated. These workshops are all online now remote. So, you know, it’s an unmoderated tool and they set it up. They set up their study and the tool, and then we, you know, wait for the results to come in the videos or whatever.
And then the third workshop is in researcher language synthesis. And that’s the, like, how do you go from all this data that you just got to actionable insights? So we look at, we talk about the data. We talk about findings that come from there. We talk about insights that are really most important. And then we talk about prioritizing those insights according to the business objective, back to the business objective, back to the decisions that need to be made. What are the most important of all of those insights? Cause you might get a lot of stuff, you know, out of even a lean study. What are the things you need to take action on? And then we talk about taking action. And I have a, again, there’s a template for summarizing their findings of their study, but there’s a table that shows like, what was the insight? Okay. It was high priority. So we’re going to take action. We’re going to do this thing. Who is going to do this thing? It’s assigned to a team. Who’s the point person. Maybe it’s the PM on the team. Maybe it’s the designer. By what date is this thing going to be done? So everybody on the team now has agreed beyond the person that ran the study. They go back to the team, have the conversation. Everybody has agreed. Here’s what we’re going to do as a result. And then that goes into that table goes into their summary. And then there’s a way to go back. If the person that’s running this study. Is not one of the people on that team in this case, they probably are because they’re the designer or the PM or whatever. But you can go back and see what was actually done. Was it, you know, when was it done? What impact was gained by that study. And you can then add the impact t your impact tracker.
Then there’s the coaching that happens after the training. And that’s really vital to help people sear in the knowledge from the training and get feedback as they go along and ask questions. So sometimes the designer will go to the researcher who led the training and ask, will you take a look at my plan ’cause I’m going off the template a bit and I wanna make sure this makes sense. Or they have a question about synthesis because they get something that they didn’t anticipate and they wanna talk through how to do it. Or they need help figuring out how to message something to somebody that wasn’t on the team but needs to get these insights. So there are things that come up in life and sometimes it’s feedback on something that they’re doing. Like when I was at Edmonds and we were doing live interviews, we’d actually have a conversation after each set of interviews with the people that were running them and how did you think that went? And if we saw something that maybe they could benefit from, we would share that with them. So I think that’s a really important part of it and something I incorporate into the workshop that I do to train, it’s not just the training, it’s the follow-up coaching as well.
Steve: I think there’s a lot of hand-wringing off and on over the years about the risks and the consequences of, what did you call them? Non-career researchers, that’s a great term.
Carol: People who do research, I think is what people are saying now.
Steve: You know, we talk about the consequences of these kinds of programs that allow non-career researchers or people who do research. If we empower them as we’re kind of sort of the gatekeepers of the skills and the knowledge to do research, which may not even be an accurate framing anyway, ’cause people are doing research anyway.
Carol: Yeah.
Steve: here’s sometimes some hand-wringing about unintended consequences or intended consequences. I don’t know, with these programs of these different organizations, were there longer term kinds of changes that you noticed?
Carol: Yeah, I think it’s a good question. And I’ll just say, I don’t have that argument anymore with people. I have stopped trying to defend how this can work and how I’ve seen it work well, because the fact is, I’m just really realistic. First of all, I’ve seen it work well in this way that we talked about where there’s this sort of training set and then this sort of coaching activity, and there’s a conversation. It’s an ongoing conversation. And so what I’ve seen work well, one of the things that I’ve seen come out of that that’s been really beneficial is that people who have gone through this program tend to have a better sense, when we’ve been in-house, tend to have a better sense of how to work with research and have a better appreciation for the research that the researchers are doing, the career researchers are doing. And that partnership is richer. I have seen it go awry. I’ve seen people go through a few workshops, refuse the coaching, and then do things like put an app in front of consumers and say, “Do you like it?” So it’s not without risk. I totally get that.
At the same time, I’ve stopped having that discussion with the researchers that are worried about the field being diluted, or I’ve stopped using the word democratize, ’cause we’re not democratizing. We’re helping people do stuff that, frankly, they’re already doing. So why wouldn’t we help them do it better? So I think what, and I see it now in the consultancy, I’m really focused, if I say that I’m focused on helping leaders in companies that maybe don’t have a research leader, or maybe they’ve got one or two researchers, or no researchers, and they’ve got all of these other people out having conversations with customers, why wouldn’t I want to help them do that in a way that it’s gonna be more effective, where they’ll get good data? Because we all know that if you just go out and put an app in front of somebody and say, “Do you like it?” You’re not gonna get, it used to be garbage in, garbage out, right? That language still applies decades later.
So yes, there are risks. I know what the risks are. I think I named one of them anyway. People just go, “Well, why can’t I do persona research?” Or whatever, probably not the best example, but just helping them realize there are things that you need to know, and I get that you need to know those things, and you’re probably not gonna get what you’re looking for with this method. And so having those conversations, it doesn’t mean that once I leave, they’re not gonna try to do that anyway. I can’t control that. Even if I’m in the company, I can’t control that.
So I think the risk of people who do research or non-career researchers doing this just without any guidance is greater than the risk of them thinking they can do something that they really need a career researcher for. And I think it’s not, this is not unrelated to, I mean, it’s a bit of a tangent, but it’s not unrelated to the thing that we see where companies think they want research and they hire someone. I’m seeing, I was seeing more of this like before the big sort of layoffs happened starting at the end of 2022, I guess. I was seeing more first researcher roles that were a player coach, kind of lead manager, which I think is great. I think that’s what I would advise clients to do if you’re gonna get one person, make sure they’re at that level.
But I do still see companies hiring more junior people. And I know what they’re thinking. They’re thinking we need someone to do some research. So they’ll get someone who’s very smart and very well-trained in their research chops, but there may be a senior researcher or maybe more junior than that. And then they’re overwhelmed with, they don’t have a sense of the landscape or how to manage in that kind of an environment. They aren’t getting mentorship in their research work. And then there’s sort of a, like there can be at the company, it kind of a, well, that didn’t really work out. So we don’t need research. You know, there’s sort of, instead of the concept of research being seen, instead of research being seen as like a concept or a practice that kind of associated with a person. And then they go, we don’t need any researchers. We just need to do this ourselves. And so I feel like that has, I’ve seen that a bit.
And I’ve seen, I mean, I’ve also been, some of the people that come to me for coaching are people in that situation because they’re researchers that are not getting mentorship and they’ve kind of been thrown into this situation where they just don’t have the experience to be able to manage all the pieces that go with it because it’s not just about running studies. And I think I totally get the excitement about being the first researcher, you know, and when someone wants you to play that role. And I mean, it’s, you know, there’s a lot of trust that goes into that. I also know people that are sort of senior researcher level, I’m just throwing these terms out. I mean, it’s all, you know, it just depends on the person, but who would say, and you know, they’re in a career search and we’re talking about their career search and they’re like, I don’t want to be the first person ’cause I know what’s involved in that. So, you know, I think it’s like, yeah, I get why someone would take that job, even if they maybe have like a couple of years of experience ’cause it’s exciting. And I also hear people are probably qualified, you know, who have been working for six or seven years. And again, those numbers are just, who knows, you know, it just depends on the person. And they’re like, I don’t want to do that ’cause I know how hard it is.
Steve: We sort of shifted in this conversation a little bit to talking about your consultancy. What did you start and why?
Carol: I had been, towards the end of my time at NerdWallet, I had been getting calls from coworkers asking for help to set up a research program. Like, how do I get started if I want to set up research? And, you know, I was just having these conversations and realizing that I was really excited about this topic and that it’s your beginning. The beginning point is really exciting to me, right? So when I left NerdWallet, I started looking at open roles at the time. And they were this, like I was saying, player/coach kind of role, right? And so it’s like you’re doing some of the bigger research while you’re setting up operations, while you’re setting up a roadmap, while you’re setting up, you know, all the infrastructure and everything. And I had already done that. I had done it a couple times. So I realized I wasn’t excited about doing that again.
And what I was excited about was the leadership components of that. And so the coaching or advising, and we can talk about what I think the differences are there, but, you know, the sort of training, helping people become more self-sufficient, either leaders feel like they’re stronger at supporting a research practice, whether they have researchers or not. Again, like we were saying earlier, helping designers, PMs, you know, et cetera, feeling confident that they can collect insights. If they’re going to do it anyway, we may as well help them do it well. So those are the pieces that I realized I was more interested in. And also just having conversations with people about the importance of operations and thinking about research ops from the beginning or the middle, wherever you are, and how that can be such a force multiplier, you know, such a way to move forward more quickly by spending some time on infrastructure, tools, templates, like having some kind of process, knowing, you know, having some way for people to capture the insights that they’re collecting and share it in whatever way that, however that looks like things that are going to help you do things better and faster later.
So those were the pieces that I was really interested in. And I decided to just go out on my own. I have, you know, I was out on my own for a while through, let’s see, like through most of the 2000s, that looked more like contract research work at that point. And I was doing that in parallel with other work that I was doing that was not tech. But at this time I was like, I’m going to go all in on this consulting model and see what happens. And that was like towards the end of 2022.
Steve: Since you teased us with coaching versus advising, I’m going to ask you to take the bait. What do you think the difference is?
Carol: I mean, I think, and this isn’t like, you know, genius. I think this is the way that a lot of people distinguish those. But to me, coaching is more, let me start with advising. Coaching is more like I’m working with the head of design or I’m working with somebody, you know, head of product or someone in that team that’s in a leadership role to help them see, you know, for themselves, like how that can, how research can be, have more impact or, you know, again, whether they have researchers or not. And so advising, I think has much more of a, like we’re in a conversation and I’m giving them ideas or tips.
Coaching is more of a, I’m working with, I don’t do the big sort of life coaching or big picture career coaching. Like, should I do this anymore necessarily? Because I’m not like trained as a coach where I would do life coaching kind of thing. It’s more like, you know, somebody is in an ops role and wants to shift to a research role and they have all the training to do that, but people aren’t seeing them as a researcher. What do they need to do with their portfolio, their resume? How do they need to talk about the work? Somebody gets laid off, you know, it’s a surprise. They’re trying to prepare for their next role. Somebody is, like I said, in a role where they’re like the only researcher and they’re not getting the mentorship. They got feedback on a specific thing and they don’t really know how to work on it. And their manager isn’t really kind of maybe helping them figure it out. Like it’s a very specific engagement around a topic that we can say, here’s the end goal and here are the steps that you can go through to get to that end goal. And what are the milestones that we can look at along the way, even if it’s just like four weeks or six weeks.
It’s a very specific set of things that we’re doing to get somebody to a particular place. Whereas advising is also there’s a set sort of, you know, sort of a set like arrangement, a number of sessions or whatever. But it’s more like me tossing out advice or ideas, maybe more than I would in a coaching model.
Steve: I’m going to use a word you haven’t used, but when you talk about coaching, I think a little about facilitation. Whereas in the advising, you have a best practice or an idea or suggestion. In the coaching, you’re kind of working along the path to get this person to articulate specific goals, that kind of thing.
Carol: It’s kind of like they are going to do the work to get to a certain place. And I am helping facilitate that. And it’s the way that I work with people in coaching, it’s like there’s actually a worksheet that we use. And the worksheet kind of starts with like, what, again, I sort of should distinguish I’m not doing this sort of big picture, like what is my life about, but I do start with like, what’s your mission statement as a researcher? And what is your broader goal over the next few years? And then what are you trying to get to in the next few months, whatever that timeframe is? And that’s a worksheet where it’s like, literally, what steps are you going to take to get there? What, you know, how are you going to know that you’ve achieved that step? So what milestones are we looking for? What does success look like? When are we going to say you’re done with that step and, you know, maybe addressing a different step?
And so it’s not super linear like that, but it really is. It’s like a, you know, a template. And I found that that worked really well. Actually developed the template when I was at NerdWallet, because I found it worked really well for the team to help them think through either the broader, like, I want to get to be a manager. How do I do that kind of thing? Or the very specific, they got feedback on a performance review about something and over the next few months they want to work on it. And so it’s a really simple template and approach, but that’s how I keep the coaching engagements to like a particular goal that people are going for.
Steve: So coaching engagements, advising engagements, what are the other ways in which you’re working for whomever?
Carol: So I do workshops and I have one workshop that’s really targeted to researchers or, I mean, it could be anybody, but mostly the people who come are like lead researcher or managers or senior researchers or designers. It could be PMs as well, but that’s Prioritizing Research for impact. And you know, there’s a lot of conversation about impact. It’s really the thing that we have had to make sure that we’re measuring, right? It’s not about, and what is impact? We can talk about that in a minute, but the workshop is about how to think through how you’re going to get to impact. It’s not just run the studies that you want. It’s not just run the studies that somebody’s telling you they want. It’s like, what’s the business objective that we’re trying to achieve? What decisions are going to be made if we have this information for a particular study? What do we already know about this? And then we sort of go through this framework based on clarity, risk and cost. So what do we already know that’s clarity? What do we still need to know? What’s the risk of going forward without more research, any research? And what’s the cost of doing research? What’s the cost of developing this?
And there’s a worksheet. It’s really a spreadsheet that we toss all of this information into and have the conversation about each of these possible research projects. And then at the end, you can see what’s high priority, what’s medium, what’s low priority. And then we also talk about how to involve, who do you involve in this prioritization process? How do you involve partners? And then when we get to the end, like who’s the ultimate decision maker for research? That may not be the person that, I mean, sometimes people come into the workshop and they’re like, well, the person who’s making the ultimate decisions, the person who should be really. So that’s a conversation to have. And then after the decisions have been made, what are some best practices to convey prioritization decisions? Transparency, you know, share the work, show people how you got to that decision. Hopefully they were either involved in the conversation up front or someone on their team was who has helped them understand the process. And so nobody is super surprised at the end, ideally.
And then sharing out the results, like literally share the worksheet with everybody that needs to have it so they can see what decisions were made, which projects were prioritized against what other projects. And then for each of the, you know, if it’s sort of low priority and you’re not going to move forward, how do you communicate that? If it’s high priority, how do you communicate that? And then we end up with a lot of things that are sort of medium, like we need to do something, but we don’t need to do a fresh study. And so maybe that’s a researcher is going to go sit through what we already know, and that will save the team time by not doing fresh research because we already know a lot about it. So we have high clarity, but it is high risk to move forward without doing anything else, you know, and the cost to do this research, meaning like go through this stuff is pretty low relative to the cost of going through development and getting it wrong, which is pretty high. So pulling those levers in, you know, in the workshop, we go through this for like three research projects so people can actually do it by the end of the workshop, they’ve prioritized three projects, then they can take that back to their organization and use that tool, the worksheet.
Yeah, it’s on Maven, which is a platform that has, it’s actually a really good platform for all kinds of workshops and leadership. There are workshops on AI now, there’s all kinds of stuff in there. So that’s the one that’s about prioritizing research for impact. I also have one that I literally call it see maximum impact from customer conversations. And that’s a it’s creating a game plan for the, it could you can call it your research program, you can call it your, you know, customer insights practice, you can, however you describe the thing you’re trying to do by having customer conversations. But the idea is that we do that one’s really tailored to like, leadership.
So the people that come are usually like head of product, head of design, product ops, you know, UX leaders, whatever, it’s leadership role. It could be somebody who’s starting a research team who’s a researcher, and they haven’t done this before the kind of player coach person we’re talking about. But the idea is at the end of that workshop, we have a game plan. We do a gap analysis, what’s the current state of research, I’m just going to call it research shorthand, you know, what’s the ultimate desired state. And then let’s make a three month, very specific three month plan to get there. And we look at infrastructure, meaning tools, processes, training, whatever’s going on there, the operational pieces, we look at staff, that could mean you have a researcher, it could mean people doing research, it could mean there’s some operations person on another team that’s helping you recruit, could be anything. And then we organically in the conversation, we start to talk about the research roadmap, because people will come in and they’ll go, well, the most important thing we need to know is X. And so it’s not a workshop to lay out your whole research roadmap.
But those pieces come in, the thing we ultimately need to know is this. Right now we need to know this other piece. So yeah, that’s also on Maven. I’ve run it internally within the company for, you know, a handful of leaders. And I’ve also started running it on Maven. The third workshop that I have right now is this training, you know, designers, PMs, content strategists, whoever, to do their own research. And that’s the thing that we talked about earlier, three parts, planning a study, executing a study, synthesizing to get to actionable insights, and then some coaching. And that one I’ve been running within companies, and I’m going to put it on Maven soon. It’s in the process of moving. I can still run it within a company, but it’s in the process of also becoming available on Maven.
Steve: The one for leaders, the title has customer conversations, not research in it.
Carol: What I’m finding, and I’m not the only one, I’ve been in conversation with a lot of people that are finding this. I mean, in this conversation, we’re talking about research, I’m using that language. But you know, my target audience really is like head of product, head of design. And so there can be, and I think in the last year and a half, become even more of a challenge with the word research in that audience that sometimes people think it means it’s going to be big, it’s going to be expensive, it’s going to take a lot of time. And yeah, sometimes it might be big, expensive and take a lot of time if what you need to know is foundationally something really important to your business, right? That you don’t know that’s going to, you know, make it or break it kind of thing, right?
But I think a lot of what people need is not necessarily that. And I don’t want people, I don’t want those leaders to think that having conversations with customers needs to be big, expensive and take a lot of time. Of course, they’re doing research, you know. But like if you look on my website right now, the word “research” does not appear in until you scroll below the fold. And so I’m experimenting with the way to talk about the offerings that go beyond the word research, because I, unfortunately, I used to be much more of a purist, like many, many years ago earlier in my career. Well, people need to know that research can be lean. Yeah, people are going to figure out that research can be read because we’re going to do it. You know that I don’t need to be preaching about it and I don’t need to be stuck on using that language. I think one of the things that’s held us up in the past as a field is that we’ve been too attached to language process ideas that aren’t necessarily current anymore. And so I’m like, call it whatever you want, you know, like we’re going to do this thing and I think it’s going to help your business and I’m not attached to the word.
Steve: When you started talking about developing this business for yourself, you kind of hinged on like what was exciting to you. And I’m wondering, you know, now that you’re kind of up and going, like do you find in a different experience for yourself when you are doing this, say, through Maven and it’s for the public, for lack of a better term, versus working with an organization and kind of going into that organization? Is there any differences for you when what you’re doing in those different kinds of venues?
Carol: You know, I have this really deep background in teaching. And so for me, leading workshops is really fun and it’s really exciting and working within the organization can also be fun and exciting. It just, they are different and I enjoy both. Yeah, I mean, I like the kind of bringing people together from different organizations and seeing the kinds of experiences they bring in to the workshop and they get a lot of benefit out of that conversation. I mean, this is the feedback that I get, like not only was it valuable to get the, you know, the material and the worksheets and whatever insights I’m bringing and facilitation, but the experience that other people are bringing in from, you know, if we do this publicly is really valuable. And frankly, sometimes I have to really rein it in because they can just start going on and trying to solve each other’s stuff, you know, help each other solve things and, which is great and I love it when they, at the end, you know, people say, let’s connect on LinkedIn and keep the conversation going. I’m actually about to set up a way for people to keep the conversation going across cohorts. So that’s something that I’m going to be doing later this year as well, because there’s so much benefit that people find from checking in, you know? So yeah, it’s different and they’re both interesting to me for very different reasons.
Steve: You had offered to give a little more definition about what impact meant. So I want to loop back to that.
Carol: I’ve been looking at and following what other research leaders are saying about this too. And I think that one thing that we seem to all agree on is that impact goes beyond what I call product impact. So, you know, pretty obvious that impact means we do some research, we come up with some insights, you know, we take the most important of those and we do something to change the existing product or we move into an area that’s new and we see some kind of impact that we can measure in terms of, you know, lift in engagement or revenue or customer satisfaction or whatever the thing is that we’re measuring, right, from a business perspective. That’s one kind of impact, but there are other types. And so I think there are three things.
One, product impact, like I just described, and organizational impact. And that’s stuff like what we were talking about earlier, seeing teams understand better how to work with career researchers by going through the process of learning how to do some research for themselves. I would call that organizational impact. Organizational impact is, you know, content strategist understands, you know, let’s cut that one. I’m going to stick with the first one. Operational impact is stuff like efficiency. So and this, again, relates back to something I said earlier, but we look at this, you know, we try to prioritize the most important and most impactful research. We look at something where we already have a lot of information. We have a lot of clarity about this problem, but maybe this team doesn’t know it.
So for example, we sort of real example, there was a new team spun up around a very important initiative. So the product manager, the designer, and the content strategist were all new, but there was a researcher that had been doing research in that area. And so the product designer, content strategist, designer thought they needed to do a six-week sprint to uncover, you know, where they needed to go with this very important thing. And researcher knew that there was a lot of information already. Researchers spent, you know, something like half a day going through all the information that they had, sat down with this trio, shared the information with them in an hour. Researchers spent like four hours. We can calculate the cost of that time, saved this trio the first three weeks of the sprint. We can calculate the cost of the time that they would have spent and do math and say, we spent X dollars. We saved X dollars here. And they were able to go straight to concept testing because there was all this foundational work that had already been done. So that’s an example of operational efficiency. And I don’t know that we, there are people talking about this, some people talking about this, but I don’t know that we’ve spent as much time on those calculations as a field as I think we could.
Steve: Are there impacts that are not measurable or not easily measurable but still kind of make your list?
Carol: I’m sure there are. I think I’ve pulled my list down to three. I mean, if you look at some of the things that people have been writing about, there are like these much more detailed models. I think it goes back to what are you going to do with this impact? Like if we want to be able to go back to leadership team, or we want to be able to put, you know, at the end of a quarter on an OKR spreadsheet, like what our impact was, we need to make it digestible by other teams and leaders. And so I think we can, I feel having studied this for a while, that everything kind of rolls up to one of those three areas. So I haven’t found something that doesn’t roll up to those three areas, let me put it that way. And I think that if we keep it simple like that, we’re much more likely to be able to say we can see, you know, like we saved X dollars by not doing a bunch of extra research on this project. And that’s something that we can talk about very clearly. I think the related to this is that it can be hard to measure impact period. And we know that, you know, so if you’re a researcher who’s a shared resource across multiple teams, you work on one thing, you go off to work with another team, how are you going to know what team A did a month later, unless someone, you know, comes back and tells you, you may have to go back and ask, hey, what happened from that study? So you know how to describe the impact that you’re having.
But we need to be making the effort to try and find out. I mean, it’s hard. As a consultant, it’s hard for me to know what the ultimate impact is of these workshops and the coaching and the advising unless people tell me. And I also know quite well from my teaching experience, sometimes people learn a thing and then it’s not until, you know, a while later that it actually kicks in for them. So I think that when we’re talking about training that doesn’t have a direct sort of relationship to work that’s happening right now, yeah, it’s hard for me to even know what impact I’m having. But I think it’s really, really important for us to continually try to make sure we can get as much as we can about that. Thank you. So, obviously, none of us knows the future, and we can’t talk about the future unless we talk about how we got where we are and where we are now, right? So I think I actually want to back up to a bit of like the difference between nine years ago and now, because I think it’s relevant to this.
So when you first invited me to do this sort of redo, have this redo conversation, one of the prompts was what’s changed and my first thought was everything. And then I went back and listened to the original conversation from nine years ago, and I realized, oh, more than everything has changed. And nine years is a long time. So we would expect that there would be shifts. But aside from the obvious, like the pandemic, remote work, that kind of stuff, just listening back and thinking about the way I talked about the work then, the way we all were talking about the work then, and the way we talk about the work now, we’ve been talking about impact. We have not, I haven’t used the word qualitative research or design thinking. And the last conversation was all about that, because that’s where we were at that point in the industry. And so that was what was making the work successful then. But we were, if we look even further back, the internet, the history of the internet, right? I was at GeoCities in 1998. We were making it up as we went along. And I remember reading the IPO paperwork and it said, we have no idea how we’re going to make money from this thing. And so that was normal. And then we had the boom and we had the bust.
And then, you know, so through like 2000s, everybody was talking about design thinking into maybe late 2010s. Now it’s all about impact. So the way that we characterize the work has really shifted. I think for me also, when I think future, being at this point in my career, I start asking myself, what is my legacy? Which sounds really fancy. It’s not like I think I’m capital L legacy, like I’m a celebrity or something, but I think we all kind of go, I’ve been doing this for a long time. Like, what am I going to leave this field? What am I contributing and what impact do I want to have now as I go along? And then what am I going to be leaving whenever I decide to stop this? So I kind of look at all of that and I go, where are we now? What does the future look like? Obviously AI. I mean, we don’t need to say much more about that. We need to figure out how that’s going to, how do we use AI tools and that’s changing every single day. How do we use those tools to help the work that we’re doing now?
I mean, when people ask me, what do I need to do? I actually had a call like this yesterday, person got laid off. What do I need to be thinking about and what do I need to do to position myself for my next role? It’s like, you need to be studying AI tools. And like, if you haven’t already done that, like get jumped in. Right. So that’s one kind of really obvious thing. I think another thing that we’re seeing now that’s not going to go away, that’s going to be in the future is this idea of people who do research, right? Non-career researchers collecting some of their own insights. We have to just, we can’t stick our heads in the sand and say, make it go away. It’s not going away. It’s here. It’s been here for a while and we need to figure out how to jump on that. We need to be mixed methods researchers.
You know, it’s funny because when I started, I came out of human factors school and that was very quantitatively focused. And then when I started working, I just started at an era when the work was very qualitatively focused. And so now we’re shifting back towards generalists. So I think everybody needs to be some kind of mixed methods researcher. And I think most people are going to end up being sort of T-shaped, like you’re very strong in some areas more than others, but I don’t think we can go out anymore and say, I only do ethnographic, deep qualitative research and I don’t know anything about writing a survey. Like I just don’t know that that’s going to be possible moving forward.
And another area that I think is really important for us is to, for people who haven’t already been doing this, because some of us have been doing this for a while, but triangulating insights across different sources. So knowing how to dive a bit into analytics data, you know, understanding something about behavioral science, if you don’t already, you know, making friends with the people who run customer support. So you know what they’re hearing, like, do you have a market research function, you know, like all of these other insights functions that I personally think and have thought and have seen work really well, where we’re like, totally working together in a very collaborative way. You know, I think at a minimum, like knowing what they’re doing, if you’re not in an environment where their culture is that collaborative, but having some way to look at things across multiple types of insights functions.
So this is a bit of a personal aside, but it’s very relevant to this question. So I went public in January with the fact that I had lung cancer late last year, and I decided to go public with it, because I thought it might be valuable to people. And I’ve gotten, I mean, you know, in terms of personally what that did for me, it’s, you know, it’s just sort of I could have gone through a full examination of my whole life and career. Oh, my God, do I want to keep doing this? And what I realized is, I’m happy making the contributions that I’m making, even though it’s hard to directly measure impact. I’m hearing from people that they’re finding value from the work that we’re doing together. And so that’s what I want to leave the world with. I want to leave people with this idea that the work that we’ve done together is valuable to them, whether it’s tomorrow, they’re taking the prioritization worksheet back to their company, or we have this coaching conversation and two years from now, they see value in it in some way that they couldn’t have anticipated. So I think that’s really vague and broad. But, you know, I’m trying to be as clear as I can about focusing in the areas where I think I can make the best contribution and have the most impact. And that what zaps my energy, what gives me energy, like you were talking about earlier, I really like teaching these public workshops, as well as doing the work internally. So I’m going to keep doing the public workshops. Yeah, just keep reexamining what, how do I feel about the work that I’m doing? And what am I getting back from people?
Steve: I think you’re saying that talking about or going public with your medical situation prompted people to reach out to you in a way that highlighted the importance to you of the impact of the work that you’re doing. Is that correct?
Carol: Yeah, it was one of the things that did that. I mean, and also just literally in terms of impact of that article. Many people have told me, oh, I went and got a checkup, because I realized I hadn’t been taking care of my health. Oh, I smoked like many years ago, I should go check that out. Or I hugged my child more closely, I called my parents, the human elements of it, as well as the physical health elements were that was really rewarding. And I don’t know what I expected. But I don’t know, for some reason, I didn’t. I don’t know why I didn’t necessarily expect all of that.
Steve: Well, yeah, you have no template for, no prior in what the response to that is going to be.
Carol: No template. I mean, just to throw this out there. And just as another, like, I didn’t write this in the article, but I didn’t even know how to tell the clients that I was working with. And I and that’s where I said, I’m going on sabbatical. And then people thought I was taking a fancy vacation. And then I said, Well, I’m taking a medical leave. And then they worried a lot and started slacking me. Are you okay? What’s going on? How are you? What do you even say when you need to take two months off or whatever it was, if you don’t want to disclose because I wasn’t ready to disclose that. So I don’t even have a template for that is what I’m saying.
Steve: Yeah, now you’ve had that experience, so you’ve learned from that experience.
Carol: Hopefully helped other people.
Steve: We’ve been talking in and around impact at various levels and yet this article, the examples you just gave from your writing of maybe it’s outcomes, not impact. I don’t know. I don’t want to jargonize it, but the kinds of things that happened as a result of your action that you found meaningful and that people reported back that they found meaningful. I don’t want to take a personal experience and try to force map it into something professional, but I guess I’m just seeing echoes throughout our conversation.
Someone saying I hugged my kid is very interesting. That was the action they took. That was something they shared with you, and that was something that had meaning for you as a result of it. When we started off talking about founding your consultancy and determining what you wanted to do for that, what kind of offerings you had, I was just struck by the fact that you used the filter of what excited you.
Now we’ve been talking about changes and even looking ahead, present moment to “future.” I guess just maybe try to tie those things together. Are there things about the near future, the distant future, whatever time horizon we have for future, are there things about that with the work that you’re doing that excite you?
Carol: The thing that I’m excited about for this year is to actually do more of the public workshops. And so I think I mentioned I’m going to roll out the research, you know, lean research for designers and PMs to be public. I’ve got some other ideas that I’m working on, like, you know, some of the pain points that I hear from customers are finding the right people finding the right participants for research, which is a lot easier and B2C than it is in B2B. But there are some things that we can talk about. That’s going to be a workshop being having more conversation around knowing when do you do this yourself? And when do you hire a career researcher? What are the operations that you need to put in place to have your conversations with customers be effective? Like, there are topics like that, that I’m exploring for either short workshops or longer ones because those are things that I’m hearing about. And I like that public forum. So I’m excited to be rolling those out later this year.
Steve: Carol, it’s really great to have this chance nine years later and talk about what’s changed more than everything and the work that you have done and that are continuing to do.
Carol: Yeah, thanks so much for including me.
Steve: Thank you for taking the time. It’s great to chat with you.
Carol: It’s been really fun.
Steve: That’s it for today. I really appreciate you listening. Find Dollars to Donuts where podcasts are podcasted, or visit portigal.com/podcast for all of the episodes, complete with show notes and transcripts. Our theme music is by Bruce Todd.
The post 41. Carol Rossi returns first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I welcome Gregg Bernstein back for a follow-up episode. He’s now Director of User Research at Hearst Magazines.
The thing that I always come back to is that there is no one way to do research. And I also think there’s no one way to do research leadership. So often when I post a video or write something, it’s a knee-jerk reaction to something somebody else might have said that I feel like is going to discourage folks or paint this industry in a negative light. I don’t want to sound like a Pollyanna, but I love this field. I think it’s invaluable. I think more companies should have a research function. And so anything that I write is usually meant to show that there’s opportunity, there is value in this work. – Gregg Bernstein
Show Links* Interviewing Users, second edition * Steve Portigal on the Understanding Users podcast * Gregg Bernstein on Dollars to Donuts, 2015 * Gregg on LinkedIn * Mailchimp * Vox Media * Condé Nast * Hearst Magazines * Research Practice: Perspectives from UX researchers in a changing field * Sian Townsend on LinkedIn * Nicole Fenton * How to Make Sense of Any Mess by Abby Covert * Gregg’s site
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Today, I’m chatting with Gregg Bernstein, nine years after he first appeared on episode one of Dollars to Donuts. For context, here’s a tiny clip from that episode.
Gregg Bernstein: And I’m a little disappointed that you didn’t start this interview off by saying this is two Jews talking about customer research.
Steve: But before that, did you know that there’s a new edition of my classic book interviewing users? The modern day book tour seems to be in fact the podcast tour. And so recently I chatted with Mike Green for his Understanding Users podcast. Here’s part of my conversation with Mike.
Mike Green: And you mentioned the world of work and how it’s changed. And the one thing that we haven’t touched on so far is obviously the pandemic and the COVID years, if I can call them that. I’m interested to get your sense of how that impacted user research as a discipline. So speaking for myself, obviously the work has continued and it’s continued at pace. But I can’t remember the last time I sat in somebody’s office or place of work or even their home and actually interviewed them face to face. Which, you know, some ways it speeds up research. You can get more done remotely. People are perhaps more relaxed if they’re sitting in their own homes on Zoom. But there’s a loss, I think. As a researcher, I find not being in the context of the individuals surrounded by what’s on their walls and what’s around them and the kind of movement of the environment. It’s harder in some ways to get the insights. But I’m interested to know kind of what’s your perspective on how the pandemic changed for good or ill, kind of what we do.
Steve: I mean, 100 percent to everything that you just said about loss. I mean, that’s the word that I use. I mean, I don’t know that it’s permanent. I think the world of work is continuing to change as we’re sitting here on this day. It’s the beginning of the year where we’re talking. I haven’t seen 800 RTO articles, return to office articles. But it seems like, you know, there’s a constant discussion about that. And it’s interesting because like for sure the pandemic changed work. But it also triggered lots of bigger and more uncomfortable sort of discussions about power like bosses and property owners that, you know, have a stake in how work takes place and where it takes place. And worker power kind of pushing back on that. And depending on where you live and what industry you’re in, you’re going to see that more or less. So like I’m saying that remote research is being affected by these much larger shifts that I don’t have any sort of brilliance on. But I think the work continues to be in the middle of.
So I have not sat with someone in their place of doing whatever it is that they’re doing and interviewed them. And I just said at the beginning of our conversation, embrace how other people see the world. Well, that’s the way to do it, right? You let go of your thing and go to their thing. And it is harder. And it’s harder for me, they’re clients. But for other people, it’s their colleagues. It’s harder for us as researchers to facilitate that, oh, kind of reaction that we’re going for. We want people to know that their assumptions are wrong. And you can get these really jaw on the floor moments that we work to facilitate. We work to create those, you know, uncover those narratives and have our teammates let go of their biases and their assumptions and their aspirations. And that’s hard to do without taking people out. It was not only what we got to do, which meant that we could connect with people. We could see stuff that we didn’t know we wanted to ask about. We could be uncomfortable. We could be forced as researchers. And then we could create, I think, effective experiences for other people to also make the work transformative. And that’s a big fancy word, but we’re all changed by doing this.
Yeah, I really miss doing that. You know, I have peers that are like, oh, someone today was talking about some overseas trip they were doing to do field work. Like, I don’t even have to go to some exotic environment, like different than my own. I just would like to sit in an office or, you know, walk around a firehouse or something like that. So I think these things are going to continue to change. But I think there’s two fronts is what I’m trying to say here, right? What do we experience in the field, but also what do we experience with our collaboration and facilitation of the people we work with? And I think this also happens after the field work. If everything that we do takes place in a remote workspace and not, you know, more often is asynchronously, we’re also having fewer of those.
I mean, I can think of just times where I’ve had like clients and colleagues and we’re off site. We’re spending several days in a room going through this stuff and trying to make sense of it and just having like life changing insights come up. And that is so grandiose, my language. I mean, when someone comes up with something that riffs off of something that someone else says, and you can just sort of like feel a bunch of ideas come into alignment. Like it’s a really powerful intellectual, creative moment. And I haven’t had that for a while since I’ve been working where my participants are in a Zoom room and my colleagues are, you know, before and after the research. And so I don’t know, personally, I’ve struggled with the work feeling a little more transactional. And I think that is sort of coincident with other pressures on the work of research.
So I don’t know, I’m throwing everything into like a big, hairy, ugly ball of smooshed stuff together. And I think when you bring up like remote and pandemic, it like, oh yeah, there’s all these things that are kind of connected to that. And I don’t know how to tease them apart in a sensible way. I think I’m, you know, I’m being buffeted by those forces, I guess the way everybody else is. But yeah, I miss it. I think that’s my bottom line is exactly what you said. Like there’s a loss there. And I hope we can evolve to a point where it is a necessary part of what the researchers do, what the team does, and to kind of have those experiences, which are so inspirational.
Again, that was me on Mike Green’s Understanding Users podcast. Check out the whole episode, and of course, pick up a copy or two of the second edition of Interviewing Users. To learn about my consulting work and the training that I offer to companies, visit portigal.com/services.
Now, let’s go to my recent conversation with Gregg. He’s the Director of User Research at Hearst Magazines.
Gregg: This is Gregg Bernstein, and you’re listening to Dollars to Donuts.
Steve: Could we get a two Jews talk about research? You think you would do that?
Gregg: Again, me from nine years ago, not the sharpest tool in the shed when it came to naming things. But sure, you’re listening to two middle-aged Jews talk about research. Everyone’s favorite podcast. [laughter]
Steve: All right, well, what a way to begin. Thank you for that. When we talked nine years ago, you were working at Mailchimp, and I think you were the maybe second or third person I interviewed for this podcast, but you were the first episode that was published. So it’s really cool to have you back and talk about what’s changed for you, what kind of things you’ve learned. So thank you. Do you want to talk maybe about some of the different places that you’ve worked and maybe compare and contrast what work was like and what you’ve kind of seen in that intervening time?
Gregg: Yeah, first of all, Steve, it’s a pleasure to be back on your podcast.
Steve: Great. Thank you.
Gregg: So when you and I first spoke nine years ago, I was the research manager at Mailchimp. And it was my first time as a research manager.
Gregg: And I also, I don’t think I realized at the time just how unique the Mailchimp situation was for a researcher. And what I mean by that is, I had an almost unlimited budget to hire videographers to film our customers and make short films. We would create these artifacts of personas that we would hang up around our office, so everybody would learn. We had a CEO who was a designer before he was CEO, who understood the value of designing for people and knowing who those people are. So he supported research. He wanted us to make the best designs, which meant knowing our customers. And I was spoiled rotten. And I realized in subsequent jobs that that was not how most research roles are.
And I think when I left Mailchimp, I joined Vox Media. And we went from being really precious about the deliverable of the research to being scrappier at Vox. And I don’t mean that we were precious. It’s not that we weren’t precious about how we did research in either organization. We were thorough. We made sure we spoke to the right people. We asked good questions. We did solid research. But I think the difference was we wouldn’t — at Vox, I spent much less time on a project. If maybe I spent a month on a project at Mailchimp, I would spend a week on it because we had a very long list of projects that needed research. We had a pretty — not aggressive, but we had a quick-paced cadence of work. And so I learned very quickly that I had to work faster. I didn’t have to spend as much time creating these amazing artifacts as long as I was answering the fundamental questions and putting them in Slack or even a very poorly formatted Google Doc. As long as people were learning from the research, that was great. That was the gold standard. Did we learn from this? Did we make good decisions from it? If yes, move on. So that was a huge change in how I thought about research.
And it also made me, I think, a better research manager or leader because I realized budget is not commensurate to quality. You can do amazing research fast, scrappy, on a budget. You don’t need those unlimited resources. And that’s not to say I would love a bucket of money to be at my disposal. If I had to choose, I would take the high budget all the time. But I had to quickly learn how to get by with less, less time, less money. And you know what? It was a great experience, great learning opportunity. And something that I feel like made me a better researcher.
Steve: What are the circumstances in which putting that effort into the deliverable is, I don’t know, necessary or appropriate? I think you’re listing the times when it’s not. There’s a big demand and people are willing to kind of consume it in the form that it comes and act on it.
Gregg: Yeah, that’s a great distinction you’re making. At Mailchimp, I think it was necessary to put so much effort into the presentation of materials because it was a young company that was growing fast.
And so, yes, we wanted people to learn from the research. But we also wanted people to understand who our users are. So if you’re an engineering manager, if you work in accounting, you still need to know who we’re serving every day. Like what is the reason we’re coming to work? And I think that that knowledge was maybe not distributed evenly. And so putting films together, creating posters, and making everything so public and investing in it sent a signal. Like you need to know who we are working for. Your job depends on it, directly or indirectly. And I think for that time in the company’s history, it was absolutely the right call. And just like at Vox when I joined, moving fast and just banging out study after study and saying this is what we need to know, okay, this is what we need to know. And saying, okay, now we know this. Let’s build a thing and move on. That was the right approach for where Vox was when I was working there.
Steve: So that’s a little about Vox and kind of the change in culture and already a big impact in your approach that was suited to how you all worked and the people you needed to have impact. What was the next sort of major role where you, maybe your practice evolved yet again?
Gregg: I think I’m going to stay in Vox because I feel like my time there — I spent four years there. And my first two years I was working on a team that was creating tools for all of our writers and editors. It was a content management system. And that was very similar to the work I was doing at Mailchimp, which was software for creating content and publishing it. At Mailchimp it was newsletters. At Vox it was content, news content. Or food content if it was for Eater. Or tech content for The Verge, to name a few of the brands we worked on.
But at the heart of it, it was how do I understand the editorial process? And how can we make a better set of tools for publishing content, whether it’s an article or a map or a video or a podcast? And that was the first time I had worked on an internal team. So recruiting was no longer difficult. I could just get in Slack and talk to anybody in the company and say, hey, I’d like to talk to you about how you write articles. You know, there was very little difficulty in setting up — in finding participants and setting that up. That was the first two years of my time there.
The second two years, my role — the mandate for my role changed from understanding how we create content to how do people discover and consume content? And, again, this was in a remote-based organization that was a little scrappier. So I really had to think about how do we build out a process of getting feedback from our hundreds of millions of visitors to our various websites? How do I work with not just my product organization but our editorial organization to understand what information would be valuable to them? How do I get support from executives to do this research to make sure that once it’s done, they will have an appetite for it and learn from it?
And so it was the first time I had to create, I guess, demand and awareness and opportunities where none existed. Because it already existed within my product organization to build the content management system. But as far as, like, doing research that would support discovery and consumption, it was research that ended up supporting marketing and sales. Because we could — if we knew more about our audiences and what they valued and what they came for, we could put ads on our pages that kind of aligned with who was coming to our sites.
And that’s not to say we didn’t have demographic data, but we didn’t really have an understanding of why is somebody coming to The Verge? What is the next action that they’re going to take after they come to The Verge? And how can we make a better experience for them? So if they’re researching headphones and they’re looking for product reviews, if we know that that’s what they’re coming for and we know that they spend a certain amount of money after the fact, we can sell ads against that. And we have a better understanding of, okay, people are — they trust us for our ads — I’m sorry, they trust us for our product reviews. We should probably think about a better product review experience. None of this really was, I guess, designed. We didn’t have a designed research process. And so for the first time, I was having to chart a new path with the support of my manager and my colleagues, but I kind of had to figure out how to make this happen within a large organization and figure out which people I needed to talk to, who I needed to get support from, who I needed buy-in from.
And it was a fantastic learning experience because there was friction. Not a lot of friction, but like I had to convince some people of why we were doing this. I had to figure out if somebody was resistant, how can I get them to support this? And then how can I ask questions that will lead to insights that don’t just benefit me, but other parts of the organization? And so I feel like that was the moment when I really understood how to — I don’t want to say lead a research function, but how to get support and buy-in for research activities where maybe that didn’t exist before. And that’s what set me up for future research leadership opportunities. I feel like that’s when the training wheels came off and I understood the bigger job of being a research leader.
Steve: Support sometimes comes out as people being blocked from doing research. And so I think you’re talking about you had your own team, your own manager, your own team that’s doing your research, but you’re trying to make connections and help people see and engage so that the work that you do is valuable and they’re gonna act on it. Am I getting it right?
Gregg: Yeah, you’re getting it totally right. So one example is on the website Eater, which is about restaurants and food culture, there is something called a map — well, it is a map, but there’s a product name for it, which is escaping me now. But you might have like the 20 hottest restaurants in New York City. Or, you know, the 10 best restaurants that you should go to in Minnesota or Minneapolis, to be more specific. And a project might be, okay, let’s make the process of building maps better. But at the same time, let’s look externally to how do people actually use these maps to understand how can we improve the user experience. So we’re trying to make a better editorial experience as well as a better user experience.
So part of that is understanding, well, why does somebody use one of these maps in the first place? What is their goal? And to do that, we might need to get support from the editorial staff at Eater, which means working with their editor-in-chief or, you know, one of the editorial directors and saying, hey, we want to put a banner on Eater that says, help us improve Eater for everyone. We need to get their buy-in so they’re not going to their website and wondering why is there a banner on the top of my page. So just taking people away from the articles that we’re publishing and pushing them to a survey or a screener to participate in an interview or usability study. So we need to get their support.
But to do that, we also need to offer some sort of carrot. Like we want to talk to people about their Eater maps experience. But while we’re talking to them, is there anything that you’re curious about? If you had an Eater reader sitting next to you, what would be on your mind? So I’m trying to throw in questions that will help my editorial colleagues, but I’m also focusing on what I need to know to improve the map experience for my team. So that’s where I need to get their buy-in and their support. And it means clearly explaining what we’re trying to do, but also saying this is also an opportunity for you to learn about your audience. And this way I’ve got their buy-in. There’s no surprises when they see some sort of banner or call to action to participate in research. And they know that they’re going to learn something. And at the same time, our product organization is going to know something and learn something. Did that make sense?
Steve: That’s a great clarification. Do you have any examples of overcoming hesitancy or uncertainty in those folks that you were needing their support from?
Gregg: The hesitancy is usually just around, let me understand what this is going to look like. So showing an example of this is what a banner might look like on your website for a mobile user or a desktop user. So that there’s not — there’s this fear that maybe there’s going to be a screen takeover. And it’ll say like, don’t read this article, click here and take a survey, which we’re not trying to create a bad user experience. So it’s, I guess, demystifying the research process and showing, hey, this is what we’re going for. This is the goal of the study. This is how we’re actually — this is what it’s actually going to look like on your website. And we’ll work with you on the language we use, you know, help us improve Eater or make Maps better for everyone.
Thinking further, like sometimes we would do these big studies of our audiences where we would need the editor-in-chief of one of our sites to write a call to action. Like, hi, I’m Nilay Patel, I’m the editor-in-chief of The Verge. We’re doing an annual survey to help us improve our site, not just the coverage that we’re writing, but also the user experience of visiting our website. And we need your help. So if you read our content or listen to our podcasts, help us out. So it’s always a matter of over-explaining and saying this is exactly what we’re going for. This is an opportunity for you to learn as well. Let’s work together. And this way we’re all going to learn something. And worst case, we get a bunch of responses that maybe they’re not exactly what we wanted to hear, but we’re still going to learn from real humans who read our content or listen to our content or watch our content. And we’ll be able to learn from it.
Steve: When you started off describing these last two years, the train of wheels came off, and you ended and I didn’t really pick up on it. And I went back to the earlier stuff, but you ended with saying something to the effect that this was really where you learned about design research leadership. Does that take us into the next role?
Gregg: I think it does because I joined Condé Nast as their research lead. Condé Nast is another publishing company. And the job I interviewed for was to be research lead for just their subscription brands, which are brands like The New Yorker, Bon Appetit, Wired. But shortly after I joined, in talking to my boss, the vice president of product design, we realized that I was the highest ranking researcher. And so if we were to have a holistic research process for the entire product design organization, we couldn’t just focus on subscriptions and subscription products. We needed to have user research across the board, across all of our brands and divisions. And I was able to articulate that this is what the role should be. You need to have somebody who is looking at subscription products, but the other parts of the company, like commerce, which means selling products through product reviews, which is something that Vogue does. Or some of the other fashion magazines where you’re not just selling a subscription to a magazine. The magazine makes money by reviewing products and saying, like, here are the 10 best bags to wear or backpacks or high heels or computers. The company makes money through that type of content.
And so going back to what I was saying, I was able to articulate that we should have research in subscriptions, but also in commerce. But also having one research leader in charge of all research means that we can instill quality control. We can make sure that the researchers are collaborating so that a researcher who’s looking at commerce and a researcher looking at subscriptions, they’re not working in a vacuum. We’re not a siloed organization. We’re one research team that can collaborate or maybe move people around as needed based on what are the most important questions of the day. So I was able to see how getting buy in, putting processes in place, managing a team, how I could take what I had done at Vox and apply it to Condé Nast and kind of create a larger role for me at Condé Nast that was really necessary to make sure that the research was holistic and that the researchers were collaborating and that insights from one part of the organization were making it to other parts.
Steve: You’re describing this point at which you go from having pockets of research, for example, to building a role that’s a leadership role where there’s a person responsible for taking care of and ensuring all those kind of qualities of research that you described. And that sounds like a point of evolution, a point of transition in the overall organization’s research maturity.
Gregg: It was because it was a point where I was able to work with my manager to look at the entire organization, all the content that we publish, and see where the gaps were in our knowledge. So I had a researcher who was embedded with The New Yorker. I had a researcher embedded with Vogue. That left something like 25 other magazines that there wasn’t research, at least not user research. And so I was able to make the case that we really should hire somebody to look at all commerce. How do we sell products? What would make for a better commerce experience for our users? I was able to make the case that we should have somebody who is looking at our subscription brands like Wired and Bon Appetit. And then make the case that we should have somebody who’s just looking at the member journey because when you have so many different titles and so many different ways people are selling content, it takes some effort to know what’s going to resonate with somebody who they’re either thinking of buying content for themselves or to buy a gift for somebody else. Do they want a digital subscription? Do they want to actually receive something in the mail? So working with my manager and with the other design leaders, it became clear exactly where we needed to have resources in order to make sure we’re learning and supporting the designers and the product managers and the engineers to build the right product for the right people. So it was an inflection point. And it was personally great because I got to hire some really awesome researchers to fill those roles.
Steve: What are some of the ingredients or elements that you are uncovering and articulating when you are making the case for those kinds of structural changes or role changes? What does that include?
Gregg: I mean, first there’s pointing out that maybe we have a number of designers and engineers and product people working on a product with little to no contact with the humans who use that product. So just pointing that out and saying there’s an imbalance here in staffing. Or pointing out that a lot of designers and product managers are asking for research but not getting it because there isn’t the headcount or enough hours in the day to support those efforts. So those are usually the two places to start.
There’s a demand or there’s an imbalance and a vacuum of user contact. I also have used interns as a way to gauge demand for research. So if I bring in a summer intern and I put them on a project with a team and then the intern goes away, the team will suddenly realize that void in their life where a user researcher used to be. So then you can make the case, hey, this team got used to working with a researcher. It’s really not ideal for them to go back to trying to do research on their own. If we were to open headcount, this is where researchers should sit as a backfill for the intern that we lost. So that’s something I’ve done that at Mailchimp, I’ve done it at Vox. It’s a good tactic to test the waters and build demand for hiring a permanent researcher.
Steve: Yeah, that’s kind of brilliant. It’s almost like a prototyping process.
Gregg: I’ve also seen it where the intern did a great job, but maybe after they left, there wasn’t as much demand as we might have guessed. And while I always love to make the case that I want to hire more people, sometimes that proves that maybe that wasn’t the right place to hire. So it is like a prototyping process.
Steve: I’m curious if you have any perspective on Condé Nast culture in terms of how work was being done, about how you were engaging with different stakeholders or anything about research that is a compare and contrast with the first two companies we talked about.
Gregg: I think what I can say about Condé Nast is it was the largest company I had worked for at that point in my career. So culture, I realized, is not set for an organization. Culture is maybe at the team level. So that was a, I don’t want to say a shock, but it was very different where you realize that other teams have very different ways of working, of communicating, of supporting each other. And so I feel like I was able to instill a really strong culture for my research team. I feel like, you know, among my design manager peers, we had a really nice relationship, but I would not want to generalize the culture based on just the people I was working with. It was large and, you know, your mileage might vary depending on who you spoke to on any given day. I’m trying to be diplomatic, Steve.
Steve: I like hearing you how you’re unpacking it because, yeah, culture is this big label we kind of stamp on things. This organization is this culture, these type of people have this culture. But it is more local than global.
Gregg: I would say that I was able to create this very supportive, warm, amazing culture. And I mean, partly it was we would get together, you know, every three to six months. So you would get to have human contact, you know, real life contact with people. But somehow I don’t even, if I could replicate it, I would. Even remotely, there was such a feeling of these folks have my back and I have theirs and I would do anything for these people. And that made its way into how we hired. Like, I don’t want to bring somebody in who is going to ruin the feeling of this organization. So let’s be really rigorous in how we hire. Not that we weren’t rigorous elsewhere, but it takes a special set of skills to communicate warmth and empathy remotely in Slack messaging, over a Zoom. And that’s something that I really cherished and something that I’ve been really mindful of ever since.
Thinking about culture is a good way to transition to where I am now. I joined Hearst Magazines, yet another publishing company, in January of 2023. So I’ve been there for a year and two months. And what stuck out immediately is the warmth of every single person I’ve spoken to or spoke to in the interview process. And since I’ve joined, it’s such a warm organization, which is a massive organization. Hearst is huge. It’s 130 some odd years old. But from our legal team to our president to our executive leadership, everyone is just, they seem to care. And that’s what stood out to me from the moment I started speaking to the people at Hearst to people I’m still meeting. I mean, it’s a giant organization. I’m still meeting new people a year and two months into this job.
But culturally, I feel it’s the closest I’ve felt to what I had at Vox, where I know that the team cares about each other. They care about the work. They’re invested in making a great employee experience, and they really want to make a great user experience. And when you can find people who care, not just about the work, but the people they’re doing the work with, it’s special. And I have a great set of colleagues, and I just, I feel like I’m in a great spot, which means you know that in like two months, people listen to this podcast and realize we have layoffs or something, and I’m no longer there now that I’ve jinxed it. I’m kidding. I’m kidding. It’s a great place, but it is weird because it is such a huge organization with so many tentacles, and I’m not quite sure how the company has managed to achieve it, but it’s a pretty special place.
Steve: As you talk about culture, I hear this attribute of, my words, not yours, like welcoming to humans. And I think a theme of this podcast and part of this conversation is culture that is welcoming to research. And I like what you’re kind of getting at, that people care about each other and they care about the product and the experience that they’re making. I’m paraphrasing you badly here.
Gregg: I think that comes from company leadership, and I realize this isn’t going to be the same across the board, but having a president of the company who says, “I really want us to know our users.” And to have the highest ranking executive in the company say that, it gets buy-in, and it makes everyone realize this is important, and it just makes embracing research that much easier to achieve.
And so that’s the reason my job opened at Hearst was our president saying we need to know our users and the company investing in a user research function. And it also means that everybody I work with is curious to know how to incorporate user research into their processes. And so for the last year, process is what I have been focusing on because we have the mandate, we have the buy-in, okay, now we need to put the pieces in place. And for me, the pressure is on to deliver because it’s different than other jobs where research had existed in Condé Nast. Research was something that we were doing at Mailchimp. At Hearst, there wasn’t user research at scale when I joined, which meant we had the mandate, but we didn’t have the ability to do it or do it well.
So I’ve spent the last year in many meetings with our legal team just to put a process in place to get consent from people who visit our websites to engage them in research activities. I’ve been having a lot of meetings with our tech team on where PII will be stored, which of our products should we use to even do research that will be secure, where recordings will not end up in somebody’s hard drive that they shouldn’t end up in, or in a cloud service where maybe it’s not locked down to our preferences. So this has also been a learning experience for me because I have spent so much time doing operations work just to make research possible. And it’s also been a little bit stressful because everybody wants research and I’m constantly having to say, let’s hold up because we don’t have all the pieces in place yet. We can’t put an intercept on our website. We can’t email a user because we shouldn’t have their PII in our individual Outlook or Gmail accounts. We need to use the right tools to engage with them that is secure, where we’re not just going to be leaking email addresses and phone numbers in the wrong places. So let’s really get buttoned up and dial this in so that we are protecting our participants, but we’re also protecting the company. And we’re not putting the entire notion of user research at this company at risk because we’re making mistakes.
Steve: At what point did you, when you sort of started on this journey of yours to build this scale, did you do so with the expectation that operations was gonna be a key order of business for you?
Gregg: I did not. I also thought maybe this was me coming in with a little too much confidence. I thought that because I had created consent forms in the past, I could just email my legal team and say, hey, we’re going to start doing user research. I’m going to put an intercept or a call to action on our websites. Here’s the consent form I created in Google Forms. And immediately my legal team said, timeout, why don’t we talk through this? And it was a setback because I thought we were ready to go, my second week on the job. And it turned out that we were many, many, many months away from actually being able to do anything at all outside of maybe a platform where we’re not using our users. We used platforms like User Testing where we could research with a panel of random people. But as far as engaging with our known users, that took a lot of logistics. But it was also a great learning experience. And I have some amazing legal colleagues who were really helpful in pointing out ways that things could go wrong and working with me to come up with a process that we’re all happy with to some degree or, you know, to the most part.
Steve: Is there anything about your industry or the culture that even though you had this support and this collaboration, is there anything that might have led to the amount of the scale of the effort that you’re describing to get there?
Gregg: I don’t know if it’s media or just legacy enterprise, you know, historic organizations. Because Vox was a media company, but it was a new media company. It started in the digital age. There was never print magazines. And so it very much operated like a startup where, you know, if there was budget and I could get my manager’s approval, we would just buy a product with a credit card. Here at Hearst, that is not how things work. Like we’re not just going to click through an agreement and agree to some random SaaS company’s terms and, you know, suddenly we’re using their product. Everything has to be examined and negotiated and approved. So things move slowly. I think that might just be because it is such an organization — such an old organization that wants to be around for another 100, 200 years. So the mindset is let’s be slow but sure. You know, it’s better to take our time rather than get sued for a million dollars because we violated somebody’s privacy or we, you know, we used a product that we shouldn’t have been using. So I think that’s the whole idea of the community thing.
Steve: Well, I love hearing you describe slow in a way that is like deliberative and collaborative. I think, you know, there’s sort of an archetype of, oh, I want to get this thing done, but I couldn’t get, I couldn’t get anyone to help me or legal drag their heels that, but you’re, and maybe you’re being diplomatic, but I guess that the perspective I’m getting from you is that it’s not resistance to overcome. It’s the natural, you know, culturally appropriate way to do things, which is, which does take time. And another company might be faster, another company might be slower, but for different reasons, less, you know, more passive resistance. And here you’ve got slow, careful support, which is an interesting kind of way to have it be.
Gregg: Yeah. It’s never no, we’re not going to do that. It’s yes, we could do that, but let’s think through every step of this process to make sure that we’re not overlooking something fundamental. So this will sound maybe tedious to people listening to this podcast. I apologize in advance. But if you think about a generic news website, okay, you go to an article. Let’s say there’s a call to action. Like you’re looking at a recipe on your favorite cooking website. We want to improve our recipes. If you have three minutes to spare to answer three questions, click here to take a survey. Okay. What survey tool are we going to use that we have an enterprise agreement with where we know that all of the data that’s collected is collected in a way where we know it’s secure? Because we have a license that we negotiated where we know exactly where the data is stored and who can access it and who has liability if there is some type of data leak. Okay. So there’s the survey tool. Okay. Maybe we want people who took the survey to opt into a follow-up interview. So we can add that question. Are you interested in joining our recipe feedback panel? If so, click here. And, you know, it takes you to a page where you can add your name and your email address. Okay. Where is that going to be stored? Who’s going to have access to it?
And I realize, like, this is not — these are not new challenges. But my simple ask of we want to do a study led my legal team to work with me to say, okay, then what happens, then what happens, then what happens? Because in previous organizations, I would just be scrappy and say, yeah, they’ll pull out a Google form, it’ll go to a spreadsheet, and then I’ll email them and I’ll send them a link to my Calendly and they’ll schedule a time. And now it’s, no, we don’t have Calendly here. You can’t use that. So what else could we use? We don’t use Gmail or Google Calendar, but we do use this other product. What are other ways that we could create an inbox and create a link to a calendar? We don’t have an enterprise license with Zoom, but we have this other thing.
So it’s really just looking at the menu of possibilities and picking the least bad options. Ideally, the better options, it would create a better user experience. But making sure that from initial contact to when we promise to expunge data, no stone is left unturned and we can account for every step of that process and know exactly what’s happening. And again, I know other people do this all the time, but for me, it was a learning experience to go from the scrappy or the let’s just throw money at this way of doing it to, okay, we really need to be buttoned up because a lawsuit is the worst possible outcome here. And I don’t want that to happen. I don’t want the company to lose money. I don’t want to ruin a reputation. So let’s make sure that whatever we’re doing is rock solid and is durable so that once we put it in place, anybody can do it and everybody can do research going forward.
Steve: Do you have a sense in your year and two months, what’s the progress indicator for you about building these processes, building these kinds of operations and infrastructure?
Gregg: I won’t claim that we have it perfect yet because it still takes a while to get an intercept on our sites just because there’s a lot of people to go through and there’s some engineering lift. It’s not just flip a switch. So I would say that doing it is not easy, but there is now a process that we can follow to do that type of research. I think the better marker of success is I’ve been able to open headcount because even with the technical ability to do research, research is still not anyone’s primary responsibility except for me and my team. So product managers are managing product. I don’t always have time to do research nor give it their 100% of their brain. Same with product designers. But because there is such demand for research, I was able to open headcount. And I think that’s the real sign that we’re making progress. Everyone wants to make better decisions, and the company has put the money into hiring humans to help us make better decisions.
Steve: I love that. Let’s switch topics a little bit as you brought us, I think, up to date and even where you’ve been successful and where it’s taking you in this current organization. Let’s talk about your book. Research Practice, Perspectives from UX Researchers in a Changing Field.
Gregg: Yay.
Gregg: That is my book, Steve. Yes, I published this book in January of 2021, so we’re now three years out from it somehow. I suddenly had a lot of time on my hands during the first year of the pandemic to publish this. But this was a book that — maybe this — I’m assuming this happens to you. So what I found is I would write a blog post or I would give a talk, but the thing that people always wanted to ask me about was how do I get a job as a user researcher coming from academia, from being a psychologist, from being a marketer? How do I make myself attractive for a user research job? What do I need to do? I’m a team of one. I don’t know how to make the case that I shouldn’t be a team of one anymore. What do I do? I’m so lonely. I also don’t have any mentorship. Or I’m a new manager at Help. Nobody knows what to tell me on how to actually be a manager in a research team. How do I make the case for more headcount? How do I manage a team? How do I hire?
Those were the questions that came in constantly. And I had my stock answers that I would give. I had articles I would point people to. I would try to anticipate what people were going to ask and write blog posts about it. But the questions kept coming. So I thought what if I created a book that just talked about what a career in UX research might look like? And I realized very quickly that I am not the person to write the all-encompassing guide to a UX research career. Because at that point, I had been a designer who transitioned into UX research. I had worked at Mailchimp. And I had worked at Vox Media. That is a small sample size to talk about all the places a UX research career might go.
On the suggestion of a very smart friend of mine named Sian Townsend, she suggested why not ask other people to contribute their own perspectives and open source this? Which was such a great idea. So the book that I had started to write about what a career in UX research might look like became a collaborative effort to get multiple perspectives on a UX research career journey. From getting the job to the challenges of the job to where you might go next. And it was a fun project to work on. I wrote my own essays. I solicited essays from many other research leaders. I worked with an amazing editor named Nicole Fenton. Nicole edited Abby Covert’s book on information architecture, which is one of my favorite books on product and research and just thinking about information. So I sought out Nicole. Nicole helped make this book fantastic, in my opinion. And it was a really good exercise in publishing content, project management, and working with a host of other research leaders to create something that would be a good and evergreen artifact for the research community.
Steve: When you’re in that role and you get these different perspectives, are there situations where you don’t agree with the guidance that’s coming in one of these essays?
Gregg: No, it might not be what I would do personally, but I also, I mean, I wasn’t trained as a researcher, first of all. So if somebody is going to talk about a rigorous quantitative research approach, then who am I to say that’s not the way I would do it? And that was the whole point of the book was you’re going to get conflicting opinions. You’re going to get different perspectives. And I guess maybe the biggest takeaway is there’s no one way to do UX research, which is also the name of a blog post I wrote last summer, because we were also seeing a lot of comments on the state of the research industry. But the more I talked to research leaders and from talking to them about this book, there really isn’t one way to do research, nor one path for UX researchers. So I wanted this book to have differing opinions and perspectives, whether I agreed with all of them or not.
Steve: You talk about evergreen, and yet Changing Field is in the title. So what does the book look like to you now, kind of three years after it’s out? Maybe that’s what you’re kind of getting at, you’re having these conversations with people later than when you wrote the book about what the world is now.
Gregg: And I think, you know, when I think about how I would tackle the book today or what I think would be different or what I’m hearing from other researchers I speak to, there is such a drive to prove the value of research and make sure that it’s worth the economic investment in hiring a research team or a research person. And I don’t love that mindset that we have to always be proving our worth. But I think that’s a theme that comes up in the conversations I have, and it’s something I would expect to — there would be more content about that if I were to publish the book again. I also think that the financial realities of today mean that people are working leaner. They don’t have as big a research budget as maybe they once had. Teams — we all saw there’s been layoffs over the last two years. Teams are smaller. They’re sacrificing headcount or being forced to sacrifice headcount. So I think teams are also seeing that they have to get by with less, less people, less tools. So I think we have more constraints and more expectations to make the investment worth it.
Steve: I mean, you made the point that the book is lots of people’s perspectives, but you are continuing to share your own expertise and guidance in, I see you on all the online things with blog posts and videos and so on. What are you focusing on in the questions that you’re trying to answer yourself?
Gregg: I think the thing that I always come back to is that there is no one way to do research. And I also think there’s no one way to do research leadership. So often when I post a video or write something, it’s a knee-jerk reaction to something somebody else might have said that I feel like is going to discourage folks or paint this industry in a negative light. I don’t know if that’s the right way to phrase it. But I often want — I don’t want to sound like a Pollyanna, but I love this field. I think it’s invaluable. I think more companies should have a research function. And so anything that I write is usually meant to show that there’s opportunity, there is value in this work, and make sure that the folks who are curious about UX research maybe aren’t being sold a jaded or maybe geographically focused perspective. I think I just want to provide balance. Whether that’s coming through or not, I’m not sure, but that’s usually what prompts me is I see something and I think I don’t know if that quite captures it. I don’t know if that’s the whole story. I wonder if I have something I can say to add a different perspective.
Steve: So in the notes to this episode, we can point people to the book, but where are you writing or creating other kinds of information and guidance for people?
Gregg: If you go to my website, gregg.io, that is my blog, which I was on a roll last year when I was, it was funny, like as I was doing all the heavy lifting of putting processes in place, I was so motivated to create content and share what I was doing. So I had this streak last year of a lot of blog posts. I’ve kind of tailed off as I’ve gotten busier. But you can go to my website and I post content there. There’s also a newsletter you can sign up for, which just takes the blog posts and sends them to your inbox. I also post on LinkedIn from time to time. Those are pretty much the main places right now. I’ve created a number of videos for the Learners app, but that too has kind of tailed off as my day job has demanded more of my time.
Steve: And do you think another book is in your future?
Gregg: I don’t know if a book is in my future, but I do think that there is an update of sorts that should happen. Like I said, there’s been layoffs. People are tightening their belts and spending less on research. So I’d be curious to talk to research leaders. Although you’re already doing that, so maybe I’ll just feed you some questions to ask people. But I do think there should be an update of sorts. I just don’t know if the book is the right vehicle for that.
Steve: So you want to be like a Daily Show correspondent, right?
Gregg: I would take that job in addition to my current job.
Steve: And by Daily Show correspondent, I meant for this podcast.
Gregg: Exactly. Just put me on spot assignments and let me help, Steve. Not that you need it.
Steve: Okay. All right, all right. The syndicated media network that is giving birth to right here, everybody. This is the moment. And Gregg, what shall we brand this larger effort?
Gregg: I’ve already been thinking about the larger extended universe. So the book is called Research Practice. My newsletter is Research Practicing. So maybe it’s Research Perfecting. Maybe it’s Research Repractice. Again, I’m terrible with names. So let’s not tie me to any of these terrible names I just threw out.
Steve: All right, well, Gregg is giggling politely, and so that might be the sign that we’re kind of coming to the end of our conversation. Any last thoughts for this time together, Gregg, or anything to kind of throw in there?
Gregg: No, Steve. I just want to say, you had me as a guest nine years ago, which at the time I thought it doesn’t get better than this. And I’m fortunate to have developed a relationship with you where you provided so much good advice and an astounding board. And so to come back nine years later and do another episode with you, I’m thankful and I’m excited. So thank you for having me.
Steve: Thank you very much for taking the time. It was really great to get your perspective, and I think people are going to learn a lot from hearing you today. So thank you.
Gregg: Thanks Steve
Steve: Yes, there we go. Thanks for listening. Tell your friends, tell your enemies about Dollars to Donuts. Give us a review on Apple Podcasts or any place that reviews podcasts. Find Dollars to Donuts in all the places that have all the things. Or visit portugal.com/podcast for all of the episodes, complete with show notes and transcripts. Our theme music is by Bruce Todd.
The post 40. Gregg Bernstein returns first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I welcome Gregg Bernstein back for a follow-up episode. He’s now Director of User Research at Hearst Magazines.
The thing that I always come back to is that there is no one way to do research. And I also think there’s no one way to do research leadership. So often when I post a video or write something, it’s a knee-jerk reaction to something somebody else might have said that I feel like is going to discourage folks or paint this industry in a negative light. I don’t want to sound like a Pollyanna, but I love this field. I think it’s invaluable. I think more companies should have a research function. And so anything that I write is usually meant to show that there’s opportunity, there is value in this work. – Gregg Bernstein
Show Links* Interviewing Users, second edition * Steve Portigal on the Understanding Users podcast * Gregg Bernstein on Dollars to Donuts, 2015 * Gregg on LinkedIn * Mailchimp * Vox Media * Condé Nast * Hearst Magazines * Research Practice: Perspectives from UX researchers in a changing field * Sian Townsend on LinkedIn * Nicole Fenton * How to Make Sense of Any Mess by Abby Covert * Gregg’s site
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Today, I’m chatting with Gregg Bernstein, nine years after he first appeared on episode one of Dollars to Donuts. For context, here’s a tiny clip from that episode.
Gregg Bernstein: And I’m a little disappointed that you didn’t start this interview off by saying this is two Jews talking about customer research.
Steve: But before that, did you know that there’s a new edition of my classic book interviewing users? The modern day book tour seems to be in fact the podcast tour. And so recently I chatted with Mike Green for his Understanding Users podcast. Here’s part of my conversation with Mike.
Mike Green: And you mentioned the world of work and how it’s changed. And the one thing that we haven’t touched on so far is obviously the pandemic and the COVID years, if I can call them that. I’m interested to get your sense of how that impacted user research as a discipline. So speaking for myself, obviously the work has continued and it’s continued at pace. But I can’t remember the last time I sat in somebody’s office or place of work or even their home and actually interviewed them face to face. Which, you know, some ways it speeds up research. You can get more done remotely. People are perhaps more relaxed if they’re sitting in their own homes on Zoom. But there’s a loss, I think. As a researcher, I find not being in the context of the individuals surrounded by what’s on their walls and what’s around them and the kind of movement of the environment. It’s harder in some ways to get the insights. But I’m interested to know kind of what’s your perspective on how the pandemic changed for good or ill, kind of what we do.
Steve: I mean, 100 percent to everything that you just said about loss. I mean, that’s the word that I use. I mean, I don’t know that it’s permanent. I think the world of work is continuing to change as we’re sitting here on this day. It’s the beginning of the year where we’re talking. I haven’t seen 800 RTO articles, return to office articles. But it seems like, you know, there’s a constant discussion about that. And it’s interesting because like for sure the pandemic changed work. But it also triggered lots of bigger and more uncomfortable sort of discussions about power like bosses and property owners that, you know, have a stake in how work takes place and where it takes place. And worker power kind of pushing back on that. And depending on where you live and what industry you’re in, you’re going to see that more or less. So like I’m saying that remote research is being affected by these much larger shifts that I don’t have any sort of brilliance on. But I think the work continues to be in the middle of.
So I have not sat with someone in their place of doing whatever it is that they’re doing and interviewed them. And I just said at the beginning of our conversation, embrace how other people see the world. Well, that’s the way to do it, right? You let go of your thing and go to their thing. And it is harder. And it’s harder for me, they’re clients. But for other people, it’s their colleagues. It’s harder for us as researchers to facilitate that, oh, kind of reaction that we’re going for. We want people to know that their assumptions are wrong. And you can get these really jaw on the floor moments that we work to facilitate. We work to create those, you know, uncover those narratives and have our teammates let go of their biases and their assumptions and their aspirations. And that’s hard to do without taking people out. It was not only what we got to do, which meant that we could connect with people. We could see stuff that we didn’t know we wanted to ask about. We could be uncomfortable. We could be forced as researchers. And then we could create, I think, effective experiences for other people to also make the work transformative. And that’s a big fancy word, but we’re all changed by doing this.
Yeah, I really miss doing that. You know, I have peers that are like, oh, someone today was talking about some overseas trip they were doing to do field work. Like, I don’t even have to go to some exotic environment, like different than my own. I just would like to sit in an office or, you know, walk around a firehouse or something like that. So I think these things are going to continue to change. But I think there’s two fronts is what I’m trying to say here, right? What do we experience in the field, but also what do we experience with our collaboration and facilitation of the people we work with? And I think this also happens after the field work. If everything that we do takes place in a remote workspace and not, you know, more often is asynchronously, we’re also having fewer of those.
I mean, I can think of just times where I’ve had like clients and colleagues and we’re off site. We’re spending several days in a room going through this stuff and trying to make sense of it and just having like life changing insights come up. And that is so grandiose, my language. I mean, when someone comes up with something that riffs off of something that someone else says, and you can just sort of like feel a bunch of ideas come into alignment. Like it’s a really powerful intellectual, creative moment. And I haven’t had that for a while since I’ve been working where my participants are in a Zoom room and my colleagues are, you know, before and after the research. And so I don’t know, personally, I’ve struggled with the work feeling a little more transactional. And I think that is sort of coincident with other pressures on the work of research.
So I don’t know, I’m throwing everything into like a big, hairy, ugly ball of smooshed stuff together. And I think when you bring up like remote and pandemic, it like, oh yeah, there’s all these things that are kind of connected to that. And I don’t know how to tease them apart in a sensible way. I think I’m, you know, I’m being buffeted by those forces, I guess the way everybody else is. But yeah, I miss it. I think that’s my bottom line is exactly what you said. Like there’s a loss there. And I hope we can evolve to a point where it is a necessary part of what the researchers do, what the team does, and to kind of have those experiences, which are so inspirational.
Again, that was me on Mike Green’s Understanding Users podcast. Check out the whole episode, and of course, pick up a copy or two of the second edition of Interviewing Users. To learn about my consulting work and the training that I offer to companies, visit portigal.com/services.
Now, let’s go to my recent conversation with Gregg. He’s the Director of User Research at Hearst Magazines.
Gregg: This is Gregg Bernstein, and you’re listening to Dollars to Donuts.
Steve: Could we get a two Jews talk about research? You think you would do that?
Gregg: Again, me from nine years ago, not the sharpest tool in the shed when it came to naming things. But sure, you’re listening to two middle-aged Jews talk about research. Everyone’s favorite podcast. [laughter]
Steve: All right, well, what a way to begin. Thank you for that. When we talked nine years ago, you were working at Mailchimp, and I think you were the maybe second or third person I interviewed for this podcast, but you were the first episode that was published. So it’s really cool to have you back and talk about what’s changed for you, what kind of things you’ve learned. So thank you. Do you want to talk maybe about some of the different places that you’ve worked and maybe compare and contrast what work was like and what you’ve kind of seen in that intervening time?
Gregg: Yeah, first of all, Steve, it’s a pleasure to be back on your podcast.
Steve: Great. Thank you.
Gregg: So when you and I first spoke nine years ago, I was the research manager at Mailchimp. And it was my first time as a research manager.
Gregg: And I also, I don’t think I realized at the time just how unique the Mailchimp situation was for a researcher. And what I mean by that is, I had an almost unlimited budget to hire videographers to film our customers and make short films. We would create these artifacts of personas that we would hang up around our office, so everybody would learn. We had a CEO who was a designer before he was CEO, who understood the value of designing for people and knowing who those people are. So he supported research. He wanted us to make the best designs, which meant knowing our customers. And I was spoiled rotten. And I realized in subsequent jobs that that was not how most research roles are.
And I think when I left Mailchimp, I joined Vox Media. And we went from being really precious about the deliverable of the research to being scrappier at Vox. And I don’t mean that we were precious. It’s not that we weren’t precious about how we did research in either organization. We were thorough. We made sure we spoke to the right people. We asked good questions. We did solid research. But I think the difference was we wouldn’t — at Vox, I spent much less time on a project. If maybe I spent a month on a project at Mailchimp, I would spend a week on it because we had a very long list of projects that needed research. We had a pretty — not aggressive, but we had a quick-paced cadence of work. And so I learned very quickly that I had to work faster. I didn’t have to spend as much time creating these amazing artifacts as long as I was answering the fundamental questions and putting them in Slack or even a very poorly formatted Google Doc. As long as people were learning from the research, that was great. That was the gold standard. Did we learn from this? Did we make good decisions from it? If yes, move on. So that was a huge change in how I thought about research.
And it also made me, I think, a better research manager or leader because I realized budget is not commensurate to quality. You can do amazing research fast, scrappy, on a budget. You don’t need those unlimited resources. And that’s not to say I would love a bucket of money to be at my disposal. If I had to choose, I would take the high budget all the time. But I had to quickly learn how to get by with less, less time, less money. And you know what? It was a great experience, great learning opportunity. And something that I feel like made me a better researcher.
Steve: What are the circumstances in which putting that effort into the deliverable is, I don’t know, necessary or appropriate? I think you’re listing the times when it’s not. There’s a big demand and people are willing to kind of consume it in the form that it comes and act on it.
Gregg: Yeah, that’s a great distinction you’re making. At Mailchimp, I think it was necessary to put so much effort into the presentation of materials because it was a young company that was growing fast.
And so, yes, we wanted people to learn from the research. But we also wanted people to understand who our users are. So if you’re an engineering manager, if you work in accounting, you still need to know who we’re serving every day. Like what is the reason we’re coming to work? And I think that that knowledge was maybe not distributed evenly. And so putting films together, creating posters, and making everything so public and investing in it sent a signal. Like you need to know who we are working for. Your job depends on it, directly or indirectly. And I think for that time in the company’s history, it was absolutely the right call. And just like at Vox when I joined, moving fast and just banging out study after study and saying this is what we need to know, okay, this is what we need to know. And saying, okay, now we know this. Let’s build a thing and move on. That was the right approach for where Vox was when I was working there.
Steve: So that’s a little about Vox and kind of the change in culture and already a big impact in your approach that was suited to how you all worked and the people you needed to have impact. What was the next sort of major role where you, maybe your practice evolved yet again?
Gregg: I think I’m going to stay in Vox because I feel like my time there — I spent four years there. And my first two years I was working on a team that was creating tools for all of our writers and editors. It was a content management system. And that was very similar to the work I was doing at Mailchimp, which was software for creating content and publishing it. At Mailchimp it was newsletters. At Vox it was content, news content. Or food content if it was for Eater. Or tech content for The Verge, to name a few of the brands we worked on.
But at the heart of it, it was how do I understand the editorial process? And how can we make a better set of tools for publishing content, whether it’s an article or a map or a video or a podcast? And that was the first time I had worked on an internal team. So recruiting was no longer difficult. I could just get in Slack and talk to anybody in the company and say, hey, I’d like to talk to you about how you write articles. You know, there was very little difficulty in setting up — in finding participants and setting that up. That was the first two years of my time there.
The second two years, my role — the mandate for my role changed from understanding how we create content to how do people discover and consume content? And, again, this was in a remote-based organization that was a little scrappier. So I really had to think about how do we build out a process of getting feedback from our hundreds of millions of visitors to our various websites? How do I work with not just my product organization but our editorial organization to understand what information would be valuable to them? How do I get support from executives to do this research to make sure that once it’s done, they will have an appetite for it and learn from it?
And so it was the first time I had to create, I guess, demand and awareness and opportunities where none existed. Because it already existed within my product organization to build the content management system. But as far as, like, doing research that would support discovery and consumption, it was research that ended up supporting marketing and sales. Because we could — if we knew more about our audiences and what they valued and what they came for, we could put ads on our pages that kind of aligned with who was coming to our sites.
And that’s not to say we didn’t have demographic data, but we didn’t really have an understanding of why is somebody coming to The Verge? What is the next action that they’re going to take after they come to The Verge? And how can we make a better experience for them? So if they’re researching headphones and they’re looking for product reviews, if we know that that’s what they’re coming for and we know that they spend a certain amount of money after the fact, we can sell ads against that. And we have a better understanding of, okay, people are — they trust us for our ads — I’m sorry, they trust us for our product reviews. We should probably think about a better product review experience. None of this really was, I guess, designed. We didn’t have a designed research process. And so for the first time, I was having to chart a new path with the support of my manager and my colleagues, but I kind of had to figure out how to make this happen within a large organization and figure out which people I needed to talk to, who I needed to get support from, who I needed buy-in from.
And it was a fantastic learning experience because there was friction. Not a lot of friction, but like I had to convince some people of why we were doing this. I had to figure out if somebody was resistant, how can I get them to support this? And then how can I ask questions that will lead to insights that don’t just benefit me, but other parts of the organization? And so I feel like that was the moment when I really understood how to — I don’t want to say lead a research function, but how to get support and buy-in for research activities where maybe that didn’t exist before. And that’s what set me up for future research leadership opportunities. I feel like that’s when the training wheels came off and I understood the bigger job of being a research leader.
Steve: Support sometimes comes out as people being blocked from doing research. And so I think you’re talking about you had your own team, your own manager, your own team that’s doing your research, but you’re trying to make connections and help people see and engage so that the work that you do is valuable and they’re gonna act on it. Am I getting it right?
Gregg: Yeah, you’re getting it totally right. So one example is on the website Eater, which is about restaurants and food culture, there is something called a map — well, it is a map, but there’s a product name for it, which is escaping me now. But you might have like the 20 hottest restaurants in New York City. Or, you know, the 10 best restaurants that you should go to in Minnesota or Minneapolis, to be more specific. And a project might be, okay, let’s make the process of building maps better. But at the same time, let’s look externally to how do people actually use these maps to understand how can we improve the user experience. So we’re trying to make a better editorial experience as well as a better user experience.
So part of that is understanding, well, why does somebody use one of these maps in the first place? What is their goal? And to do that, we might need to get support from the editorial staff at Eater, which means working with their editor-in-chief or, you know, one of the editorial directors and saying, hey, we want to put a banner on Eater that says, help us improve Eater for everyone. We need to get their buy-in so they’re not going to their website and wondering why is there a banner on the top of my page. So just taking people away from the articles that we’re publishing and pushing them to a survey or a screener to participate in an interview or usability study. So we need to get their support.
But to do that, we also need to offer some sort of carrot. Like we want to talk to people about their Eater maps experience. But while we’re talking to them, is there anything that you’re curious about? If you had an Eater reader sitting next to you, what would be on your mind? So I’m trying to throw in questions that will help my editorial colleagues, but I’m also focusing on what I need to know to improve the map experience for my team. So that’s where I need to get their buy-in and their support. And it means clearly explaining what we’re trying to do, but also saying this is also an opportunity for you to learn about your audience. And this way I’ve got their buy-in. There’s no surprises when they see some sort of banner or call to action to participate in research. And they know that they’re going to learn something. And at the same time, our product organization is going to know something and learn something. Did that make sense?
Steve: That’s a great clarification. Do you have any examples of overcoming hesitancy or uncertainty in those folks that you were needing their support from?
Gregg: The hesitancy is usually just around, let me understand what this is going to look like. So showing an example of this is what a banner might look like on your website for a mobile user or a desktop user. So that there’s not — there’s this fear that maybe there’s going to be a screen takeover. And it’ll say like, don’t read this article, click here and take a survey, which we’re not trying to create a bad user experience. So it’s, I guess, demystifying the research process and showing, hey, this is what we’re going for. This is the goal of the study. This is how we’re actually — this is what it’s actually going to look like on your website. And we’ll work with you on the language we use, you know, help us improve Eater or make Maps better for everyone.
Thinking further, like sometimes we would do these big studies of our audiences where we would need the editor-in-chief of one of our sites to write a call to action. Like, hi, I’m Nilay Patel, I’m the editor-in-chief of The Verge. We’re doing an annual survey to help us improve our site, not just the coverage that we’re writing, but also the user experience of visiting our website. And we need your help. So if you read our content or listen to our podcasts, help us out. So it’s always a matter of over-explaining and saying this is exactly what we’re going for. This is an opportunity for you to learn as well. Let’s work together. And this way we’re all going to learn something. And worst case, we get a bunch of responses that maybe they’re not exactly what we wanted to hear, but we’re still going to learn from real humans who read our content or listen to our content or watch our content. And we’ll be able to learn from it.
Steve: When you started off describing these last two years, the train of wheels came off, and you ended and I didn’t really pick up on it. And I went back to the earlier stuff, but you ended with saying something to the effect that this was really where you learned about design research leadership. Does that take us into the next role?
Gregg: I think it does because I joined Condé Nast as their research lead. Condé Nast is another publishing company. And the job I interviewed for was to be research lead for just their subscription brands, which are brands like The New Yorker, Bon Appetit, Wired. But shortly after I joined, in talking to my boss, the vice president of product design, we realized that I was the highest ranking researcher. And so if we were to have a holistic research process for the entire product design organization, we couldn’t just focus on subscriptions and subscription products. We needed to have user research across the board, across all of our brands and divisions. And I was able to articulate that this is what the role should be. You need to have somebody who is looking at subscription products, but the other parts of the company, like commerce, which means selling products through product reviews, which is something that Vogue does. Or some of the other fashion magazines where you’re not just selling a subscription to a magazine. The magazine makes money by reviewing products and saying, like, here are the 10 best bags to wear or backpacks or high heels or computers. The company makes money through that type of content.
And so going back to what I was saying, I was able to articulate that we should have research in subscriptions, but also in commerce. But also having one research leader in charge of all research means that we can instill quality control. We can make sure that the researchers are collaborating so that a researcher who’s looking at commerce and a researcher looking at subscriptions, they’re not working in a vacuum. We’re not a siloed organization. We’re one research team that can collaborate or maybe move people around as needed based on what are the most important questions of the day. So I was able to see how getting buy in, putting processes in place, managing a team, how I could take what I had done at Vox and apply it to Condé Nast and kind of create a larger role for me at Condé Nast that was really necessary to make sure that the research was holistic and that the researchers were collaborating and that insights from one part of the organization were making it to other parts.
Steve: You’re describing this point at which you go from having pockets of research, for example, to building a role that’s a leadership role where there’s a person responsible for taking care of and ensuring all those kind of qualities of research that you described. And that sounds like a point of evolution, a point of transition in the overall organization’s research maturity.
Gregg: It was because it was a point where I was able to work with my manager to look at the entire organization, all the content that we publish, and see where the gaps were in our knowledge. So I had a researcher who was embedded with The New Yorker. I had a researcher embedded with Vogue. That left something like 25 other magazines that there wasn’t research, at least not user research. And so I was able to make the case that we really should hire somebody to look at all commerce. How do we sell products? What would make for a better commerce experience for our users? I was able to make the case that we should have somebody who is looking at our subscription brands like Wired and Bon Appetit. And then make the case that we should have somebody who’s just looking at the member journey because when you have so many different titles and so many different ways people are selling content, it takes some effort to know what’s going to resonate with somebody who they’re either thinking of buying content for themselves or to buy a gift for somebody else. Do they want a digital subscription? Do they want to actually receive something in the mail? So working with my manager and with the other design leaders, it became clear exactly where we needed to have resources in order to make sure we’re learning and supporting the designers and the product managers and the engineers to build the right product for the right people. So it was an inflection point. And it was personally great because I got to hire some really awesome researchers to fill those roles.
Steve: What are some of the ingredients or elements that you are uncovering and articulating when you are making the case for those kinds of structural changes or role changes? What does that include?
Gregg: I mean, first there’s pointing out that maybe we have a number of designers and engineers and product people working on a product with little to no contact with the humans who use that product. So just pointing that out and saying there’s an imbalance here in staffing. Or pointing out that a lot of designers and product managers are asking for research but not getting it because there isn’t the headcount or enough hours in the day to support those efforts. So those are usually the two places to start.
There’s a demand or there’s an imbalance and a vacuum of user contact. I also have used interns as a way to gauge demand for research. So if I bring in a summer intern and I put them on a project with a team and then the intern goes away, the team will suddenly realize that void in their life where a user researcher used to be. So then you can make the case, hey, this team got used to working with a researcher. It’s really not ideal for them to go back to trying to do research on their own. If we were to open headcount, this is where researchers should sit as a backfill for the intern that we lost. So that’s something I’ve done that at Mailchimp, I’ve done it at Vox. It’s a good tactic to test the waters and build demand for hiring a permanent researcher.
Steve: Yeah, that’s kind of brilliant. It’s almost like a prototyping process.
Gregg: I’ve also seen it where the intern did a great job, but maybe after they left, there wasn’t as much demand as we might have guessed. And while I always love to make the case that I want to hire more people, sometimes that proves that maybe that wasn’t the right place to hire. So it is like a prototyping process.
Steve: I’m curious if you have any perspective on Condé Nast culture in terms of how work was being done, about how you were engaging with different stakeholders or anything about research that is a compare and contrast with the first two companies we talked about.
Gregg: I think what I can say about Condé Nast is it was the largest company I had worked for at that point in my career. So culture, I realized, is not set for an organization. Culture is maybe at the team level. So that was a, I don’t want to say a shock, but it was very different where you realize that other teams have very different ways of working, of communicating, of supporting each other. And so I feel like I was able to instill a really strong culture for my research team. I feel like, you know, among my design manager peers, we had a really nice relationship, but I would not want to generalize the culture based on just the people I was working with. It was large and, you know, your mileage might vary depending on who you spoke to on any given day. I’m trying to be diplomatic, Steve.
Steve: I like hearing you how you’re unpacking it because, yeah, culture is this big label we kind of stamp on things. This organization is this culture, these type of people have this culture. But it is more local than global.
Gregg: I would say that I was able to create this very supportive, warm, amazing culture. And I mean, partly it was we would get together, you know, every three to six months. So you would get to have human contact, you know, real life contact with people. But somehow I don’t even, if I could replicate it, I would. Even remotely, there was such a feeling of these folks have my back and I have theirs and I would do anything for these people. And that made its way into how we hired. Like, I don’t want to bring somebody in who is going to ruin the feeling of this organization. So let’s be really rigorous in how we hire. Not that we weren’t rigorous elsewhere, but it takes a special set of skills to communicate warmth and empathy remotely in Slack messaging, over a Zoom. And that’s something that I really cherished and something that I’ve been really mindful of ever since.
Thinking about culture is a good way to transition to where I am now. I joined Hearst Magazines, yet another publishing company, in January of 2023. So I’ve been there for a year and two months. And what stuck out immediately is the warmth of every single person I’ve spoken to or spoke to in the interview process. And since I’ve joined, it’s such a warm organization, which is a massive organization. Hearst is huge. It’s 130 some odd years old. But from our legal team to our president to our executive leadership, everyone is just, they seem to care. And that’s what stood out to me from the moment I started speaking to the people at Hearst to people I’m still meeting. I mean, it’s a giant organization. I’m still meeting new people a year and two months into this job.
But culturally, I feel it’s the closest I’ve felt to what I had at Vox, where I know that the team cares about each other. They care about the work. They’re invested in making a great employee experience, and they really want to make a great user experience. And when you can find people who care, not just about the work, but the people they’re doing the work with, it’s special. And I have a great set of colleagues, and I just, I feel like I’m in a great spot, which means you know that in like two months, people listen to this podcast and realize we have layoffs or something, and I’m no longer there now that I’ve jinxed it. I’m kidding. I’m kidding. It’s a great place, but it is weird because it is such a huge organization with so many tentacles, and I’m not quite sure how the company has managed to achieve it, but it’s a pretty special place.
Steve: As you talk about culture, I hear this attribute of, my words, not yours, like welcoming to humans. And I think a theme of this podcast and part of this conversation is culture that is welcoming to research. And I like what you’re kind of getting at, that people care about each other and they care about the product and the experience that they’re making. I’m paraphrasing you badly here.
Gregg: I think that comes from company leadership, and I realize this isn’t going to be the same across the board, but having a president of the company who says, “I really want us to know our users.” And to have the highest ranking executive in the company say that, it gets buy-in, and it makes everyone realize this is important, and it just makes embracing research that much easier to achieve.
And so that’s the reason my job opened at Hearst was our president saying we need to know our users and the company investing in a user research function. And it also means that everybody I work with is curious to know how to incorporate user research into their processes. And so for the last year, process is what I have been focusing on because we have the mandate, we have the buy-in, okay, now we need to put the pieces in place. And for me, the pressure is on to deliver because it’s different than other jobs where research had existed in Condé Nast. Research was something that we were doing at Mailchimp. At Hearst, there wasn’t user research at scale when I joined, which meant we had the mandate, but we didn’t have the ability to do it or do it well.
So I’ve spent the last year in many meetings with our legal team just to put a process in place to get consent from people who visit our websites to engage them in research activities. I’ve been having a lot of meetings with our tech team on where PII will be stored, which of our products should we use to even do research that will be secure, where recordings will not end up in somebody’s hard drive that they shouldn’t end up in, or in a cloud service where maybe it’s not locked down to our preferences. So this has also been a learning experience for me because I have spent so much time doing operations work just to make research possible. And it’s also been a little bit stressful because everybody wants research and I’m constantly having to say, let’s hold up because we don’t have all the pieces in place yet. We can’t put an intercept on our website. We can’t email a user because we shouldn’t have their PII in our individual Outlook or Gmail accounts. We need to use the right tools to engage with them that is secure, where we’re not just going to be leaking email addresses and phone numbers in the wrong places. So let’s really get buttoned up and dial this in so that we are protecting our participants, but we’re also protecting the company. And we’re not putting the entire notion of user research at this company at risk because we’re making mistakes.
Steve: At what point did you, when you sort of started on this journey of yours to build this scale, did you do so with the expectation that operations was gonna be a key order of business for you?
Gregg: I did not. I also thought maybe this was me coming in with a little too much confidence. I thought that because I had created consent forms in the past, I could just email my legal team and say, hey, we’re going to start doing user research. I’m going to put an intercept or a call to action on our websites. Here’s the consent form I created in Google Forms. And immediately my legal team said, timeout, why don’t we talk through this? And it was a setback because I thought we were ready to go, my second week on the job. And it turned out that we were many, many, many months away from actually being able to do anything at all outside of maybe a platform where we’re not using our users. We used platforms like User Testing where we could research with a panel of random people. But as far as engaging with our known users, that took a lot of logistics. But it was also a great learning experience. And I have some amazing legal colleagues who were really helpful in pointing out ways that things could go wrong and working with me to come up with a process that we’re all happy with to some degree or, you know, to the most part.
Steve: Is there anything about your industry or the culture that even though you had this support and this collaboration, is there anything that might have led to the amount of the scale of the effort that you’re describing to get there?
Gregg: I don’t know if it’s media or just legacy enterprise, you know, historic organizations. Because Vox was a media company, but it was a new media company. It started in the digital age. There was never print magazines. And so it very much operated like a startup where, you know, if there was budget and I could get my manager’s approval, we would just buy a product with a credit card. Here at Hearst, that is not how things work. Like we’re not just going to click through an agreement and agree to some random SaaS company’s terms and, you know, suddenly we’re using their product. Everything has to be examined and negotiated and approved. So things move slowly. I think that might just be because it is such an organization — such an old organization that wants to be around for another 100, 200 years. So the mindset is let’s be slow but sure. You know, it’s better to take our time rather than get sued for a million dollars because we violated somebody’s privacy or we, you know, we used a product that we shouldn’t have been using. So I think that’s the whole idea of the community thing.
Steve: Well, I love hearing you describe slow in a way that is like deliberative and collaborative. I think, you know, there’s sort of an archetype of, oh, I want to get this thing done, but I couldn’t get, I couldn’t get anyone to help me or legal drag their heels that, but you’re, and maybe you’re being diplomatic, but I guess that the perspective I’m getting from you is that it’s not resistance to overcome. It’s the natural, you know, culturally appropriate way to do things, which is, which does take time. And another company might be faster, another company might be slower, but for different reasons, less, you know, more passive resistance. And here you’ve got slow, careful support, which is an interesting kind of way to have it be.
Gregg: Yeah. It’s never no, we’re not going to do that. It’s yes, we could do that, but let’s think through every step of this process to make sure that we’re not overlooking something fundamental. So this will sound maybe tedious to people listening to this podcast. I apologize in advance. But if you think about a generic news website, okay, you go to an article. Let’s say there’s a call to action. Like you’re looking at a recipe on your favorite cooking website. We want to improve our recipes. If you have three minutes to spare to answer three questions, click here to take a survey. Okay. What survey tool are we going to use that we have an enterprise agreement with where we know that all of the data that’s collected is collected in a way where we know it’s secure? Because we have a license that we negotiated where we know exactly where the data is stored and who can access it and who has liability if there is some type of data leak. Okay. So there’s the survey tool. Okay. Maybe we want people who took the survey to opt into a follow-up interview. So we can add that question. Are you interested in joining our recipe feedback panel? If so, click here. And, you know, it takes you to a page where you can add your name and your email address. Okay. Where is that going to be stored? Who’s going to have access to it?
And I realize, like, this is not — these are not new challenges. But my simple ask of we want to do a study led my legal team to work with me to say, okay, then what happens, then what happens, then what happens? Because in previous organizations, I would just be scrappy and say, yeah, they’ll pull out a Google form, it’ll go to a spreadsheet, and then I’ll email them and I’ll send them a link to my Calendly and they’ll schedule a time. And now it’s, no, we don’t have Calendly here. You can’t use that. So what else could we use? We don’t use Gmail or Google Calendar, but we do use this other product. What are other ways that we could create an inbox and create a link to a calendar? We don’t have an enterprise license with Zoom, but we have this other thing.
So it’s really just looking at the menu of possibilities and picking the least bad options. Ideally, the better options, it would create a better user experience. But making sure that from initial contact to when we promise to expunge data, no stone is left unturned and we can account for every step of that process and know exactly what’s happening. And again, I know other people do this all the time, but for me, it was a learning experience to go from the scrappy or the let’s just throw money at this way of doing it to, okay, we really need to be buttoned up because a lawsuit is the worst possible outcome here. And I don’t want that to happen. I don’t want the company to lose money. I don’t want to ruin a reputation. So let’s make sure that whatever we’re doing is rock solid and is durable so that once we put it in place, anybody can do it and everybody can do research going forward.
Steve: Do you have a sense in your year and two months, what’s the progress indicator for you about building these processes, building these kinds of operations and infrastructure?
Gregg: I won’t claim that we have it perfect yet because it still takes a while to get an intercept on our sites just because there’s a lot of people to go through and there’s some engineering lift. It’s not just flip a switch. So I would say that doing it is not easy, but there is now a process that we can follow to do that type of research. I think the better marker of success is I’ve been able to open headcount because even with the technical ability to do research, research is still not anyone’s primary responsibility except for me and my team. So product managers are managing product. I don’t always have time to do research nor give it their 100% of their brain. Same with product designers. But because there is such demand for research, I was able to open headcount. And I think that’s the real sign that we’re making progress. Everyone wants to make better decisions, and the company has put the money into hiring humans to help us make better decisions.
Steve: I love that. Let’s switch topics a little bit as you brought us, I think, up to date and even where you’ve been successful and where it’s taking you in this current organization. Let’s talk about your book. Research Practice, Perspectives from UX Researchers in a Changing Field.
Gregg: Yay.
Gregg: That is my book, Steve. Yes, I published this book in January of 2021, so we’re now three years out from it somehow. I suddenly had a lot of time on my hands during the first year of the pandemic to publish this. But this was a book that — maybe this — I’m assuming this happens to you. So what I found is I would write a blog post or I would give a talk, but the thing that people always wanted to ask me about was how do I get a job as a user researcher coming from academia, from being a psychologist, from being a marketer? How do I make myself attractive for a user research job? What do I need to do? I’m a team of one. I don’t know how to make the case that I shouldn’t be a team of one anymore. What do I do? I’m so lonely. I also don’t have any mentorship. Or I’m a new manager at Help. Nobody knows what to tell me on how to actually be a manager in a research team. How do I make the case for more headcount? How do I manage a team? How do I hire?
Those were the questions that came in constantly. And I had my stock answers that I would give. I had articles I would point people to. I would try to anticipate what people were going to ask and write blog posts about it. But the questions kept coming. So I thought what if I created a book that just talked about what a career in UX research might look like? And I realized very quickly that I am not the person to write the all-encompassing guide to a UX research career. Because at that point, I had been a designer who transitioned into UX research. I had worked at Mailchimp. And I had worked at Vox Media. That is a small sample size to talk about all the places a UX research career might go.
On the suggestion of a very smart friend of mine named Sian Townsend, she suggested why not ask other people to contribute their own perspectives and open source this? Which was such a great idea. So the book that I had started to write about what a career in UX research might look like became a collaborative effort to get multiple perspectives on a UX research career journey. From getting the job to the challenges of the job to where you might go next. And it was a fun project to work on. I wrote my own essays. I solicited essays from many other research leaders. I worked with an amazing editor named Nicole Fenton. Nicole edited Abby Covert’s book on information architecture, which is one of my favorite books on product and research and just thinking about information. So I sought out Nicole. Nicole helped make this book fantastic, in my opinion. And it was a really good exercise in publishing content, project management, and working with a host of other research leaders to create something that would be a good and evergreen artifact for the research community.
Steve: When you’re in that role and you get these different perspectives, are there situations where you don’t agree with the guidance that’s coming in one of these essays?
Gregg: No, it might not be what I would do personally, but I also, I mean, I wasn’t trained as a researcher, first of all. So if somebody is going to talk about a rigorous quantitative research approach, then who am I to say that’s not the way I would do it? And that was the whole point of the book was you’re going to get conflicting opinions. You’re going to get different perspectives. And I guess maybe the biggest takeaway is there’s no one way to do UX research, which is also the name of a blog post I wrote last summer, because we were also seeing a lot of comments on the state of the research industry. But the more I talked to research leaders and from talking to them about this book, there really isn’t one way to do research, nor one path for UX researchers. So I wanted this book to have differing opinions and perspectives, whether I agreed with all of them or not.
Steve: You talk about evergreen, and yet Changing Field is in the title. So what does the book look like to you now, kind of three years after it’s out? Maybe that’s what you’re kind of getting at, you’re having these conversations with people later than when you wrote the book about what the world is now.
Gregg: And I think, you know, when I think about how I would tackle the book today or what I think would be different or what I’m hearing from other researchers I speak to, there is such a drive to prove the value of research and make sure that it’s worth the economic investment in hiring a research team or a research person. And I don’t love that mindset that we have to always be proving our worth. But I think that’s a theme that comes up in the conversations I have, and it’s something I would expect to — there would be more content about that if I were to publish the book again. I also think that the financial realities of today mean that people are working leaner. They don’t have as big a research budget as maybe they once had. Teams — we all saw there’s been layoffs over the last two years. Teams are smaller. They’re sacrificing headcount or being forced to sacrifice headcount. So I think teams are also seeing that they have to get by with less, less people, less tools. So I think we have more constraints and more expectations to make the investment worth it.
Steve: I mean, you made the point that the book is lots of people’s perspectives, but you are continuing to share your own expertise and guidance in, I see you on all the online things with blog posts and videos and so on. What are you focusing on in the questions that you’re trying to answer yourself?
Gregg: I think the thing that I always come back to is that there is no one way to do research. And I also think there’s no one way to do research leadership. So often when I post a video or write something, it’s a knee-jerk reaction to something somebody else might have said that I feel like is going to discourage folks or paint this industry in a negative light. I don’t know if that’s the right way to phrase it. But I often want — I don’t want to sound like a Pollyanna, but I love this field. I think it’s invaluable. I think more companies should have a research function. And so anything that I write is usually meant to show that there’s opportunity, there is value in this work, and make sure that the folks who are curious about UX research maybe aren’t being sold a jaded or maybe geographically focused perspective. I think I just want to provide balance. Whether that’s coming through or not, I’m not sure, but that’s usually what prompts me is I see something and I think I don’t know if that quite captures it. I don’t know if that’s the whole story. I wonder if I have something I can say to add a different perspective.
Steve: So in the notes to this episode, we can point people to the book, but where are you writing or creating other kinds of information and guidance for people?
Gregg: If you go to my website, gregg.io, that is my blog, which I was on a roll last year when I was, it was funny, like as I was doing all the heavy lifting of putting processes in place, I was so motivated to create content and share what I was doing. So I had this streak last year of a lot of blog posts. I’ve kind of tailed off as I’ve gotten busier. But you can go to my website and I post content there. There’s also a newsletter you can sign up for, which just takes the blog posts and sends them to your inbox. I also post on LinkedIn from time to time. Those are pretty much the main places right now. I’ve created a number of videos for the Learners app, but that too has kind of tailed off as my day job has demanded more of my time.
Steve: And do you think another book is in your future?
Gregg: I don’t know if a book is in my future, but I do think that there is an update of sorts that should happen. Like I said, there’s been layoffs. People are tightening their belts and spending less on research. So I’d be curious to talk to research leaders. Although you’re already doing that, so maybe I’ll just feed you some questions to ask people. But I do think there should be an update of sorts. I just don’t know if the book is the right vehicle for that.
Steve: So you want to be like a Daily Show correspondent, right?
Gregg: I would take that job in addition to my current job.
Steve: And by Daily Show correspondent, I meant for this podcast.
Gregg: Exactly. Just put me on spot assignments and let me help, Steve. Not that you need it.
Steve: Okay. All right, all right. The syndicated media network that is giving birth to right here, everybody. This is the moment. And Gregg, what shall we brand this larger effort?
Gregg: I’ve already been thinking about the larger extended universe. So the book is called Research Practice. My newsletter is Research Practicing. So maybe it’s Research Perfecting. Maybe it’s Research Repractice. Again, I’m terrible with names. So let’s not tie me to any of these terrible names I just threw out.
Steve: All right, well, Gregg is giggling politely, and so that might be the sign that we’re kind of coming to the end of our conversation. Any last thoughts for this time together, Gregg, or anything to kind of throw in there?
Gregg: No, Steve. I just want to say, you had me as a guest nine years ago, which at the time I thought it doesn’t get better than this. And I’m fortunate to have developed a relationship with you where you provided so much good advice and an astounding board. And so to come back nine years later and do another episode with you, I’m thankful and I’m excited. So thank you for having me.
Steve: Thank you very much for taking the time. It was really great to get your perspective, and I think people are going to learn a lot from hearing you today. So thank you.
Gregg: Thanks Steve
Steve: Yes, there we go. Thanks for listening. Tell your friends, tell your enemies about Dollars to Donuts. Give us a review on Apple Podcasts or any place that reviews podcasts. Find Dollars to Donuts in all the places that have all the things. Or visit portugal.com/podcast for all of the episodes, complete with show notes and transcripts. Our theme music is by Bruce Todd.
The post 40. Gregg Bernstein returns first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Mani Pande, Director and Head of Research at Cisco Meraki.
We used to do these immersion events where we would bring everybody who worked on, who was our stakeholder, to come and listen and talk to our customers. And we would do these focus groups, they were like a whole day event. There were folks from marketing and ops team who ran some of these focus groups. And when we got feedback about the immersion, it was very clear that everybody realized that when researchers are not doing the moderation, the kind of data that you get is not good. And the conversations were not that interesting. They didn’t feel that it was a good use of their time. So I think you can have your stakeholders experience it, that it’s not that easy to do moderation. – Mani Pande
Show Links* Interviewing Users, 2nd Edition * UI Breakfast Podcast. Episode 280: User Interviewing Techniques with Steve Portigal * Mani on LinkedIn * Cisco Meraki * the Double Diamond * CSAT * On-Prem * SuccessFactors * How To Influence Without Authority In The Workplace * structural equation modeling * latent class analysis * FigJam * Miro * Institute for the Future * Comscore
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. In case you don’t already know, I recently released a second edition of my classic book, Interviewing Users. This new edition is the product of 10 more years of me working as a researcher and teaching other people. It’s bigger and better. It’s got two new chapters, a lot of updated content, new examples, new guest essays, and more.
As part of releasing this new edition, I’ve been on a number of podcasts myself, including a conversation with Jane Portman that was part of her podcast, UI Breakfast. Here’s a quick excerpt from that conversation.
Jane Portman: As you’re training other researchers, you’re training experienced researchers, I’m thinking, what do you feel is common knowledge that they’re mastering well and that we’re all like good at and what things are surprisingly difficult?
Steve: You know, I think especially for people that are, they have a little bit of experience, but they’re kind of starting to blossom a little bit. One of the things they often need the most is confidence. And so people will often describe a scenario that they were in. People are messy and people are unpredictable and so all these things happen and so they go in with the best of intentions and plans and then things change a little bit. Somebody mentions their divorce, they don’t know what to do. And so I feel like my job isn’t to tell people that they’ve screwed up and that’s not how you do it. I think my job is to tell people that the thing that you encountered is very common. It’s a thing that a lot of researchers struggle with. I try to handle it this way, but there are situations when I handle it this way. Like I think I have a lot of like specific guidance and best practices, but all of those come with a lot of subjectivity and that it’s sort of the nature of the work to be a little confused or uncertain and to have to try things.
And by the way, there’s no right choice. When someone mentions their divorce, you know, not even me versus you, like me versus me. The next time I would do that interview, I would handle it differently. There’s a moment and if the divorce was brought up in minute three versus minute thirty, like it would play out differently. We’re not algorithms, we are, I think improvisation is a big part of it. So I think I want to help junior researchers feel okay about that there’s no one right way to handle this and that their way of, the fact that they felt confused and uncertain in a situation and they made this kind of, here’s how they addressed it, it’s, I rarely tell them like, well that’s the worst thing you could have done. It’s usually, they usually are doing their best. That confidence to make a different choice is kind of what I want to help somebody with.
You know, more experienced interviewers, I think I like, I like working with them because I think we can have a better, richer conversation about what are all these choices and what are the differences between them and you know, I love being in a workshop where I’ve got people with different experience levels because then I might give some guidance and then we can have different people suggest, well you know, here’s what I’ve done. Everyone can learn from each other and sometimes we can debate and I don’t mean that in like a right and wrong way but I think someone with experience, we can have a really interesting conversation where we look at one scenario and four or five different ways to handle it and we might disagree on what is sort of the optimal way.
I think what that surfaces is that as individuals, as interviewers, we’re all wired differently and we all have different instincts and different personalities and you can get away with things that I can’t because of my age or my gender or my energy and I can get away with things that you can’t and getting away is maybe there’s the wrong framing but there’s just so many interesting choices and it’s I love hearing about other people’s things and thinking like, oh could I have that amount of friendliness or that amount of stillness or that amount of curiosity or that amount of empathy, you know, could I present those things in different amounts? You know, that’s part of being an expert interviewer is you have your own personality and your own strengths and you can exercise different facets of that as the situation requires. I think, you know, a more experienced interviewer has more adaptability to that, has their core, you know, but also can put on different authentic, true to themselves faces and energies to kind of support different kinds of situations that are going to come up in these interviews.
Again, that was me speaking with Jane Portman on UI Breakfast. You can check out the whole episode and you should totally buy a copy of this new edition of Interviewing Users. You can also check out portigal.com/services to read more about how I work with teams and companies.
But now, let’s get to my conversation with Mani Pande. She’s the Director and Head of Research at Cisco Meraki.
Well Mani, thanks so much for coming on the podcast. It’s great to get the chance to chat with you today.
Mani Pande: Thank you, Steve, for having me on the podcast. I’ve been listening to your podcast for several years, so it’s great to be a guest on it.
Steve: Excellent. Do you want to start us off with kind of a little introduction to you and then we can build a conversation from there?
Mani: Sure. So my name is Mani Pande, and currently I lead the UXR team for Cisco Meraki. And, you know, Cisco is a really big company, and I am part of the networking division. And within the networking division, like, there are two types of primary products, I would say. Meraki, which is their SaaS offering, and then enterprise networking, which has primarily been their on-prem offering. And my team works across both Meraki and enterprise networking. So it’s a pretty big team, and it’s a big part of Cisco’s business because for Cisco, networking is still bread and butter. So there’s a lot of interesting work that the team does, and a lot of the work that they do does impact the products that we ship and also has a — hopefully has a lot of positive impact on Cisco’s bottom line.
Steve: Do you have any examples you can share about situations where research impacted something that the product was doing?
Mani: Yeah, so one of the — my manager is a big believer in the double diamond approach, you know, starting from doing foundational work and then moving on to doing — you know, once you have the design, testing it, doing concept testing, doing usability testing, and once you have shipped it, you are also — you also have some kind of metrics that you have used to define success and trying to measure those. So there are several examples that — where the team worked throughout the double diamond process and was able to make an impact not only just in defining, you know, what kind of product do we want to ship, but once we had some concepts, they helped define what are some of the hypotheses that we want to test, did a lot of concept validation, did a lot of usability testing because we had a lot of designs that we wanted to test, so pressure tested them in front of our customers.
And then, you know, once we had shipped the product, like we wanted to see that, you know, our customer’s happy with it. Do they really like it? Was it worth the effort? So, you know, doing some kind of like customer satisfaction surveys as part of getting continuous feedback from customers. And also, you know, even when we did those customer CSAP surveys, like we got a lot of open-ended comments, and we got some feedback from customers such, “Oh, you know, this product, like this is not working or that is not working.” So those were good early signals of what we needed to change before it escalated into a support issue.
So there are several projects that we have worked on. At Meraki, we always call our projects by planet names. So there’s a project called Jupiter. There’s a project called Aurora, which is all around, you know, providing more visibility for on-prem devices to show up on the Meraki dashboard so you can see them as well as manage them through the SaaS product.
Steve: And so is it the same, roughly the same, group of researchers that are working through the stages of the double diamond like you described?
Mani: Yes, I would say like it’s like, for example, for one of the projects that I mentioned, Aurora, it was a same researcher who worked through the whole process and obviously in very, very close collaboration with the designer. So they were a tight-knit team that worked throughout the double diamond process. So, you know, yes, they are.
Steve: Is that an example of a researcher being embedded? I know that’s kind of a buzzword. Is the researcher embedded with that team in that case?
Mani: And I know there are a lot of people have a lot of, especially research leaders have a lot of opinion about whether you should have an embedded model versus not an embedded model. In our case, I think just because of the complexity of the domain, having an embedded model is extremely important. I, you know, I have worked across, like I’ve worked at Wikipedia, I’ve worked with that Lyft, I have some B2B experience, I used to work at Success Factor where we made HR software, I worked at Samsung, I worked as a consultant where, you know, you just need to know a little bit to be effective.
But I have never, ever worked in such a technical domain, which is networking. Like every day at work, I feel like I, sometimes I feel like, okay, I know a little bit, but then there are days where I feel like I know my thing. So for us, I think the embedded model is extremely important because you need to have a little bit of domain expertise to be able to do research a little more intelligently and more meaningfully.
And another thing like I feel, this is my perspective that an embedded model works better. And, you know, even when I worked at Lyft, which I would say in terms of complexity is nothing compared to enterprise networking, we still had an embedded model because what an embedded model enables is relationships, which are harder to form if you’re not in an embedded model. Like for researchers, one of the things that, you know, we do is that we lead or we bring about change without authority. So for that, I feel like having relationships is extremely important.
In fact, you know, like there’s this article that I came across recently from Harvard Business Review, which, you know, they had listed like the three things that you need to do to lead without authority. One of them was relationships. Like having relationships with your PM partners, with your design counterpart, engineering, data science, it’s extremely important to be able to bring about change, to be able to show that, you know, what you are hearing from your customers matter, to be able to change hearts and minds. Because I sometimes feel that we are in the business of changing hearts and minds. You know, a lot of people, like I have worked with a lot of PM partners, they have very, you know, some of them have very strong opinions. So to be able to bring about change, I feel that having strong relationships is extremely important. So I am a big believer in the embedded model.
Steve: Are there other things that they have to do or that you encourage them to do to build the relationships in the way that you’re talking about?
Mani: There are various things like I have done all my career and then I also encourage my team to do. So one of the things is I feel like to have a good relationship, you need to bring, and this is more important if you’re in IT and you’re doing the research yourself, is to bring your design, to bring your stakeholders along with the right when you are conducting research. So that’s one thing that I always encourage my teams is invite people to come to your research sessions. Make sure that they are involved in helping you come up with the insights. Like obviously the researchers are going to do the heavy lifting. Like you don’t expect the PMs or the engineers or the data science to do the heavy lifting. But do a workshop with them and ask them, like what did you hear from some of the interviews that you attended? Like what resonated with you?
And also the other thing that comes out with that is that you have less resistance towards the end. People are less likely to challenge you. So it ensures that everyone is kind of on the same page from the beginning. So I feel that it’s also good for relationship building. So that’s one thing that I always tell the ICs to do.
And myself as a research leader, like I obviously try and build relationship with whoever are my counterparts. And I also try, I’ve always done is like build relationship, especially with, you know, PMs like who are like the head of the product that your team leads. So for example, when I was at Lyft, like I worked on the Driver app. So I used to meet with our head of product management for Driver at least one supporter to make sure that I also had a good relationship with them.
Another thing that I learned through that was working with them, it was easier to figure out what we should do long term because a lot of product managers are only thinking about, you know, what they have to deliver within the quarter or maximum the six months. Like if you build relationships with the leadership, like it enables your team to work on more long term projects. So that was just a learning that I had. And I always try and do that is like have a good relationship with the head of product, like people, you know, two or three levels above me. I mean, they have a lot of ideas.
Steve: How does the relationship support the longer term conversation?
Mani: Like they are thinking more about, you know, like where the business needs to be. They’re not so focused on the product roadmap. They are not so much thinking about, you know, this is the feature that I need to ship tomorrow. And also, you know, you can get them to say yes to something that you feel that the team should be working on, but that they might be a little bit of pushback from the product team. So if you get their blessings, you know, you can be working on projects.
Like, you know, as researchers, we have a lot of opinion. And I always tell people like you have to have a point of view. You are spending so much of time with the customers. You’re talking to them. Like if you don’t have a point of view on what research we need to do or what matters to our customers, then you’re probably not doing a good job. So like let’s say if you have a point of view of some research that needs to be done, but it’s much more long term. You know, you would probably not going to see the impact or nobody is thinking about in terms of their roadmap for the next quarter or the half. Like it’s easier to get a buy-in from the executive. And to be able to get that buy-in, like you have to be able to get that buy-in like you have to have a relationship with them. That’s what I have experienced and it has worked for me in the past. That’s what I do, but I also encourage like my ICs to do it.
Steve: So it’s you as the leader are having the relationship and the buy-in for the longer term pieces. This is not things that your ICs and researchers are focused on. This is your role as the leader.
Mani: But you know, a lot of people feel intimidated to go and meet the VP. So I encourage them to do it. And in fact, you know, like one of the other things when I used to do this was I would take like if I did this conversation, I would invite the ICs who were relevant for that conversation to be part of that conversation. So they didn’t feel intimidated to be having that conversation and they could also participate in that conversation and hopefully meaning going forward can do it themselves without me being there.
Steve: So when you talk about relationships, you’re creating them yourself. You’re encouraging others to do that. And then you’re, I guess, enabling or facilitating relationships between other people. You’re talking about a number of different fronts. for building these relationships– the workshops, inviting people to come to sessions, having these sort of, I guess, planning meetings or discussion meetings.
Mani: Yeah, ultimately my role as a research leader is to help my, enable my team. That’s how I think about it. That is one of my important goals. I would say not the only goal. So whatever I can do to enable that, I always try and facilitate that.
Steve: One thing that I’ve heard a lot and that I experienced myself is that the kind of relationship building you’re talking about, whether that’s workshops, participating in interviews, or just kind of meeting, is harder or it’s at least different when work is remote. And I wonder, have you seen changes in how you’re doing this relationship building or how you’re helping others to do it over the past few years?
Mani: Yeah, I mean you can think of it as a glass half full or half empty. That’s how I think about it. I think it’s still possible. In today’s world, like we have so many more tools. Like I would say this would have been so much harder 10 years back. Like doing a remote workshop is so much easier. Like we are all like you can use FigJam, you can use Miro. Like even our stakeholders, like everybody knows how to use those tools. And there are so many templates that you can leverage. Actually, you know, in some ways it’s easier to do it remotely versus do it in person.
But I agree like, you know, in person there’s a lot more energy to it. There is something about being in person at the same time. Like the wife is very different. But with the tools that we have, I think it’s just so much easier to do that. The last two jobs I have started, I’ve started them remotely.
Steve: That’s a good reframe on my question. I think glass half full, glass half empty is a lovely way to look at it.
Mani: And honestly, like if I compare them to the last two jobs that I have, like did I feel any difference? I would say it’s a little harder. It takes a little longer, right? But it’s not impossible. And you can get to that same level of relationship over time. So the one thing that I have done, the one thing that I have done is I have done a lot of work. And the one thing that I have done though is like I’ve also had this privilege like, you know, when I worked at Lyft or at Meraki, I live in the Bay Area. The offices are in the Bay Area. So when I go to the office, like I don’t, I think of those days as days of relationship building. So I go and I meet a lot of people. And at home it’s obviously a little more focused work or you could be in bigger meetings. And I think about work on how I spend my time in the office has also changed a little bit in the last four years.
Steve: Right, we might not have done explicitly relationship building, maybe not even having that as an intention, going into the office to build relationships.
Mani: Yeah. So not only just with stakeholders, I think it’s also important for the team. Like when I was at Lyft, our team, you know, like when we went to the office, we all went to the office. So it became more like a team day. Not very productive, I would say, but it was good for, you know, team bonding and it was good for us in terms of, you know, coming to know each other and just building our relationship and figuring out what kind of a team we are together.
Steve: You mentioned this article about change without authority and you said that they listed three things and one of them was relationships. Putting you on the spot, do you remember what the other two were?
Mani: I do actually because I did a blog about it. So I’ve given it a lot of thought. The other one was expertise. I think that’s also an important one for researchers because in the last, you know, few years, there’s all this, everybody talks about DIY research, you know, you can farm out research, everybody can do research, and there’s also this perception, you know, how hard is it to do research? Like, you only have to talk to people, right? Like, you and I are talking. We could be doing research right now. So I think it’s important for us as researchers to show our stakeholders that, you know, research is not easy as it appears. It’s very easy for us to do bad research and get bad insights from that.
So one of the things that I did at one of my previous jobs was we used to do these immersion events where we would bring, you know, everybody who worked on, who was our stakeholder, to come and listen and talk to our customers. And we would do these focus groups if they were like a whole day event. And in the beginning when we started that, there were folks from marketing and ops team who ran some of these focus groups. And when we got feedback about the immersion, it was very clear that everybody realized that when researchers are not doing the moderation, the kind of data that you get is not good. And the conversations were not that interesting. They didn’t feel that it was a good use of their time. So I think you can have your stakeholders experience it, that it’s not that easy to do moderation.
In fact, I feel like when I was in IC and if I times, like I would do four to five interviews in a day, I would be mentally exhausted. You know, doing moderation is one of the hardest things to do because you’re multitasking at another level. Like you’re trying to, I mean, I’m a little old school, I would take notes, I would try and listen to what the person is saying and figure out like, do I follow my interview script? Do I change it? So just having people experience that is important. So I think show and tell could be one way of showing that, you know, research is hard.
And then when it comes to quantitative research, like, you know, writing surveys, I think that’s a pretty specialized skill. I have seen people, you know, think that they can write surveys, but there’s a lot that goes into it. You can get absolutely wrong responses if your question is not well designed, if your scale is not well designed. So for quant research, like I have a little bit of a strong opinion on that, that I don’t think it should ever be DIY. It shouldn’t ever be unskilled stakeholders to write a survey. So that’s one thing, like, you know, just showing your expertise is one thing that you can do to leave without authority.
The other one they mentioned was about business. Organizational understanding is the third one. That is very important. That, you know, one of the things I will, as I said, you know, I have a long career, I worked for 20 years after grad school. I have seen often like researchers resist or don’t want to have a good understanding of how the company makes revenue, earns profit. They’re a little wary of that part of the business. I think it’s also important for us to understand, like, how does the company make profit? Like, for example, if you work for a B2B company, like, I would say that you should have a relationship with the sales team to understand how do they sell, what do they sell, like, what is some of the feedback that they get from customers.
So I think having a little bit of understanding of revenue and profitability is important. Like, for example, like, if you work for a company that’s in the gig economy, which has, you know, which has a marketplace, like, it could be Lyft, Uber or DoorDash, when that is the crux of the business marketplace. That’s how the company makes money. So understanding the dynamics of the marketplace is an important thing that a researcher should do. Let’s say if you are a researcher at Uber and you’re working on the Rider app, you should still have an understanding of the marketplace. Usually, you know, it’s the marketplace is kind of on its own, but irrespective of whether you work on the Driver app or the Rider app, you should have an understanding of how, you know, Riders and Drivers are matched, because ultimately that’s where the secret sauce happens and that’s where the company makes money.
I think a lot of us come from academic background and maybe a little bit of purist background. So maybe it could, this is just a hypothesis and that is where it comes from. I mean, I would say to a certain extent, even I had it, like, early in my career. It took a while for me to realize that understanding revenue, profitability is important. One thing that I would say, like, most researchers do agree on is understanding business priorities and, you know, doing research that aligns with business priority. I don’t see, I have never seen that resistance on that, but when it comes to revenue and profitability, I have seen researchers, like, have a little bit of resistance to that. Like, for example, I have some friends who work at Facebook and Google, you know, how they make money is ads, but many of them did not want to work on the ads team.
Steve: That seems different to me and just thinking about myself. That seems different to me than understanding like, how does the system match riders and drivers? When you say that, that kind of sparks my researcher curiosity. Like I think we would want to know that because what’s behind that secret wall and how do the gears all mesh? But again, my own bias here when you say working on ads is not, I kind of get that feeling too. Like I just think like, ugh, ads. And so to me that seems different. And I don’t know, I’m not trying to pin you down on something because you’re talking kind of subjectively what we each have our impressions. But I think this idea that, to your point about changing without authority, that there are these important things to understand. I guess there’s a difference between wanting to work on Facebook ads as your project and having enough of an understanding of the revenue model, which impacts everything you would ever do research on at Facebook I think.
Mani: So I think both. What I’m trying to get at is understanding the money part of the business is important. Like, and the money part of the business for different companies is different.
Steve: Yeah.
Mani: For a gig economy, it’s the marketplace. For Google and Facebook, it’s ads. So having some understanding of how your company makes money is important for researchers.
Steve: And not to flog a dead horse here, but I feel like companies like Google and Meta or Facebook, it seems like their culture is such that how the company makes money is sort of kept in a separate box and we’re going to come here and work on whatever the latest amazing thing that’s going to change the world Internet through balloons, you know, some amazing project. And so speaking out of my hat here, but I have some empathy for people that don’t want to think about the money because they’re not being sold that as an employee. I don’t know if that’s true. I’m hypothesizing that the company culture kind of keeps those things in separate buckets. But if you work at Lyft or Uber or DoorDash or Meraki, there’s a product and a service that’s much more essential to the conversation they’re having. Again, this is not my direct experience. I’m just kind of — I’m just giving you my biased interpretation of what you’re describing.
Mani: I think just the basic level is probably, I’m sure, like, you know, they keep it under wraps to a certain extent, but just like having a basic understanding and not having an aversion to it is important.
Steve: So when you talk about these factors, right, the understanding the business, so there’s some specificity, the expertise and the relationships, and you talked about Cisco and Meraki being just the level of complexity, the sort of technological and I guess industry specific stuff. Does that complexity become a compounding factor or something in trying to achieve those levels of those three factors that you’ve brought up?
Mani: I think especially for something this technical, having some domain expertise matters. And that goes back to having some, that goes back to the first one that I was talking about, which is expertise. Like having some domain expertise becomes important because if you want to have a meaningful conversation, you know, if you’re doing an interview, like you need to have some basic level of understanding to have a meaningful conversation. I know in research we say that, you know, no question is stupid, but if you have zero understanding, you’ll only have stupid questions, at least in this kind of a domain. So I think having some understanding is important, and that’s what I tell all my researchers.
Like after I joined Meraki, I have this cheat sheet I call “Mani’s Cheat Sheet” about networking, and every time I hear something that I don’t know, I get a version from Google and then I get a chat GPT version, like please explain it to a middle schooler version, which actually is the version that works for me. And that cheat sheet, as I’m like, “Huh, like 20, 30 pages long?” It just keeps on increasing. I’m not going to be an expert on networking. I don’t want to be, but I just want to know a little bit to be able to have meaningful conversations and to be also able to provide, you know, feedback to my team. Like sometimes, you know, when we have an important presentation, like I work with them to figure out like what are the insights, like what are some of our action items, but if I have no understanding of the domain, I can’t do that.
Steve: Can you talk about from the period of time that you came to Meraki, like what that progress has been and what research has gone from to where it is now?
Mani: So Meraki has seen incredible growth in design as well as research in, I would say, in the last one, two years, which is the opposite of where, you know, design as a field and UXR as a field is going at other companies. So we have, it’s also because, you know, as I was saying, like the company has really adopted the double diamond approach. So they see research as being an integral part of how you build products. So that’s one of the reasons why we have seen this big, massive growth in the last one year. And the reason why I came here was, you know, as I was telling you earlier, you know, like Cisco had these two products, like the SaaS product, the on-prem, and the strategy now is to, you know, convert, which is, you know, have like some of the on-products be able to be managed in the cloud.
So they brought these two teams together, the design team for the on-prem side and the design team from Meraki. So that’s how I got hired as the head of UXR because then, you know, the team grew because you had two different teams that merged and the complexity of the business and the complexity of the problems that the team was expected to answer grew because now you had two different businesses that you had to support. So that’s how I got hired and, right, and, you know, our team has grown quite a lot even in the last one year. Like I’ve hired, like just in March, three people have joined the team. And it’s also because, you know, everyone agrees that, you know, you need to do research if you want to build better products. So there is this hunger for research, so that’s the other reason why we’ve been able to grow despite, you know, the market going in the other direction.
Steve: So you came into this newly formed, newly merged organization that already had an appetite for a belief and a commitment to research.
Mani: Yeah, and that appetite has been growing. I would say it’s been a steady state because there are more and more teams that we are working with because Cisco networking is huge. As I was saying earlier, it’s like it’s their bread and butter. It’s the major chunk of their revenue. So there are more and more teams that we are working with and that’s one of the reasons why I’ve been able to hire researchers to support teams that did not do research previously but want to do research now.
Steve: Are there things that you are doing in your role that are behind that? In addition to there being these other teams, but do you think you’re responsible for the increasing the demand in any way?
Mani: I would say it’s a team effort. I would not put — I won’t say that it’s just me by myself. But obviously, you know, as a research leader — or I would say as a research professional, we have to evangelize research as much as we can all the time.
I can talk about my time at SuccessFactors. You know, when I joined SuccessFactors, they already had an IPO. Then they were bought by SAP in 2012 for like $3.5 billion. So the company was pretty successful, but they never had research at that time. And one of my — like what I had to do in the beginning was, you know, just evangelize research and make sure that it became part of how we build products. So in the beginning, I did a lot of like evaluated research to show impact and then, you know, slowly as our team grew, we — I mean, continued to do obviously evaluated research, but we did a lot of generated research as part of new product launches.
Steve: So you talk about that situation, they were bought into research enough that they hired you, but it doesn’t sound like the hunger that you’re talking about now wasn’t there at that time.
Mani: And also it’s like, you know, we’re talking 2013. I know the field has changed quite a lot in that time. But yes, the hunger wasn’t there. So my job was to create a hunger for research. And it was like very different tactics. Like I did a lot of usability research because that gets you — it’s a quick hit. That’s how I would describe it at times, you know, usability research. If you are looking for quick impact, you can get it pretty fast.
And you can get — you can find people who are on your side who become evangelists of research pretty easily. So that’s what I did a lot when, you know, when I had to build a team there because, you know, there was no research team. I wouldn’t say that — I mean, there was no research majority.
Steve: I hear relationships and expertise at least in that story.
Mani: Yes, there was because I remember when I joined, there was some usability testing that had been done. But whoever did it, they did not know how to define usability tasks. So when it was an unmoderated test and user testing, people were very confused about what was expected from them. And it was very clear that you needed somebody who knew the basics of usability testing to have — who should have set up the test.
Steve: Just switching topics slightly, you’ve kind of brought us back in time to some different roles that you’ve had and I wonder if we could go further back and maybe could you talk about how you entered the field of UX research.
Mani: I did not think I was going to become a UX researcher, honestly. Like I have a PhD in sociology. I thought I was going to become a college professor. That was my goal. But I’m — my husband is an engineer. So we moved to the Bay Area. And I looked for teaching jobs in the Bay Area. There were none. So I moved — I went — I got a job at Institute for the Future. I don’t know if you know about them, but they are a forecasting research company. So they do a lot of tech research. And I started working there for five years. And I did not even think the kind of research that I was doing was, you know, user experience research, like product strategy research. But that’s the kind of work that I did there.
One of the biggest clients that we had was Nokia. And I did a lot of ethnographic research, you know, going across — traveling to people’s home in India, U.S., and Brazil, and, you know, trying to understand, you know, how do they use mobile phones? Because this was like 2006, 2008. It was still a new phenomenon. And that helped — that was to help define Nokia’s strategy, especially for these markets for the next one to three years. And from that, I kind of transitioned into UXR. And in fact, you know, when I finished my Ph.D., I did not even know that there was an HCI field I completed in 2004. So it was still pretty early days of — at least from my perspective for the field.
Steve: When you found yourself doing global ethnographic fieldwork for Nokia, did you see any difference between how you were working in that context and the skills and approach that you had developed in your PhD?
Mani: There are cultural differences that you have to be aware of. One of the biggest things is you should be ready for anything. There are a lot of unknowns. I remember, like, we were doing this project, but we were working for Nokia. Like, we had — I was working with our clients, and we were supposed to go and interview someone, and the person didn’t show up. So there’s a lot of those things that you got to be ready for, like, you did not prepare for. Like, very rarely it’s happened to me that, you know, I did ethnographic research in the U.S., and we had no shows. But it’s happened in India so many times. It’s happened in Brazil. So you got to have these contingency plans and go with the flow. And I think those are some of the things that you have to be aware of.
And also, you know, especially, like, when I worked in Brazil, like, I obviously don’t speak the language. Like, I do not — like, for me, obviously, India is easier because I have lived in India. I know, you know, what the culture is, like, what are the things that you do. Like, for example, if you go to India — to anyone’s house in India, they are going to offer you tea, coffee, something to eat, and it’s polite to eat it. And it’s impolite if you don’t eat it. And when I went to Brazil, like, I had to work with translators, which I had never worked with before. So one of my big learnings was that it takes you twice as much to do the same interview because they are translating it for you.
So I think there are a lot of cultural nuances that you have to be familiar with when you go — especially when you do research outside of, you know, U.S. and maybe to a certain extent even Europe.
Steve: And did you see differences at that point in your career where you’re coming from an academic environment to a commercial environment? Were there points of transition for you as you moved from one to the other?
Mani: Yes. In terms of, like, I remember when I wrote my PhD dissertation, I took a year to do the analysis after I had finished my qualitative interviews. So I — and, you know, when I — this Nokia one that I was talking about, like, we were sharing research inside from every interview every day. So I think that is the big transition that you have to make. Like, if you come from academia is the pace. And also thinking about, you know, how — like, how does this impact the business? Like, how does this impact the product? And also working with so many stakeholders. Like, those are some of the big changes. Like, when I was in academia, the only time, like, I worked with others is my advisor provided me feedback for my dissertation. And when I published, I had these three reviewers who gave me feedback for some of the articles that I published. But, yeah, you know, the way you work with stakeholders is so different. And also you got to be, you — okay, we’re doing a lot of quick and dirty work, which obviously, you know, you don’t do it in academia. Especially if you do that, it won’t get published.
Steve: So after this Institute for the Future experience, was user research that was your career path at that point?
Mani: Yeah, and then I joined Wikipedia, which was actually a very amazing experience in terms of what I was able to do because — I mean, Wikipedia is like I have never met anybody who doesn’t like Wikipedia. I, you know, I’ve worked in different — as I said, you know, I have worked in different industries. Often we more than often, like, meet customers who say, like, I don’t like this about your product. I don’t like that about your product. But for Wikipedia, it was very different.
So one of the things that I did when I joined Wikipedia was I did the first ever survey of Wikipedia editors. So Wikipedia is, as you can probably guess, from their mission, unlike most companies, they barely do any tracking. Like, they don’t use, for example, cookies. So we used to rely on ComScore to even get the numbers for active readers for Wikipedia because they were not using cookies. So they did not know at that time, like, what percentage of editors are women because Wikipedia, even today, has a gender problem. So I did the first ever surveys of Wikipedia editors, and the answer at that time was 9%. The only 9% of people editing Wikipedia were women. And as a result of that, a lot of — there are fewer articles about women on Wikipedia versus men. So there’s a big gender bias in Wikipedia. So that is some of the work that I did when I joined Wikipedia, which I thought — and I also worked on the redesign of the mobile app. So I again did ethnographic research in the U.S., India, and Brazil for the redesign of the mobile app for Wikipedia. But all that work was very fulfilling just because of the amount of impact that you could have on users. Because I don’t remember the numbers now, but at that time there were 250 million active readers on Wikipedia every month, and there used to be like 7 to 8 billion page views.
Steve: As a user researcher over your career and maybe as a leader now, I don’t know, how do you think about different kinds of methods or different kinds of approaches to getting the data that you want to bring your team?
Mani: I mean, I’m trained as a mixed methods researcher, and when you think about mixed methods, I like to believe it’s on a continuum. There are some people who do small surveys and think that they are mixed methods researcher only doing descriptive statistics. There are some people who do regression analysis, who will do, I don’t know, structural equation modeling, latent class analysis, who are also mixed methods researchers. So it’s a little bit on a continuum, even for qualitative, right? Like you can be an ethnographer. You can be somebody who really believes in participant observation, or you can just do qualitative one-on-one interviews. With that said, like I do both because I just happen to be trained. When I did my PhD, I specialized in research methods, which meant that I had to prove that I could do both quant and work. So I did a lot of statistics. I did a lot, you know, I took a lot of qualitative courses. And in fact, you know, my dissertation was a qualitative dissertation, despite, you know, having a very deep background in stats. So I do both.
But with that said, I would say you should be methodologically agnostic. Like it should not, depending on what’s the problem that you’re trying to solve, you should figure out what’s the research, right research method. So I think that is what is more important. And that’s why I feel like having a mixed methods background is good, because then you just have more methods in your toolkit. So you can figure out like, you know, for this problem, probably a survey is a good solution. Or for if I want to find the answer to this problem, maybe doing ethnographic research, qualitative interviews is the right thing to do. And, you know, you can use all these methods throughout the process. Like you can do a survey, you know, along with some foundational research.
Like I can give you an example. When I worked at Wikipedia, I’m very proud of the feature that we released. I’ll talk about a little bit about that. So when I worked at Wikipedia, I interviewed this person in Salvador, which is, you know, in north of Brazil, in the Bahia district. And he was a Wikipedia editor. And obviously, like you’re a Wikipedia editor, you’re a big reader too. And at that time, he told me that, you know, he wanted to read Portuguese Wikipedia, but he felt that it was not mature enough. Some of the articles did not have the kind of detail that he was looking for. So he would often flip to English Wikipedia. So I thought that was an interesting thing that he mentioned. So after we did the ethnographic research, we followed it up with the survey and we asked people like, you know, do you, how many languages Wikipedia do you read? And, you know, we got data. Like if you read English, what else do you read? So obviously it was clear that, you know, English is the primary one. But more often than not, people read other language Wikipedia’s too. Especially outside of, you know, US I would say, even in Europe, like German Wikipedia is very big, but the German readers are also reading English Wikipedia. So, and that was also very clear in the survey data that we got. So we introduced, it’s even there today in the app. We introduced what we call the inter-language Wikilinks, which made it really easy to toggle between different language Wikipedia. So, for example, like if you search for an article, it tells you that, you know, this article is also available in this other language. You can set your primary languages, you can set your secondary languages. And all this came out through this, you know, ethnographic research that was done, followed it up with a survey, gave us more confidence that, you know, this would be a useful feature. And it still exists like 10, 12 years after we did that research.
Steve: That’s a great story and one detail that excites me is that small data point from one method. You find new questions in research. You found something and then you did some more research to kind of understand that phenomenon that you didn’t know.
Mani: Yeah.
Steve: That’s such a great example of that. That the ethnography was not sufficient to make something new, but it did point to a whole new effort to understand something, which then led to that feature.
Mani: Yeah, but ethnographic is that insight, which I don’t think I would have got from a survey, right? So it was, the survey just gave us more confidence that, yes, I think that’s worth building.
Steve: Yeah, that’s a good clarification. We’re going to talk about mixed methods and about different kinds of tools kind of feeding together. You talked early on about relationships, so it just makes me think about some of the different roles that we have around data science as a big part of what so many companies are doing. How do you see kind of the relationship between UXRs, mixed methods and otherwise folks and data science?
Mani: I think there’s a lot of synergy between what UX cards do and what data scientists do, because ultimately we are both researchers. It’s just that we are looking at different type of data. But if we bring it together, I strongly feel that we can have a much more holistic understanding of our users. And, you know, in my career, like I’ve had, especially at Wikipedia and Lyft, I was lucky to work very closely and actually even at Meraki, work with data scientists. So there are various ways we can work with data scientists. Like, for example, like if you’re doing segmentation, you know, often we do segmentation of our users just based on attitudinal data. But you can get the behavioral data from data scientists and bring that together to have, to just have a better segmentation model of your users. Various companies, they have dashboards to, you know, look at how their customers are doing and tracking. Many of them, especially in B2C, tend to be more behavioral based versus attitudinal. So I think there’s an opportunity there to have dashboards that not only just tell you what are the users doing, but also how are they feeling. I think that’s an important story to tell. So there’s that one can do.
Then there’s also an opportunity to work with data science during A/B experimentations. Like in my past companies, I have worked with data scientists when they have an A/B experiment to, you know, do a survey along with the experiment to see like if there are any differences between the control and the experimental group on not only what they’re doing, but how they are feeling. And the one thing that I learned is that there’s a little bit of lag. Like if people are unhappy, it takes a little while before they stop doing the thing. So like, you know, doing that survey gives you an early pulse that, okay, maybe this experiment is not working as well as it should. Maybe we should tweak something so that the users are happier. So I think that is one thing. So those are some of the opportunities that I see working with data scientists.
Steve: Is there anything else today in our conversation that I should have asked you about or you want to make sure that we talk about?
Mani: I can’t think of anything else right now. I’m sure I will think of something later.
Steve: Well, Mani, it’s been really lovely to speak with you and learn from you. I want to thank you again for taking the time to chat and share all your stories and experiences. It’s been great.
Mani: And thank you, Steve, for having me on the podcast.
Steve: There we go. Thanks for listening. Tell your friends. Tell your enemies about Dollars to Donuts. Give us a review on Apple Podcasts or any place that reviews podcasts.
Find Dollars to Donuts in all the places that have all the things or visit portigal.com/podcast for all of the episodes complete with show notes and transcripts. Our theme music is by Bruce Todd.
The post 39. Mani Pande of Cisco Meraki first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Mani Pande, Director and Head of Research at Cisco Meraki.
We used to do these immersion events where we would bring everybody who worked on, who was our stakeholder, to come and listen and talk to our customers. And we would do these focus groups, they were like a whole day event. There were folks from marketing and ops team who ran some of these focus groups. And when we got feedback about the immersion, it was very clear that everybody realized that when researchers are not doing the moderation, the kind of data that you get is not good. And the conversations were not that interesting. They didn’t feel that it was a good use of their time. So I think you can have your stakeholders experience it, that it’s not that easy to do moderation. – Mani Pande
Show Links* Interviewing Users, 2nd Edition * UI Breakfast Podcast. Episode 280: User Interviewing Techniques with Steve Portigal * Mani on LinkedIn * Cisco Meraki * the Double Diamond * CSAT * On-Prem * SuccessFactors * How To Influence Without Authority In The Workplace * structural equation modeling * latent class analysis * FigJam * Miro * Institute for the Future * Comscore
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. In case you don’t already know, I recently released a second edition of my classic book, Interviewing Users. This new edition is the product of 10 more years of me working as a researcher and teaching other people. It’s bigger and better. It’s got two new chapters, a lot of updated content, new examples, new guest essays, and more.
As part of releasing this new edition, I’ve been on a number of podcasts myself, including a conversation with Jane Portman that was part of her podcast, UI Breakfast. Here’s a quick excerpt from that conversation.
Jane Portman: As you’re training other researchers, you’re training experienced researchers, I’m thinking, what do you feel is common knowledge that they’re mastering well and that we’re all like good at and what things are surprisingly difficult?
Steve: You know, I think especially for people that are, they have a little bit of experience, but they’re kind of starting to blossom a little bit. One of the things they often need the most is confidence. And so people will often describe a scenario that they were in. People are messy and people are unpredictable and so all these things happen and so they go in with the best of intentions and plans and then things change a little bit. Somebody mentions their divorce, they don’t know what to do. And so I feel like my job isn’t to tell people that they’ve screwed up and that’s not how you do it. I think my job is to tell people that the thing that you encountered is very common. It’s a thing that a lot of researchers struggle with. I try to handle it this way, but there are situations when I handle it this way. Like I think I have a lot of like specific guidance and best practices, but all of those come with a lot of subjectivity and that it’s sort of the nature of the work to be a little confused or uncertain and to have to try things.
And by the way, there’s no right choice. When someone mentions their divorce, you know, not even me versus you, like me versus me. The next time I would do that interview, I would handle it differently. There’s a moment and if the divorce was brought up in minute three versus minute thirty, like it would play out differently. We’re not algorithms, we are, I think improvisation is a big part of it. So I think I want to help junior researchers feel okay about that there’s no one right way to handle this and that their way of, the fact that they felt confused and uncertain in a situation and they made this kind of, here’s how they addressed it, it’s, I rarely tell them like, well that’s the worst thing you could have done. It’s usually, they usually are doing their best. That confidence to make a different choice is kind of what I want to help somebody with.
You know, more experienced interviewers, I think I like, I like working with them because I think we can have a better, richer conversation about what are all these choices and what are the differences between them and you know, I love being in a workshop where I’ve got people with different experience levels because then I might give some guidance and then we can have different people suggest, well you know, here’s what I’ve done. Everyone can learn from each other and sometimes we can debate and I don’t mean that in like a right and wrong way but I think someone with experience, we can have a really interesting conversation where we look at one scenario and four or five different ways to handle it and we might disagree on what is sort of the optimal way.
I think what that surfaces is that as individuals, as interviewers, we’re all wired differently and we all have different instincts and different personalities and you can get away with things that I can’t because of my age or my gender or my energy and I can get away with things that you can’t and getting away is maybe there’s the wrong framing but there’s just so many interesting choices and it’s I love hearing about other people’s things and thinking like, oh could I have that amount of friendliness or that amount of stillness or that amount of curiosity or that amount of empathy, you know, could I present those things in different amounts? You know, that’s part of being an expert interviewer is you have your own personality and your own strengths and you can exercise different facets of that as the situation requires. I think, you know, a more experienced interviewer has more adaptability to that, has their core, you know, but also can put on different authentic, true to themselves faces and energies to kind of support different kinds of situations that are going to come up in these interviews.
Again, that was me speaking with Jane Portman on UI Breakfast. You can check out the whole episode and you should totally buy a copy of this new edition of Interviewing Users. You can also check out portigal.com/services to read more about how I work with teams and companies.
But now, let’s get to my conversation with Mani Pande. She’s the Director and Head of Research at Cisco Meraki.
Well Mani, thanks so much for coming on the podcast. It’s great to get the chance to chat with you today.
Mani Pande: Thank you, Steve, for having me on the podcast. I’ve been listening to your podcast for several years, so it’s great to be a guest on it.
Steve: Excellent. Do you want to start us off with kind of a little introduction to you and then we can build a conversation from there?
Mani: Sure. So my name is Mani Pande, and currently I lead the UXR team for Cisco Meraki. And, you know, Cisco is a really big company, and I am part of the networking division. And within the networking division, like, there are two types of primary products, I would say. Meraki, which is their SaaS offering, and then enterprise networking, which has primarily been their on-prem offering. And my team works across both Meraki and enterprise networking. So it’s a pretty big team, and it’s a big part of Cisco’s business because for Cisco, networking is still bread and butter. So there’s a lot of interesting work that the team does, and a lot of the work that they do does impact the products that we ship and also has a — hopefully has a lot of positive impact on Cisco’s bottom line.
Steve: Do you have any examples you can share about situations where research impacted something that the product was doing?
Mani: Yeah, so one of the — my manager is a big believer in the double diamond approach, you know, starting from doing foundational work and then moving on to doing — you know, once you have the design, testing it, doing concept testing, doing usability testing, and once you have shipped it, you are also — you also have some kind of metrics that you have used to define success and trying to measure those. So there are several examples that — where the team worked throughout the double diamond process and was able to make an impact not only just in defining, you know, what kind of product do we want to ship, but once we had some concepts, they helped define what are some of the hypotheses that we want to test, did a lot of concept validation, did a lot of usability testing because we had a lot of designs that we wanted to test, so pressure tested them in front of our customers.
And then, you know, once we had shipped the product, like we wanted to see that, you know, our customer’s happy with it. Do they really like it? Was it worth the effort? So, you know, doing some kind of like customer satisfaction surveys as part of getting continuous feedback from customers. And also, you know, even when we did those customer CSAP surveys, like we got a lot of open-ended comments, and we got some feedback from customers such, “Oh, you know, this product, like this is not working or that is not working.” So those were good early signals of what we needed to change before it escalated into a support issue.
So there are several projects that we have worked on. At Meraki, we always call our projects by planet names. So there’s a project called Jupiter. There’s a project called Aurora, which is all around, you know, providing more visibility for on-prem devices to show up on the Meraki dashboard so you can see them as well as manage them through the SaaS product.
Steve: And so is it the same, roughly the same, group of researchers that are working through the stages of the double diamond like you described?
Mani: Yes, I would say like it’s like, for example, for one of the projects that I mentioned, Aurora, it was a same researcher who worked through the whole process and obviously in very, very close collaboration with the designer. So they were a tight-knit team that worked throughout the double diamond process. So, you know, yes, they are.
Steve: Is that an example of a researcher being embedded? I know that’s kind of a buzzword. Is the researcher embedded with that team in that case?
Mani: And I know there are a lot of people have a lot of, especially research leaders have a lot of opinion about whether you should have an embedded model versus not an embedded model. In our case, I think just because of the complexity of the domain, having an embedded model is extremely important. I, you know, I have worked across, like I’ve worked at Wikipedia, I’ve worked with that Lyft, I have some B2B experience, I used to work at Success Factor where we made HR software, I worked at Samsung, I worked as a consultant where, you know, you just need to know a little bit to be effective.
But I have never, ever worked in such a technical domain, which is networking. Like every day at work, I feel like I, sometimes I feel like, okay, I know a little bit, but then there are days where I feel like I know my thing. So for us, I think the embedded model is extremely important because you need to have a little bit of domain expertise to be able to do research a little more intelligently and more meaningfully.
And another thing like I feel, this is my perspective that an embedded model works better. And, you know, even when I worked at Lyft, which I would say in terms of complexity is nothing compared to enterprise networking, we still had an embedded model because what an embedded model enables is relationships, which are harder to form if you’re not in an embedded model. Like for researchers, one of the things that, you know, we do is that we lead or we bring about change without authority. So for that, I feel like having relationships is extremely important.
In fact, you know, like there’s this article that I came across recently from Harvard Business Review, which, you know, they had listed like the three things that you need to do to lead without authority. One of them was relationships. Like having relationships with your PM partners, with your design counterpart, engineering, data science, it’s extremely important to be able to bring about change, to be able to show that, you know, what you are hearing from your customers matter, to be able to change hearts and minds. Because I sometimes feel that we are in the business of changing hearts and minds. You know, a lot of people, like I have worked with a lot of PM partners, they have very, you know, some of them have very strong opinions. So to be able to bring about change, I feel that having strong relationships is extremely important. So I am a big believer in the embedded model.
Steve: Are there other things that they have to do or that you encourage them to do to build the relationships in the way that you’re talking about?
Mani: There are various things like I have done all my career and then I also encourage my team to do. So one of the things is I feel like to have a good relationship, you need to bring, and this is more important if you’re in IT and you’re doing the research yourself, is to bring your design, to bring your stakeholders along with the right when you are conducting research. So that’s one thing that I always encourage my teams is invite people to come to your research sessions. Make sure that they are involved in helping you come up with the insights. Like obviously the researchers are going to do the heavy lifting. Like you don’t expect the PMs or the engineers or the data science to do the heavy lifting. But do a workshop with them and ask them, like what did you hear from some of the interviews that you attended? Like what resonated with you?
And also the other thing that comes out with that is that you have less resistance towards the end. People are less likely to challenge you. So it ensures that everyone is kind of on the same page from the beginning. So I feel that it’s also good for relationship building. So that’s one thing that I always tell the ICs to do.
And myself as a research leader, like I obviously try and build relationship with whoever are my counterparts. And I also try, I’ve always done is like build relationship, especially with, you know, PMs like who are like the head of the product that your team leads. So for example, when I was at Lyft, like I worked on the Driver app. So I used to meet with our head of product management for Driver at least one supporter to make sure that I also had a good relationship with them.
Another thing that I learned through that was working with them, it was easier to figure out what we should do long term because a lot of product managers are only thinking about, you know, what they have to deliver within the quarter or maximum the six months. Like if you build relationships with the leadership, like it enables your team to work on more long term projects. So that was just a learning that I had. And I always try and do that is like have a good relationship with the head of product, like people, you know, two or three levels above me. I mean, they have a lot of ideas.
Steve: How does the relationship support the longer term conversation?
Mani: Like they are thinking more about, you know, like where the business needs to be. They’re not so focused on the product roadmap. They are not so much thinking about, you know, this is the feature that I need to ship tomorrow. And also, you know, you can get them to say yes to something that you feel that the team should be working on, but that they might be a little bit of pushback from the product team. So if you get their blessings, you know, you can be working on projects.
Like, you know, as researchers, we have a lot of opinion. And I always tell people like you have to have a point of view. You are spending so much of time with the customers. You’re talking to them. Like if you don’t have a point of view on what research we need to do or what matters to our customers, then you’re probably not doing a good job. So like let’s say if you have a point of view of some research that needs to be done, but it’s much more long term. You know, you would probably not going to see the impact or nobody is thinking about in terms of their roadmap for the next quarter or the half. Like it’s easier to get a buy-in from the executive. And to be able to get that buy-in, like you have to be able to get that buy-in like you have to have a relationship with them. That’s what I have experienced and it has worked for me in the past. That’s what I do, but I also encourage like my ICs to do it.
Steve: So it’s you as the leader are having the relationship and the buy-in for the longer term pieces. This is not things that your ICs and researchers are focused on. This is your role as the leader.
Mani: But you know, a lot of people feel intimidated to go and meet the VP. So I encourage them to do it. And in fact, you know, like one of the other things when I used to do this was I would take like if I did this conversation, I would invite the ICs who were relevant for that conversation to be part of that conversation. So they didn’t feel intimidated to be having that conversation and they could also participate in that conversation and hopefully meaning going forward can do it themselves without me being there.
Steve: So when you talk about relationships, you’re creating them yourself. You’re encouraging others to do that. And then you’re, I guess, enabling or facilitating relationships between other people. You’re talking about a number of different fronts. for building these relationships– the workshops, inviting people to come to sessions, having these sort of, I guess, planning meetings or discussion meetings.
Mani: Yeah, ultimately my role as a research leader is to help my, enable my team. That’s how I think about it. That is one of my important goals. I would say not the only goal. So whatever I can do to enable that, I always try and facilitate that.
Steve: One thing that I’ve heard a lot and that I experienced myself is that the kind of relationship building you’re talking about, whether that’s workshops, participating in interviews, or just kind of meeting, is harder or it’s at least different when work is remote. And I wonder, have you seen changes in how you’re doing this relationship building or how you’re helping others to do it over the past few years?
Mani: Yeah, I mean you can think of it as a glass half full or half empty. That’s how I think about it. I think it’s still possible. In today’s world, like we have so many more tools. Like I would say this would have been so much harder 10 years back. Like doing a remote workshop is so much easier. Like we are all like you can use FigJam, you can use Miro. Like even our stakeholders, like everybody knows how to use those tools. And there are so many templates that you can leverage. Actually, you know, in some ways it’s easier to do it remotely versus do it in person.
But I agree like, you know, in person there’s a lot more energy to it. There is something about being in person at the same time. Like the wife is very different. But with the tools that we have, I think it’s just so much easier to do that. The last two jobs I have started, I’ve started them remotely.
Steve: That’s a good reframe on my question. I think glass half full, glass half empty is a lovely way to look at it.
Mani: And honestly, like if I compare them to the last two jobs that I have, like did I feel any difference? I would say it’s a little harder. It takes a little longer, right? But it’s not impossible. And you can get to that same level of relationship over time. So the one thing that I have done, the one thing that I have done is I have done a lot of work. And the one thing that I have done though is like I’ve also had this privilege like, you know, when I worked at Lyft or at Meraki, I live in the Bay Area. The offices are in the Bay Area. So when I go to the office, like I don’t, I think of those days as days of relationship building. So I go and I meet a lot of people. And at home it’s obviously a little more focused work or you could be in bigger meetings. And I think about work on how I spend my time in the office has also changed a little bit in the last four years.
Steve: Right, we might not have done explicitly relationship building, maybe not even having that as an intention, going into the office to build relationships.
Mani: Yeah. So not only just with stakeholders, I think it’s also important for the team. Like when I was at Lyft, our team, you know, like when we went to the office, we all went to the office. So it became more like a team day. Not very productive, I would say, but it was good for, you know, team bonding and it was good for us in terms of, you know, coming to know each other and just building our relationship and figuring out what kind of a team we are together.
Steve: You mentioned this article about change without authority and you said that they listed three things and one of them was relationships. Putting you on the spot, do you remember what the other two were?
Mani: I do actually because I did a blog about it. So I’ve given it a lot of thought. The other one was expertise. I think that’s also an important one for researchers because in the last, you know, few years, there’s all this, everybody talks about DIY research, you know, you can farm out research, everybody can do research, and there’s also this perception, you know, how hard is it to do research? Like, you only have to talk to people, right? Like, you and I are talking. We could be doing research right now. So I think it’s important for us as researchers to show our stakeholders that, you know, research is not easy as it appears. It’s very easy for us to do bad research and get bad insights from that.
So one of the things that I did at one of my previous jobs was we used to do these immersion events where we would bring, you know, everybody who worked on, who was our stakeholder, to come and listen and talk to our customers. And we would do these focus groups if they were like a whole day event. And in the beginning when we started that, there were folks from marketing and ops team who ran some of these focus groups. And when we got feedback about the immersion, it was very clear that everybody realized that when researchers are not doing the moderation, the kind of data that you get is not good. And the conversations were not that interesting. They didn’t feel that it was a good use of their time. So I think you can have your stakeholders experience it, that it’s not that easy to do moderation.
In fact, I feel like when I was in IC and if I times, like I would do four to five interviews in a day, I would be mentally exhausted. You know, doing moderation is one of the hardest things to do because you’re multitasking at another level. Like you’re trying to, I mean, I’m a little old school, I would take notes, I would try and listen to what the person is saying and figure out like, do I follow my interview script? Do I change it? So just having people experience that is important. So I think show and tell could be one way of showing that, you know, research is hard.
And then when it comes to quantitative research, like, you know, writing surveys, I think that’s a pretty specialized skill. I have seen people, you know, think that they can write surveys, but there’s a lot that goes into it. You can get absolutely wrong responses if your question is not well designed, if your scale is not well designed. So for quant research, like I have a little bit of a strong opinion on that, that I don’t think it should ever be DIY. It shouldn’t ever be unskilled stakeholders to write a survey. So that’s one thing, like, you know, just showing your expertise is one thing that you can do to leave without authority.
The other one they mentioned was about business. Organizational understanding is the third one. That is very important. That, you know, one of the things I will, as I said, you know, I have a long career, I worked for 20 years after grad school. I have seen often like researchers resist or don’t want to have a good understanding of how the company makes revenue, earns profit. They’re a little wary of that part of the business. I think it’s also important for us to understand, like, how does the company make profit? Like, for example, if you work for a B2B company, like, I would say that you should have a relationship with the sales team to understand how do they sell, what do they sell, like, what is some of the feedback that they get from customers.
So I think having a little bit of understanding of revenue and profitability is important. Like, for example, like, if you work for a company that’s in the gig economy, which has, you know, which has a marketplace, like, it could be Lyft, Uber or DoorDash, when that is the crux of the business marketplace. That’s how the company makes money. So understanding the dynamics of the marketplace is an important thing that a researcher should do. Let’s say if you are a researcher at Uber and you’re working on the Rider app, you should still have an understanding of the marketplace. Usually, you know, it’s the marketplace is kind of on its own, but irrespective of whether you work on the Driver app or the Rider app, you should have an understanding of how, you know, Riders and Drivers are matched, because ultimately that’s where the secret sauce happens and that’s where the company makes money.
I think a lot of us come from academic background and maybe a little bit of purist background. So maybe it could, this is just a hypothesis and that is where it comes from. I mean, I would say to a certain extent, even I had it, like, early in my career. It took a while for me to realize that understanding revenue, profitability is important. One thing that I would say, like, most researchers do agree on is understanding business priorities and, you know, doing research that aligns with business priority. I don’t see, I have never seen that resistance on that, but when it comes to revenue and profitability, I have seen researchers, like, have a little bit of resistance to that. Like, for example, I have some friends who work at Facebook and Google, you know, how they make money is ads, but many of them did not want to work on the ads team.
Steve: That seems different to me and just thinking about myself. That seems different to me than understanding like, how does the system match riders and drivers? When you say that, that kind of sparks my researcher curiosity. Like I think we would want to know that because what’s behind that secret wall and how do the gears all mesh? But again, my own bias here when you say working on ads is not, I kind of get that feeling too. Like I just think like, ugh, ads. And so to me that seems different. And I don’t know, I’m not trying to pin you down on something because you’re talking kind of subjectively what we each have our impressions. But I think this idea that, to your point about changing without authority, that there are these important things to understand. I guess there’s a difference between wanting to work on Facebook ads as your project and having enough of an understanding of the revenue model, which impacts everything you would ever do research on at Facebook I think.
Mani: So I think both. What I’m trying to get at is understanding the money part of the business is important. Like, and the money part of the business for different companies is different.
Steve: Yeah.
Mani: For a gig economy, it’s the marketplace. For Google and Facebook, it’s ads. So having some understanding of how your company makes money is important for researchers.
Steve: And not to flog a dead horse here, but I feel like companies like Google and Meta or Facebook, it seems like their culture is such that how the company makes money is sort of kept in a separate box and we’re going to come here and work on whatever the latest amazing thing that’s going to change the world Internet through balloons, you know, some amazing project. And so speaking out of my hat here, but I have some empathy for people that don’t want to think about the money because they’re not being sold that as an employee. I don’t know if that’s true. I’m hypothesizing that the company culture kind of keeps those things in separate buckets. But if you work at Lyft or Uber or DoorDash or Meraki, there’s a product and a service that’s much more essential to the conversation they’re having. Again, this is not my direct experience. I’m just kind of — I’m just giving you my biased interpretation of what you’re describing.
Mani: I think just the basic level is probably, I’m sure, like, you know, they keep it under wraps to a certain extent, but just like having a basic understanding and not having an aversion to it is important.
Steve: So when you talk about these factors, right, the understanding the business, so there’s some specificity, the expertise and the relationships, and you talked about Cisco and Meraki being just the level of complexity, the sort of technological and I guess industry specific stuff. Does that complexity become a compounding factor or something in trying to achieve those levels of those three factors that you’ve brought up?
Mani: I think especially for something this technical, having some domain expertise matters. And that goes back to having some, that goes back to the first one that I was talking about, which is expertise. Like having some domain expertise becomes important because if you want to have a meaningful conversation, you know, if you’re doing an interview, like you need to have some basic level of understanding to have a meaningful conversation. I know in research we say that, you know, no question is stupid, but if you have zero understanding, you’ll only have stupid questions, at least in this kind of a domain. So I think having some understanding is important, and that’s what I tell all my researchers.
Like after I joined Meraki, I have this cheat sheet I call “Mani’s Cheat Sheet” about networking, and every time I hear something that I don’t know, I get a version from Google and then I get a chat GPT version, like please explain it to a middle schooler version, which actually is the version that works for me. And that cheat sheet, as I’m like, “Huh, like 20, 30 pages long?” It just keeps on increasing. I’m not going to be an expert on networking. I don’t want to be, but I just want to know a little bit to be able to have meaningful conversations and to be also able to provide, you know, feedback to my team. Like sometimes, you know, when we have an important presentation, like I work with them to figure out like what are the insights, like what are some of our action items, but if I have no understanding of the domain, I can’t do that.
Steve: Can you talk about from the period of time that you came to Meraki, like what that progress has been and what research has gone from to where it is now?
Mani: So Meraki has seen incredible growth in design as well as research in, I would say, in the last one, two years, which is the opposite of where, you know, design as a field and UXR as a field is going at other companies. So we have, it’s also because, you know, as I was saying, like the company has really adopted the double diamond approach. So they see research as being an integral part of how you build products. So that’s one of the reasons why we have seen this big, massive growth in the last one year. And the reason why I came here was, you know, as I was telling you earlier, you know, like Cisco had these two products, like the SaaS product, the on-prem, and the strategy now is to, you know, convert, which is, you know, have like some of the on-products be able to be managed in the cloud.
So they brought these two teams together, the design team for the on-prem side and the design team from Meraki. So that’s how I got hired as the head of UXR because then, you know, the team grew because you had two different teams that merged and the complexity of the business and the complexity of the problems that the team was expected to answer grew because now you had two different businesses that you had to support. So that’s how I got hired and, right, and, you know, our team has grown quite a lot even in the last one year. Like I’ve hired, like just in March, three people have joined the team. And it’s also because, you know, everyone agrees that, you know, you need to do research if you want to build better products. So there is this hunger for research, so that’s the other reason why we’ve been able to grow despite, you know, the market going in the other direction.
Steve: So you came into this newly formed, newly merged organization that already had an appetite for a belief and a commitment to research.
Mani: Yeah, and that appetite has been growing. I would say it’s been a steady state because there are more and more teams that we are working with because Cisco networking is huge. As I was saying earlier, it’s like it’s their bread and butter. It’s the major chunk of their revenue. So there are more and more teams that we are working with and that’s one of the reasons why I’ve been able to hire researchers to support teams that did not do research previously but want to do research now.
Steve: Are there things that you are doing in your role that are behind that? In addition to there being these other teams, but do you think you’re responsible for the increasing the demand in any way?
Mani: I would say it’s a team effort. I would not put — I won’t say that it’s just me by myself. But obviously, you know, as a research leader — or I would say as a research professional, we have to evangelize research as much as we can all the time.
I can talk about my time at SuccessFactors. You know, when I joined SuccessFactors, they already had an IPO. Then they were bought by SAP in 2012 for like $3.5 billion. So the company was pretty successful, but they never had research at that time. And one of my — like what I had to do in the beginning was, you know, just evangelize research and make sure that it became part of how we build products. So in the beginning, I did a lot of like evaluated research to show impact and then, you know, slowly as our team grew, we — I mean, continued to do obviously evaluated research, but we did a lot of generated research as part of new product launches.
Steve: So you talk about that situation, they were bought into research enough that they hired you, but it doesn’t sound like the hunger that you’re talking about now wasn’t there at that time.
Mani: And also it’s like, you know, we’re talking 2013. I know the field has changed quite a lot in that time. But yes, the hunger wasn’t there. So my job was to create a hunger for research. And it was like very different tactics. Like I did a lot of usability research because that gets you — it’s a quick hit. That’s how I would describe it at times, you know, usability research. If you are looking for quick impact, you can get it pretty fast.
And you can get — you can find people who are on your side who become evangelists of research pretty easily. So that’s what I did a lot when, you know, when I had to build a team there because, you know, there was no research team. I wouldn’t say that — I mean, there was no research majority.
Steve: I hear relationships and expertise at least in that story.
Mani: Yes, there was because I remember when I joined, there was some usability testing that had been done. But whoever did it, they did not know how to define usability tasks. So when it was an unmoderated test and user testing, people were very confused about what was expected from them. And it was very clear that you needed somebody who knew the basics of usability testing to have — who should have set up the test.
Steve: Just switching topics slightly, you’ve kind of brought us back in time to some different roles that you’ve had and I wonder if we could go further back and maybe could you talk about how you entered the field of UX research.
Mani: I did not think I was going to become a UX researcher, honestly. Like I have a PhD in sociology. I thought I was going to become a college professor. That was my goal. But I’m — my husband is an engineer. So we moved to the Bay Area. And I looked for teaching jobs in the Bay Area. There were none. So I moved — I went — I got a job at Institute for the Future. I don’t know if you know about them, but they are a forecasting research company. So they do a lot of tech research. And I started working there for five years. And I did not even think the kind of research that I was doing was, you know, user experience research, like product strategy research. But that’s the kind of work that I did there.
One of the biggest clients that we had was Nokia. And I did a lot of ethnographic research, you know, going across — traveling to people’s home in India, U.S., and Brazil, and, you know, trying to understand, you know, how do they use mobile phones? Because this was like 2006, 2008. It was still a new phenomenon. And that helped — that was to help define Nokia’s strategy, especially for these markets for the next one to three years. And from that, I kind of transitioned into UXR. And in fact, you know, when I finished my Ph.D., I did not even know that there was an HCI field I completed in 2004. So it was still pretty early days of — at least from my perspective for the field.
Steve: When you found yourself doing global ethnographic fieldwork for Nokia, did you see any difference between how you were working in that context and the skills and approach that you had developed in your PhD?
Mani: There are cultural differences that you have to be aware of. One of the biggest things is you should be ready for anything. There are a lot of unknowns. I remember, like, we were doing this project, but we were working for Nokia. Like, we had — I was working with our clients, and we were supposed to go and interview someone, and the person didn’t show up. So there’s a lot of those things that you got to be ready for, like, you did not prepare for. Like, very rarely it’s happened to me that, you know, I did ethnographic research in the U.S., and we had no shows. But it’s happened in India so many times. It’s happened in Brazil. So you got to have these contingency plans and go with the flow. And I think those are some of the things that you have to be aware of.
And also, you know, especially, like, when I worked in Brazil, like, I obviously don’t speak the language. Like, I do not — like, for me, obviously, India is easier because I have lived in India. I know, you know, what the culture is, like, what are the things that you do. Like, for example, if you go to India — to anyone’s house in India, they are going to offer you tea, coffee, something to eat, and it’s polite to eat it. And it’s impolite if you don’t eat it. And when I went to Brazil, like, I had to work with translators, which I had never worked with before. So one of my big learnings was that it takes you twice as much to do the same interview because they are translating it for you.
So I think there are a lot of cultural nuances that you have to be familiar with when you go — especially when you do research outside of, you know, U.S. and maybe to a certain extent even Europe.
Steve: And did you see differences at that point in your career where you’re coming from an academic environment to a commercial environment? Were there points of transition for you as you moved from one to the other?
Mani: Yes. In terms of, like, I remember when I wrote my PhD dissertation, I took a year to do the analysis after I had finished my qualitative interviews. So I — and, you know, when I — this Nokia one that I was talking about, like, we were sharing research inside from every interview every day. So I think that is the big transition that you have to make. Like, if you come from academia is the pace. And also thinking about, you know, how — like, how does this impact the business? Like, how does this impact the product? And also working with so many stakeholders. Like, those are some of the big changes. Like, when I was in academia, the only time, like, I worked with others is my advisor provided me feedback for my dissertation. And when I published, I had these three reviewers who gave me feedback for some of the articles that I published. But, yeah, you know, the way you work with stakeholders is so different. And also you got to be, you — okay, we’re doing a lot of quick and dirty work, which obviously, you know, you don’t do it in academia. Especially if you do that, it won’t get published.
Steve: So after this Institute for the Future experience, was user research that was your career path at that point?
Mani: Yeah, and then I joined Wikipedia, which was actually a very amazing experience in terms of what I was able to do because — I mean, Wikipedia is like I have never met anybody who doesn’t like Wikipedia. I, you know, I’ve worked in different — as I said, you know, I have worked in different industries. Often we more than often, like, meet customers who say, like, I don’t like this about your product. I don’t like that about your product. But for Wikipedia, it was very different.
So one of the things that I did when I joined Wikipedia was I did the first ever survey of Wikipedia editors. So Wikipedia is, as you can probably guess, from their mission, unlike most companies, they barely do any tracking. Like, they don’t use, for example, cookies. So we used to rely on ComScore to even get the numbers for active readers for Wikipedia because they were not using cookies. So they did not know at that time, like, what percentage of editors are women because Wikipedia, even today, has a gender problem. So I did the first ever surveys of Wikipedia editors, and the answer at that time was 9%. The only 9% of people editing Wikipedia were women. And as a result of that, a lot of — there are fewer articles about women on Wikipedia versus men. So there’s a big gender bias in Wikipedia. So that is some of the work that I did when I joined Wikipedia, which I thought — and I also worked on the redesign of the mobile app. So I again did ethnographic research in the U.S., India, and Brazil for the redesign of the mobile app for Wikipedia. But all that work was very fulfilling just because of the amount of impact that you could have on users. Because I don’t remember the numbers now, but at that time there were 250 million active readers on Wikipedia every month, and there used to be like 7 to 8 billion page views.
Steve: As a user researcher over your career and maybe as a leader now, I don’t know, how do you think about different kinds of methods or different kinds of approaches to getting the data that you want to bring your team?
Mani: I mean, I’m trained as a mixed methods researcher, and when you think about mixed methods, I like to believe it’s on a continuum. There are some people who do small surveys and think that they are mixed methods researcher only doing descriptive statistics. There are some people who do regression analysis, who will do, I don’t know, structural equation modeling, latent class analysis, who are also mixed methods researchers. So it’s a little bit on a continuum, even for qualitative, right? Like you can be an ethnographer. You can be somebody who really believes in participant observation, or you can just do qualitative one-on-one interviews. With that said, like I do both because I just happen to be trained. When I did my PhD, I specialized in research methods, which meant that I had to prove that I could do both quant and work. So I did a lot of statistics. I did a lot, you know, I took a lot of qualitative courses. And in fact, you know, my dissertation was a qualitative dissertation, despite, you know, having a very deep background in stats. So I do both.
But with that said, I would say you should be methodologically agnostic. Like it should not, depending on what’s the problem that you’re trying to solve, you should figure out what’s the research, right research method. So I think that is what is more important. And that’s why I feel like having a mixed methods background is good, because then you just have more methods in your toolkit. So you can figure out like, you know, for this problem, probably a survey is a good solution. Or for if I want to find the answer to this problem, maybe doing ethnographic research, qualitative interviews is the right thing to do. And, you know, you can use all these methods throughout the process. Like you can do a survey, you know, along with some foundational research.
Like I can give you an example. When I worked at Wikipedia, I’m very proud of the feature that we released. I’ll talk about a little bit about that. So when I worked at Wikipedia, I interviewed this person in Salvador, which is, you know, in north of Brazil, in the Bahia district. And he was a Wikipedia editor. And obviously, like you’re a Wikipedia editor, you’re a big reader too. And at that time, he told me that, you know, he wanted to read Portuguese Wikipedia, but he felt that it was not mature enough. Some of the articles did not have the kind of detail that he was looking for. So he would often flip to English Wikipedia. So I thought that was an interesting thing that he mentioned. So after we did the ethnographic research, we followed it up with the survey and we asked people like, you know, do you, how many languages Wikipedia do you read? And, you know, we got data. Like if you read English, what else do you read? So obviously it was clear that, you know, English is the primary one. But more often than not, people read other language Wikipedia’s too. Especially outside of, you know, US I would say, even in Europe, like German Wikipedia is very big, but the German readers are also reading English Wikipedia. So, and that was also very clear in the survey data that we got. So we introduced, it’s even there today in the app. We introduced what we call the inter-language Wikilinks, which made it really easy to toggle between different language Wikipedia. So, for example, like if you search for an article, it tells you that, you know, this article is also available in this other language. You can set your primary languages, you can set your secondary languages. And all this came out through this, you know, ethnographic research that was done, followed it up with a survey, gave us more confidence that, you know, this would be a useful feature. And it still exists like 10, 12 years after we did that research.
Steve: That’s a great story and one detail that excites me is that small data point from one method. You find new questions in research. You found something and then you did some more research to kind of understand that phenomenon that you didn’t know.
Mani: Yeah.
Steve: That’s such a great example of that. That the ethnography was not sufficient to make something new, but it did point to a whole new effort to understand something, which then led to that feature.
Mani: Yeah, but ethnographic is that insight, which I don’t think I would have got from a survey, right? So it was, the survey just gave us more confidence that, yes, I think that’s worth building.
Steve: Yeah, that’s a good clarification. We’re going to talk about mixed methods and about different kinds of tools kind of feeding together. You talked early on about relationships, so it just makes me think about some of the different roles that we have around data science as a big part of what so many companies are doing. How do you see kind of the relationship between UXRs, mixed methods and otherwise folks and data science?
Mani: I think there’s a lot of synergy between what UX cards do and what data scientists do, because ultimately we are both researchers. It’s just that we are looking at different type of data. But if we bring it together, I strongly feel that we can have a much more holistic understanding of our users. And, you know, in my career, like I’ve had, especially at Wikipedia and Lyft, I was lucky to work very closely and actually even at Meraki, work with data scientists. So there are various ways we can work with data scientists. Like, for example, like if you’re doing segmentation, you know, often we do segmentation of our users just based on attitudinal data. But you can get the behavioral data from data scientists and bring that together to have, to just have a better segmentation model of your users. Various companies, they have dashboards to, you know, look at how their customers are doing and tracking. Many of them, especially in B2C, tend to be more behavioral based versus attitudinal. So I think there’s an opportunity there to have dashboards that not only just tell you what are the users doing, but also how are they feeling. I think that’s an important story to tell. So there’s that one can do.
Then there’s also an opportunity to work with data science during A/B experimentations. Like in my past companies, I have worked with data scientists when they have an A/B experiment to, you know, do a survey along with the experiment to see like if there are any differences between the control and the experimental group on not only what they’re doing, but how they are feeling. And the one thing that I learned is that there’s a little bit of lag. Like if people are unhappy, it takes a little while before they stop doing the thing. So like, you know, doing that survey gives you an early pulse that, okay, maybe this experiment is not working as well as it should. Maybe we should tweak something so that the users are happier. So I think that is one thing. So those are some of the opportunities that I see working with data scientists.
Steve: Is there anything else today in our conversation that I should have asked you about or you want to make sure that we talk about?
Mani: I can’t think of anything else right now. I’m sure I will think of something later.
Steve: Well, Mani, it’s been really lovely to speak with you and learn from you. I want to thank you again for taking the time to chat and share all your stories and experiences. It’s been great.
Mani: And thank you, Steve, for having me on the podcast.
Steve: There we go. Thanks for listening. Tell your friends. Tell your enemies about Dollars to Donuts. Give us a review on Apple Podcasts or any place that reviews podcasts.
Find Dollars to Donuts in all the places that have all the things or visit portigal.com/podcast for all of the episodes complete with show notes and transcripts. Our theme music is by Bruce Todd.
The post 39. Mani Pande of Cisco Meraki first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Vanessa Whatley, UX Director – Research & Documentation at Twilio.
For many years, I had anxiety and regret around not starting my career in the field that I’m in sooner because I felt very very lost stumbling through all of the different fields and roles, and only in hindsight do the dots connect. I’m better at what I do now because I learned the lessons in all of the different jobs. Even something like being an executive assistant, I was able to sit in on more senior leadership meetings, and I really early picked up on short attention span, How do you get your point across concisely, What do they care about? And I think that made me a better researcher right away, even as I was still learning the practice because it taught me something about communication…I think all of those little pieces along the way just shaped how I interact with people and I think has made me better at what I do today. Maybe just know that it’s all connected somehow. – Vanessa Whatley
Show Links* Interviewing Users Workshop at Advancing Research 2024
* Interviewing Users,
2nd Edition
* Steve on Darren Hood’s World of UX
* Vanessa on LinkedIn
* Twilio
* Segment
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. As part of the upcoming Advancing Research Conference, I’m teaching a full day in-person workshop about user research. It’s March 27th, 2024 in New York City. This is a rare opportunity to learn about interviewing users from me in person. You’ll also have the chance to engage with other researchers at different levels of experience in an interactive environment. I’ll put the link in the show notes for the Advancing Research Conference with more info and information about how to register. If you know someone who would benefit from this workshop, then please pass this info along.
The newest version of my workshop makes use of the writing and rewriting I did on the very recent second edition of Interviewing Users, which you should absolutely buy several copies of.
Shortly after the book came out, I had a conversation with Darren Hood for his World of UX podcast. We got into the intricacies of asking questions. I’ll link to the whole episode, but I’m going to include an excerpt right here.
Darren Hood: And the topic of chapter six, the title is the intricacies of asking questions. And I love this because this is probably when I’m teaching people about research, it’s one of the outside of the classroom. When I’m talking to people about research, this is the topic for me that comes up the most. This one particular thing that you mentioned, and I’m going to read another excerpt from the book. And there’s a heading here, “There’s Power in Your Silence.” Oh my God, how many times have I talked to people about this? Steve says, “After you ask a question, be silent. This is tricky because you are speaking with someone you’ve never spoken to before. You’re learning about their conversational rhythm, how receptive they are to your questions, and what cues they give when thinking about an answer. These tiny moments from part of a second to several seconds are nerve-wracking.” And I love that because it’s one of the things that I see and people will ask a question. And it’s funny to watch people grind their teeth when the participant is silent. To watch people, hem and haw, the researcher, wants to help the participant and things get out of hand sometimes. I have seen people jump practically across the table to try to guide somebody because they just couldn’t stand the silence. But the title says it, the subheading there or the heading in the chapter, “There’s Power in Your Silence.” And so I’ll hand it over to you to elaborate on this topic.
Steve: Yeah, and I think you described some of the phenomenon pretty well. And there was a moment in this conversation, I think you, because we’re on video, even though we’re recording audio, we are in video and we’re looking at each other and nodding and doing all the, as best we can over video kind of feedback. There’s a point in which you said, “Oh, I see the gears in your head are turning, Steve, and I’m going to turn it over to you.” And I think, as a trained interviewer, as an experienced podcast, I was like, “You learn what that is.” And there’s been moments where I’m asking a question and I’ll just stop. I don’t need to finish my question. The person is ready to talk. And I’m going to ask a follow-up question. I’m going to ask dozens of follow-up questions.
So if the thing that they want to say is not exactly what I want to ask them about, it’s better for the whole dynamic to have them go and then me follow on and follow on and follow on as opposed to like, “No, no, no, wait, let me make sure you understand exactly what my question is so the information that you give me perfectly conforms to the parameters which I am articulating.” Like that’s not how research works. It’s this sloppy interface between people that kind of goes back and forth. And so you have to understand your role is to kind of draw that out of them. And so that hem and haw thing, or even trying to help them, as you say, I think is really important because if you can’t allow for that silence, the anti-pattern or the bad behavior that comes out is asking these run-on questions. And the run-on questions are deadly. And in those run-on questions, people start suggesting, so you’re going to ask what could be an open-ended question. Like what kind of microphone do you have for your video calls? But the run-on question is what kind of microphone do you have? Is that a USB mic or is that a Shure microphone or is there that part of your headset? Like you start suggesting possible answers.
Darren: Yep, yep.
Steve: And the motivation for doing that, I think there’s a lot of, you have to pay attention to yourself. It feels like you can kid yourself that you’re being helpful when you do that. I’m being helpful. I’m just showing them what examples are. But in fact, it’s because you are, and I shouldn’t say you, I should say we, this is, I’m in this all the time. It’s uncomfortable to stop and just say, what kind of microphone is that? For all the reasons in that quote that you described, like, I don’t know what’s going to happen. I’m going to lose space. I’m going to be seen as an idiot. That person’s not tracking with me. My boss is going to watch this video. There’s all this risk in that moment. It’s kind of like a little, it’s a little abyss that you’re kind of beeping into. But if you start suggesting things, it messes up the power dynamic. It says that for one thing, the participant is required to listen to the, to the interviewer going on and on and on.
Darren: Right.
Steve: And also starts to say over time that their answers should be, it’s multiple choice question. So you participant should be giving answers within the format that I have outlined. So you might think that’s ridiculous, Steve and Darren. If it’s none of those mics, the person’s just going to say, no, this is just an old karaoke mic that I brought up from the basement. They’re going to give you an answer that’s outside that list. And the first time they will, and maybe the second time they will, but eventually you are training them as to how to do a good job, which they want to do. They want to do a good job for you. And so they’re not being squelched from sharing their truth about microphones. They’re just trying to get through this interview and do a good job.
So the more you teach them indirectly what a good job looks like, in other words, one of the following, then the more you risk not hearing from them and not kind of getting stuff. And you don’t, we don’t realize the power we have over people despite being kind and self-deprecating and telling them at the beginning, I just want to hear from you. Just tell me your truth. And, you know, there’s no wrong answers. You can do all that. It doesn’t matter once you start training them what good looks like, then that’s what you’re going to get. So I think it’s, it’s hard. And the quote you read kind of explains why it’s hard and we trick ourselves that we’re helping. So that makes it harder. The risk in this, I think is significant because it accrues to changing the dynamic and the interview and changing what it is you’re going to hear.
Darren: Yes, yes, absolutely.
Steve: Again, that was Darren Hood’s World of UX podcast. Now let’s get to my conversation with Vanessa Whatley. She’s the UX director of research and documentation at Twilio. Vanessa thanks so much for being on Dollars to Donuts.
Vanessa Whatley: Thank you for having me.
Steve: It’s really great to have you here. So I’m going to just do my cliched opening. The only thing that’s really yet effective, I think, is just to throw it over to you right: away and ask you to give a bit of an introduction to yourself.
Vanessa: Sure. My name is Vanessa Whatley. I lead our research service design documentation team at Twilio. So my title is UX director, but I have all the UX functions outside of design is how I like to explain it.
Steve: And what’s Twilio for those of us like me that don’t know?
Vanessa: Yeah, great question. Usually people think it’s Trulia, like the real estate company. So we’re not that. Twilio is a communications API company, and specifically the portion that my team works on is second So that was a product that was acquired by Twilio a few years back, and we are essentially a customer data platform.
Steve: What companies or people or roles use Segment and what do they do with it?
Vanessa: Yeah, we are B2B, so lots of large as well small SMB companies use Segment to essentially get a better picture of their data. So a lot of times companies are collecting data in a lot of different tools, a lot of different systems, and then it’s siloed and they have a hard time reconciling. So if you are, you know, Steve on the mobile app and Steve on the website, and then you’re interacting by replying to an email, you might look like three different people. And so Segment really helps bring all of the data sources together, unify it into a single profile, and then from there it lets companies interact with you in a, I guess, a more intelligent way because they know who you are as a single person rather than three different IDs.
Steve: Can you briefly put Segment in the context of sort of the larger set of things that Twilio as a company does?
Vanessa: Yeah, so if you think about Segment as kind of the data layer, so that can be your foundation of you understand who your customer is really well, what they’ve done with you in the past, or even predict what they might do with you in the future. And then from the data piece, a lot of companies want to actually activate on that data. So they might want to send you a text message or send you an email, and so the rest of the Twilio portfolio kind of has more of the communications APIs so that you can go on ahead and like use that data to actually engage your customer. Another good customer example could be someone like a DoorDash where DoorDash is trying to connect a restaurant, a driver, and the person who ordered the food. Instead of them building everything from scratch in their app, the communication piece of me being able to text a driver without having to give my own number and the driver seeing my number and vice versa, you can use an API so that they can communicate without DoorDash as an app having to build all of that native into their platform.
Steve: But so if you’re a company like Twilio and just thinking about, you know, user research point of view, you’ve got those kinds of users that you’re describing in like that DoorDash example, the end user and the driver, the food purchaser and the driver. But you also have the — I guess it’s some kind of IT or development team that’s using Twilio tools to build that so that their end users or their drivers can all communicate. Are you — from a research point of view, where are you focused, if at all, on any of that?
Vanessa: Yeah, so on the Segment side, like I said, we’re more so the data piece that is powering a lot of different things. And so you’re absolutely correct. A lot of our customers end up actually being either the data team or engineering team within a company because they’re the ones that are essentially most likely collecting the data, manipulating the data so that there’s protocols and that it’s actionable.
And then ideally it goes all the way through to a business use case. So that might be a product manager or marketer then making decisions on that data set and deciding, okay, we want to run a marketing campaign or we want to analyze this cohort or this audience.
Steve: I want to go back to the beginning and you were describing sort of the structure a little bit or the areas of the organization that you’re focused on. And I’m sorry, you had a great catchphrase, which I should have written down because now I’ve forgotten it. Can you go back to that?
Vanessa: Oh, I was saying everything in UX outside of design. Yes.
Steve: Okay.
Vanessa: So there is a different person that manages our full design org, but content design, service design, technical writing, and research all sit within my team.
Steve: Can you say a little bit about the research team?
Vanessa: Yeah, so we currently have a team of about five and the makeup of that team has changed a lot over time. So we’ve grown with layoffs. We unfortunately lost a few people on the team but overall I think as the size of the team changed, our operating model has kind of shifted along with that. So we started out being a little bit more embedded and really aligning each researcher to a specific area or product area features. And then I would say last year we really decided to go all in on more foundational work and take a little bit less of our demand from product to try to answer like larger strategic questions. And now we probably sit somewhere in the middle where we do a mix of product work as well as foundational work.
Steve: I was expecting that embedded and centralized would be the sort of contrasting terms, but as you’re kind of relaying it, I kind of hear you. I think you’re contrasting embedded and foundational. And maybe you could explain what those endpoints look like as you’re kind of moving between them.
Vanessa: Yeah, I think you’re right. I never really thought about the fact that usually people say centralized. I don’t love centralized research orgs. I’ve worked in that manner before, but I’m a little hesitant to call it that with how we’re currently structured because oftentimes, at least my experience, I’m sure there’s many ways to do centralized, but in my experience that often means it’s a little bit more like an intake request type of situation where you act more like an internal consultant almost and you can cover a lot of different breadth.
And what I try to do with my team and why I call it foundational is a lot of times the researcher might have still had a focus area that they are stronger in or just cover an entire flow, like a user journey and sit within that part of the product. Or maybe they specialize in a set of personas, but they’re not necessarily bouncing around to any project that comes up and we’re managing bandwidth that way. It’s a little bit more driven by where we think product strategy is going to go and then trying to still align people to spaces that they can gain deeper knowledge in just because of the complexity of our space too. It’s really, really hard to bounce around and be the expert in the marketing persona, but then also the data engineer. And yeah, I think that’s why I use those two terms even though they’re not actually polar opposites per se.
Steve: So centralized might mean an intake process, which is challenging if that means that anyone gets assigned to anything. I’m kind of steamrolling over the nuance that you were depicting to kind of check and see. Because I feel like when you’re talking, there’s sort of a couple of aspects. One is like what projects are we going to do? And the other is who’s going to do them?
Vanessa: Correct.
Steve: So I don’t know. Are you — does the idea of intake, is that in itself limiting or something that you would try to avoid?
Vanessa: I think it’s a mix. We definitely talk to all of our stakeholders and try to understand what feature level work and even what foundational work do they want us to kind of produce or participate in, collaborate on. But I actually sit down with my team every quarter and we end up doing anywhere from like 90 minutes to two, three hours of brainstorming where I really encourage them. What are the gaps that you see? What are kind of like big strategic questions or areas where you feel like there hasn’t been enough emphasis or we’re not connecting the dots properly because I do think the risk of operating at the feature level means everything’s a little bit more siloed. And as we know, customers experience things in a series of steps or flows or have an entire journey they need to navigate. And so I just try to position my team so that it can really think at that level.
And I find that sometimes when we do more of the intake model from design and from product that they tend to focus on their scope and their area of focus which might more so sit at the feature level than it does cross-product.
Steve: Right. So it’s the proactive versus reactive aspect. So people that ask for help from research that aren’t — that don’t know about research as much as you do or your team does are going to ask for the problems to be solved that they think research can help with. But if your team brainstorms, here’s what we’re seeing, here’s where the gaps are, here’s how we can get ahead of what’s going on, then — so now I think the more we talk, the more I understand kind of how you — why you characterize that as foundational.
That it’s not reactive, feature level. And you haven’t said this, but I feel like when those questions come, sometimes they come late or when you do that intake model, there’s other ways that you could have helped if you were, like you said, reaching out to those stakeholders and talking about what they’re doing and how you can help them. And you’re saying that you’re kind of in the middle now, you’re somewhere in between if embedded in foundational or sort of endpoints, you’re kind of in the middle right now.
Vanessa: Correct. Because I think at the end of the day what I’m trying to balance for is impact. And so there are some, I guess, areas of the product or even feature level things that we know are very critical for us to get done this year. They’re highly complex. They need someone that is thinking in that space day in and day out. And so there are often one or two researchers that are embedded in those spaces. And then there’s broader strategic questions that maybe you’re not getting directly from the PMs.
Maybe that’s coming even from senior leadership where they’re thinking about the landscape and where do we need to go and broader, less scope, less defined questions. And so some of that will be, I guess, covered by us or we’ll just create bandwidth for so that we can operate at those different altitudes.
And then we’ve also kind of launched a bunch of internal programs so that the feature level work that does need to get done can still be supported. So we have office hours. We have rolling research. We encourage designers to do their own research or PMs to do their own research. I know that is often a hot topic in the industry. I think we try our best to make sure all of the things that would benefit from a researcher’s kind of attention actually gets that attention. But then we also try not to gatekeep due to the size of our team. We’re just not able to get to everything.
Steve: I want to follow up on that, but I want to just go back to the more you’re describing. I’m having another reflection, I guess, because you started off saying that you started off in one mode and then you kind of shifted to another and then you made another shift. And, you know, as people try to ask, like, what’s the right model, you know, to hear how over a fairly short period of time, you know, you have — you’ve iterated or evolved and that it makes me think that the answer — there’s so many — there’s so many it depends on, you know, what model to have, and it depends on your company, depends on your team. But also that these are things that change and that there’s no reason to pick one and stick with it, but to adapt as it sounds like you have to changing conditions and right, you know, in another X amount of time, you might go back to fully embedded because the company’s here or your — I guess other factors like your team or other changes in the strategy might lead you to choose a different model.
Vanessa: A hundred percent. I think those are some of the deciding variables. Team size. So at one point the team grew to 12 researchers, which is why we were able to embed because we had enough to go around almost. And then when we reduced in size, that meant, okay, we need to find the highest value areas. And then I think, yeah, company strategy, thinking about whether your product is in a place where it’s more stable or if we’re in a place where we have a lot of pressure to innovate. All of those things matter. And kind of I think I took into consideration on how can we work to still make it sustainable for the researchers too. Because of course we could have played the volume game and tried to crank out three to five projects every quarter.
But I’ve really been emphasizing let’s choose quality over quantity. And we need time for deep work and for thinking, even if that means you’re cutting down to one to two projects for the quarter and you’re spending much more time synthesizing across past work and doing more foundational or longitudinal work. And luckily we’ve been very much supported by our cross-functional partners and leadership because I know for some companies it’s like, no, everything needs to be tested before it ships. There’s many different reasons why you might get blocked from something like that. I think we’ve been able to answer questions and show areas where we should pivot because of how we choose to work and the types of insights and the clarity that that provides
that we’re continuing to see support and we’re not really being forced to play the volume game.
Steve: Is there anything you can point to that helped you establish that footing where there is that support?
Vanessa: I think a project I talk about internally often happened about a year and a half ago where it was my first time to really decide to go rogue a little bit and just grab two people on the team and say this is the area that I think we should investigate and publish research around. And then because I think with research sometimes there’s an ask or there’s like a push model. Like no one asked for this, but we’re telling them anyways. And so I think about a year and a half ago was the first time I did that for a larger scale project and we just found different avenues to communicate the information. Essentially the more we got it in front of the relevant stakeholders and leadership I think the more the problem became clear and people were bought in and then over time we also saw in one particular area like oh the quantitative data starting to support that story too because it was a newly launched product. And so we were almost ahead of the trend in being able to point out here’s some of the challenges we’re going to encounter. And I think just having a few case studies like that helped us prove the value and kind of earn the respect, get the traction for having that flexibility. Of course that could have backfired. It could have not been received well, but I think in that particular instance that to me was almost my personal proof point for we should keep going down this path and like helping also me gain confidence that this is the right path to take the team down.
Steve: The path in this case is referring to what kind of work and how you’re helping the company.
Vanessa: Exactly. The path is essentially choosing our own topics to go investigate even if no one’s asking for the work. And I mean there’s still ways like we don’t go away for three to five months and then come back and like ta-da we have something cool to show you. There’s still ways to gain buy-in along the way so we are doing our due diligence by crafting a proposal, shopping it around, seeing who at the company could be a good stakeholder to actually implement some of the changes that we’re suggesting. So I’m not saying just go rogue and hope it works out, but I guess it’s more so again the proactive versus reactive model like really taking ownership and saying we actually have things that we think are really important to go after and then advocating for that and pursuing it.
Steve: And in that first example, the first Rogue project a year and a half ago, I heard you talk about, you know, finding this area and choosing to spend resources and people’s time to go do it, but then you, I think also are highlighting, communicating that to some group of people to kind of highlight it. Yeah. Is there a way to sort of compare the proportion of effort in the — and I don’t know if it cleanly breaks this way, like doing that research and communicating that research?
Vanessa: Yeah, I don’t know if there’s a split. I will say for research that other people ask for, the effort and energy to advocate for it is way less. Like if you’re being pulled into something then people are organically just going to show up more, be interested in the findings. So I think there’s a lot more effort and design up front that goes into really making sure that what you do learn is strategically still a place that we can go. Because I think that’s the other thing you want to make sure of, right, is you don’t want to pursue a project and either everyone already knows the information or people didn’t know the information but were already locked into whatever plan when it comes to the product.
So I think the up-front work is really important and takes a lot more energy as well as the communication of findings because now you have to create your own forums and audience whereas other work just is very organic. Like yeah, you go to the product manager who owns this feature.
Steve: I’m hearing in your answer, and now the second time you’ve kind of explained this, that my question was flawed. My question was kind of about research and then communication, but you’re really emphasizing, even when it’s a study or especially when it’s a study of your own sort of discovery or advocacy, there’s that upfront due diligence. There’s doing the research and there’s the communicating. But you’re doing all those three pieces and I guess just to repeat what you’re saying, when it’s a rogue-style project, the upfront and the after part are significantly more effort.
Vanessa: Yeah, I would say so. And I’ve encouraged my team to do this across any project but I think we also do more work to design the artifact ahead of time to think about how this information could be presented or even just have something tangible for someone to react to before we kind of double down and we’re like, again, for foundational work it might be a much higher end than like testing a little feature and so we don’t want to be 15 plus interviews in and realize like, oh, this is not going to work out or people already know this.
Steve: So what kinds of things — you’re talking about what kind of output you create as a result of the research? What kinds of things might your team create as output?
Vanessa: Correct. Yeah, I think a lot of the traditional artifacts, so we do create a lot of decks. I think beyond decks we work in Figma a lot so we try to prototype different styles of outputs. So sometimes it might get really visual and we’re trying to bring in more graphs and charts or if we’re doing persona work, designing templates and stuff ahead of time to think about what data do we want to collect along the way. I mentioned we also have a service designer on the team so sometimes that is like a full-blown journey map where we’re bringing in all of the layers. We’re bringing in the product touchpoint and like the external guides and people like touchpoint when they’re talking to a salesperson or AE, account executive.
So I think we try to remain pretty open, sometimes try to get creative in terms of, you know, what the medium should be. Should it be pre-recorded? Should it be video? But I think, yeah, we really try to do some like content design ahead of time to also use that as part of the conversations when we shop around a project.
Steve: So to clarify, this is happening in the upfront portion of the project, so you’re thinking about what the output might be in order to have these conversations with people about this research, which you have identified as important, but they haven’t asked for. So what are you showing them? These are sort of — these are samples of what a deliverable might contain, but there’s no — there’s nothing in it because you haven’t done the research yet.
Vanessa: Yeah, either there’s nothing in it and it’s essentially like a template just to show like placeholders of the type of information we can show. But sometimes again we’re not starting from scratch. Like the reason we’re initiating this project is because we kind of have like sprinkled evidence popping up across the team, but we’ve never deliberately investigated this area. And so part of it is almost like an early synthesis of like we have a hunch because we saw this, we saw this.
And so sometimes the story just emerges from that already where at high level we can kind of come with an outline of like, okay, these are the group of people. We think they’re experiencing this problem. So of course that’s not like how you do research all of the time, but I think it could be very effective to do that some of the time, especially in an environment where it’s like people are looking to make decisions. If our research is just like, oh, that was cool and people move on, then we didn’t really do our job that well.
Steve: So it’s interesting. It’s almost like a — I don’t know, like a trailer for a movie or something. Like, here’s the decision you’re going to be able to make. Here’s where we are today. If we do this, then we can fill in these gaps. And so you’re not selling them research. You’re selling them the thing that they care about, the decision they need to make.
Vanessa: Correct.
Steve: Way back when I said I wanted to go back to something and then we got into this interesting thread, you said that part of what your team does is, you know, help other cross-functional partners and folks do their own research. And I think you said this is kind of a hot topic right now. But yeah, what’s been effective for you? What have you seen work well?
Vanessa: That’s a good question because sometimes it still scares me too. I think the things that have worked are especially stakeholders that have previously either had experience with research or have been close to previous work that we’ve done, tend to already have a better sense of like how to go about it. We’ve created templates just around like here’s how to write a study guide. And the ideal version of that would be they take a stab at it and then bring it to something like an office hours or talk to a researcher they know so that we can at least coach them a little bit on like are these the right questions and like is that the right set of people that you need to talk to? How do you recruit properly? Things like that.
And so I think if all of the setup goes well, I sometimes have like less concern about the actual sessions themselves if they’re able to just follow a script. I think the analysis piece then again becomes a little bit risky in terms of how people analyze, synthesize information. And I think we could probably do more there internally to like guide people through that process because I think it’s just yeah, challenging if you haven’t experienced it often enough or I’ve seen people overgeneralize or grab that quote that supports exactly what they want to push or need to do and kind of ignore the rest. So that’s why it’s definitely mixed, my feelings towards it. But at the same time, I have seen it also be very useful when it’s something we just can’t support or take on and they do have kind of the proper resources and guidance to get to some of those answers themselves.
Steve: size and availability of your team weren’t an issue, what would you do to help, you know, people who do research be effective with analysis and synthesis?
Vanessa: Oh, good question. I think I would probably be in the form of a workshop or just live debriefing. Multiple jobs go. We used to do a lot of that as in groups like in person with sticky notes and really go deep on that front. I think now that everyone’s remote, we try to do that more in digital tools. But I think having more guidance around how to approach that or even providing them with a framework or a plan on what notes do you want to capture, how do you properly set that up and capture them, especially the more participants you have. I think that’s where I’ve seen lots of people waste lots of time because they didn’t have a plan. They just went through, captured all the data, and then they’re kind of in a now what situation where it’s like, do I rewatch 20 videos? I’m like, please don’t do that.
And so I don’t know. I’ve joked with my team this week that I’ve equated research to party planning, but it’s like the more you plan up front on all the things that are going to be happening and go well and to have a schedule around it and have a plan for how to get to the end, I think the smoother it goes.
Steve: I really like party planning as a kind of a framing. And so yeah, analysis and synthesis is part of the party plan.
Vanessa: Yep.
Steve: I think it’s interesting to hear you and this is not meant to be presented as a disagreement or not, just a reflection that when you think about kind of how to help people move forward with that are not familiar with analysis and synthesis, you’re talking about the, I don’t know, like the tools and tactics of managing that data. That’s I think what I heard you kind of emphasize. I like that because those are things that can be described and be enacted. But there is this part to me of analysis and synthesis that feels, it scares me to, even though I do train people to do this, it still scares me because it feels like it’s creative. I’m even like sheepishly using that word and speaking to you, but it feels like it’s creative and a little bit magic and a little bit hard to describe.
Vanessa: 100%, which is why I still say it’s also the scariest part for me to let people go on that part of the journey. So I don’t know. Like when I was first entering research, I think a lot of things helped structure my thinking, at least, around the difference between hearing something verbatim versus having your own interpretation of it and kind of going back and trying to be a little bit more rigorous around what did you hear, what does it mean, and how do you navigate that? But I love hearing you talk about that it is a creative process because I think, yes, once all of the kind of analysis is done and the data is on the page, understanding what’s important, what story you want to tell, and how to put it all back together is, I think, an extremely creative process because regurgitating everything you heard is not going to work. Like you have to make it compelling and you have to find a point.
And I actually think that’s the thing that most junior folks struggle with is they want to share everything they learned. And like what are your top three things? Because in reality, we can only action probably one to two, if anything. So I’m a huge fan of pushing for prescriptive findings and having ideas around what should happen next.
Steve: Can you explain prescriptive findings?
Vanessa: Yeah. I think so. Rather than having a recommendation like XYZ needs to be clearer on this page or like users were confused by X, which is a little bit more just like describing what happened and where the problem lies, telling the rest of the team, here’s how I think we should solve it. Like, are you saying you should write something differently? Are you saying a human needs to help them? Are we going to introduce AI because that makes it better for them? Like, I think there’s so many different ways to kind of get into solutioning. And I mean, maybe one could argue by just presenting the problem, you’re leaving that solutioning piece open.
And in that case, I would say follow up with a workshop and get a room full of people together to do that in a more deliberate way. But if you’re limiting yourself to a readout and you have reasonable confidence that you actually know the next step forward, I ask people to just say it rather than hold back and just try to be more neutral.
Steve: Yeah.
Vanessa: Yeah, that’s kind of where I land on, being prescriptive. Curious if that resonates with you.
Steve: Yeah.
Vanessa: [Laughter]
Steve: I mean, I think that, you know, you’re kind of getting at what are some of the hot topics in research and I think do researchers give recommendations is a hot topic. At least for me, it feels something I’m sensitive about. But you’re providing some nuance here and you’re kind of giving some of the, it depends. And I think I’m hearing you saying like, if it’s clear what the thing is like what the solution is right.
Then, then yes, there’s no reason to hold back on that. I think where I, if I were to think about my own practice. I would, I agree with you. People are confused by X is, it’s just a description. But I, and again, we’re speaking very about generalities here but I think what I try to do is say, people are confused by X, because they understand this word to mean this and you’re using it in a way that means that.
Vanessa: Yeah.
Steve: So, you know, you can go further and say like if you want people to understand this, you know, the language has to line up. And I think that’s very different than change the label on this button from A to B so people know how to use X, I tend to not. And this is also, I’m a consultant I don’t have the same relationship with the product team that your folks do.
Vanessa: Yeah.
Steve: And I might be more likely to like, to like, give them the whole to decompose it a lot so that they understand, and maybe what to do is obvious. But I think I don’t ever know enough to say, I shouldn’t say ever I often don’t know enough or what all the possible solutions are what the roots of that are. And I want to sort of hand them. You know that thing about helping somebody telling somebody what to decide but making it feel like they’re the ones that are deciding it.
Vanessa: Yeah.
Steve: You know that’s easier for consultant or that’s maybe more appropriate for an outside person for in house in house person.
Vanessa: I love that, though. I think I’ve actually done a mix. Like, back when I was still doing a lot of research myself, I would probably initially land at kind of the fidelity that you’re talking about, of like being very specific about what’s wrong and how they’re misunderstanding it, and even the prevalence of the problem and just really making it concrete of, where are people getting hung up? And then I would just use a little idea, you know, bubble icon, and then separate that and be like, “What idea is this?” So that I’m still getting it in there, but they can at least anchor on, “Okay, this is the finding, and then I’m taking it one step further, but they’re not so tightly coupled.” Because I think if I would jump straight into, “Hey, you need to do this, this, this, and this,” and they don’t have the context behind it, one, it loses credibility, and then two, if I was off, now they don’t have the anchoring problem to actually address it differently. So I kind of like the combo.
Steve: I really like the putting the recommendation or the suggestion in a separate area the, the idea bulb, or the light bulb kind of call out. Because I, you know, I think sometimes what I’m trying to activate is to get them thinking about that transition between research and action like, and that there are, you know, I love your example of the workshop and we can always get the workshop so can we at least put this forward as, for instance, for instance, if we did this this would we’re not saying this is the only way but for instance this would. This is a way that you could use the tools of design or copy or whatever to solve this problem. I mean, I had an experience a year or so ago where, you know, I put some for instances into something for a client that I really wanted them to riff on what was possible because I just didn’t know what was possible.
And I was really torn between like, you know, as a researcher if somebody build something based on what you’ve recommended like, that’s a tremendous win. So I was really proud of that and, but also, I wasn’t right, it wasn’t a recommendation go to x it was like start thinking about solutions in this area. So I had this sort of mixed reaction, you know, in that experience.
Vanessa: That’s understandable, but I agree. I think that’s always like, counted as a success of like, “Hey, my idea made it into the product.” And ideally, due to our background and our proximity to customers, it’s like a pretty legitimate idea. It’s not like we just pulled it out of thin air. So I like that.
Steve: Maybe we’ll switch topics a little bit. You’ve, you talked a little bit about, you know, you’re just talking now about some ways that you practice in previous roles. You know, now you’re in a leadership position and thinking about right driving you’re advocating for the practice. But if we could rewind, however far back we want to go but you know, can you talk a little bit about, you know, what was your path to get to the role that you’re in now.
Vanessa: Yeah, my path was definitely not a conventional one, but I think we are in a field where that is often the case. So I think my first real entry into, I guess, being in a tech environment was after I quit my personal training job. So I had a anthropology degree from Berkeley as my undergrad degree, and I did not know what to do with that. I would go to job interviews, and they barely knew what that meant. Someone asked me if that means studying dinosaurs, and I was like, “Not quite. It’s actually the study of people and cultures.” So I struggled quite a bit after college and kind of had like this passion for fitness on the outside, was a personal trainer, and did not love the working hours and how that schedule tends to play out because you have to train when everyone else is not working.
And so I found a temp agency that had this office manager job at a startup called BrightRoll, and within a few months of me being hired by Yahoo. So that was my first exposure on talking to people that were working in product and UX and marketing because I had ambitions of other things but I just did not know what I wanted to do. And then fast forward, I actually worked at another company that got acquired by Capital One, and I was still kind of in an office manager role and then shifted to an administrative role. So I ended up becoming the executive assistant to a product VP and that was really where I figured out that UX is what I wanted to do because I essentially told them I wanted to learn the business. I wanted to be close to the work that was happening and they had a very heavy design thinking culture. So I was participating in trainings and design sprints and home visits and just like really being immersed in UX and decided that research felt a little bit more aligned with my undergrad experience kind of in anthropology.
And so about a year after doing that, still at Capital One, I was able to connect with a few folks in UX and get a research operation job. I wanted to jump straight to research but the head of the department said you need a master’s degree for that. So I ended up doing a lot of research operations which gave me a lot of proximity to research and pursuing my master’s at the same time. So I was, you know, doing a lot of the recruiting, writing screeners, managing our tools, like procuring new tools. So really it was like all of the research. I almost felt like a research assistant in many ways while I was getting my master’s in human factors and information design and then from there once I did have my degree I formally became a researcher at Capital One. So there I was working a little bit on the mobile app and then on the website Capital One dot com. So that was like a really fun time in my career and then after when I ended up at Google I switched very quickly from being an IC to kind of first managing a few contractors to managing a qual team to also managing folks that were quant researchers to now my role kind of expanding even beyond research.
So it all kind of fell in place very, very quickly as I yeah, kind of navigated my career and didn’t spend as many years doing IC work as I expected. Like a lot of the time when I was at Google I was kind of in a hybrid role so I was still conducting my own research, managing a few vendors that were doing kind of consulting projects for us and then had a team.
Steve: How did you learn the managing part you kind of moved into that and that’s a different skill set than what you had been got your master’s degree and I’m assuming. How did you learn that.
Vanessa: Yeah, I think very, very different skill set. But I think I was gravitated towards that. Like even when I was younger I think kind of leadership came natural to me like in high school I was like president of this club or captain of the volleyball team so I always kind of gravitated towards that and I was a tutor in high school for many, many years. So I think there was always the spirit of like helping people, coaching, collaborating that came natural to me and then on top of that I think on the job training of just like understanding how to navigate different situations.
I mean I don’t think I could have ever imagined the situations you find yourself in because of course there’s performance related stuff. There’s stakeholder related stuff, politics, and then there’s personal stuff and it all kind of comes together and then there’s the actual like looking at your team as a whole are we actually operating and performing in a way that is like helpful to the individual but also to the company and so I feel like there’s a lot of different variables to juggle, but I would say I kind of picked it up from like observing my former managers, realizing what wasn’t working for me, and then just like workshops, trainings, and on the job experience.
Steve: How did you distinguish between management and leadership were sort of using both those words in this conversation I wonder, do you have a definition or an explanation for either.
Vanessa: I mean I guess I tend to agree with like the ones you typically hear where it’s like anyone can lead or be a leader and then management is kind of described more as like a distinct set of facilities, so right now I’m using them pretty interchangeably but I know there are formal distinctions on how that works.
Steve: When you and I talked in anticipation of having this conversation. One thing that you brought up was both an inclusive research practice and things like inclusive hiring and I guess inclusive as a big term here, but it seemed like you had had some experience.
I had some experience with that and some perspective on that, not just a Twilio but I think throughout your career and I’d love to hear you illuminate some of what you’ve done and how you approach it.
Vanessa: Yeah. I’m glad we’re touching on this topic. I think kind of inclusive research, workplace, all the things has always been really important to me. Unfortunately I think in 2024 we still see lots of forms of discrimination whether intentional or not. I think it’s a big conversation again now that all of the companies are thinking about AI and thinking about okay how are these data sets, like where are they coming from, like how are models being trained, and a lot of it is always looking back, and so I think just throughout my career I have been very mindful of the folks that are marginalized whether it’s like in the workplace or as consumers of certain products that you know sometimes things are not designed with women in mind or people of color in mind or certain disabilities are not thought about which often leads to problems for again the people who are supposed to be building these products for but then also I think in the workplace like I have seen these things manifest and I think these things go hand in hand as people from different backgrounds or different walks of life. I think there has been so many studies now published on you know having a more diverse team actually leads to more creativity and better solutions because you’re able to see things from different angles and bringing all those perspectives together kind of creates a better outcome and so yeah that’s just been something that has been important to me throughout my career, that I have been mindful of with any of the teams that I have been on that we take that to heart and also design our research to reflect that.
Steve: So when you say design your research, that makes me think about sampling but that may be a very small view on what you’re talking about, how does it show up when designing a project?
Vanessa: Yeah, I definitely think sampling is a big part of it. I think where tech companies are located it’s easy to do convenience sampling and just say you know we will do something here in a lab. Luckily now remote research is more popular but back in the day where like a lot of physical labs were being used to you know test an app or show a new site that you just don’t want out there on the internet. That often meant you know you are sampling a geographic area where the income might be higher or there’s a certain distribution like whether it’s a gender ratio or a race ratio that might not reflect the full population. I think being very U.S. centric is often a thing with companies and research that we do and there’s a lot of additional hurdles sometimes to be inclusive. Like if you want to do a study that is conducted in people’s native language and you’re an international company there goes many dollars for recruiting and translation and kind of the scale that it now takes rather than doing something that’s local in your area. So that’s kind of what I mean by like the design, but then it can also mean you know all the way down to who is actually conducting the research. Again like if it has to be a native speaker or if it’s better with sensitive populations who might have mistrust when it comes to testing medical products or things like that like it all then requires a different level of planning and consideration in order to also make the participants feel comfortable and to collect that data in a way that’s respectful and yeah I mean I think I could go on and on. That’s kind of what comes to mind for me and what I have kind of pushed for and I think for the most part there hasn’t been any push back on whether it’s the right thing to do per se, but it really comes down to then the timing constraints and the financial constraints that push a lot of companies to say you know what? Not right now. So I have been a pretty strong advocate for when it makes sense to let’s take that time and kind of broaden our scope in order to make sure it works for the broader set of people, not just the convenient sample.
Steve: And you made a point about right diverse teams are more creative and then even thinking about who does the research.
Vanessa: Yeah, definitely. I think right now I’m in a context where we’re actually B2B and so I would say some of those factors have been a little bit less prominent for us and we tend to you know kind of go for the people who are using the product and try to just like expand our sample to that, but I’m sure that’s your experience too with B2B often you just have a way smaller population than in consumer where you have often hundreds of thousands of people you can contact like B2B might be. You have a list of 5,000 and by the time you add a few criteria now you have like 30 people that qualify. So that makes it a little tougher, but I think in the past when I’ve worked in SMB spaces or consumer type products then that has been more of a consideration and I think that’s also where vendors have sometimes come in for us where maybe we’re not the appropriate set of people to have this conversation or we might miss some of the nuance that exists in this conversation, so we’ve like when I was at Google we definitely would get outside help to kind of round out some of those conversations or kind of share like the interviews across a native Spanish speaker if we don’t have that on the team.
Steve: Does this come into how you approach hiring.
Vanessa: That’s a good question because I think with hiring I’ve not deliberately been in a situation where I’m like saying we need this race or this gender or this language because I think that can get very tricky and also potentially lead to discrimination in other ways, so hiring has largely been like merit based, but I think the part of hiring that I still find problematic is often people talk about pipeline issues and not being able to source candidates from different backgrounds and I often find like that there is actually something we can do about that and I think that’s where I focused more efforts is if I’m seeing candidates come through that are a little bit more narrow in kind of let’s say education. They’re all coming from a set of Ivy League schools or they’re kind of like located in a specific part of the country then I have kind of like worked with recruiting to say hey, can we kind of reach different hiring pools? Do you have access to like different communities where you can plug these jobs to kind of diversify the pipeline so we can make sure that at least candidates that have different levels of education or different background can at least have their resumes reviewed, be interviewed, things like that.
Steve: You know, in the time that you spent in your career so far. Have we made progress on these issues like how, how do you contrast what you see 2024. Like you said there’s still discrimination is not gone away.
Vanessa: Yeah. It’s hard to say. I’m not sure if we’ve made progress or not. I think during the pandemic when George Floyd was murdered there was definitely a heightened interest from companies to think about a lot of these topics. So one way that worked out is a lot of companies including Google and I was actually participating in like some of the hiring initiatives was to think about product inclusion specific roles and like really carve out space to say like we have people in the company who are kind of like trained and qualified to deal with maybe like more sensitive populations and then who are also actively putting programs together to like teach the broader company like how do you make sure that your product especially again when it comes to like language or anything AI related or kind of like anything that’s like visual is not discriminating against like oh the photos can’t pick up on darker skin or they can’t understand people with a certain dialect. Like how can we get that education out there? So I have seen a push in that. There was more consulting firms, more job postings that were specifically around product inclusion and bringing that academic knowledge into the tech field, but as far as like broader hiring trends I don’t know the data well enough across the full population, but I don’t know if I have seen like a uptick per se.
Steve: So maybe one last area to loop back to something else that we were talking about it and you described, you know, going from personal trader to attempt job that gave you exposure to, you know, things that led you into you accidentally research. In some ways it seems, I don’t know sort of circumstantial or opportunistic or something like it’s hard, it’s hard to plan for creating the conditions for that. And that’s, you know, what, is there any lesson or, you know, advice to people that that are listening or that you come across in your in your travels anyway about, you know, going from to me they seem like very different worlds to go to going from one bridging from one world into the other, you know, for you is, it was, I think a certain amount of happenstance and having the, you know, having the right lens on it. But I know does this lead to any guidance or advice for other people from your experience.
Vanessa: I think for many years, I had anxiety and regret around not starting my career in the field that I’m in sooner because I felt very very lost stumbling through all of the different fields and roles, and only in hindsight do the dots connect. As you were saying, they seem very, very different, but I think I’m better at what I do now because I learned the lessons in all of the different jobs. Even something like, again, being a personal trainer or executive assistant, being an executive assistant, I was able to sit in on more senior leadership meetings, and I really early picked up on short attention span, How do you get your point across concisely, What do they care about? And I think that made me a better researcher right away, even as I was still learning the practice because it taught me something about communication.
Then even reaching back to personal training, I think that made me a better manager because I like to think of myself very much as a peer to my team, where we are thought partners. They come to me with things, I come to them with things, and because they are all so driven, we don’t really have a lot of issues where I just have to enforce things or tell anyone to do anything.
But circling back to the personal training piece, I think that really put me in a headspace of okay, someone has a goal and they’re struggling with this, or they need support through this. How do you tap into the human psychology of – they want to get from A to B, we want to make it as sustainable as possible, do something that hopefully they enjoy enough that they can do it on their own.
And so really just getting into that mindset of how do I collaborate with another person and find common ways to address a problem and align with them so it doesn’t feel like I’m pushing you or forcing you, we’re kind of going towards this goal together. I think all of those little pieces along the way just shaped how I interact with people and I think has made me better at what I do today. Maybe just know that it’s all connected somehow.
Steve: Yeah. Yeah, I want to normalize part of what you said at least that, yeah, I also had regrets for not starting my career earlier I felt like I was late. And, you know, I’m, but that was long enough ago that many people I know didn’t know me then or wouldn’t know that about me but, you know, I had a complexly directed path to do what I do now. And, yeah, I think you’re right that we’re sort of all products of all of our experiences is one of the things I like about research but I think you’re right to extend it towards management and everything that there are a lot of different paths and then there are a lot of different ways of being. And that, yeah, if I wasn’t who I was I wouldn’t have gone on those paths and I wouldn’t be able to be all the weird mixes of things that makes me whatever kind of researcher or leader or, you know, all the things that you’re kind of bringing up. I like that about, I’ve always enjoyed that about our field that we’re just, we all come from different places. Even over the generations I think there’s still an interesting mix of that and then means like oh the next researcher that you meet, you know, done something that you didn’t know about like, and you get to work with them and like, you know, there’s something about personal training that shows up in an analysis or in a planning meeting or something and you get all these great stories from people. So I enjoy that. But yeah, I just, I started off by talking about myself here that I have, I know what that regret for me is like that I didn’t come into it out the gate early on and had to play catch up in a lot of ways too.
And maybe that’s a great place to kind of wrap up our conversation. Thank you so much for taking the time, sharing your own experiences and your own perspectives on the work that you’ve been doing all the way along and yeah it’s great to get to chat with you. Thank you.
Vanessa: Thank you so much for having me. it’s really an honor. And even when I was telling folks that Dollars to Donuts was coming back on my team everyone was really excited because a lot of us have your books and have listened to you and learned from you so I really appreciate being here.
Steve: All right, well, great. I hope they enjoy listening to you talk about all the great work that you’re doing.
Vanessa: Thanks.
Steve: Thank you. All right, that’s it for today. I really appreciate you listening to this episode. If you like Dollars to Donuts, recommend it to a friend or colleague or post about it on the social medias. You can find Dollars to Donuts in most of the places that you find podcasts. Review the show on Apple podcasts and go to portigal.com/podcast to find all the episodes including show notes and transcripts. our theme music is by Bruce Todd.
The post 38. Vanessa Whatley of Twilio first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Vanessa Whatley, UX Director – Research & Documentation at Twilio.
For many years, I had anxiety and regret around not starting my career in the field that I’m in sooner because I felt very very lost stumbling through all of the different fields and roles, and only in hindsight do the dots connect. I’m better at what I do now because I learned the lessons in all of the different jobs. Even something like being an executive assistant, I was able to sit in on more senior leadership meetings, and I really early picked up on short attention span, How do you get your point across concisely, What do they care about? And I think that made me a better researcher right away, even as I was still learning the practice because it taught me something about communication…I think all of those little pieces along the way just shaped how I interact with people and I think has made me better at what I do today. Maybe just know that it’s all connected somehow. – Vanessa Whatley
Show Links* Interviewing Users Workshop at Advancing Research 2024
* Interviewing Users,
2nd Edition
* Steve on Darren Hood’s World of UX
* Vanessa on LinkedIn
* Twilio
* Segment
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. As part of the upcoming Advancing Research Conference, I’m teaching a full day in-person workshop about user research. It’s March 27th, 2024 in New York City. This is a rare opportunity to learn about interviewing users from me in person. You’ll also have the chance to engage with other researchers at different levels of experience in an interactive environment. I’ll put the link in the show notes for the Advancing Research Conference with more info and information about how to register. If you know someone who would benefit from this workshop, then please pass this info along.
The newest version of my workshop makes use of the writing and rewriting I did on the very recent second edition of Interviewing Users, which you should absolutely buy several copies of.
Shortly after the book came out, I had a conversation with Darren Hood for his World of UX podcast. We got into the intricacies of asking questions. I’ll link to the whole episode, but I’m going to include an excerpt right here.
Darren Hood: And the topic of chapter six, the title is the intricacies of asking questions. And I love this because this is probably when I’m teaching people about research, it’s one of the outside of the classroom. When I’m talking to people about research, this is the topic for me that comes up the most. This one particular thing that you mentioned, and I’m going to read another excerpt from the book. And there’s a heading here, “There’s Power in Your Silence.” Oh my God, how many times have I talked to people about this? Steve says, “After you ask a question, be silent. This is tricky because you are speaking with someone you’ve never spoken to before. You’re learning about their conversational rhythm, how receptive they are to your questions, and what cues they give when thinking about an answer. These tiny moments from part of a second to several seconds are nerve-wracking.” And I love that because it’s one of the things that I see and people will ask a question. And it’s funny to watch people grind their teeth when the participant is silent. To watch people, hem and haw, the researcher, wants to help the participant and things get out of hand sometimes. I have seen people jump practically across the table to try to guide somebody because they just couldn’t stand the silence. But the title says it, the subheading there or the heading in the chapter, “There’s Power in Your Silence.” And so I’ll hand it over to you to elaborate on this topic.
Steve: Yeah, and I think you described some of the phenomenon pretty well. And there was a moment in this conversation, I think you, because we’re on video, even though we’re recording audio, we are in video and we’re looking at each other and nodding and doing all the, as best we can over video kind of feedback. There’s a point in which you said, “Oh, I see the gears in your head are turning, Steve, and I’m going to turn it over to you.” And I think, as a trained interviewer, as an experienced podcast, I was like, “You learn what that is.” And there’s been moments where I’m asking a question and I’ll just stop. I don’t need to finish my question. The person is ready to talk. And I’m going to ask a follow-up question. I’m going to ask dozens of follow-up questions.
So if the thing that they want to say is not exactly what I want to ask them about, it’s better for the whole dynamic to have them go and then me follow on and follow on and follow on as opposed to like, “No, no, no, wait, let me make sure you understand exactly what my question is so the information that you give me perfectly conforms to the parameters which I am articulating.” Like that’s not how research works. It’s this sloppy interface between people that kind of goes back and forth. And so you have to understand your role is to kind of draw that out of them. And so that hem and haw thing, or even trying to help them, as you say, I think is really important because if you can’t allow for that silence, the anti-pattern or the bad behavior that comes out is asking these run-on questions. And the run-on questions are deadly. And in those run-on questions, people start suggesting, so you’re going to ask what could be an open-ended question. Like what kind of microphone do you have for your video calls? But the run-on question is what kind of microphone do you have? Is that a USB mic or is that a Shure microphone or is there that part of your headset? Like you start suggesting possible answers.
Darren: Yep, yep.
Steve: And the motivation for doing that, I think there’s a lot of, you have to pay attention to yourself. It feels like you can kid yourself that you’re being helpful when you do that. I’m being helpful. I’m just showing them what examples are. But in fact, it’s because you are, and I shouldn’t say you, I should say we, this is, I’m in this all the time. It’s uncomfortable to stop and just say, what kind of microphone is that? For all the reasons in that quote that you described, like, I don’t know what’s going to happen. I’m going to lose space. I’m going to be seen as an idiot. That person’s not tracking with me. My boss is going to watch this video. There’s all this risk in that moment. It’s kind of like a little, it’s a little abyss that you’re kind of beeping into. But if you start suggesting things, it messes up the power dynamic. It says that for one thing, the participant is required to listen to the, to the interviewer going on and on and on.
Darren: Right.
Steve: And also starts to say over time that their answers should be, it’s multiple choice question. So you participant should be giving answers within the format that I have outlined. So you might think that’s ridiculous, Steve and Darren. If it’s none of those mics, the person’s just going to say, no, this is just an old karaoke mic that I brought up from the basement. They’re going to give you an answer that’s outside that list. And the first time they will, and maybe the second time they will, but eventually you are training them as to how to do a good job, which they want to do. They want to do a good job for you. And so they’re not being squelched from sharing their truth about microphones. They’re just trying to get through this interview and do a good job.
So the more you teach them indirectly what a good job looks like, in other words, one of the following, then the more you risk not hearing from them and not kind of getting stuff. And you don’t, we don’t realize the power we have over people despite being kind and self-deprecating and telling them at the beginning, I just want to hear from you. Just tell me your truth. And, you know, there’s no wrong answers. You can do all that. It doesn’t matter once you start training them what good looks like, then that’s what you’re going to get. So I think it’s, it’s hard. And the quote you read kind of explains why it’s hard and we trick ourselves that we’re helping. So that makes it harder. The risk in this, I think is significant because it accrues to changing the dynamic and the interview and changing what it is you’re going to hear.
Darren: Yes, yes, absolutely.
Steve: Again, that was Darren Hood’s World of UX podcast. Now let’s get to my conversation with Vanessa Whatley. She’s the UX director of research and documentation at Twilio. Vanessa thanks so much for being on Dollars to Donuts.
Vanessa Whatley: Thank you for having me.
Steve: It’s really great to have you here. So I’m going to just do my cliched opening. The only thing that’s really yet effective, I think, is just to throw it over to you right: away and ask you to give a bit of an introduction to yourself.
Vanessa: Sure. My name is Vanessa Whatley. I lead our research service design documentation team at Twilio. So my title is UX director, but I have all the UX functions outside of design is how I like to explain it.
Steve: And what’s Twilio for those of us like me that don’t know?
Vanessa: Yeah, great question. Usually people think it’s Trulia, like the real estate company. So we’re not that. Twilio is a communications API company, and specifically the portion that my team works on is second So that was a product that was acquired by Twilio a few years back, and we are essentially a customer data platform.
Steve: What companies or people or roles use Segment and what do they do with it?
Vanessa: Yeah, we are B2B, so lots of large as well small SMB companies use Segment to essentially get a better picture of their data. So a lot of times companies are collecting data in a lot of different tools, a lot of different systems, and then it’s siloed and they have a hard time reconciling. So if you are, you know, Steve on the mobile app and Steve on the website, and then you’re interacting by replying to an email, you might look like three different people. And so Segment really helps bring all of the data sources together, unify it into a single profile, and then from there it lets companies interact with you in a, I guess, a more intelligent way because they know who you are as a single person rather than three different IDs.
Steve: Can you briefly put Segment in the context of sort of the larger set of things that Twilio as a company does?
Vanessa: Yeah, so if you think about Segment as kind of the data layer, so that can be your foundation of you understand who your customer is really well, what they’ve done with you in the past, or even predict what they might do with you in the future. And then from the data piece, a lot of companies want to actually activate on that data. So they might want to send you a text message or send you an email, and so the rest of the Twilio portfolio kind of has more of the communications APIs so that you can go on ahead and like use that data to actually engage your customer. Another good customer example could be someone like a DoorDash where DoorDash is trying to connect a restaurant, a driver, and the person who ordered the food. Instead of them building everything from scratch in their app, the communication piece of me being able to text a driver without having to give my own number and the driver seeing my number and vice versa, you can use an API so that they can communicate without DoorDash as an app having to build all of that native into their platform.
Steve: But so if you’re a company like Twilio and just thinking about, you know, user research point of view, you’ve got those kinds of users that you’re describing in like that DoorDash example, the end user and the driver, the food purchaser and the driver. But you also have the — I guess it’s some kind of IT or development team that’s using Twilio tools to build that so that their end users or their drivers can all communicate. Are you — from a research point of view, where are you focused, if at all, on any of that?
Vanessa: Yeah, so on the Segment side, like I said, we’re more so the data piece that is powering a lot of different things. And so you’re absolutely correct. A lot of our customers end up actually being either the data team or engineering team within a company because they’re the ones that are essentially most likely collecting the data, manipulating the data so that there’s protocols and that it’s actionable.
And then ideally it goes all the way through to a business use case. So that might be a product manager or marketer then making decisions on that data set and deciding, okay, we want to run a marketing campaign or we want to analyze this cohort or this audience.
Steve: I want to go back to the beginning and you were describing sort of the structure a little bit or the areas of the organization that you’re focused on. And I’m sorry, you had a great catchphrase, which I should have written down because now I’ve forgotten it. Can you go back to that?
Vanessa: Oh, I was saying everything in UX outside of design. Yes.
Steve: Okay.
Vanessa: So there is a different person that manages our full design org, but content design, service design, technical writing, and research all sit within my team.
Steve: Can you say a little bit about the research team?
Vanessa: Yeah, so we currently have a team of about five and the makeup of that team has changed a lot over time. So we’ve grown with layoffs. We unfortunately lost a few people on the team but overall I think as the size of the team changed, our operating model has kind of shifted along with that. So we started out being a little bit more embedded and really aligning each researcher to a specific area or product area features. And then I would say last year we really decided to go all in on more foundational work and take a little bit less of our demand from product to try to answer like larger strategic questions. And now we probably sit somewhere in the middle where we do a mix of product work as well as foundational work.
Steve: I was expecting that embedded and centralized would be the sort of contrasting terms, but as you’re kind of relaying it, I kind of hear you. I think you’re contrasting embedded and foundational. And maybe you could explain what those endpoints look like as you’re kind of moving between them.
Vanessa: Yeah, I think you’re right. I never really thought about the fact that usually people say centralized. I don’t love centralized research orgs. I’ve worked in that manner before, but I’m a little hesitant to call it that with how we’re currently structured because oftentimes, at least my experience, I’m sure there’s many ways to do centralized, but in my experience that often means it’s a little bit more like an intake request type of situation where you act more like an internal consultant almost and you can cover a lot of different breadth.
And what I try to do with my team and why I call it foundational is a lot of times the researcher might have still had a focus area that they are stronger in or just cover an entire flow, like a user journey and sit within that part of the product. Or maybe they specialize in a set of personas, but they’re not necessarily bouncing around to any project that comes up and we’re managing bandwidth that way. It’s a little bit more driven by where we think product strategy is going to go and then trying to still align people to spaces that they can gain deeper knowledge in just because of the complexity of our space too. It’s really, really hard to bounce around and be the expert in the marketing persona, but then also the data engineer. And yeah, I think that’s why I use those two terms even though they’re not actually polar opposites per se.
Steve: So centralized might mean an intake process, which is challenging if that means that anyone gets assigned to anything. I’m kind of steamrolling over the nuance that you were depicting to kind of check and see. Because I feel like when you’re talking, there’s sort of a couple of aspects. One is like what projects are we going to do? And the other is who’s going to do them?
Vanessa: Correct.
Steve: So I don’t know. Are you — does the idea of intake, is that in itself limiting or something that you would try to avoid?
Vanessa: I think it’s a mix. We definitely talk to all of our stakeholders and try to understand what feature level work and even what foundational work do they want us to kind of produce or participate in, collaborate on. But I actually sit down with my team every quarter and we end up doing anywhere from like 90 minutes to two, three hours of brainstorming where I really encourage them. What are the gaps that you see? What are kind of like big strategic questions or areas where you feel like there hasn’t been enough emphasis or we’re not connecting the dots properly because I do think the risk of operating at the feature level means everything’s a little bit more siloed. And as we know, customers experience things in a series of steps or flows or have an entire journey they need to navigate. And so I just try to position my team so that it can really think at that level.
And I find that sometimes when we do more of the intake model from design and from product that they tend to focus on their scope and their area of focus which might more so sit at the feature level than it does cross-product.
Steve: Right. So it’s the proactive versus reactive aspect. So people that ask for help from research that aren’t — that don’t know about research as much as you do or your team does are going to ask for the problems to be solved that they think research can help with. But if your team brainstorms, here’s what we’re seeing, here’s where the gaps are, here’s how we can get ahead of what’s going on, then — so now I think the more we talk, the more I understand kind of how you — why you characterize that as foundational.
That it’s not reactive, feature level. And you haven’t said this, but I feel like when those questions come, sometimes they come late or when you do that intake model, there’s other ways that you could have helped if you were, like you said, reaching out to those stakeholders and talking about what they’re doing and how you can help them. And you’re saying that you’re kind of in the middle now, you’re somewhere in between if embedded in foundational or sort of endpoints, you’re kind of in the middle right now.
Vanessa: Correct. Because I think at the end of the day what I’m trying to balance for is impact. And so there are some, I guess, areas of the product or even feature level things that we know are very critical for us to get done this year. They’re highly complex. They need someone that is thinking in that space day in and day out. And so there are often one or two researchers that are embedded in those spaces. And then there’s broader strategic questions that maybe you’re not getting directly from the PMs.
Maybe that’s coming even from senior leadership where they’re thinking about the landscape and where do we need to go and broader, less scope, less defined questions. And so some of that will be, I guess, covered by us or we’ll just create bandwidth for so that we can operate at those different altitudes.
And then we’ve also kind of launched a bunch of internal programs so that the feature level work that does need to get done can still be supported. So we have office hours. We have rolling research. We encourage designers to do their own research or PMs to do their own research. I know that is often a hot topic in the industry. I think we try our best to make sure all of the things that would benefit from a researcher’s kind of attention actually gets that attention. But then we also try not to gatekeep due to the size of our team. We’re just not able to get to everything.
Steve: I want to follow up on that, but I want to just go back to the more you’re describing. I’m having another reflection, I guess, because you started off saying that you started off in one mode and then you kind of shifted to another and then you made another shift. And, you know, as people try to ask, like, what’s the right model, you know, to hear how over a fairly short period of time, you know, you have — you’ve iterated or evolved and that it makes me think that the answer — there’s so many — there’s so many it depends on, you know, what model to have, and it depends on your company, depends on your team. But also that these are things that change and that there’s no reason to pick one and stick with it, but to adapt as it sounds like you have to changing conditions and right, you know, in another X amount of time, you might go back to fully embedded because the company’s here or your — I guess other factors like your team or other changes in the strategy might lead you to choose a different model.
Vanessa: A hundred percent. I think those are some of the deciding variables. Team size. So at one point the team grew to 12 researchers, which is why we were able to embed because we had enough to go around almost. And then when we reduced in size, that meant, okay, we need to find the highest value areas. And then I think, yeah, company strategy, thinking about whether your product is in a place where it’s more stable or if we’re in a place where we have a lot of pressure to innovate. All of those things matter. And kind of I think I took into consideration on how can we work to still make it sustainable for the researchers too. Because of course we could have played the volume game and tried to crank out three to five projects every quarter.
But I’ve really been emphasizing let’s choose quality over quantity. And we need time for deep work and for thinking, even if that means you’re cutting down to one to two projects for the quarter and you’re spending much more time synthesizing across past work and doing more foundational or longitudinal work. And luckily we’ve been very much supported by our cross-functional partners and leadership because I know for some companies it’s like, no, everything needs to be tested before it ships. There’s many different reasons why you might get blocked from something like that. I think we’ve been able to answer questions and show areas where we should pivot because of how we choose to work and the types of insights and the clarity that that provides
that we’re continuing to see support and we’re not really being forced to play the volume game.
Steve: Is there anything you can point to that helped you establish that footing where there is that support?
Vanessa: I think a project I talk about internally often happened about a year and a half ago where it was my first time to really decide to go rogue a little bit and just grab two people on the team and say this is the area that I think we should investigate and publish research around. And then because I think with research sometimes there’s an ask or there’s like a push model. Like no one asked for this, but we’re telling them anyways. And so I think about a year and a half ago was the first time I did that for a larger scale project and we just found different avenues to communicate the information. Essentially the more we got it in front of the relevant stakeholders and leadership I think the more the problem became clear and people were bought in and then over time we also saw in one particular area like oh the quantitative data starting to support that story too because it was a newly launched product. And so we were almost ahead of the trend in being able to point out here’s some of the challenges we’re going to encounter. And I think just having a few case studies like that helped us prove the value and kind of earn the respect, get the traction for having that flexibility. Of course that could have backfired. It could have not been received well, but I think in that particular instance that to me was almost my personal proof point for we should keep going down this path and like helping also me gain confidence that this is the right path to take the team down.
Steve: The path in this case is referring to what kind of work and how you’re helping the company.
Vanessa: Exactly. The path is essentially choosing our own topics to go investigate even if no one’s asking for the work. And I mean there’s still ways like we don’t go away for three to five months and then come back and like ta-da we have something cool to show you. There’s still ways to gain buy-in along the way so we are doing our due diligence by crafting a proposal, shopping it around, seeing who at the company could be a good stakeholder to actually implement some of the changes that we’re suggesting. So I’m not saying just go rogue and hope it works out, but I guess it’s more so again the proactive versus reactive model like really taking ownership and saying we actually have things that we think are really important to go after and then advocating for that and pursuing it.
Steve: And in that first example, the first Rogue project a year and a half ago, I heard you talk about, you know, finding this area and choosing to spend resources and people’s time to go do it, but then you, I think also are highlighting, communicating that to some group of people to kind of highlight it. Yeah. Is there a way to sort of compare the proportion of effort in the — and I don’t know if it cleanly breaks this way, like doing that research and communicating that research?
Vanessa: Yeah, I don’t know if there’s a split. I will say for research that other people ask for, the effort and energy to advocate for it is way less. Like if you’re being pulled into something then people are organically just going to show up more, be interested in the findings. So I think there’s a lot more effort and design up front that goes into really making sure that what you do learn is strategically still a place that we can go. Because I think that’s the other thing you want to make sure of, right, is you don’t want to pursue a project and either everyone already knows the information or people didn’t know the information but were already locked into whatever plan when it comes to the product.
So I think the up-front work is really important and takes a lot more energy as well as the communication of findings because now you have to create your own forums and audience whereas other work just is very organic. Like yeah, you go to the product manager who owns this feature.
Steve: I’m hearing in your answer, and now the second time you’ve kind of explained this, that my question was flawed. My question was kind of about research and then communication, but you’re really emphasizing, even when it’s a study or especially when it’s a study of your own sort of discovery or advocacy, there’s that upfront due diligence. There’s doing the research and there’s the communicating. But you’re doing all those three pieces and I guess just to repeat what you’re saying, when it’s a rogue-style project, the upfront and the after part are significantly more effort.
Vanessa: Yeah, I would say so. And I’ve encouraged my team to do this across any project but I think we also do more work to design the artifact ahead of time to think about how this information could be presented or even just have something tangible for someone to react to before we kind of double down and we’re like, again, for foundational work it might be a much higher end than like testing a little feature and so we don’t want to be 15 plus interviews in and realize like, oh, this is not going to work out or people already know this.
Steve: So what kinds of things — you’re talking about what kind of output you create as a result of the research? What kinds of things might your team create as output?
Vanessa: Correct. Yeah, I think a lot of the traditional artifacts, so we do create a lot of decks. I think beyond decks we work in Figma a lot so we try to prototype different styles of outputs. So sometimes it might get really visual and we’re trying to bring in more graphs and charts or if we’re doing persona work, designing templates and stuff ahead of time to think about what data do we want to collect along the way. I mentioned we also have a service designer on the team so sometimes that is like a full-blown journey map where we’re bringing in all of the layers. We’re bringing in the product touchpoint and like the external guides and people like touchpoint when they’re talking to a salesperson or AE, account executive.
So I think we try to remain pretty open, sometimes try to get creative in terms of, you know, what the medium should be. Should it be pre-recorded? Should it be video? But I think, yeah, we really try to do some like content design ahead of time to also use that as part of the conversations when we shop around a project.
Steve: So to clarify, this is happening in the upfront portion of the project, so you’re thinking about what the output might be in order to have these conversations with people about this research, which you have identified as important, but they haven’t asked for. So what are you showing them? These are sort of — these are samples of what a deliverable might contain, but there’s no — there’s nothing in it because you haven’t done the research yet.
Vanessa: Yeah, either there’s nothing in it and it’s essentially like a template just to show like placeholders of the type of information we can show. But sometimes again we’re not starting from scratch. Like the reason we’re initiating this project is because we kind of have like sprinkled evidence popping up across the team, but we’ve never deliberately investigated this area. And so part of it is almost like an early synthesis of like we have a hunch because we saw this, we saw this.
And so sometimes the story just emerges from that already where at high level we can kind of come with an outline of like, okay, these are the group of people. We think they’re experiencing this problem. So of course that’s not like how you do research all of the time, but I think it could be very effective to do that some of the time, especially in an environment where it’s like people are looking to make decisions. If our research is just like, oh, that was cool and people move on, then we didn’t really do our job that well.
Steve: So it’s interesting. It’s almost like a — I don’t know, like a trailer for a movie or something. Like, here’s the decision you’re going to be able to make. Here’s where we are today. If we do this, then we can fill in these gaps. And so you’re not selling them research. You’re selling them the thing that they care about, the decision they need to make.
Vanessa: Correct.
Steve: Way back when I said I wanted to go back to something and then we got into this interesting thread, you said that part of what your team does is, you know, help other cross-functional partners and folks do their own research. And I think you said this is kind of a hot topic right now. But yeah, what’s been effective for you? What have you seen work well?
Vanessa: That’s a good question because sometimes it still scares me too. I think the things that have worked are especially stakeholders that have previously either had experience with research or have been close to previous work that we’ve done, tend to already have a better sense of like how to go about it. We’ve created templates just around like here’s how to write a study guide. And the ideal version of that would be they take a stab at it and then bring it to something like an office hours or talk to a researcher they know so that we can at least coach them a little bit on like are these the right questions and like is that the right set of people that you need to talk to? How do you recruit properly? Things like that.
And so I think if all of the setup goes well, I sometimes have like less concern about the actual sessions themselves if they’re able to just follow a script. I think the analysis piece then again becomes a little bit risky in terms of how people analyze, synthesize information. And I think we could probably do more there internally to like guide people through that process because I think it’s just yeah, challenging if you haven’t experienced it often enough or I’ve seen people overgeneralize or grab that quote that supports exactly what they want to push or need to do and kind of ignore the rest. So that’s why it’s definitely mixed, my feelings towards it. But at the same time, I have seen it also be very useful when it’s something we just can’t support or take on and they do have kind of the proper resources and guidance to get to some of those answers themselves.
Steve: size and availability of your team weren’t an issue, what would you do to help, you know, people who do research be effective with analysis and synthesis?
Vanessa: Oh, good question. I think I would probably be in the form of a workshop or just live debriefing. Multiple jobs go. We used to do a lot of that as in groups like in person with sticky notes and really go deep on that front. I think now that everyone’s remote, we try to do that more in digital tools. But I think having more guidance around how to approach that or even providing them with a framework or a plan on what notes do you want to capture, how do you properly set that up and capture them, especially the more participants you have. I think that’s where I’ve seen lots of people waste lots of time because they didn’t have a plan. They just went through, captured all the data, and then they’re kind of in a now what situation where it’s like, do I rewatch 20 videos? I’m like, please don’t do that.
And so I don’t know. I’ve joked with my team this week that I’ve equated research to party planning, but it’s like the more you plan up front on all the things that are going to be happening and go well and to have a schedule around it and have a plan for how to get to the end, I think the smoother it goes.
Steve: I really like party planning as a kind of a framing. And so yeah, analysis and synthesis is part of the party plan.
Vanessa: Yep.
Steve: I think it’s interesting to hear you and this is not meant to be presented as a disagreement or not, just a reflection that when you think about kind of how to help people move forward with that are not familiar with analysis and synthesis, you’re talking about the, I don’t know, like the tools and tactics of managing that data. That’s I think what I heard you kind of emphasize. I like that because those are things that can be described and be enacted. But there is this part to me of analysis and synthesis that feels, it scares me to, even though I do train people to do this, it still scares me because it feels like it’s creative. I’m even like sheepishly using that word and speaking to you, but it feels like it’s creative and a little bit magic and a little bit hard to describe.
Vanessa: 100%, which is why I still say it’s also the scariest part for me to let people go on that part of the journey. So I don’t know. Like when I was first entering research, I think a lot of things helped structure my thinking, at least, around the difference between hearing something verbatim versus having your own interpretation of it and kind of going back and trying to be a little bit more rigorous around what did you hear, what does it mean, and how do you navigate that? But I love hearing you talk about that it is a creative process because I think, yes, once all of the kind of analysis is done and the data is on the page, understanding what’s important, what story you want to tell, and how to put it all back together is, I think, an extremely creative process because regurgitating everything you heard is not going to work. Like you have to make it compelling and you have to find a point.
And I actually think that’s the thing that most junior folks struggle with is they want to share everything they learned. And like what are your top three things? Because in reality, we can only action probably one to two, if anything. So I’m a huge fan of pushing for prescriptive findings and having ideas around what should happen next.
Steve: Can you explain prescriptive findings?
Vanessa: Yeah. I think so. Rather than having a recommendation like XYZ needs to be clearer on this page or like users were confused by X, which is a little bit more just like describing what happened and where the problem lies, telling the rest of the team, here’s how I think we should solve it. Like, are you saying you should write something differently? Are you saying a human needs to help them? Are we going to introduce AI because that makes it better for them? Like, I think there’s so many different ways to kind of get into solutioning. And I mean, maybe one could argue by just presenting the problem, you’re leaving that solutioning piece open.
And in that case, I would say follow up with a workshop and get a room full of people together to do that in a more deliberate way. But if you’re limiting yourself to a readout and you have reasonable confidence that you actually know the next step forward, I ask people to just say it rather than hold back and just try to be more neutral.
Steve: Yeah.
Vanessa: Yeah, that’s kind of where I land on, being prescriptive. Curious if that resonates with you.
Steve: Yeah.
Vanessa: [Laughter]
Steve: I mean, I think that, you know, you’re kind of getting at what are some of the hot topics in research and I think do researchers give recommendations is a hot topic. At least for me, it feels something I’m sensitive about. But you’re providing some nuance here and you’re kind of giving some of the, it depends. And I think I’m hearing you saying like, if it’s clear what the thing is like what the solution is right.
Then, then yes, there’s no reason to hold back on that. I think where I, if I were to think about my own practice. I would, I agree with you. People are confused by X is, it’s just a description. But I, and again, we’re speaking very about generalities here but I think what I try to do is say, people are confused by X, because they understand this word to mean this and you’re using it in a way that means that.
Vanessa: Yeah.
Steve: So, you know, you can go further and say like if you want people to understand this, you know, the language has to line up. And I think that’s very different than change the label on this button from A to B so people know how to use X, I tend to not. And this is also, I’m a consultant I don’t have the same relationship with the product team that your folks do.
Vanessa: Yeah.
Steve: And I might be more likely to like, to like, give them the whole to decompose it a lot so that they understand, and maybe what to do is obvious. But I think I don’t ever know enough to say, I shouldn’t say ever I often don’t know enough or what all the possible solutions are what the roots of that are. And I want to sort of hand them. You know that thing about helping somebody telling somebody what to decide but making it feel like they’re the ones that are deciding it.
Vanessa: Yeah.
Steve: You know that’s easier for consultant or that’s maybe more appropriate for an outside person for in house in house person.
Vanessa: I love that, though. I think I’ve actually done a mix. Like, back when I was still doing a lot of research myself, I would probably initially land at kind of the fidelity that you’re talking about, of like being very specific about what’s wrong and how they’re misunderstanding it, and even the prevalence of the problem and just really making it concrete of, where are people getting hung up? And then I would just use a little idea, you know, bubble icon, and then separate that and be like, “What idea is this?” So that I’m still getting it in there, but they can at least anchor on, “Okay, this is the finding, and then I’m taking it one step further, but they’re not so tightly coupled.” Because I think if I would jump straight into, “Hey, you need to do this, this, this, and this,” and they don’t have the context behind it, one, it loses credibility, and then two, if I was off, now they don’t have the anchoring problem to actually address it differently. So I kind of like the combo.
Steve: I really like the putting the recommendation or the suggestion in a separate area the, the idea bulb, or the light bulb kind of call out. Because I, you know, I think sometimes what I’m trying to activate is to get them thinking about that transition between research and action like, and that there are, you know, I love your example of the workshop and we can always get the workshop so can we at least put this forward as, for instance, for instance, if we did this this would we’re not saying this is the only way but for instance this would. This is a way that you could use the tools of design or copy or whatever to solve this problem. I mean, I had an experience a year or so ago where, you know, I put some for instances into something for a client that I really wanted them to riff on what was possible because I just didn’t know what was possible.
And I was really torn between like, you know, as a researcher if somebody build something based on what you’ve recommended like, that’s a tremendous win. So I was really proud of that and, but also, I wasn’t right, it wasn’t a recommendation go to x it was like start thinking about solutions in this area. So I had this sort of mixed reaction, you know, in that experience.
Vanessa: That’s understandable, but I agree. I think that’s always like, counted as a success of like, “Hey, my idea made it into the product.” And ideally, due to our background and our proximity to customers, it’s like a pretty legitimate idea. It’s not like we just pulled it out of thin air. So I like that.
Steve: Maybe we’ll switch topics a little bit. You’ve, you talked a little bit about, you know, you’re just talking now about some ways that you practice in previous roles. You know, now you’re in a leadership position and thinking about right driving you’re advocating for the practice. But if we could rewind, however far back we want to go but you know, can you talk a little bit about, you know, what was your path to get to the role that you’re in now.
Vanessa: Yeah, my path was definitely not a conventional one, but I think we are in a field where that is often the case. So I think my first real entry into, I guess, being in a tech environment was after I quit my personal training job. So I had a anthropology degree from Berkeley as my undergrad degree, and I did not know what to do with that. I would go to job interviews, and they barely knew what that meant. Someone asked me if that means studying dinosaurs, and I was like, “Not quite. It’s actually the study of people and cultures.” So I struggled quite a bit after college and kind of had like this passion for fitness on the outside, was a personal trainer, and did not love the working hours and how that schedule tends to play out because you have to train when everyone else is not working.
And so I found a temp agency that had this office manager job at a startup called BrightRoll, and within a few months of me being hired by Yahoo. So that was my first exposure on talking to people that were working in product and UX and marketing because I had ambitions of other things but I just did not know what I wanted to do. And then fast forward, I actually worked at another company that got acquired by Capital One, and I was still kind of in an office manager role and then shifted to an administrative role. So I ended up becoming the executive assistant to a product VP and that was really where I figured out that UX is what I wanted to do because I essentially told them I wanted to learn the business. I wanted to be close to the work that was happening and they had a very heavy design thinking culture. So I was participating in trainings and design sprints and home visits and just like really being immersed in UX and decided that research felt a little bit more aligned with my undergrad experience kind of in anthropology.
And so about a year after doing that, still at Capital One, I was able to connect with a few folks in UX and get a research operation job. I wanted to jump straight to research but the head of the department said you need a master’s degree for that. So I ended up doing a lot of research operations which gave me a lot of proximity to research and pursuing my master’s at the same time. So I was, you know, doing a lot of the recruiting, writing screeners, managing our tools, like procuring new tools. So really it was like all of the research. I almost felt like a research assistant in many ways while I was getting my master’s in human factors and information design and then from there once I did have my degree I formally became a researcher at Capital One. So there I was working a little bit on the mobile app and then on the website Capital One dot com. So that was like a really fun time in my career and then after when I ended up at Google I switched very quickly from being an IC to kind of first managing a few contractors to managing a qual team to also managing folks that were quant researchers to now my role kind of expanding even beyond research.
So it all kind of fell in place very, very quickly as I yeah, kind of navigated my career and didn’t spend as many years doing IC work as I expected. Like a lot of the time when I was at Google I was kind of in a hybrid role so I was still conducting my own research, managing a few vendors that were doing kind of consulting projects for us and then had a team.
Steve: How did you learn the managing part you kind of moved into that and that’s a different skill set than what you had been got your master’s degree and I’m assuming. How did you learn that.
Vanessa: Yeah, I think very, very different skill set. But I think I was gravitated towards that. Like even when I was younger I think kind of leadership came natural to me like in high school I was like president of this club or captain of the volleyball team so I always kind of gravitated towards that and I was a tutor in high school for many, many years. So I think there was always the spirit of like helping people, coaching, collaborating that came natural to me and then on top of that I think on the job training of just like understanding how to navigate different situations.
I mean I don’t think I could have ever imagined the situations you find yourself in because of course there’s performance related stuff. There’s stakeholder related stuff, politics, and then there’s personal stuff and it all kind of comes together and then there’s the actual like looking at your team as a whole are we actually operating and performing in a way that is like helpful to the individual but also to the company and so I feel like there’s a lot of different variables to juggle, but I would say I kind of picked it up from like observing my former managers, realizing what wasn’t working for me, and then just like workshops, trainings, and on the job experience.
Steve: How did you distinguish between management and leadership were sort of using both those words in this conversation I wonder, do you have a definition or an explanation for either.
Vanessa: I mean I guess I tend to agree with like the ones you typically hear where it’s like anyone can lead or be a leader and then management is kind of described more as like a distinct set of facilities, so right now I’m using them pretty interchangeably but I know there are formal distinctions on how that works.
Steve: When you and I talked in anticipation of having this conversation. One thing that you brought up was both an inclusive research practice and things like inclusive hiring and I guess inclusive as a big term here, but it seemed like you had had some experience.
I had some experience with that and some perspective on that, not just a Twilio but I think throughout your career and I’d love to hear you illuminate some of what you’ve done and how you approach it.
Vanessa: Yeah. I’m glad we’re touching on this topic. I think kind of inclusive research, workplace, all the things has always been really important to me. Unfortunately I think in 2024 we still see lots of forms of discrimination whether intentional or not. I think it’s a big conversation again now that all of the companies are thinking about AI and thinking about okay how are these data sets, like where are they coming from, like how are models being trained, and a lot of it is always looking back, and so I think just throughout my career I have been very mindful of the folks that are marginalized whether it’s like in the workplace or as consumers of certain products that you know sometimes things are not designed with women in mind or people of color in mind or certain disabilities are not thought about which often leads to problems for again the people who are supposed to be building these products for but then also I think in the workplace like I have seen these things manifest and I think these things go hand in hand as people from different backgrounds or different walks of life. I think there has been so many studies now published on you know having a more diverse team actually leads to more creativity and better solutions because you’re able to see things from different angles and bringing all those perspectives together kind of creates a better outcome and so yeah that’s just been something that has been important to me throughout my career, that I have been mindful of with any of the teams that I have been on that we take that to heart and also design our research to reflect that.
Steve: So when you say design your research, that makes me think about sampling but that may be a very small view on what you’re talking about, how does it show up when designing a project?
Vanessa: Yeah, I definitely think sampling is a big part of it. I think where tech companies are located it’s easy to do convenience sampling and just say you know we will do something here in a lab. Luckily now remote research is more popular but back in the day where like a lot of physical labs were being used to you know test an app or show a new site that you just don’t want out there on the internet. That often meant you know you are sampling a geographic area where the income might be higher or there’s a certain distribution like whether it’s a gender ratio or a race ratio that might not reflect the full population. I think being very U.S. centric is often a thing with companies and research that we do and there’s a lot of additional hurdles sometimes to be inclusive. Like if you want to do a study that is conducted in people’s native language and you’re an international company there goes many dollars for recruiting and translation and kind of the scale that it now takes rather than doing something that’s local in your area. So that’s kind of what I mean by like the design, but then it can also mean you know all the way down to who is actually conducting the research. Again like if it has to be a native speaker or if it’s better with sensitive populations who might have mistrust when it comes to testing medical products or things like that like it all then requires a different level of planning and consideration in order to also make the participants feel comfortable and to collect that data in a way that’s respectful and yeah I mean I think I could go on and on. That’s kind of what comes to mind for me and what I have kind of pushed for and I think for the most part there hasn’t been any push back on whether it’s the right thing to do per se, but it really comes down to then the timing constraints and the financial constraints that push a lot of companies to say you know what? Not right now. So I have been a pretty strong advocate for when it makes sense to let’s take that time and kind of broaden our scope in order to make sure it works for the broader set of people, not just the convenient sample.
Steve: And you made a point about right diverse teams are more creative and then even thinking about who does the research.
Vanessa: Yeah, definitely. I think right now I’m in a context where we’re actually B2B and so I would say some of those factors have been a little bit less prominent for us and we tend to you know kind of go for the people who are using the product and try to just like expand our sample to that, but I’m sure that’s your experience too with B2B often you just have a way smaller population than in consumer where you have often hundreds of thousands of people you can contact like B2B might be. You have a list of 5,000 and by the time you add a few criteria now you have like 30 people that qualify. So that makes it a little tougher, but I think in the past when I’ve worked in SMB spaces or consumer type products then that has been more of a consideration and I think that’s also where vendors have sometimes come in for us where maybe we’re not the appropriate set of people to have this conversation or we might miss some of the nuance that exists in this conversation, so we’ve like when I was at Google we definitely would get outside help to kind of round out some of those conversations or kind of share like the interviews across a native Spanish speaker if we don’t have that on the team.
Steve: Does this come into how you approach hiring.
Vanessa: That’s a good question because I think with hiring I’ve not deliberately been in a situation where I’m like saying we need this race or this gender or this language because I think that can get very tricky and also potentially lead to discrimination in other ways, so hiring has largely been like merit based, but I think the part of hiring that I still find problematic is often people talk about pipeline issues and not being able to source candidates from different backgrounds and I often find like that there is actually something we can do about that and I think that’s where I focused more efforts is if I’m seeing candidates come through that are a little bit more narrow in kind of let’s say education. They’re all coming from a set of Ivy League schools or they’re kind of like located in a specific part of the country then I have kind of like worked with recruiting to say hey, can we kind of reach different hiring pools? Do you have access to like different communities where you can plug these jobs to kind of diversify the pipeline so we can make sure that at least candidates that have different levels of education or different background can at least have their resumes reviewed, be interviewed, things like that.
Steve: You know, in the time that you spent in your career so far. Have we made progress on these issues like how, how do you contrast what you see 2024. Like you said there’s still discrimination is not gone away.
Vanessa: Yeah. It’s hard to say. I’m not sure if we’ve made progress or not. I think during the pandemic when George Floyd was murdered there was definitely a heightened interest from companies to think about a lot of these topics. So one way that worked out is a lot of companies including Google and I was actually participating in like some of the hiring initiatives was to think about product inclusion specific roles and like really carve out space to say like we have people in the company who are kind of like trained and qualified to deal with maybe like more sensitive populations and then who are also actively putting programs together to like teach the broader company like how do you make sure that your product especially again when it comes to like language or anything AI related or kind of like anything that’s like visual is not discriminating against like oh the photos can’t pick up on darker skin or they can’t understand people with a certain dialect. Like how can we get that education out there? So I have seen a push in that. There was more consulting firms, more job postings that were specifically around product inclusion and bringing that academic knowledge into the tech field, but as far as like broader hiring trends I don’t know the data well enough across the full population, but I don’t know if I have seen like a uptick per se.
Steve: So maybe one last area to loop back to something else that we were talking about it and you described, you know, going from personal trader to attempt job that gave you exposure to, you know, things that led you into you accidentally research. In some ways it seems, I don’t know sort of circumstantial or opportunistic or something like it’s hard, it’s hard to plan for creating the conditions for that. And that’s, you know, what, is there any lesson or, you know, advice to people that that are listening or that you come across in your in your travels anyway about, you know, going from to me they seem like very different worlds to go to going from one bridging from one world into the other, you know, for you is, it was, I think a certain amount of happenstance and having the, you know, having the right lens on it. But I know does this lead to any guidance or advice for other people from your experience.
Vanessa: I think for many years, I had anxiety and regret around not starting my career in the field that I’m in sooner because I felt very very lost stumbling through all of the different fields and roles, and only in hindsight do the dots connect. As you were saying, they seem very, very different, but I think I’m better at what I do now because I learned the lessons in all of the different jobs. Even something like, again, being a personal trainer or executive assistant, being an executive assistant, I was able to sit in on more senior leadership meetings, and I really early picked up on short attention span, How do you get your point across concisely, What do they care about? And I think that made me a better researcher right away, even as I was still learning the practice because it taught me something about communication.
Then even reaching back to personal training, I think that made me a better manager because I like to think of myself very much as a peer to my team, where we are thought partners. They come to me with things, I come to them with things, and because they are all so driven, we don’t really have a lot of issues where I just have to enforce things or tell anyone to do anything.
But circling back to the personal training piece, I think that really put me in a headspace of okay, someone has a goal and they’re struggling with this, or they need support through this. How do you tap into the human psychology of – they want to get from A to B, we want to make it as sustainable as possible, do something that hopefully they enjoy enough that they can do it on their own.
And so really just getting into that mindset of how do I collaborate with another person and find common ways to address a problem and align with them so it doesn’t feel like I’m pushing you or forcing you, we’re kind of going towards this goal together. I think all of those little pieces along the way just shaped how I interact with people and I think has made me better at what I do today. Maybe just know that it’s all connected somehow.
Steve: Yeah. Yeah, I want to normalize part of what you said at least that, yeah, I also had regrets for not starting my career earlier I felt like I was late. And, you know, I’m, but that was long enough ago that many people I know didn’t know me then or wouldn’t know that about me but, you know, I had a complexly directed path to do what I do now. And, yeah, I think you’re right that we’re sort of all products of all of our experiences is one of the things I like about research but I think you’re right to extend it towards management and everything that there are a lot of different paths and then there are a lot of different ways of being. And that, yeah, if I wasn’t who I was I wouldn’t have gone on those paths and I wouldn’t be able to be all the weird mixes of things that makes me whatever kind of researcher or leader or, you know, all the things that you’re kind of bringing up. I like that about, I’ve always enjoyed that about our field that we’re just, we all come from different places. Even over the generations I think there’s still an interesting mix of that and then means like oh the next researcher that you meet, you know, done something that you didn’t know about like, and you get to work with them and like, you know, there’s something about personal training that shows up in an analysis or in a planning meeting or something and you get all these great stories from people. So I enjoy that. But yeah, I just, I started off by talking about myself here that I have, I know what that regret for me is like that I didn’t come into it out the gate early on and had to play catch up in a lot of ways too.
And maybe that’s a great place to kind of wrap up our conversation. Thank you so much for taking the time, sharing your own experiences and your own perspectives on the work that you’ve been doing all the way along and yeah it’s great to get to chat with you. Thank you.
Vanessa: Thank you so much for having me. it’s really an honor. And even when I was telling folks that Dollars to Donuts was coming back on my team everyone was really excited because a lot of us have your books and have listened to you and learned from you so I really appreciate being here.
Steve: All right, well, great. I hope they enjoy listening to you talk about all the great work that you’re doing.
Vanessa: Thanks.
Steve: Thank you. All right, that’s it for today. I really appreciate you listening to this episode. If you like Dollars to Donuts, recommend it to a friend or colleague or post about it on the social medias. You can find Dollars to Donuts in most of the places that you find podcasts. Review the show on Apple podcasts and go to portigal.com/podcast to find all the episodes including show notes and transcripts. our theme music is by Bruce Todd.
The post 38. Vanessa Whatley of Twilio first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Nizar Saqqar, the Head of User Research at Snowflake.
For a domain that takes a lot of pride and empathy and how we can represent the end user, there’s a component that sometimes gets overshadowed, which is the empathy with cross-functional partners. With every domain, product design, research, there’s people that are better at their job than others. I dobelieve that everybody comes from a good place. Everybody’s trying to do their best work. And if we have some empathy to what their constraints, what they’re going through, what their success criteria is, how they’re being measured and what pressures they’re under, it makes it much, much easier for them to want to seek the help of a researcher to say, “Help me get out of this. Let’s work together and let me use research for those goals that are shared.” – Nizar Saqqar
Show Links* Interviewing Users, second edition * Nizar on LinkedIn * Snowflake * How to use MaxDiff analysis * You Might Not Like What Jon Stewart Has to Tell * Old Man Yells at Cloud * Noam Segal returns
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. Today’s guest, Nizar Saqqar, actually brings this up in our conversation, but I’m going to remind you myself that there is a new edition of my classic book, Interviewing Users. It’s now available 10 years after the first edition came out. Of course I’m biased, but I highly recommend it. And hey, it makes a great gift for the researcher or research-adjacent person in your life, or persons. If you haven’t got the second edition yet, you can use the offer code “donuts” that’s D-O-N-U-T-S for a limited time to get a 10% discount from Rosenfeld Media.
But now, let’s get to my conversation with Nizar Saqqar. He’s the head of user research at Snowflake. Well, Nizar, thank you for coming on the podcast. It’s great to get to chat with you.
Nizar Saqqar: Thank you for having me. I’m really excited for it.
Steve: Let’s start with an introduction from you. You want to say a little bit about your role, your context, anything to kind of get us rolling? And we’ll go from there.
Nizar: Absolutely. I’m Nizar. I lead user research at Snowflake. I’ve been here for about three years. It’s been a pretty exciting adventure. When I started first researcher at a company that’s been around for 10 years and really doubling down on showcasing the impact of research, why we need to scale, and we’ve been scaling nonstop in today’s environment, which has been a pretty exciting challenge. It comes with the fun of it, but comes with the challenges as well, and I think more to come. To take a step back and try to simplify it as much as I can,
Steve: What kind of company is Snowflake?
Nizar: Snowflake enables organizations to store huge amounts of data from many sources in one place. So it empowers organizations to make the most out of that data. And as we’ve scaled the company, we’re continuing to push the envelope on platform-level offerings that try to enable native app developers to do the development of data applications.
Steve: What are some examples or vertical scenarios that we might know about?
Nizar: So the simplification of it is get your data in a place, make the most out of it. Snowflake will help companies do that as efficiently as possible with as many use cases as we can. It’s definitely not in the day-to-day conversation.
Steve: What are some examples of what Snowflake is?
Nizar: It’s not a B2C product, but at the core of it, it really starts with the data warehousing of getting the data engineer brings in all of the data from many different places, many different sources into one place for storage, and then making it usable for other users, maybe like the data analyst or the data scientist who makes something happen out of it as an outcome. And as I mentioned earlier, building the native app development framework, it’s been exciting to see all of the, think of them as the more classical kind of software developers that are now into our ecosystems to get closer to the data. So it’s a pretty complex ecosystem.
We also have a marketplace that then kind of introduces the dynamic of a provider and a consumer and the business decision makers who are coming in for that transaction. So it’s a pretty intense ecosystem that magically all connects into just making the most out of your data.
Steve: So when you came in as a researcher, what did you observe about how this company was thinking about its users or thinking about what it knew or didn’t know? Do you remember that early process?
Nizar: Yeah, and to be honest, that process started before I even started. Even in the interview process, I really wanted to be sure that the company is thinking a lot about their users. They’re thinking a lot about how research can integrate. They’re thinking about challenging some of maybe the perceptions of what research can bring to the table and just having some of these tough conversations even before. And I will say that where we are definitely lucky is that Snowflake does put, do what we can to make it as great of a product for our users as like a core value of the company.
The interesting thing with that is that it brings in a lot of data points from a lot of pieces. Now you have a lot of user perspectives from the sales team, directly from the product team. Then you have all of the metrics and dashboards that you’re following. So you actually get a lot of, you get a lot of data. You get a lot of points that might actually make it harder for the product teams to action on or prioritize.
So as I started, I kind of wanted to first take a moment to better understand the domain, really kind of find my footing, know what’s going on, build the right relationships and start with something that’s very low-hanging fruit of saying, “Hey, let me just build credibility. Let me just come in and say I can add value very quickly and then scale that up.” It’s been interesting to see how the role has continued to evolve since that day one. It really started off with, we’ve been here for 10 years. This guy is here. And it’s really kind of evolving into user research is just a critical component of how we think about product development.
But it’s taken many phases that we’ve had to adapt as we continue to go, starting off with the very tactical, then zooming out into something that perhaps is more strategic, then shifting focus into our hiring strategy and our hiring rubrics and how we interview, going all the way into what we define as success criteria, performance evaluation and how we integrate research into the overall product process. And it just doesn’t stop. So the role itself has been changing over the past three years and I perceive it to continue to do that.
Steve: Can you give an example of a tactical, sort of quick win that you would approach kind of coming in, in those early days?
Nizar: Yeah, absolutely. And I think for me it really starts with what is something that is tactical enough, close enough to wanting to launch. There’s enough resourcing, but there’s some level of disagreement in the organization how to proceed. And seeing if there’s an appetite to make research kind of be a tiebreaker of sort or really find the right balance between the two. I found that it’s very rare that there’s option A or option B wins and there’s always like components of each that kind of resonate that when you bring them together, you kind of find something that really resonates to the actual flows.
And if I remember correctly, one of my very, very, very early contributions was kind of around something as simple as a concept evaluation. And I think those are the methods that are just going back to the basics and some people take for granted, but you’re just coming in and you’re saying, “Let’s just test it and see what’s resonating and what’s not.” And coming up with some actionable steps that align multiple teams that might have dependencies on each other to find the solution that may not make everyone happy, but at least everyone is aligned that, “Okay, this seems to align on a path forward from a user lens.” So then it just continues to evolve.
I was talking recently about how coming off the bat, just seeing that there’s an overwhelmed product manager who says, “Hey, I have 20 features that I’m asked to ship.” And my role there was to come in and say, “Let me help you do a MaxDiff survey to just make a case for some things that you should actually deprioritize so you can make progress towards some of the top features that you want to run through.” And I think that was part of the evolution of, “Okay, we could use research for many different use cases and in different areas where we could integrate with the product roadmap.”
Steve: So I think it’s super interesting that you’re using the interview process, I guess, to understand the context that you’d be coming into.
Yeah, I’m wondering, and I don’t know, I’m going to ask it like a binary, obviously it’s not, but, you know, how much of a mandate were you given versus how much you were trying to figure out what the needs were and, you know, make recommendations appropriately? That’s a terribly leading question.
Nizar: It’s not, though. It’s kind of interesting, because there’s– when someone opens a position, when somebody asks for a headcount, for the most part, they have an idea of what they’re looking for. They have an idea of what they think the success criteria is. In my case, I was hired by a head of design who had an idea that I could help elevate the design team. That was kind of like the primary premise. And the interview process, that comes out, and it’s a really exciting thing. And a head of design is really excited to be like, “Now I get to finally get to support design with research.” I started digging into the appetite of, “But how do we kind of expand beyond the design?” If we’re to look at the pie and we’re to say, “Instead of making that piece more efficient, how do we just make the pie bigger? How do we get the design team holistically more involved and have that impact from earlier stages as well?”
So look beyond the design research component and don’t worry about it. I’ll make sure you have a good story for where your organization is growing in terms of impact beyond the pixels. And I think that was a really good back and forth that early on showcased that there’s a lot of action and appetite for, “Hey, if you can define something outside of what I have in mind that you perceive could be even more value-adding for the organization, that’s what we’re optimizing for.” And I think that was a good start of saying, “Okay, I won’t be in a situation where somebody comes in and says, ‘This is what you need to do.'”
I do hear a lot of stories of, “The researcher comes in and all you can do is one-week sprint and stuff, just do a study every week, and it’s non-negotiable.” And it was pretty important to gauge that, “Hey, can we just align on value for users and value for the company and value for the team as the criteria, and let me do what I need to do without a specific framework of how I should be operating within those objectives?”
Steve: I love the phrase impact beyond the pixels. That’s like, that’s a pull quote or that should be a title of your next talk. So that sets you up then to find that overwhelmed PM. And if I understand it correctly, you’re kind of saying to them, like, did you know, hey, this is a situation you’re in, like, here’s an approach that would help you. That’s kind of where my, maybe where my mandate versus discovery question comes from. It sounds like you are finding opportunities or finding places to have impact where that PM is not going to ask you, hey, can you do a MaxDiff survey? You’re coming in, seeing the situation saying, yeah, here’s a way that research can unblock you.
Nizar: This was a fascinating story altogether, and there’s some more context behind it, which is kind of funny when you look at it. That PM was super excited before I started, messaged me before I started, started telling everybody that me joining is going to be a game changer, was the friendliest PM I’ve ever met when I started. And then back then, my manager was saying, “Hey, we think this is the most ambiguous thing. We need to redefine a roadmap. You need to put a really found– I think this is a really foundational research problem.” So when I went to the PM and I told him, “Hey, we can work together on this,” and I’m actually excited to team up, he actually said, “No, I’m not interested.” And I told him, “Let’s take a step back and let’s speak about why you’re not interested, what’s–just what’s on your mind. Let’s not talk about the research. Let’s talk about the problem you’re solving.” And his take was, from my experience with research in the past, a lot of the times it does take a lot of attention to keep up with all that’s happening, be part of the interviews, and then you come back with a lot of insights that I frankly don’t have resources to do anything with.
So if you come to me and you say, “Here’s 10 things you need to build,” I’m just going to put it at the end of the JIRA board as items 21 through 30, and I’m not going to get to them. So the key learning for me back then was, okay, everybody kind of perceives my role and how I can solve this problem very differently, and I really need to set some shared language and shared expectations of why I’m here. So that’s when I was like, “Hey, how about we do this? How about I’m just going to go into your JIRA board? I’m just going to steal the things that you have there. I don’t need you to be involved, and let’s make a case of why you don’t need to pursue all of these features at once. And let me do the heavy lifting. We’ll team up on it.” And then going back to my manager back then and saying, “I don’t think I need a multi-month, huge effort to start. Let’s just kind of help him get out of the weeds for a bit and just aligning the expectations over what we can do with the research.”
And in this case, it was a core example of research really used to de-scope, to de-prioritize, to say that not everything is equally important. At a high level, when you take single-off, one-off stories, they all come up as high needs, but are they all at the same level of importance for when you look at our user needs and the business value that they bring? And that’s essentially what came out of that Maxx a lot of these are way below when you compare it to what’s really bubbling up to the top. And how do I make the case that doesn’t say everything is there for a reason? But let’s make a case that with the limited engineering resources that we have, we can drive the most value if we really focus most of it on those very specific areas and get those to a place where our end users are really happy with the experience that we’re offering. And that was a different mindset. That was a different principle for that PM where it’s like, I didn’t know that we could do that. I didn’t know we could do research that helps me tell a story to executives of why I shouldn’t be doing work or why I should say no to some of the work that’s coming up.
And then that led to me really wanting to define some of the languages used around why research exists and why research is at that company. And the wording that I tend to use, which may not apply for everyone, but I try to take it around driving the allocation of limited resources into the most impactful efforts for our users or organizations. And if we can have that be a shared language mandate for what research is optimizing for, it takes away some of the misconceptions here and there and where there comes some of the tactical things like changing the title names from UX researcher to user researcher or changing some of the way that we present decks or reports or internal documentation. So there’s some tactical things that come with it, but at the core of it really linking it to the intersection of user value and organizational value.
Steve: So to get somebody unstuck, when we get so overwhelmed, we can’t even see our way out of something. I don’t have time to do your solution. I’m just, you know, I’m treading water here. So I love that aspect of the story that you found an approach that also is about limited resources and that took that person where they were at. And I don’t hear complaining about a stakeholder that wouldn’t commit to the project, you found an approach that you had a 10,000 foot view and it could kind of see how you could add value. And still, I think you were fairly new to the organization at that point. Is that right?
Nizar: I almost had no idea what was going on. I needed to rely on them to make sure that the items in my MaxDiff actually made sense. Even when I was mentioning it earlier, and I come from a B2C company. I come from a place where that ecosystem was extremely new to me. So of course there was some collaboration there, but I tried to keep it as lightweight as possible as I make sure that I have the right pieces but without overwhelming them.
And I love what you’re saying. I love the way you’re describing it. For a domain that takes a lot of pride and empathy and how we can represent the end user, there’s a component that sometimes gets overshadowed, which is the empathy with cross-functional partners. With every domain, product design, research, there’s people that are better at their job than others. Sure, for the most part, I do believe that everybody comes from a good place. Everybody’s trying to do their best work. And if we have some empathy to what their constraints, what they’re going through, what their success criteria is, to be honest, how they’re being measured and what pressures they’re under, it makes it much, much easier for them to want to seek the help of a researcher to say, “Help me get out of this. Let’s work together and let me use research for those goals that are shared.” And at the end of the day, it is still user-driven. It’s still based on the data that we’re getting and we’re able to drive direction. Finding ways to go with the flow while still having a strong perspective over what’s best for the users, rather than feeling that the role of research is always to be on the opposing end of cross-functional partners, could be a really powerful tool. And in these cases, all it leads to is that intersection of product impact and user impact, which I think is the end goal.
Steve: It is a great story that this person was enthusiastic for you and reaching out and was excited about research. And when you think about what opposition is sometimes, I think it’s easy to sort of demonize someone and say, well, they don’t, they don’t get it. They don’t believe in research. They don’t like me, whatever kind of escalate. And here you started with the best possible out the gate dynamic with this other person. They were a fan. They were like welcoming you and they couldn’t wait. And still they had a concern. And so by identifying that and coming up with the right approach that suited all those constraints, you get to them the kind of impact that you’re looking to have.
Nizar: And it kind of makes sense. I mean, if you really think about it, the definition of what a designer does, the definition of what an engineer does, for the most part, is pretty material. You finish your effort, you pass it on. Eventually, the thing that that designer or engineer touched is the product that you end up using. What that does is that for areas like research, there’s more fluidity, and the perception of what you’re here for. So that fluidity could be a great thing and could be an awful thing, because at the end of the day, it opens up a lot of opportunity to set the expectation of why the researcher is here.
But it takes a lot of work to get people to align, because they’re also basing it on their past experiences, basing it on their biases, basing it on whatever good experiences they had, but also whatever bad experiences they’ve had with research, and a lot of that bias of how the work may or may not be precisely presented to what the end user sees just opens a lot of these gaps and unknowns that I think just plugging these holes and making sure that the narrative is clear around why the researcher is coming in, to me, I see it as an opportunity. It’s not always the most fun process to cover some of these holes and make sure that there’s no gap in the perception of why the researcher is here.
Steve: We started with the foundational, this tactical stuff that you’re doing, but maybe we can look at the whole arc of creating that more evolved understanding in the organization across all these folks about what research is here to do.
Nizar: Building up on that first story that I was saying, I made my success criteria be less about the research output and more about how’s the research being used, how’s the research actually integrated directly to the roadmaps, and some of that lives till today with the team. You’ll see a lot more emphasis on how often is your research referenced in a cross-functional document. Then you’ll see about what is the quality of your report, for example, as some of the success criteria that we have.
But taking it back to that initial journey, there’s a disadvantage and advantage of being the only researcher back then. The disadvantage is that it’s overwhelming. There’s a lot to cover. The advantage is that gives you the ability to say, “I’m not going to do it all. I can’t do it all.” And you get to pick and choose a bit in terms of where you can foresee some of the most opportunity is.
And at the same time, you look at where some of the path of least resistance could exist, too. So if there was a huge problem where the path of resistance is pretty significant, the question that I need to ask myself is, is this where I want to continue proceeding? Should I continue butting heads to be included there, or should I go find that place that is a bit more welcoming to changing their processes and their approach and just kind of use that as a case study? Once that case study lands, how are we showcasing that case study?
And I’ve never been a fan of visibility for the sake of visibility, but especially in the earlier days of research, there’s a lot of advantages for visibility as a case study of how research could work to empower those around. And that became key. And basically what happened right off the bat is we started hearing the sentence, “Well, I want research. Where’s my research? Why don’t I have a researcher?” And the demand for research support started coming more organically from the cross-functional teams. So it wasn’t on me to necessarily say, “I need people. I need to grow the team. I need to do this.” It wasn’t an ask from the research department to grow. It was an ask from cross-functional partners who have seen how much more effective and how much more efficient they could be with the appropriate level of research support. And that just kind of creates more of that shared language, the shared narrative of what the organization is looking to do with the research and work closely with it, but also what’s the success criteria for the research team.
And as we started to scale, it became more and more important to set pretty stable goalposts to gauge what success looks like and what our objectives are and being very intentional about kind of not falling into the trap of making research the end goal where you’re out of the loop of what decisions are actually being made and you’re trying to do like a one-size-fits-all approach to research that says, as you get more senior, your research gets more complex, which I don’t believe is the best definition of researcher seniority, but really kind of anchoring it in how we’re able to continue driving the product roadmap forward. Even then, there’s a lot of back and forth that goes into it.
Steve: When you started to get these requests from people, we want research, where’s my research, the kinds of things that folks were hoping for or asking for, did that line up with what you would want to or hope to support them with?
Nizar: When you’re starting the team from scratch, the default is actually about, hey, since you only have one researcher, there’s only two of you. Do you need to do some intake form to take everybody’s input and then try to cover as much as possible? And I put my foot down that I don’t think this works. I don’t think that’s the most effective way to do it, and I don’t think hiring a researcher starting off with a service model that says, hey, you’re not part of the team. You’re an outsider who will do research and come back will be the most effective way to drive meaningful change.
So I approached it with a lot of support and I approached it from a point of view of let’s continue that as a proof of concept. Let me hire a researcher and embed them directly in one of our most critical strategic teams that has significant strategic significance for the company as a whole, but also has a lot of open questions and a lot of things that we can benefit from a researcher, and that’s all they’re working on.
And of course, you get the pushback of, well, what about these other teams? And my take is you’ve been operating with that research for 10 years. We can weigh it a bit more, and let’s continue gauging how things go there. And that kind of starts it off with setting up the researcher for success and empowering them as a core member of the team, challenging the notion that they’re there to take requests or answer questions and have them be able to actively predict where there will be blockers and how they can get their research ahead, maybe like three months ahead, six months ahead, to be able to actually be ready for the decisions for when the actual time to make the call comes.
So it’s essentially making sure that research is proactive rather than reactive. And that model worked. That model worked great with that team. We hired a phenomenal researcher. Until today, you’re always excited about the first hire being a phenomenal person on the team, and you start replicating that model with different teams for areas that also are strategic to the company and have a lot of ambiguity, and that kind of becomes the framing in terms of unblocking, creating alignment, efficiency, and how we can just continue to scale from there.
Steve: At the risk of oversimplifying, I guess I’m hearing in your answer where I was going wrong on my question, it’s the difference between this team needs a researcher and this team needs research. I was starting with this team needs research and you’re putting a researcher in there and they are figuring out the questions, being proactive, that’s very different than that service model intake form thing.
Nizar: Yeah, correct, and generally I think teams that start off being user-centered at times think they’re doing research. There’s a lot of types of research, right? So sometimes you’re talking to, for us, you know, we’re a B2B company that has great relationship with our customers. So you think that, hey, I’m talking to their sales engineer or somebody called me for a meeting. Like, I’m doing some level of research. I kind of have a take of, you know, I’m not here to keep, go do your thing, but at the same time, I’m not here to demarcate. I’m not here to, like, empower as many people to do research as possible.
You know, my role is to be a cross-functional stakeholder, and I will jump in with what the problems are that we need to solve together, and let me find ways to deal with them. So I think in this specific case, there was always an acknowledgement of, hey, there’s stuff that we don’t know, and we need some form of research. The definition of what research is and how it’s going to be incorporated is the thing that needed to be tightened up a bit more, and then integrating the researcher as in the framing of this is a cross-functional partner, not a source of research, if that makes sense.
I started changing the language around the expectations of even when they’re invited, even when people go to them, even when what topics they’re having in their one-on-ones. So it’s less about, like, here are some questions that I want and more about, hey, I’m struggling with this thing and we talk through it. So they all tie in together. I think to my point earlier where there’s a little bit of the path of least resistance when you’re starting and you can pick one team, you know, that was, you know, you can call it a lucky privilege of saying, okay, there’s a team that could be ready. I’m seeing conditions that are priming a researcher to be successful here. Let’s go with that model, with that team, and continue scaling from there.
Steve: I mean that reminds me of just your interview process, you’re looking for those conditions to understand what that is before you start and now as you grow your team, you keep looking for those conditions within different parts of the organization to see where research is going to, could go next and have the most, again you’re really focused on the impact and the product and the experience in the company.
Nizar: That’s a good summary, and it feeds into our interview process. I do, you know, we try our best to make our interview process as applicant-friendly as possible where it’s not convoluted, but at the same time, it covers a lot. So a key part of it is who joins the team and what’s their approach as well. We do tend to see, like, there’s a specific type of researcher that tends to do best, and usually we look at researchers who do have the depth and soundness of research methodology as kind of like a core expectation. But then they layer on top of it the user-centric process and thinking, you know, when do you integrate at different stages of product development?
You start seeing the kind of the business sense of really wanting to be integrated deeply with the team and solving the problem at heart rather than solving the open question. And then cross-functional collaboration as a core area. I do think that every researcher needs to fully understand what resources the team is working with, whether it be engineering, design, any other blockers, to be able to come forward with the most effective size of recommendation.
And then we always have that over-layering, overarching umbrella of leadership and teamwork, really kind of looking for people who have a growth mindset, who are looking to help others succeed, who don’t necessarily see it as like their world and like their thing, but just really collectively looking for everybody to succeed together, which I think has been pretty key as we’ve scaled the culture of our team into a team that’s pretty collaborative, a team that’s looking to help each other, and a team where people aren’t competing. And there’s no incentive for people on the team to compete. There’s actually an incentive for them to make each other better and learn from each other. So that’s been an exciting part of scaling from a cultural perspective within the research team.
Steve: I want to ask you to clarify, you used the phrase the problem at heart versus the open question.
Nizar: Absolutely.
Steve: Can you explain what that looks like for, what does that mean for any particular problem?
Nizar: One thing I’ve become really sensitive to, maybe too much so, is when I see kind of a research plan that says our objective is to answer these five questions or six questions, and my take is that’s actually a step removed from what you’re going to do with the answers that you’re going to get. So I like to start it off with what’s the perceived outcome? What’s the perceived objective? What are you looking to learn and why in terms of what’s actionable? And then take that to go a step backwards and say, okay, to get to that effectively, let’s now go into what questions do we need to ask? And based on that leverage, what method is the most efficient and appropriate for what we’re trying to accomplish?
What I’ve seen a lot in the past is even your stakeholders think they’re asking you the right questions. How many people have been asked, can you create personas? Can you tell me the different types of users? And a researcher goes off, does this for a month or two, and then they come back and nobody knows how to use them. And to me, that’s the problem that I’m trying to avoid as much as possible and just saying, okay, you want personas. What are you going to do with them? What’s the decision you’re trying to make? And often coming to the conclusion that you don’t need that at all. What you need is something much more simplified. Or we could actually get a pulse check to start getting you some signal of the answers that you’re looking for that will help with that decision-making process. And then we can decide when to iterate or if it’s necessary to iterate.
With open questions, I find that there’s sometimes the danger of over-scoping research efforts. For what you’re trying to do with it. It’s just that outcome of you show up with a deck that has 100 slides, but the team can only act on the first two. And so my question becomes, was this the best use of the researcher’s time versus trying to focus on those first two slides and then connecting that to a longer-term program that we can then create follow-ups on as we continue to learn throughout the process. So in a way, it’s forced efficiency and early hypotheses of how we connect to the impact before even starting off with prioritizing the effort.
Steve: I want to follow up something else that you said, you were describing a lot of the qualities that you’re looking for, the mindsets and the kind of abilities, you know, how do applicants demonstrate that information in your process?
Nizar: I could talk about that for a long, long time. So I generally don’t believe that these buckets are a pass or a fail. I don’t think it’s are you good at user-centered thinking or not. How I see it is everything kind of sits on a continuum. And what I’m trying to optimize for is for the level of seniority that this role will require specifically, am I seeing enough of ability to handle different situations effectively that then puts them in a position where they’re going to be able to know what they need to do regardless of what’s thrown at them.
So, for example, let’s say somebody is, we hear this a lot with the break apart of the tactical versus foundational or that, where it’s like, I do this and not the other. And my question becomes why? Why create that separation between this form of research and the other if you’re able to tell a story around pretty much your ability to come in at multiple different stages of product development and say that I can help you across every stage and I know exactly how to do it. And I can help you across every sort of limitation that you have and I know how to do it. And I can help you address multiple different type of issues that we’re facing, whether they be needing of some qualitative research or something that’s more quantified or something that’s quick and dirty or something that just needs a brainstorming workshop and I’m able to just be flexible in where I integrate with the team.
So I know that was a long-winded answer that went all over the place, but it’s really hard to — the reason it’s hard to describe is, I really don’t think research is good or bad, yes or no kind of domain in general. And what I’m really trying to optimize for as much as possible is, does the research applicant have the breadth to be able to tackle as many problems as possible? To me, that’s a much better predictor of seniority and success than somebody coming and saying, “I did this multi-country 12-month research project that was really complex logistically,” which is impressive in its own way. It’s great, but for me, it’s not what we’re trying to optimize for in general.
Steve: And so you’re looking at past experiences that the applicant can, I guess, describe to you, or those kinds of clues to the breadth.
Nizar: We look both at past experiences and we look at some of the hypotheticals as well. So we do have some scenario-based questions where we try to gauge some of the thought process. It’s the thing that I tell people that there’s no right or wrong answer. You’re just going to get a hypothetical and I just want to hear how you think about it. And I want to hear what are the different considerations that you take into account or into place when you’re making your mind up about the best approach and what you’re going to do. How often are you coming in and having the hard conversations about what needs to be done versus steering the conversation in a completely different way versus just saying, “You know what? This isn’t worth the back and forth. Let me just do something quick and move forward.” So at the end of it, when we’re combining the hypothetical with the past experiences, I’m really looking for effectiveness and efficiency under the umbrella of strong and sound research.
Steve: Those are words that are sometimes seen as at odds with each other, but I think you’re talking about how they’re in support of each other, that effective and efficient doesn’t mean that you’re not sound, doesn’t mean that you’re not, like you said, solving the problem at heart versus the open question. That seems like a key mindset that you’re bringing to this.
Nizar: A hundred percent. And I hear that sentiment every now and then. I hear the sentiment of, “Oh, if you go too scrappy, you’re doing really terrible research.” Or, “It either has to be great research or it’s terrible research that is fast.” And I don’t agree with that mindset or that context. For me, it really depends around what you’re trying to learn, what your objectives are. If you’re trying to do something that is an extremely small pulse check, for example, you don’t need to boil the ocean.
I still remember earlier in my career, I joined a team and they just had no idea, they knew nothing about their users. Absolutely nothing. And I was telling them, “Do you have any hypothesis? Do you have any open questions? Do you have anything there?” And they’re like, “We just don’t know. We just know that nobody’s using this feature. That’s all we know.” And we look at our dashboards, we look at our metrics, we have a target addressable market of millions, and we have tens using it. So we don’t know why. And I just came in and I said, “Look, the best use of time right now for me is just to do some sort of a small single pulse check survey. One question, pretty much trying to understand the state of everything. Just for me to have context to get started, just give me some perspective. Am I planning to use this in road-napping? Probably not, but I need some form of context from end users to be able to tell me, “Okay, I have an idea of what’s happening there, and I have an idea of the value add, I have an idea of why they’re churning, I have an idea of why maybe they’re not seeing some value. Let it be scrappy.”
And you get the pushback of, “Well, this is qualitative. You need to do in-depth interviews for that.” I’m like, “No, I don’t. I really don’t. I don’t need to invest 40, 80 hours just to get an idea of what’s going on if you can do this in 24 hours, and then take that as an entry point into something that’s more detailed, that’s more rigorous.” So for me, it just goes back into linking the amount of effort to the projected outcome and really just finding the thing that works for next steps. And in this case, we did end up actually needing to go very in-depth with foundational interviews and a full design sprint, and then going to concept evaluations and stack rank.
It ended up being a really complex process over maybe the course of a year that really turned around a product that wasn’t used, a product that had millions of users. But at the start of it, I did not have the luxury of saying, “I just need to go away and do in-depth interviews,” because the research domain says that qualitative is not allowed in a survey. So sometimes I think just breaking the rules in our domain is very okay, as long as you know why you’re breaking the rules and what you’re going to do with the insights that you have.
Steve: This research effort that you’re scoping at any point may not be, or probably isn’t, the only time you’re ever going to learn anything. And so, you know, as I’m taking that away from you, then I can sort of feel some of my own anxiety just like ebbing away, like, of course, right? If you think of research as a longer-term thing, like, what’s the question we need to ask now? What’s the right amount of effort for right now? Okay, everybody, we’re not going to get everything. We’re not going to boil the ocean, as you said. There’s more to do, but here’s where we are right now. And so, yeah, that good versus bad research thing is it says we’re only going to do it once, and it’s kind of this monolith that’s either going to answer everything or not answer anything. And these gray areas you’re describing are, it’s a gentle reframe for me, I think, about where I sometimes feel anxious about trying to tamp down the commitment or the investment.
Nizar: As long as we have the right data point for the right decision that’s being made, if we’re coming in and saying, “We need to invest all of our engineering team of this one single customer satisfaction open-ended box,” I get an anxiety attack. I get it. But sometimes that’s not the decision that you’re making. Sometimes the decision — the pros and cons that you weigh, the cons of having something that’s scrappy and fast are justified when you look at the pros of being able to get ahead and then establish a research roadmap that actually gets you ahead of the product team. So that’s the consideration for me.
And to your point earlier, product development is iterative, and I think people forget that sometimes. People forget that even if you launch something, that team is still there, and that team will still continue to want to optimize it in some shape or form. So if anything, I feel researchers should take some comfort in that and saying that, “Okay, if I miss the boat now, how do I get ahead and say, ‘Okay, for the next iteration, for the next thing that’s happening, I’m able to get ahead and have some things ready in time?'” And acknowledge that the same way that product development is iterative, even the most foundational research efforts, you’ll end up having to iterate on in some capacity. I haven’t seen a world where a research deck is still relevant years later, and nobody has ever touched that topic again.
Of course, you want to minimize how often we redo work that’s already been done, but everything changes. Once you start having a user base, the kind of data that you have is different. In this case, when you have tens of users and you go into the thousands, the kind of feedback you’re starting to hear is already different. The usage data that you’re starting to get is different. You’re able to use telemetry a little bit more than when you had nobody. You start being able to triangulate in a way that you just weren’t able to earlier. So iterations are good, and that doesn’t mean don’t do really deep foundational generative efforts. They’re just a time and place to say, “This is my time to get scrappy, and this is my time to dive deeper into the topic.”
Steve: If I didn’t think about it too deeply, I might, you know, sort of have this reflex that says, well, when we know nothing, that’s when we have to learn everything, that the foundational work comes at the beginning. But you’ve got a number of examples where you’re coming in and seeing a big gap and saying, no, it’s not, this is not the boil the ocean time, it’s the quick win or the thing that we can act on or the scrappy thing. And no one believes that A is B. No one believes that the scrappy quick thing is, in fact, going to answer all the questions. But you’re helping take action, you know, within the constraints that are there.
Nizar: I’d say caveats. Tell people there are limitations to what I’m doing. We are aware of that. Every research method we do has limitations, and I’m yet to come across any research study that has solved everything or is now claiming that we have learned everything about our entire user base or our entire feature area. And that’s the reason researchers continue to be in the same role or on the same team for years. There’s always a lot to uncover. And a lot of the time, just really weighing the cons of coming in early and saying, “You know what? I just started. I’ve been here for a week. Let me go disappear for three months or so.”
And I get it. There are ways that you can incorporate your cross-functional partners. But for the most part, especially as somebody’s building credibility, starting with data is much more effective than starting with raw data and giving you some form of, “Here are some next steps,” where often the next steps are research. When I did that Pulse survey early on, the next steps were research. Now I needed to go and actually do in-depth interviews to learn more, but at least I had some litmus of, “What am I talking about? What is my script going to have?” So I’m not finding myself in a situation where I’m interviewing, even if it’s 10 end users, I’d be like, “Can you tell me anything? I don’t know where to start. The team doesn’t know where to start.” But I had something. And for me, the value of that effort, even if it just fed into the definition of a research template for the next steps, that was value-adding, and that saved me a lot of time.
Steve: I’m going to switch topics a little bit here and go back to something you said, I don’t know, before. And I just — maybe you can unpack this or just clarify it. I think you were saying that, you know, that you look at, for people on the research team, you look at number of references or citations of research work in the work product of other cross-functional teams. I’m saying this really poorly, but did I capture that at all?
Nizar: It’s a good summary, and I’ll also give it an asterisk and say, “Not as the only signal, but it’s an effective signal.”
Steve: Yeah. So I have a bias against that. And that’s, of course, coming from someone that doesn’t work inside an organization. So my bias is maybe just hypothetical, but — or just from conversations. And maybe that — maybe that asterisk is really, really important. I agree it’s a signal. I do worry about researchers either getting external pressure or pressuring themselves that this thing, which is essentially out of their control, whether somebody else does something or not, is kind of — is a measure of their worth. Where there’s lots of reasons why people don’t do things and don’t listen to things. And I think you’re talking so much about how to — how to prevent that from happening. Right? The right work at the right time, with the right collaboration, with the right understanding, and all that stuff being kind of scaled appropriately. But, you know, just having spent my career giving people stuff that we’ve agreed was going to be important. And then seeing all kinds of things happen and don’t happen. And to a certain point, there’s a certain amount of surrender, right? Like, I’m going to give you everything that we agreed you need and maybe more, but I can’t control what happens.
So I don’t know. I don’t want to frame this as a debate or anything like that. But I’m open to you telling me that, like, I’m wrong, that I’m framing that wrong. I’m just curious what — you know, how you think about this. It’s not the only signal, but how do you think about sort of how to use that signal or how we should all think about that signal of what somebody else does?
Nizar: The conversation is super valid. That’s where the asterisk comes in. And if anything, I always love the counter perspectives here as well. The reason that I added asterisks here, too, is exactly what you’re saying, that you can’t really control what somebody else does, where it ends up putting some emphasis is encouraging research teams to be very strategic in terms of where they’re prioritizing their time and how they have ownership over the product direction as well. But that doesn’t only go on the researcher. A lot of the conversations that have to take place as well do have to happen at the leadership level. And kind of talking about if we’re to say that the researcher also is to be held accountable for what’s going on there, what’s the collaboration model, and where are they coming in, and are they left out of being able to have that, or is the expectation set that there’s some form of path for them to do it?
And there are also multiple ways from my perspective to showcase that. I think when you look at the referencing, it’s as direct as it gets usually, but even that, it can be optimized. Sometimes you have to take it the other way. Don’t optimize for making sure that your research isn’t docs. That’s not what we’re optimizing for either. But what are the ways in which, as a research team, that for better or worse needs to continue kind of driving the narrative of the value that we bring and connecting the dots to the different decisions that have been made because of the research and the leadership role that each researcher is taking and actually guiding the product roadmap? How could we make sure that we are being very intentional about collecting the evidence and documenting it and being sure that we’re telling our story in a way that does a service to the team members? And often you’ll find researchers in environments where that’s just really hard with their teams. That’s just not how their stakeholders are wired.
And when that happens, my question becomes, what’s the role of leadership and me in a lot of cases in streamlining that, but also what are other ways that are effective in gauging the success of that researcher that do not rely on that being the only mechanism? And that’s where that asterisk plays a huge role. And yeah, absolutely. There are some full quarters. We do performance reviews quarterly, which is pretty intense, and sometimes you don’t get to finish things that are meaningful in a quarter. So we want to give people the benefit of the doubt as well into how the research efforts bleed into the quarters after. But there are some quarters where you’re deep in the research, the team itself doesn’t even have a document, and there’s no way to say, “Hey, this is what’s happening.” But we look for other ways to continue connecting the dots there.
But for me, one thing that I do genuinely care about, and maybe it’s just from previous experiences in the past of seeing where research can get thrown under the bus sometimes, I think for the past many years, I’ve been very intentional about just telling the story of the ROI of the researcher themselves. So not the ROI of research, which I think sometimes gets confused of the ROI of the researcher. I find it that often, and of course it depends environment to environment, company to company, but I find it that often people don’t debate the value of research. They sometimes debate the value of the researcher doing the research. That’s the topic that comes up here and there, and I try to be as intentional as possible to position the researcher and position the organization to give the space for the researcher to be a product leader, not only a research delivery mechanism, if that makes sense.
And with that comes some of the expectations that end up changing. Fingers crossed it worked for me throughout my career. At the same time, I always want to acknowledge that when I say it works for me, there’s also a right time at the right place component of it, and it’s not always on the way that the research is conducted or what the researcher is doing.
Steve: Let’s just switch topics again. We haven’t talked at all about, you know, your overall trajectory. And it’d be great maybe to get a — maybe a summary of how you found user research, what you started off doing, what some things that you did that kind of led you to this role. Maybe have set the context that we haven’t talked about for what you have been sharing.
Nizar: Sure, yeah, I could take it many, many years back. So I actually went to the University of Jordan to study industrial engineering and I was one of the few people who actually cared to have an emphasis on human factors. I don’t know why, but I was always fascinated by the intersection of humans, computers, and business and psychology and all of these together, and I didn’t really know what you could do with it. And it was as early as undergrad that I thought that, okay, this domain seems to cover a lot of those areas, graduated and worked at a company in Jordan under a title that back then was something along the lines of process engineer or something, but in reality it was more understand the inefficiencies in the process and how people are coming in and out of their day to day and how we can make it more efficient. So it had a big kind of research component, and I was aggressively reading about what are some of those programs that I could continue learning in that space, and because where I lived, nobody knew what that was. That wasn’t the thing. And you couldn’t even say, there’s visual design, but you couldn’t really say user experience or UX research.
And moving on to grad school, I went to San Jose State for the master’s program there, and the beauty of that was just the amount of exposure that I had to a lot of different companies and different people who are doing some of the user research work, and it was pretty much a straight shot from there. I went into consulting in a user researcher role, went into a startup where I built up from the ground up into a research and design org. So I was managing research and design, moved on to Google, at YouTube specifically, where I spent about three years, and then when the opportunity came out at Snowflake, it was just too hard to say no to that opportunity. So moved on. I’ve been there for about almost three years now, which is crazy to think about.
Steve: Do you think of yourself as someone that has a superpower?
Nizar: It’s a humbling question, to be honest. The thing I take pride in is I’m always open to being wrong. I’m always open to challenging the status quo and being told that there’s a better way to do it. And I think where I take some of that pride is a lot of the times I hear people that even you look up to throughout your career, and then you get to a point where you’re like, “I kind of disagree. I see a different way.” And trying to challenge status quo for something that could be better is just something that gets me pretty excited.
Is it a superpower? I don’t know. Maybe it hinders me at times, but at the same time, especially in the conversation that’s taking place around research right now, you start seeing a lot of the consistent perspectives of this is right, this is wrong, this is what you do, this is what you don’t. And I try to be very intentional in hearing what are those different perspectives and why are they seeing things differently and what works for me and how do I acknowledge that what works for me at the environments that I’m in may not work for somebody else in the environment that they’re in as well and give people the benefit of the doubt and keep running with what I’m doing.
Steve: There’s sort of two facets, I think. You started off saying that you’re okay being wrong yourself, but you’re also looking for when the conventional wisdom is wrong. Did I get that right? There’s sort of two aspects. It’s like you’re willing to forgo needing to be right, but you’re also embracing or curious about, hey, maybe something out there that’s established as right, the status quo, like you said. Maybe that’s wrong and you’re like challenging that.
Nizar: And think of it like they’re intertwined. Think of it as somebody who is a user researcher. For the most part, you’re kind of looking for best practices, you’re looking for the perspective, you’re looking for the voice of the crowd. They’re kind of intertwined a bit. And I think it’s a solid starting point. It’s much better than starting from zero. Learning from someone is always significantly better than just figuring it out on your own. I mentor upcoming researchers every now and then and I say, “If you can avoid being the first researcher out of grad school, I would avoid that.” But I want to acknowledge that not everybody has the luxury of picking and choosing especially their first job. So take it and learn on the job is better than not having anything.
But it becomes a starting point of, okay, we think this is the best practice. I guess then there’s a tough conversation of, does the best practice make sense? Does the best practice work for me and my approach? Does the best practice work for the environment that I’m within? And where do we continue optimizing it? And how do I continue doing the internal reflection, the internal research on what’s working in the processes that I’m establishing and what’s not? And how do we continue treating honestly my career trajectory as a product that you continue learning, iterating and hopefully making it better?
Often it leaves you at odds with what a lot of voices have in place, but it’s okay accepting that as well and being like there’s no reason for everybody to align on one topic. And it’s always fun. It’s always fascinating when you’re the, I want to look at things from both perspectives. Jon Stewart came back on The Daily Show last week and he got a lot of hate because he was in between two sides. But from my perspective, these are the voices that often bring in a lot of reason and just say let’s just call out everything as it is and see how we can look inwards of how we can continue to be better. But it’s always fascinating because you’re going to get some pushback from the side that agrees on one thing and then pushback from the side that disagrees on the other if you’re optimizing in your own way.
Steve: Are you seeing patterns in the people that you’re mentoring in terms of what topics or questions you’re helping them with?
Nizar: I think the biggest one is there’s an obsession with the research as the end goal. I think that’s the one that’s just becoming more and more and more apparent. There’s different reasons that when somebody’s starting their career, it makes sense that they think, okay, I’m here to do research. There are some people that are more mid or later career where that’s what they’ve learned how to optimize throughout their career because that was their success criteria. So there’s various flavors of that same thing. But you often hear a lot about like I want my methodology to be, like I’m focused on my methodology or I want to do more foundational research or it’s very anchored on research as the end goal.
Even in our interview process, we interview a lot of amazing candidates with amazing resumes and as they’re presenting their case studies, they gloss over why they did the research or they gloss over what happened after it. But they take a lot of pride in the thing that they did, kind of the actions that they took as a researcher. I think that’s the biggest, for me, gap that I see between a lot of the conversations that I have and where I believe research should be positioned as more of a tool to drive decision-making rather than an end goal.
Steve: I don’t know a ton about how mentorship could or should work, but are there things that you are able to say or do in these interactions to help somebody shift their perspective to what you’re talking about?
Nizar: It depends on the relationship I have with the person too. So to be honest, that kind of dictates a lot of the conversations that happen and honestly, like how hard I push back. There are some people that I used to manage in the past who I’m very comfortable telling, “You’re just absolutely wrong and stop doing it the way you’re doing it and here’s how you can be more effective.” You can’t do that with somebody you barely know. And you try to nudge it in terms of like how do you expand your thought process beyond what you’re doing into why you’re doing it? And how do we kind of like reset your tone in terms of the perceived outcomes of the work that you’re doing?
And I do a lot of resume reviews and I think that’s a place where people seek feedback and I usually call out that a lot of resumes that I see for researchers read like job descriptions. And I try to tell them, “What’s your superpower? What’s your story?” When I read your resume and I see conduct tactical and strategic research, conduct qualitative and quantitative research, that reads like the job description that doesn’t give you an edge over other applicants and I don’t have any context over why you’re doing it, why you’re doing it, what you’re able to do, move forward. And of course, at the end of the day, the research in itself is core. That’s table stakes. Being a strong researcher with broad methodology and being able to tackle, again, different types of problems is core to the job. So we don’t want to hire a researcher who doesn’t know how to do research. But how does that researcher connect that great research to why they’re doing it and what impact that it’s having is the biggest gap that I tend to see across resumes, some interviews, some mentoring calls, that I think there’s a continued opportunity there.
Steve: What didn’t we talk about yet today that you think we might want to cover?
Nizar: I can interview you.
Steve: I’m willing to try.
Nizar: The question I have for you is you have a new edition of a phenomenal book, so congratulations. That’s something that I hope every researcher has read the book. I recommend every researcher read that book as well. Ten years later, what are the areas that you’ve seen evolve or change?
Steve: Yeah, the context in which research takes place is totally different. You know, we didn’t have language around operations, for example, and operations as being separate. Like it took me a long time and very recently even it sort of distinguished operations from logistics. And I was, in fact, I was resistant to the idea of research ops because it takes away some of what the researcher needs to do. You know, if you try to recruit a part, recruit participants, you understand the space just by going through that. And so I had this naive view.
Now I’m not even answering your question, but I used to have this naive view that like, oh, well, that, you know, recruiting is part of figuring out how this population thinks and works and how to work with them and so on. And that ops is going to take that away. And I think I’ve just only recently sort of started to understand that research operations is about supporting the organization to do research, not to take the burden of tactics and logistics away. It’s about sort of infrastructure and so on. So, you know, trying to have a more sophisticated conversation in the second edition about logistics and operations. These were things that like things stuff that’s really important from a legal and compliance perspective right now in research was just kind of, let’s just see if we can avoid legal finding out about this kind of in the past. I think, you know, like this podcast couldn’t really have existed 10 years ago. There were, well, I shouldn’t say that I’ve been doing it for a long time. Maybe that’s the wrong number. I don’t know.
At some point, there were far fewer people who were doing what you’re doing, who are building teams, you know, bringing leadership and management to research. Researchers were abandoned or worked for design manager or something like that. The idea that research could be a peer. I mean, all the things started off talking about could be a peer to another function and work proactively. Those were sort of ideas and aspirations, but you didn’t see that as much. So I think, you know, the profession has matured and there are more teams with leaders with career ladders and, you know, clear ideas how to interview what they’re looking for that as an in-house profession. It’s just much, much more mature.
And I was just thinking today about, you know, that phrase that Kate Towsey came up with, People who Do Research, like that’s, she came up with that term in 2018 as far as I was able to determine. And that’s fairly recent, but I feel like, oh, giving it a name, like there are researchers and you’ve mentioned a few times, like researchers and research. We kind of come back and forth on that there. And like you said, there’s all sorts of customer contact going on, but give, you know, creating a name that sort of says, here’s what, who’s who researchers are. And here’s this other category of people that’s also are doing research like that. Having that label, I think, clarifies a lot. A
nd yes, we have democratization debates. And it’s not like the problem is solved by giving it a name, but we’re clearly at a point where we can say there’s different types of research happening, which is a point that you’ve made. And there’s different people doing research, whatever we mean by that. And that those are all sort of different considerations. So it’s not a solved issue, but it’s a much more clarified issue than it was 10 years ago. You know, and again, not really what you asked, but, you know, I think a lot of the fundamentals that my write about, like how to ask a question, how to ask a follow up question, how to listen. You know, I’ve had 10 more years of doing that and 10 more years of teaching it. So I’m, I have just more examples and more stories and more clarification and more nuance. So that those are the fundamentals I think of interviewing, but I think I can explain it better than I could before because it’s just practice, practice, practice.
Nizar: This is great. You talked a lot about kind of the evolution of research and some of the things that are different. What’s the hot take about the world of user research that you find yourself disagreeing with?
Steve: Wow.
Nizar: Putting you on the spot there.
Steve: Yeah, no, that’s good. I mean, I am kind of just like, I feel like that Grandpa Simpson meme of like, you know, old man, shout, yells at cloud, whatever that is, I feel like, I don’t know, just being my age and my grumpiness like I am. I don’t know. And I guess I could justify that, you know, you just the longer you live, the more sort of hype cycles you kind of go through. And I’m just sort of reluctant to engage with, with, with hype stuff. So yeah, AI is a big hype topic. I’m amazed that I’m the one bringing it up in this conversation, because usually everybody else that I talk to has to bring it up right away. So we’ve gone. So thanks for making me do that. And sometimes I’m sort of resentful about, it’s like, I don’t want to have a hot take on research. And I’m resentful about the overwhelming amount of hot takes about research.
The most recent episode of this podcast that I posted is all about Noam Segal’s hot take on research. And so no disrespect to him for that he thought about it a lot. So I don’t know, my hot take is almost like an anti-hot take, like, can we all can everybody just chill out about AI or about the end of everything? I think we have a lot of, like short term ism or just like immediacy, we don’t we see what’s right in front of us, and we overreact. I include myself in that. And that’s not just that’s just human nature, I think. And you know, whatever, like the LinkedIn pundit info cycle, you know, entertainment complex, whatever that is, we all have to have an opinion about something and see what’s happening with research right now, but we’re at an inflection point. And so we don’t know. And maybe it’s okay not to know, which something that you’ve said a few different ways.
So I don’t know, my hot take is like to be anti-hot take is like, it’s maybe okay. And maybe that’s just my privilege speaking, like, you know, if I was younger, trying to make a certain kind of name or get a job, I might feel like I need to come in with an opinion about something. But I think there’s a little bit of peace and calmness that I would like to, you know, nurture within myself to like not react so much to kind of the change around us, the world feels very dynamic and uncertain and complex and worrisome. And, you know, I would I like when our conversations that we have in our professions collectively, sort of soothe that and not add to that.
Nizar: I love what you’re describing and just to add to it a little bit. I’m hearing a lot of the takes on how research is now fundamentally different. It’s doing things wrong. There’s definitely an over-exaggerating, an over-exaggerated sentiment behind what’s going on there. There’s a macroeconomic condition. You hear a lot of over statements across domains. I think when you’re in research, you just feel it more because a lot of your circles. At the heart of it, a lot of different domains got hit pretty hard. If you ask many people, they’ll tell you that it’s their area that got hit the hardest. I hear the same from product managers. I hear the same from software engineers. Let’s not even get started with the recruiting and some of the operational support.
At the same time, I see it as always a good opportunity to just reflect on what we’re doing and what works and what doesn’t, and we continue to iterate. It doesn’t need a big research is dying kind of header to encourage some of the discussions and the conversations, how we continue to evolve. It wasn’t too long ago where every researcher was called a usability engineer, and things will continue to evolve and things will continue to change, and that’s okay. This was part of the course. I’m just excited for the continued trajectory of researchers becoming key drivers and business leaders who represent users as their core mission. But it takes a few steps, takes a few optimizations, and I think it’s just part of the exciting journey of a domain that’s still relatively young. When you look at the grand scheme of, at least in the tech world for example, the other areas that have been around as fundamental to the product development process.
Steve: I think that’s us ending on a high note. Thank you so much for a great conversation, turning the tables a little bit and sharing so much. It was lovely to get to chat with you.
Nizar: I really appreciate it. It was a wonderful chat, and thank you so much.
Steve: There you go. That’s our episode. If you made it all the way to the very end, give yourself a hearty pat on the back for listening. Please spread the word about Dollars to Donuts. You can find Dollars to Donuts in most of the places that you find podcasts. You can raise awareness even more by reviewing the show on Apple podcasts or wherever it is that you’re finding it. Check out Portigal.com/podcast to find all the episodes, including show notes and transcripts. Our theme music is by Bruce Todd.
The post 37. Nizar Saqqar of Snowflake first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Nizar Saqqar, the Head of User Research at Snowflake.
For a domain that takes a lot of pride and empathy and how we can represent the end user, there’s a component that sometimes gets overshadowed, which is the empathy with cross-functional partners. With every domain, product design, research, there’s people that are better at their job than others. I dobelieve that everybody comes from a good place. Everybody’s trying to do their best work. And if we have some empathy to what their constraints, what they’re going through, what their success criteria is, how they’re being measured and what pressures they’re under, it makes it much, much easier for them to want to seek the help of a researcher to say, “Help me get out of this. Let’s work together and let me use research for those goals that are shared.” – Nizar Saqqar
Show Links* Interviewing Users, second edition * Nizar on LinkedIn * Snowflake * How to use MaxDiff analysis * You Might Not Like What Jon Stewart Has to Tell * Old Man Yells at Cloud * Noam Segal returns
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. Today’s guest, Nizar Saqqar, actually brings this up in our conversation, but I’m going to remind you myself that there is a new edition of my classic book, Interviewing Users. It’s now available 10 years after the first edition came out. Of course I’m biased, but I highly recommend it. And hey, it makes a great gift for the researcher or research-adjacent person in your life, or persons. If you haven’t got the second edition yet, you can use the offer code “donuts” that’s D-O-N-U-T-S for a limited time to get a 10% discount from Rosenfeld Media.
But now, let’s get to my conversation with Nizar Saqqar. He’s the head of user research at Snowflake. Well, Nizar, thank you for coming on the podcast. It’s great to get to chat with you.
Nizar Saqqar: Thank you for having me. I’m really excited for it.
Steve: Let’s start with an introduction from you. You want to say a little bit about your role, your context, anything to kind of get us rolling? And we’ll go from there.
Nizar: Absolutely. I’m Nizar. I lead user research at Snowflake. I’ve been here for about three years. It’s been a pretty exciting adventure. When I started first researcher at a company that’s been around for 10 years and really doubling down on showcasing the impact of research, why we need to scale, and we’ve been scaling nonstop in today’s environment, which has been a pretty exciting challenge. It comes with the fun of it, but comes with the challenges as well, and I think more to come. To take a step back and try to simplify it as much as I can,
Steve: What kind of company is Snowflake?
Nizar: Snowflake enables organizations to store huge amounts of data from many sources in one place. So it empowers organizations to make the most out of that data. And as we’ve scaled the company, we’re continuing to push the envelope on platform-level offerings that try to enable native app developers to do the development of data applications.
Steve: What are some examples or vertical scenarios that we might know about?
Nizar: So the simplification of it is get your data in a place, make the most out of it. Snowflake will help companies do that as efficiently as possible with as many use cases as we can. It’s definitely not in the day-to-day conversation.
Steve: What are some examples of what Snowflake is?
Nizar: It’s not a B2C product, but at the core of it, it really starts with the data warehousing of getting the data engineer brings in all of the data from many different places, many different sources into one place for storage, and then making it usable for other users, maybe like the data analyst or the data scientist who makes something happen out of it as an outcome. And as I mentioned earlier, building the native app development framework, it’s been exciting to see all of the, think of them as the more classical kind of software developers that are now into our ecosystems to get closer to the data. So it’s a pretty complex ecosystem.
We also have a marketplace that then kind of introduces the dynamic of a provider and a consumer and the business decision makers who are coming in for that transaction. So it’s a pretty intense ecosystem that magically all connects into just making the most out of your data.
Steve: So when you came in as a researcher, what did you observe about how this company was thinking about its users or thinking about what it knew or didn’t know? Do you remember that early process?
Nizar: Yeah, and to be honest, that process started before I even started. Even in the interview process, I really wanted to be sure that the company is thinking a lot about their users. They’re thinking a lot about how research can integrate. They’re thinking about challenging some of maybe the perceptions of what research can bring to the table and just having some of these tough conversations even before. And I will say that where we are definitely lucky is that Snowflake does put, do what we can to make it as great of a product for our users as like a core value of the company.
The interesting thing with that is that it brings in a lot of data points from a lot of pieces. Now you have a lot of user perspectives from the sales team, directly from the product team. Then you have all of the metrics and dashboards that you’re following. So you actually get a lot of, you get a lot of data. You get a lot of points that might actually make it harder for the product teams to action on or prioritize.
So as I started, I kind of wanted to first take a moment to better understand the domain, really kind of find my footing, know what’s going on, build the right relationships and start with something that’s very low-hanging fruit of saying, “Hey, let me just build credibility. Let me just come in and say I can add value very quickly and then scale that up.” It’s been interesting to see how the role has continued to evolve since that day one. It really started off with, we’ve been here for 10 years. This guy is here. And it’s really kind of evolving into user research is just a critical component of how we think about product development.
But it’s taken many phases that we’ve had to adapt as we continue to go, starting off with the very tactical, then zooming out into something that perhaps is more strategic, then shifting focus into our hiring strategy and our hiring rubrics and how we interview, going all the way into what we define as success criteria, performance evaluation and how we integrate research into the overall product process. And it just doesn’t stop. So the role itself has been changing over the past three years and I perceive it to continue to do that.
Steve: Can you give an example of a tactical, sort of quick win that you would approach kind of coming in, in those early days?
Nizar: Yeah, absolutely. And I think for me it really starts with what is something that is tactical enough, close enough to wanting to launch. There’s enough resourcing, but there’s some level of disagreement in the organization how to proceed. And seeing if there’s an appetite to make research kind of be a tiebreaker of sort or really find the right balance between the two. I found that it’s very rare that there’s option A or option B wins and there’s always like components of each that kind of resonate that when you bring them together, you kind of find something that really resonates to the actual flows.
And if I remember correctly, one of my very, very, very early contributions was kind of around something as simple as a concept evaluation. And I think those are the methods that are just going back to the basics and some people take for granted, but you’re just coming in and you’re saying, “Let’s just test it and see what’s resonating and what’s not.” And coming up with some actionable steps that align multiple teams that might have dependencies on each other to find the solution that may not make everyone happy, but at least everyone is aligned that, “Okay, this seems to align on a path forward from a user lens.” So then it just continues to evolve.
I was talking recently about how coming off the bat, just seeing that there’s an overwhelmed product manager who says, “Hey, I have 20 features that I’m asked to ship.” And my role there was to come in and say, “Let me help you do a MaxDiff survey to just make a case for some things that you should actually deprioritize so you can make progress towards some of the top features that you want to run through.” And I think that was part of the evolution of, “Okay, we could use research for many different use cases and in different areas where we could integrate with the product roadmap.”
Steve: So I think it’s super interesting that you’re using the interview process, I guess, to understand the context that you’d be coming into.
Yeah, I’m wondering, and I don’t know, I’m going to ask it like a binary, obviously it’s not, but, you know, how much of a mandate were you given versus how much you were trying to figure out what the needs were and, you know, make recommendations appropriately? That’s a terribly leading question.
Nizar: It’s not, though. It’s kind of interesting, because there’s– when someone opens a position, when somebody asks for a headcount, for the most part, they have an idea of what they’re looking for. They have an idea of what they think the success criteria is. In my case, I was hired by a head of design who had an idea that I could help elevate the design team. That was kind of like the primary premise. And the interview process, that comes out, and it’s a really exciting thing. And a head of design is really excited to be like, “Now I get to finally get to support design with research.” I started digging into the appetite of, “But how do we kind of expand beyond the design?” If we’re to look at the pie and we’re to say, “Instead of making that piece more efficient, how do we just make the pie bigger? How do we get the design team holistically more involved and have that impact from earlier stages as well?”
So look beyond the design research component and don’t worry about it. I’ll make sure you have a good story for where your organization is growing in terms of impact beyond the pixels. And I think that was a really good back and forth that early on showcased that there’s a lot of action and appetite for, “Hey, if you can define something outside of what I have in mind that you perceive could be even more value-adding for the organization, that’s what we’re optimizing for.” And I think that was a good start of saying, “Okay, I won’t be in a situation where somebody comes in and says, ‘This is what you need to do.'”
I do hear a lot of stories of, “The researcher comes in and all you can do is one-week sprint and stuff, just do a study every week, and it’s non-negotiable.” And it was pretty important to gauge that, “Hey, can we just align on value for users and value for the company and value for the team as the criteria, and let me do what I need to do without a specific framework of how I should be operating within those objectives?”
Steve: I love the phrase impact beyond the pixels. That’s like, that’s a pull quote or that should be a title of your next talk. So that sets you up then to find that overwhelmed PM. And if I understand it correctly, you’re kind of saying to them, like, did you know, hey, this is a situation you’re in, like, here’s an approach that would help you. That’s kind of where my, maybe where my mandate versus discovery question comes from. It sounds like you are finding opportunities or finding places to have impact where that PM is not going to ask you, hey, can you do a MaxDiff survey? You’re coming in, seeing the situation saying, yeah, here’s a way that research can unblock you.
Nizar: This was a fascinating story altogether, and there’s some more context behind it, which is kind of funny when you look at it. That PM was super excited before I started, messaged me before I started, started telling everybody that me joining is going to be a game changer, was the friendliest PM I’ve ever met when I started. And then back then, my manager was saying, “Hey, we think this is the most ambiguous thing. We need to redefine a roadmap. You need to put a really found– I think this is a really foundational research problem.” So when I went to the PM and I told him, “Hey, we can work together on this,” and I’m actually excited to team up, he actually said, “No, I’m not interested.” And I told him, “Let’s take a step back and let’s speak about why you’re not interested, what’s–just what’s on your mind. Let’s not talk about the research. Let’s talk about the problem you’re solving.” And his take was, from my experience with research in the past, a lot of the times it does take a lot of attention to keep up with all that’s happening, be part of the interviews, and then you come back with a lot of insights that I frankly don’t have resources to do anything with.
So if you come to me and you say, “Here’s 10 things you need to build,” I’m just going to put it at the end of the JIRA board as items 21 through 30, and I’m not going to get to them. So the key learning for me back then was, okay, everybody kind of perceives my role and how I can solve this problem very differently, and I really need to set some shared language and shared expectations of why I’m here. So that’s when I was like, “Hey, how about we do this? How about I’m just going to go into your JIRA board? I’m just going to steal the things that you have there. I don’t need you to be involved, and let’s make a case of why you don’t need to pursue all of these features at once. And let me do the heavy lifting. We’ll team up on it.” And then going back to my manager back then and saying, “I don’t think I need a multi-month, huge effort to start. Let’s just kind of help him get out of the weeds for a bit and just aligning the expectations over what we can do with the research.”
And in this case, it was a core example of research really used to de-scope, to de-prioritize, to say that not everything is equally important. At a high level, when you take single-off, one-off stories, they all come up as high needs, but are they all at the same level of importance for when you look at our user needs and the business value that they bring? And that’s essentially what came out of that Maxx a lot of these are way below when you compare it to what’s really bubbling up to the top. And how do I make the case that doesn’t say everything is there for a reason? But let’s make a case that with the limited engineering resources that we have, we can drive the most value if we really focus most of it on those very specific areas and get those to a place where our end users are really happy with the experience that we’re offering. And that was a different mindset. That was a different principle for that PM where it’s like, I didn’t know that we could do that. I didn’t know we could do research that helps me tell a story to executives of why I shouldn’t be doing work or why I should say no to some of the work that’s coming up.
And then that led to me really wanting to define some of the languages used around why research exists and why research is at that company. And the wording that I tend to use, which may not apply for everyone, but I try to take it around driving the allocation of limited resources into the most impactful efforts for our users or organizations. And if we can have that be a shared language mandate for what research is optimizing for, it takes away some of the misconceptions here and there and where there comes some of the tactical things like changing the title names from UX researcher to user researcher or changing some of the way that we present decks or reports or internal documentation. So there’s some tactical things that come with it, but at the core of it really linking it to the intersection of user value and organizational value.
Steve: So to get somebody unstuck, when we get so overwhelmed, we can’t even see our way out of something. I don’t have time to do your solution. I’m just, you know, I’m treading water here. So I love that aspect of the story that you found an approach that also is about limited resources and that took that person where they were at. And I don’t hear complaining about a stakeholder that wouldn’t commit to the project, you found an approach that you had a 10,000 foot view and it could kind of see how you could add value. And still, I think you were fairly new to the organization at that point. Is that right?
Nizar: I almost had no idea what was going on. I needed to rely on them to make sure that the items in my MaxDiff actually made sense. Even when I was mentioning it earlier, and I come from a B2C company. I come from a place where that ecosystem was extremely new to me. So of course there was some collaboration there, but I tried to keep it as lightweight as possible as I make sure that I have the right pieces but without overwhelming them.
And I love what you’re saying. I love the way you’re describing it. For a domain that takes a lot of pride and empathy and how we can represent the end user, there’s a component that sometimes gets overshadowed, which is the empathy with cross-functional partners. With every domain, product design, research, there’s people that are better at their job than others. Sure, for the most part, I do believe that everybody comes from a good place. Everybody’s trying to do their best work. And if we have some empathy to what their constraints, what they’re going through, what their success criteria is, to be honest, how they’re being measured and what pressures they’re under, it makes it much, much easier for them to want to seek the help of a researcher to say, “Help me get out of this. Let’s work together and let me use research for those goals that are shared.” And at the end of the day, it is still user-driven. It’s still based on the data that we’re getting and we’re able to drive direction. Finding ways to go with the flow while still having a strong perspective over what’s best for the users, rather than feeling that the role of research is always to be on the opposing end of cross-functional partners, could be a really powerful tool. And in these cases, all it leads to is that intersection of product impact and user impact, which I think is the end goal.
Steve: It is a great story that this person was enthusiastic for you and reaching out and was excited about research. And when you think about what opposition is sometimes, I think it’s easy to sort of demonize someone and say, well, they don’t, they don’t get it. They don’t believe in research. They don’t like me, whatever kind of escalate. And here you started with the best possible out the gate dynamic with this other person. They were a fan. They were like welcoming you and they couldn’t wait. And still they had a concern. And so by identifying that and coming up with the right approach that suited all those constraints, you get to them the kind of impact that you’re looking to have.
Nizar: And it kind of makes sense. I mean, if you really think about it, the definition of what a designer does, the definition of what an engineer does, for the most part, is pretty material. You finish your effort, you pass it on. Eventually, the thing that that designer or engineer touched is the product that you end up using. What that does is that for areas like research, there’s more fluidity, and the perception of what you’re here for. So that fluidity could be a great thing and could be an awful thing, because at the end of the day, it opens up a lot of opportunity to set the expectation of why the researcher is here.
But it takes a lot of work to get people to align, because they’re also basing it on their past experiences, basing it on their biases, basing it on whatever good experiences they had, but also whatever bad experiences they’ve had with research, and a lot of that bias of how the work may or may not be precisely presented to what the end user sees just opens a lot of these gaps and unknowns that I think just plugging these holes and making sure that the narrative is clear around why the researcher is coming in, to me, I see it as an opportunity. It’s not always the most fun process to cover some of these holes and make sure that there’s no gap in the perception of why the researcher is here.
Steve: We started with the foundational, this tactical stuff that you’re doing, but maybe we can look at the whole arc of creating that more evolved understanding in the organization across all these folks about what research is here to do.
Nizar: Building up on that first story that I was saying, I made my success criteria be less about the research output and more about how’s the research being used, how’s the research actually integrated directly to the roadmaps, and some of that lives till today with the team. You’ll see a lot more emphasis on how often is your research referenced in a cross-functional document. Then you’ll see about what is the quality of your report, for example, as some of the success criteria that we have.
But taking it back to that initial journey, there’s a disadvantage and advantage of being the only researcher back then. The disadvantage is that it’s overwhelming. There’s a lot to cover. The advantage is that gives you the ability to say, “I’m not going to do it all. I can’t do it all.” And you get to pick and choose a bit in terms of where you can foresee some of the most opportunity is.
And at the same time, you look at where some of the path of least resistance could exist, too. So if there was a huge problem where the path of resistance is pretty significant, the question that I need to ask myself is, is this where I want to continue proceeding? Should I continue butting heads to be included there, or should I go find that place that is a bit more welcoming to changing their processes and their approach and just kind of use that as a case study? Once that case study lands, how are we showcasing that case study?
And I’ve never been a fan of visibility for the sake of visibility, but especially in the earlier days of research, there’s a lot of advantages for visibility as a case study of how research could work to empower those around. And that became key. And basically what happened right off the bat is we started hearing the sentence, “Well, I want research. Where’s my research? Why don’t I have a researcher?” And the demand for research support started coming more organically from the cross-functional teams. So it wasn’t on me to necessarily say, “I need people. I need to grow the team. I need to do this.” It wasn’t an ask from the research department to grow. It was an ask from cross-functional partners who have seen how much more effective and how much more efficient they could be with the appropriate level of research support. And that just kind of creates more of that shared language, the shared narrative of what the organization is looking to do with the research and work closely with it, but also what’s the success criteria for the research team.
And as we started to scale, it became more and more important to set pretty stable goalposts to gauge what success looks like and what our objectives are and being very intentional about kind of not falling into the trap of making research the end goal where you’re out of the loop of what decisions are actually being made and you’re trying to do like a one-size-fits-all approach to research that says, as you get more senior, your research gets more complex, which I don’t believe is the best definition of researcher seniority, but really kind of anchoring it in how we’re able to continue driving the product roadmap forward. Even then, there’s a lot of back and forth that goes into it.
Steve: When you started to get these requests from people, we want research, where’s my research, the kinds of things that folks were hoping for or asking for, did that line up with what you would want to or hope to support them with?
Nizar: When you’re starting the team from scratch, the default is actually about, hey, since you only have one researcher, there’s only two of you. Do you need to do some intake form to take everybody’s input and then try to cover as much as possible? And I put my foot down that I don’t think this works. I don’t think that’s the most effective way to do it, and I don’t think hiring a researcher starting off with a service model that says, hey, you’re not part of the team. You’re an outsider who will do research and come back will be the most effective way to drive meaningful change.
So I approached it with a lot of support and I approached it from a point of view of let’s continue that as a proof of concept. Let me hire a researcher and embed them directly in one of our most critical strategic teams that has significant strategic significance for the company as a whole, but also has a lot of open questions and a lot of things that we can benefit from a researcher, and that’s all they’re working on.
And of course, you get the pushback of, well, what about these other teams? And my take is you’ve been operating with that research for 10 years. We can weigh it a bit more, and let’s continue gauging how things go there. And that kind of starts it off with setting up the researcher for success and empowering them as a core member of the team, challenging the notion that they’re there to take requests or answer questions and have them be able to actively predict where there will be blockers and how they can get their research ahead, maybe like three months ahead, six months ahead, to be able to actually be ready for the decisions for when the actual time to make the call comes.
So it’s essentially making sure that research is proactive rather than reactive. And that model worked. That model worked great with that team. We hired a phenomenal researcher. Until today, you’re always excited about the first hire being a phenomenal person on the team, and you start replicating that model with different teams for areas that also are strategic to the company and have a lot of ambiguity, and that kind of becomes the framing in terms of unblocking, creating alignment, efficiency, and how we can just continue to scale from there.
Steve: At the risk of oversimplifying, I guess I’m hearing in your answer where I was going wrong on my question, it’s the difference between this team needs a researcher and this team needs research. I was starting with this team needs research and you’re putting a researcher in there and they are figuring out the questions, being proactive, that’s very different than that service model intake form thing.
Nizar: Yeah, correct, and generally I think teams that start off being user-centered at times think they’re doing research. There’s a lot of types of research, right? So sometimes you’re talking to, for us, you know, we’re a B2B company that has great relationship with our customers. So you think that, hey, I’m talking to their sales engineer or somebody called me for a meeting. Like, I’m doing some level of research. I kind of have a take of, you know, I’m not here to keep, go do your thing, but at the same time, I’m not here to demarcate. I’m not here to, like, empower as many people to do research as possible.
You know, my role is to be a cross-functional stakeholder, and I will jump in with what the problems are that we need to solve together, and let me find ways to deal with them. So I think in this specific case, there was always an acknowledgement of, hey, there’s stuff that we don’t know, and we need some form of research. The definition of what research is and how it’s going to be incorporated is the thing that needed to be tightened up a bit more, and then integrating the researcher as in the framing of this is a cross-functional partner, not a source of research, if that makes sense.
I started changing the language around the expectations of even when they’re invited, even when people go to them, even when what topics they’re having in their one-on-ones. So it’s less about, like, here are some questions that I want and more about, hey, I’m struggling with this thing and we talk through it. So they all tie in together. I think to my point earlier where there’s a little bit of the path of least resistance when you’re starting and you can pick one team, you know, that was, you know, you can call it a lucky privilege of saying, okay, there’s a team that could be ready. I’m seeing conditions that are priming a researcher to be successful here. Let’s go with that model, with that team, and continue scaling from there.
Steve: I mean that reminds me of just your interview process, you’re looking for those conditions to understand what that is before you start and now as you grow your team, you keep looking for those conditions within different parts of the organization to see where research is going to, could go next and have the most, again you’re really focused on the impact and the product and the experience in the company.
Nizar: That’s a good summary, and it feeds into our interview process. I do, you know, we try our best to make our interview process as applicant-friendly as possible where it’s not convoluted, but at the same time, it covers a lot. So a key part of it is who joins the team and what’s their approach as well. We do tend to see, like, there’s a specific type of researcher that tends to do best, and usually we look at researchers who do have the depth and soundness of research methodology as kind of like a core expectation. But then they layer on top of it the user-centric process and thinking, you know, when do you integrate at different stages of product development?
You start seeing the kind of the business sense of really wanting to be integrated deeply with the team and solving the problem at heart rather than solving the open question. And then cross-functional collaboration as a core area. I do think that every researcher needs to fully understand what resources the team is working with, whether it be engineering, design, any other blockers, to be able to come forward with the most effective size of recommendation.
And then we always have that over-layering, overarching umbrella of leadership and teamwork, really kind of looking for people who have a growth mindset, who are looking to help others succeed, who don’t necessarily see it as like their world and like their thing, but just really collectively looking for everybody to succeed together, which I think has been pretty key as we’ve scaled the culture of our team into a team that’s pretty collaborative, a team that’s looking to help each other, and a team where people aren’t competing. And there’s no incentive for people on the team to compete. There’s actually an incentive for them to make each other better and learn from each other. So that’s been an exciting part of scaling from a cultural perspective within the research team.
Steve: I want to ask you to clarify, you used the phrase the problem at heart versus the open question.
Nizar: Absolutely.
Steve: Can you explain what that looks like for, what does that mean for any particular problem?
Nizar: One thing I’ve become really sensitive to, maybe too much so, is when I see kind of a research plan that says our objective is to answer these five questions or six questions, and my take is that’s actually a step removed from what you’re going to do with the answers that you’re going to get. So I like to start it off with what’s the perceived outcome? What’s the perceived objective? What are you looking to learn and why in terms of what’s actionable? And then take that to go a step backwards and say, okay, to get to that effectively, let’s now go into what questions do we need to ask? And based on that leverage, what method is the most efficient and appropriate for what we’re trying to accomplish?
What I’ve seen a lot in the past is even your stakeholders think they’re asking you the right questions. How many people have been asked, can you create personas? Can you tell me the different types of users? And a researcher goes off, does this for a month or two, and then they come back and nobody knows how to use them. And to me, that’s the problem that I’m trying to avoid as much as possible and just saying, okay, you want personas. What are you going to do with them? What’s the decision you’re trying to make? And often coming to the conclusion that you don’t need that at all. What you need is something much more simplified. Or we could actually get a pulse check to start getting you some signal of the answers that you’re looking for that will help with that decision-making process. And then we can decide when to iterate or if it’s necessary to iterate.
With open questions, I find that there’s sometimes the danger of over-scoping research efforts. For what you’re trying to do with it. It’s just that outcome of you show up with a deck that has 100 slides, but the team can only act on the first two. And so my question becomes, was this the best use of the researcher’s time versus trying to focus on those first two slides and then connecting that to a longer-term program that we can then create follow-ups on as we continue to learn throughout the process. So in a way, it’s forced efficiency and early hypotheses of how we connect to the impact before even starting off with prioritizing the effort.
Steve: I want to follow up something else that you said, you were describing a lot of the qualities that you’re looking for, the mindsets and the kind of abilities, you know, how do applicants demonstrate that information in your process?
Nizar: I could talk about that for a long, long time. So I generally don’t believe that these buckets are a pass or a fail. I don’t think it’s are you good at user-centered thinking or not. How I see it is everything kind of sits on a continuum. And what I’m trying to optimize for is for the level of seniority that this role will require specifically, am I seeing enough of ability to handle different situations effectively that then puts them in a position where they’re going to be able to know what they need to do regardless of what’s thrown at them.
So, for example, let’s say somebody is, we hear this a lot with the break apart of the tactical versus foundational or that, where it’s like, I do this and not the other. And my question becomes why? Why create that separation between this form of research and the other if you’re able to tell a story around pretty much your ability to come in at multiple different stages of product development and say that I can help you across every stage and I know exactly how to do it. And I can help you across every sort of limitation that you have and I know how to do it. And I can help you address multiple different type of issues that we’re facing, whether they be needing of some qualitative research or something that’s more quantified or something that’s quick and dirty or something that just needs a brainstorming workshop and I’m able to just be flexible in where I integrate with the team.
So I know that was a long-winded answer that went all over the place, but it’s really hard to — the reason it’s hard to describe is, I really don’t think research is good or bad, yes or no kind of domain in general. And what I’m really trying to optimize for as much as possible is, does the research applicant have the breadth to be able to tackle as many problems as possible? To me, that’s a much better predictor of seniority and success than somebody coming and saying, “I did this multi-country 12-month research project that was really complex logistically,” which is impressive in its own way. It’s great, but for me, it’s not what we’re trying to optimize for in general.
Steve: And so you’re looking at past experiences that the applicant can, I guess, describe to you, or those kinds of clues to the breadth.
Nizar: We look both at past experiences and we look at some of the hypotheticals as well. So we do have some scenario-based questions where we try to gauge some of the thought process. It’s the thing that I tell people that there’s no right or wrong answer. You’re just going to get a hypothetical and I just want to hear how you think about it. And I want to hear what are the different considerations that you take into account or into place when you’re making your mind up about the best approach and what you’re going to do. How often are you coming in and having the hard conversations about what needs to be done versus steering the conversation in a completely different way versus just saying, “You know what? This isn’t worth the back and forth. Let me just do something quick and move forward.” So at the end of it, when we’re combining the hypothetical with the past experiences, I’m really looking for effectiveness and efficiency under the umbrella of strong and sound research.
Steve: Those are words that are sometimes seen as at odds with each other, but I think you’re talking about how they’re in support of each other, that effective and efficient doesn’t mean that you’re not sound, doesn’t mean that you’re not, like you said, solving the problem at heart versus the open question. That seems like a key mindset that you’re bringing to this.
Nizar: A hundred percent. And I hear that sentiment every now and then. I hear the sentiment of, “Oh, if you go too scrappy, you’re doing really terrible research.” Or, “It either has to be great research or it’s terrible research that is fast.” And I don’t agree with that mindset or that context. For me, it really depends around what you’re trying to learn, what your objectives are. If you’re trying to do something that is an extremely small pulse check, for example, you don’t need to boil the ocean.
I still remember earlier in my career, I joined a team and they just had no idea, they knew nothing about their users. Absolutely nothing. And I was telling them, “Do you have any hypothesis? Do you have any open questions? Do you have anything there?” And they’re like, “We just don’t know. We just know that nobody’s using this feature. That’s all we know.” And we look at our dashboards, we look at our metrics, we have a target addressable market of millions, and we have tens using it. So we don’t know why. And I just came in and I said, “Look, the best use of time right now for me is just to do some sort of a small single pulse check survey. One question, pretty much trying to understand the state of everything. Just for me to have context to get started, just give me some perspective. Am I planning to use this in road-napping? Probably not, but I need some form of context from end users to be able to tell me, “Okay, I have an idea of what’s happening there, and I have an idea of the value add, I have an idea of why they’re churning, I have an idea of why maybe they’re not seeing some value. Let it be scrappy.”
And you get the pushback of, “Well, this is qualitative. You need to do in-depth interviews for that.” I’m like, “No, I don’t. I really don’t. I don’t need to invest 40, 80 hours just to get an idea of what’s going on if you can do this in 24 hours, and then take that as an entry point into something that’s more detailed, that’s more rigorous.” So for me, it just goes back into linking the amount of effort to the projected outcome and really just finding the thing that works for next steps. And in this case, we did end up actually needing to go very in-depth with foundational interviews and a full design sprint, and then going to concept evaluations and stack rank.
It ended up being a really complex process over maybe the course of a year that really turned around a product that wasn’t used, a product that had millions of users. But at the start of it, I did not have the luxury of saying, “I just need to go away and do in-depth interviews,” because the research domain says that qualitative is not allowed in a survey. So sometimes I think just breaking the rules in our domain is very okay, as long as you know why you’re breaking the rules and what you’re going to do with the insights that you have.
Steve: This research effort that you’re scoping at any point may not be, or probably isn’t, the only time you’re ever going to learn anything. And so, you know, as I’m taking that away from you, then I can sort of feel some of my own anxiety just like ebbing away, like, of course, right? If you think of research as a longer-term thing, like, what’s the question we need to ask now? What’s the right amount of effort for right now? Okay, everybody, we’re not going to get everything. We’re not going to boil the ocean, as you said. There’s more to do, but here’s where we are right now. And so, yeah, that good versus bad research thing is it says we’re only going to do it once, and it’s kind of this monolith that’s either going to answer everything or not answer anything. And these gray areas you’re describing are, it’s a gentle reframe for me, I think, about where I sometimes feel anxious about trying to tamp down the commitment or the investment.
Nizar: As long as we have the right data point for the right decision that’s being made, if we’re coming in and saying, “We need to invest all of our engineering team of this one single customer satisfaction open-ended box,” I get an anxiety attack. I get it. But sometimes that’s not the decision that you’re making. Sometimes the decision — the pros and cons that you weigh, the cons of having something that’s scrappy and fast are justified when you look at the pros of being able to get ahead and then establish a research roadmap that actually gets you ahead of the product team. So that’s the consideration for me.
And to your point earlier, product development is iterative, and I think people forget that sometimes. People forget that even if you launch something, that team is still there, and that team will still continue to want to optimize it in some shape or form. So if anything, I feel researchers should take some comfort in that and saying that, “Okay, if I miss the boat now, how do I get ahead and say, ‘Okay, for the next iteration, for the next thing that’s happening, I’m able to get ahead and have some things ready in time?'” And acknowledge that the same way that product development is iterative, even the most foundational research efforts, you’ll end up having to iterate on in some capacity. I haven’t seen a world where a research deck is still relevant years later, and nobody has ever touched that topic again.
Of course, you want to minimize how often we redo work that’s already been done, but everything changes. Once you start having a user base, the kind of data that you have is different. In this case, when you have tens of users and you go into the thousands, the kind of feedback you’re starting to hear is already different. The usage data that you’re starting to get is different. You’re able to use telemetry a little bit more than when you had nobody. You start being able to triangulate in a way that you just weren’t able to earlier. So iterations are good, and that doesn’t mean don’t do really deep foundational generative efforts. They’re just a time and place to say, “This is my time to get scrappy, and this is my time to dive deeper into the topic.”
Steve: If I didn’t think about it too deeply, I might, you know, sort of have this reflex that says, well, when we know nothing, that’s when we have to learn everything, that the foundational work comes at the beginning. But you’ve got a number of examples where you’re coming in and seeing a big gap and saying, no, it’s not, this is not the boil the ocean time, it’s the quick win or the thing that we can act on or the scrappy thing. And no one believes that A is B. No one believes that the scrappy quick thing is, in fact, going to answer all the questions. But you’re helping take action, you know, within the constraints that are there.
Nizar: I’d say caveats. Tell people there are limitations to what I’m doing. We are aware of that. Every research method we do has limitations, and I’m yet to come across any research study that has solved everything or is now claiming that we have learned everything about our entire user base or our entire feature area. And that’s the reason researchers continue to be in the same role or on the same team for years. There’s always a lot to uncover. And a lot of the time, just really weighing the cons of coming in early and saying, “You know what? I just started. I’ve been here for a week. Let me go disappear for three months or so.”
And I get it. There are ways that you can incorporate your cross-functional partners. But for the most part, especially as somebody’s building credibility, starting with data is much more effective than starting with raw data and giving you some form of, “Here are some next steps,” where often the next steps are research. When I did that Pulse survey early on, the next steps were research. Now I needed to go and actually do in-depth interviews to learn more, but at least I had some litmus of, “What am I talking about? What is my script going to have?” So I’m not finding myself in a situation where I’m interviewing, even if it’s 10 end users, I’d be like, “Can you tell me anything? I don’t know where to start. The team doesn’t know where to start.” But I had something. And for me, the value of that effort, even if it just fed into the definition of a research template for the next steps, that was value-adding, and that saved me a lot of time.
Steve: I’m going to switch topics a little bit here and go back to something you said, I don’t know, before. And I just — maybe you can unpack this or just clarify it. I think you were saying that, you know, that you look at, for people on the research team, you look at number of references or citations of research work in the work product of other cross-functional teams. I’m saying this really poorly, but did I capture that at all?
Nizar: It’s a good summary, and I’ll also give it an asterisk and say, “Not as the only signal, but it’s an effective signal.”
Steve: Yeah. So I have a bias against that. And that’s, of course, coming from someone that doesn’t work inside an organization. So my bias is maybe just hypothetical, but — or just from conversations. And maybe that — maybe that asterisk is really, really important. I agree it’s a signal. I do worry about researchers either getting external pressure or pressuring themselves that this thing, which is essentially out of their control, whether somebody else does something or not, is kind of — is a measure of their worth. Where there’s lots of reasons why people don’t do things and don’t listen to things. And I think you’re talking so much about how to — how to prevent that from happening. Right? The right work at the right time, with the right collaboration, with the right understanding, and all that stuff being kind of scaled appropriately. But, you know, just having spent my career giving people stuff that we’ve agreed was going to be important. And then seeing all kinds of things happen and don’t happen. And to a certain point, there’s a certain amount of surrender, right? Like, I’m going to give you everything that we agreed you need and maybe more, but I can’t control what happens.
So I don’t know. I don’t want to frame this as a debate or anything like that. But I’m open to you telling me that, like, I’m wrong, that I’m framing that wrong. I’m just curious what — you know, how you think about this. It’s not the only signal, but how do you think about sort of how to use that signal or how we should all think about that signal of what somebody else does?
Nizar: The conversation is super valid. That’s where the asterisk comes in. And if anything, I always love the counter perspectives here as well. The reason that I added asterisks here, too, is exactly what you’re saying, that you can’t really control what somebody else does, where it ends up putting some emphasis is encouraging research teams to be very strategic in terms of where they’re prioritizing their time and how they have ownership over the product direction as well. But that doesn’t only go on the researcher. A lot of the conversations that have to take place as well do have to happen at the leadership level. And kind of talking about if we’re to say that the researcher also is to be held accountable for what’s going on there, what’s the collaboration model, and where are they coming in, and are they left out of being able to have that, or is the expectation set that there’s some form of path for them to do it?
And there are also multiple ways from my perspective to showcase that. I think when you look at the referencing, it’s as direct as it gets usually, but even that, it can be optimized. Sometimes you have to take it the other way. Don’t optimize for making sure that your research isn’t docs. That’s not what we’re optimizing for either. But what are the ways in which, as a research team, that for better or worse needs to continue kind of driving the narrative of the value that we bring and connecting the dots to the different decisions that have been made because of the research and the leadership role that each researcher is taking and actually guiding the product roadmap? How could we make sure that we are being very intentional about collecting the evidence and documenting it and being sure that we’re telling our story in a way that does a service to the team members? And often you’ll find researchers in environments where that’s just really hard with their teams. That’s just not how their stakeholders are wired.
And when that happens, my question becomes, what’s the role of leadership and me in a lot of cases in streamlining that, but also what are other ways that are effective in gauging the success of that researcher that do not rely on that being the only mechanism? And that’s where that asterisk plays a huge role. And yeah, absolutely. There are some full quarters. We do performance reviews quarterly, which is pretty intense, and sometimes you don’t get to finish things that are meaningful in a quarter. So we want to give people the benefit of the doubt as well into how the research efforts bleed into the quarters after. But there are some quarters where you’re deep in the research, the team itself doesn’t even have a document, and there’s no way to say, “Hey, this is what’s happening.” But we look for other ways to continue connecting the dots there.
But for me, one thing that I do genuinely care about, and maybe it’s just from previous experiences in the past of seeing where research can get thrown under the bus sometimes, I think for the past many years, I’ve been very intentional about just telling the story of the ROI of the researcher themselves. So not the ROI of research, which I think sometimes gets confused of the ROI of the researcher. I find it that often, and of course it depends environment to environment, company to company, but I find it that often people don’t debate the value of research. They sometimes debate the value of the researcher doing the research. That’s the topic that comes up here and there, and I try to be as intentional as possible to position the researcher and position the organization to give the space for the researcher to be a product leader, not only a research delivery mechanism, if that makes sense.
And with that comes some of the expectations that end up changing. Fingers crossed it worked for me throughout my career. At the same time, I always want to acknowledge that when I say it works for me, there’s also a right time at the right place component of it, and it’s not always on the way that the research is conducted or what the researcher is doing.
Steve: Let’s just switch topics again. We haven’t talked at all about, you know, your overall trajectory. And it’d be great maybe to get a — maybe a summary of how you found user research, what you started off doing, what some things that you did that kind of led you to this role. Maybe have set the context that we haven’t talked about for what you have been sharing.
Nizar: Sure, yeah, I could take it many, many years back. So I actually went to the University of Jordan to study industrial engineering and I was one of the few people who actually cared to have an emphasis on human factors. I don’t know why, but I was always fascinated by the intersection of humans, computers, and business and psychology and all of these together, and I didn’t really know what you could do with it. And it was as early as undergrad that I thought that, okay, this domain seems to cover a lot of those areas, graduated and worked at a company in Jordan under a title that back then was something along the lines of process engineer or something, but in reality it was more understand the inefficiencies in the process and how people are coming in and out of their day to day and how we can make it more efficient. So it had a big kind of research component, and I was aggressively reading about what are some of those programs that I could continue learning in that space, and because where I lived, nobody knew what that was. That wasn’t the thing. And you couldn’t even say, there’s visual design, but you couldn’t really say user experience or UX research.
And moving on to grad school, I went to San Jose State for the master’s program there, and the beauty of that was just the amount of exposure that I had to a lot of different companies and different people who are doing some of the user research work, and it was pretty much a straight shot from there. I went into consulting in a user researcher role, went into a startup where I built up from the ground up into a research and design org. So I was managing research and design, moved on to Google, at YouTube specifically, where I spent about three years, and then when the opportunity came out at Snowflake, it was just too hard to say no to that opportunity. So moved on. I’ve been there for about almost three years now, which is crazy to think about.
Steve: Do you think of yourself as someone that has a superpower?
Nizar: It’s a humbling question, to be honest. The thing I take pride in is I’m always open to being wrong. I’m always open to challenging the status quo and being told that there’s a better way to do it. And I think where I take some of that pride is a lot of the times I hear people that even you look up to throughout your career, and then you get to a point where you’re like, “I kind of disagree. I see a different way.” And trying to challenge status quo for something that could be better is just something that gets me pretty excited.
Is it a superpower? I don’t know. Maybe it hinders me at times, but at the same time, especially in the conversation that’s taking place around research right now, you start seeing a lot of the consistent perspectives of this is right, this is wrong, this is what you do, this is what you don’t. And I try to be very intentional in hearing what are those different perspectives and why are they seeing things differently and what works for me and how do I acknowledge that what works for me at the environments that I’m in may not work for somebody else in the environment that they’re in as well and give people the benefit of the doubt and keep running with what I’m doing.
Steve: There’s sort of two facets, I think. You started off saying that you’re okay being wrong yourself, but you’re also looking for when the conventional wisdom is wrong. Did I get that right? There’s sort of two aspects. It’s like you’re willing to forgo needing to be right, but you’re also embracing or curious about, hey, maybe something out there that’s established as right, the status quo, like you said. Maybe that’s wrong and you’re like challenging that.
Nizar: And think of it like they’re intertwined. Think of it as somebody who is a user researcher. For the most part, you’re kind of looking for best practices, you’re looking for the perspective, you’re looking for the voice of the crowd. They’re kind of intertwined a bit. And I think it’s a solid starting point. It’s much better than starting from zero. Learning from someone is always significantly better than just figuring it out on your own. I mentor upcoming researchers every now and then and I say, “If you can avoid being the first researcher out of grad school, I would avoid that.” But I want to acknowledge that not everybody has the luxury of picking and choosing especially their first job. So take it and learn on the job is better than not having anything.
But it becomes a starting point of, okay, we think this is the best practice. I guess then there’s a tough conversation of, does the best practice make sense? Does the best practice work for me and my approach? Does the best practice work for the environment that I’m within? And where do we continue optimizing it? And how do I continue doing the internal reflection, the internal research on what’s working in the processes that I’m establishing and what’s not? And how do we continue treating honestly my career trajectory as a product that you continue learning, iterating and hopefully making it better?
Often it leaves you at odds with what a lot of voices have in place, but it’s okay accepting that as well and being like there’s no reason for everybody to align on one topic. And it’s always fun. It’s always fascinating when you’re the, I want to look at things from both perspectives. Jon Stewart came back on The Daily Show last week and he got a lot of hate because he was in between two sides. But from my perspective, these are the voices that often bring in a lot of reason and just say let’s just call out everything as it is and see how we can look inwards of how we can continue to be better. But it’s always fascinating because you’re going to get some pushback from the side that agrees on one thing and then pushback from the side that disagrees on the other if you’re optimizing in your own way.
Steve: Are you seeing patterns in the people that you’re mentoring in terms of what topics or questions you’re helping them with?
Nizar: I think the biggest one is there’s an obsession with the research as the end goal. I think that’s the one that’s just becoming more and more and more apparent. There’s different reasons that when somebody’s starting their career, it makes sense that they think, okay, I’m here to do research. There are some people that are more mid or later career where that’s what they’ve learned how to optimize throughout their career because that was their success criteria. So there’s various flavors of that same thing. But you often hear a lot about like I want my methodology to be, like I’m focused on my methodology or I want to do more foundational research or it’s very anchored on research as the end goal.
Even in our interview process, we interview a lot of amazing candidates with amazing resumes and as they’re presenting their case studies, they gloss over why they did the research or they gloss over what happened after it. But they take a lot of pride in the thing that they did, kind of the actions that they took as a researcher. I think that’s the biggest, for me, gap that I see between a lot of the conversations that I have and where I believe research should be positioned as more of a tool to drive decision-making rather than an end goal.
Steve: I don’t know a ton about how mentorship could or should work, but are there things that you are able to say or do in these interactions to help somebody shift their perspective to what you’re talking about?
Nizar: It depends on the relationship I have with the person too. So to be honest, that kind of dictates a lot of the conversations that happen and honestly, like how hard I push back. There are some people that I used to manage in the past who I’m very comfortable telling, “You’re just absolutely wrong and stop doing it the way you’re doing it and here’s how you can be more effective.” You can’t do that with somebody you barely know. And you try to nudge it in terms of like how do you expand your thought process beyond what you’re doing into why you’re doing it? And how do we kind of like reset your tone in terms of the perceived outcomes of the work that you’re doing?
And I do a lot of resume reviews and I think that’s a place where people seek feedback and I usually call out that a lot of resumes that I see for researchers read like job descriptions. And I try to tell them, “What’s your superpower? What’s your story?” When I read your resume and I see conduct tactical and strategic research, conduct qualitative and quantitative research, that reads like the job description that doesn’t give you an edge over other applicants and I don’t have any context over why you’re doing it, why you’re doing it, what you’re able to do, move forward. And of course, at the end of the day, the research in itself is core. That’s table stakes. Being a strong researcher with broad methodology and being able to tackle, again, different types of problems is core to the job. So we don’t want to hire a researcher who doesn’t know how to do research. But how does that researcher connect that great research to why they’re doing it and what impact that it’s having is the biggest gap that I tend to see across resumes, some interviews, some mentoring calls, that I think there’s a continued opportunity there.
Steve: What didn’t we talk about yet today that you think we might want to cover?
Nizar: I can interview you.
Steve: I’m willing to try.
Nizar: The question I have for you is you have a new edition of a phenomenal book, so congratulations. That’s something that I hope every researcher has read the book. I recommend every researcher read that book as well. Ten years later, what are the areas that you’ve seen evolve or change?
Steve: Yeah, the context in which research takes place is totally different. You know, we didn’t have language around operations, for example, and operations as being separate. Like it took me a long time and very recently even it sort of distinguished operations from logistics. And I was, in fact, I was resistant to the idea of research ops because it takes away some of what the researcher needs to do. You know, if you try to recruit a part, recruit participants, you understand the space just by going through that. And so I had this naive view.
Now I’m not even answering your question, but I used to have this naive view that like, oh, well, that, you know, recruiting is part of figuring out how this population thinks and works and how to work with them and so on. And that ops is going to take that away. And I think I’ve just only recently sort of started to understand that research operations is about supporting the organization to do research, not to take the burden of tactics and logistics away. It’s about sort of infrastructure and so on. So, you know, trying to have a more sophisticated conversation in the second edition about logistics and operations. These were things that like things stuff that’s really important from a legal and compliance perspective right now in research was just kind of, let’s just see if we can avoid legal finding out about this kind of in the past. I think, you know, like this podcast couldn’t really have existed 10 years ago. There were, well, I shouldn’t say that I’ve been doing it for a long time. Maybe that’s the wrong number. I don’t know.
At some point, there were far fewer people who were doing what you’re doing, who are building teams, you know, bringing leadership and management to research. Researchers were abandoned or worked for design manager or something like that. The idea that research could be a peer. I mean, all the things started off talking about could be a peer to another function and work proactively. Those were sort of ideas and aspirations, but you didn’t see that as much. So I think, you know, the profession has matured and there are more teams with leaders with career ladders and, you know, clear ideas how to interview what they’re looking for that as an in-house profession. It’s just much, much more mature.
And I was just thinking today about, you know, that phrase that Kate Towsey came up with, People who Do Research, like that’s, she came up with that term in 2018 as far as I was able to determine. And that’s fairly recent, but I feel like, oh, giving it a name, like there are researchers and you’ve mentioned a few times, like researchers and research. We kind of come back and forth on that there. And like you said, there’s all sorts of customer contact going on, but give, you know, creating a name that sort of says, here’s what, who’s who researchers are. And here’s this other category of people that’s also are doing research like that. Having that label, I think, clarifies a lot. A
nd yes, we have democratization debates. And it’s not like the problem is solved by giving it a name, but we’re clearly at a point where we can say there’s different types of research happening, which is a point that you’ve made. And there’s different people doing research, whatever we mean by that. And that those are all sort of different considerations. So it’s not a solved issue, but it’s a much more clarified issue than it was 10 years ago. You know, and again, not really what you asked, but, you know, I think a lot of the fundamentals that my write about, like how to ask a question, how to ask a follow up question, how to listen. You know, I’ve had 10 more years of doing that and 10 more years of teaching it. So I’m, I have just more examples and more stories and more clarification and more nuance. So that those are the fundamentals I think of interviewing, but I think I can explain it better than I could before because it’s just practice, practice, practice.
Nizar: This is great. You talked a lot about kind of the evolution of research and some of the things that are different. What’s the hot take about the world of user research that you find yourself disagreeing with?
Steve: Wow.
Nizar: Putting you on the spot there.
Steve: Yeah, no, that’s good. I mean, I am kind of just like, I feel like that Grandpa Simpson meme of like, you know, old man, shout, yells at cloud, whatever that is, I feel like, I don’t know, just being my age and my grumpiness like I am. I don’t know. And I guess I could justify that, you know, you just the longer you live, the more sort of hype cycles you kind of go through. And I’m just sort of reluctant to engage with, with, with hype stuff. So yeah, AI is a big hype topic. I’m amazed that I’m the one bringing it up in this conversation, because usually everybody else that I talk to has to bring it up right away. So we’ve gone. So thanks for making me do that. And sometimes I’m sort of resentful about, it’s like, I don’t want to have a hot take on research. And I’m resentful about the overwhelming amount of hot takes about research.
The most recent episode of this podcast that I posted is all about Noam Segal’s hot take on research. And so no disrespect to him for that he thought about it a lot. So I don’t know, my hot take is almost like an anti-hot take, like, can we all can everybody just chill out about AI or about the end of everything? I think we have a lot of, like short term ism or just like immediacy, we don’t we see what’s right in front of us, and we overreact. I include myself in that. And that’s not just that’s just human nature, I think. And you know, whatever, like the LinkedIn pundit info cycle, you know, entertainment complex, whatever that is, we all have to have an opinion about something and see what’s happening with research right now, but we’re at an inflection point. And so we don’t know. And maybe it’s okay not to know, which something that you’ve said a few different ways.
So I don’t know, my hot take is like to be anti-hot take is like, it’s maybe okay. And maybe that’s just my privilege speaking, like, you know, if I was younger, trying to make a certain kind of name or get a job, I might feel like I need to come in with an opinion about something. But I think there’s a little bit of peace and calmness that I would like to, you know, nurture within myself to like not react so much to kind of the change around us, the world feels very dynamic and uncertain and complex and worrisome. And, you know, I would I like when our conversations that we have in our professions collectively, sort of soothe that and not add to that.
Nizar: I love what you’re describing and just to add to it a little bit. I’m hearing a lot of the takes on how research is now fundamentally different. It’s doing things wrong. There’s definitely an over-exaggerating, an over-exaggerated sentiment behind what’s going on there. There’s a macroeconomic condition. You hear a lot of over statements across domains. I think when you’re in research, you just feel it more because a lot of your circles. At the heart of it, a lot of different domains got hit pretty hard. If you ask many people, they’ll tell you that it’s their area that got hit the hardest. I hear the same from product managers. I hear the same from software engineers. Let’s not even get started with the recruiting and some of the operational support.
At the same time, I see it as always a good opportunity to just reflect on what we’re doing and what works and what doesn’t, and we continue to iterate. It doesn’t need a big research is dying kind of header to encourage some of the discussions and the conversations, how we continue to evolve. It wasn’t too long ago where every researcher was called a usability engineer, and things will continue to evolve and things will continue to change, and that’s okay. This was part of the course. I’m just excited for the continued trajectory of researchers becoming key drivers and business leaders who represent users as their core mission. But it takes a few steps, takes a few optimizations, and I think it’s just part of the exciting journey of a domain that’s still relatively young. When you look at the grand scheme of, at least in the tech world for example, the other areas that have been around as fundamental to the product development process.
Steve: I think that’s us ending on a high note. Thank you so much for a great conversation, turning the tables a little bit and sharing so much. It was lovely to get to chat with you.
Nizar: I really appreciate it. It was a wonderful chat, and thank you so much.
Steve: There you go. That’s our episode. If you made it all the way to the very end, give yourself a hearty pat on the back for listening. Please spread the word about Dollars to Donuts. You can find Dollars to Donuts in most of the places that you find podcasts. You can raise awareness even more by reviewing the show on Apple podcasts or wherever it is that you’re finding it. Check out Portigal.com/podcast to find all the episodes, including show notes and transcripts. Our theme music is by Bruce Todd.
The post 37. Nizar Saqqar of Snowflake first appeared on Portigal Consulting.
This episode of Dollars to Donuts features a return visit from Noam Segal, now a Senior Research Manager at Upwork.
AI will help us see opportunities for research that we haven’t seen. It will help us settle a bunch of debates that maybe we’ve struggled to settle before. It will help us to connect with more users, more customers, more clients, whatever you call them, from all over the world in a way which vastly improves how equitably and how inclusively we build technology products, which is something that we’ve struggled with traditionally, if we’re being honest here. – Noam Segal
Show Links* Dental Hygienist Explains Ultrasonic Scaling * Interviewing Users, second edition * Steve Portigal on the Content Strategy Insights podcast * Noam Segal on Dollars to Donuts, 2020 * Noam on LinkedIn * Upwork * CatGPT shirt * Young trader dies by suicide after thinking he racked up big losses on Robinhood * Meta, TikTok and other social media CEOs testify in Senate hearing on child exploitation * X is blocking Taylor Swift searches… barely * Trust and safety * Temperature in Generative AI * The UX Research Reckoning is Here (Judd Antin) * The Waves of Research Practice (Dave Hora) * Strategic UX Research is the next big thing (Jared Spool) * Sam Ladner * Big Data needs Thick Data (Tricia Wang) * Multipliers: How the Best Leaders Make Everyone Smarter * Genway
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I went to the dentist recently for my regular teeth cleaning. I was in the chair while the hygienist was working away. This obviously wasn’t the best situation to ask a question, but I had a moment of curiosity and I found a chance between implements in my mouth. I should say that at this dentist, their cleaning process, first go over your teeth with something called an ultrasonic scaler. I had assumed this was just like an industrial strength water pick, or like a tightly focused pressure washer for the mouth. After that, they follow up with a metal scraping pick. So during the metal scraping pick portion, I asked the hygienist, “Does the water soften it?” I was wondering if the first stage softens up whatever gets removed by this mechanical pick. Somehow my weird question prompted her to give me a 101 lesson on how teeth cleaning works, what is being cleaned and how the tools are used to accomplish that. Anyway, she starts off by telling me that the water is just to cool the cleaning head. The water isn’t doing the cleaning. There’s a vibrating cleaning head that does that work. I was very excited to learn this because I had the entirely wrong mental model. I had assumed that this device was just water and I hadn’t ever perceived any mechanical tip. Of course, I’ve never seen what this device looks like, other than when it’s coming right at my face when I’m the patient. And I had made all these assumptions based on what I experienced from being in that role.
It was a lovely reminder about how we build mental models based on incomplete information, based on our role or interaction and how powerful those mental models are. And of course, this was also a reminder of the power of asking questions, where even this simple question in non-ideal circumstances led to a lot of information that really changed how I understood a process that I was involved in.
It was a great reminder about one aspect of why I do this work and some of the process that makes it interesting and insightful. Speaking of interesting and insightful, we’ll get to my guest, Noam Siegel, in a few minutes, but I wanted to make sure you know that I recently released a second edition of my book, “Interviewing Users.” It’s bigger and better. Two new chapters, a lot of updated content, new examples, new guest essays, and more. It’s the result of ten more years of me working as a researcher and teaching other people, as well as the changes that have happened in that time.
As part of the “book tour,” I’ve had a lot of great conversations about user research and interviewing users, and I want to share an excerpt from my discussion with Larry Swanson that was part of his podcast, Content Strategy Insights. Let’s go to that now.
Larry Swanson: It also reminds me, as you’re talking about that, it’s like you show up at a place like in the old days, you drive up and you’re in the car with a team, and that’s a good reminder that this is like a business activity. In fact, you open the book with a chapter about business and the business intent of your interviews, and I also like that you close the book with a chapter on impact, which I assume is about the measurement and the assessment of that satisfying that business intent. Was that bookending intentional, or am I just reading into that?
Steve: This is where I just laugh confidently and say, “Oh, of course, you saw my organizing scheme.” I hadn’t thought about it as bookending, which that’s a little bit of nice reflecting back. In some ways, I think I was just sort of following a chronology, like why are we doing this, how do we do it, and then what happens with that? So no, but sure.
Larry: Yeah, sorry, I didn’t mean to project on that. But anyhow, that’s sort of the — maybe just focusing on the business part of it, because I think that’s something that’s come to the fore in the content design world, and particularly the last couple years. I think it might have to do with the sort of economic environment we’re in, but also even before that, there were people talking about increasing concern with the ROI of our work and alignment with business values, and maybe we’re focusing too much on the customer and not balancing that. But how do you balance or kind of plant your business intent in your head as you go into an interviewing project?
Steve: I think it kind of — maybe it’s like a sine wave where it kind of comes in and out. We were just talking about transitioning into talking to Marnie, a hypothetical person, for 30 minutes. I really want people’s business intent to be absent during that interview, so that’s maybe the lower part of a curve. But leading up to that, who are we going to talk to, what are we going to talk to them about, who’s going to come with us? That’s very much rooted in — I don’t know why I made up this metaphor of the sine wave, but we’re very highly indexed on the business aspect of it.
We designed this project to address some context that we see in the business, either a request or an opportunity that we proactively identify. So we think about what decisions have to be made, what knowledge gaps are there, what’s being produced, and what will we need to help inform decisions between different paths kind of coming up.
I talk in the book about a business opportunity or a business question and a research question. So what do we as an organization — what decisions or tasks are kind of coming up for us? So what do we have to do? We want to launch a new X. We’re revising our queue. We need to make sure that people doing these and these things have this kind of information. That’s about us. Then from that, you can produce a research question. We need to learn from other people, our users, our customers, people downstream from them, whatever that is. We need to learn from them this information so that we can then make an intelligent response to this business challenge that we’re faced with. So all the planning, all the logistics, all the tactics, what method are we going to use, what sample are we going to create, what questions are we going to ask, what collateral are we going to produce to evaluate or to use as stimulus or prompting?All that is all coming from what is the business need and how we can go at it. Yes, there still is a sideways. So then we set that aside to talk to Marnie, to talk to everybody. We really embrace them. we have all this data. We have to make sense of this data. And then here, I think we sort of straddle a little bit because you’re going to answer the questions you started out with. I think if you do a reasonable job, you’re going to have a point of view about all the things that you wanted to learn about. But you always learn something that you didn’t know that you didn’t know beforehand.
And I think this goes to the impact piece. This goes to sort of the business thing that’s behind all this. What do you do with what we didn’t know that we didn’t know? I want there to be this universal truth like, oh, if you just show people the real opportunity, then they’ll embrace it. And then everybody makes a billion dollars and the product is successful. I think that principle from improv is of yes and. I think we have to meet our brief. We’re asked to have a perspective on something. Part of the politics or the compassion way of having impact is to not leave our teammates and stakeholders in the lurch.
So we have these questions. We have answers to these questions. And also, we feel like there’s some other questions that we should have been asking. We we want to challenge how we framed this business question to begin with. We see there’s new opportunities. We see there’s insights here that other teams outside the scope of this initiative can benefit from.
There’s all sorts of other stuff that you get. And I think it behooves us to be kind about how we bring that up, because no one necessarily wants a project or a thing to think about that they didn’t ask for. So how do you sort of find the learning ready moment or create that moment or find the advocate that can utilize the more that you learn that can have even more kind of impact on the business? That’s not a single moment. That’s an ongoing effort. Part of the dynamic that you have for the rest of the organization.
Again, that was me in conversation with Larry Swanson on the Content Strategy Insights podcast. Check out the whole episode. And if you haven’t got the second edition of interviewing users yet, I encourage you to check it out. If you use the offer code donuts, that’s D O N U T S, you can get a 10 percent discount from Rosenfeld Media. You can also check out portigal.com/services to read more about the consulting work that I do for teams and organizations.
But now let’s get to my conversation with Noam Siegel. He’s a research manager at Upwork, and he’s returning as a guest after four years. You can check out the original episode for a lot more about Noam’s background. Well, Noam, thank you for coming back to the podcast. It’s great to chat with you again.
Noam Segal: It’s absolutely my pleasure, Steve. Great to see you and great to be here.
Steve: Yes, if you are listening, we can see each other, but you can’t see us.
So that’s the magic of technology, although Noam is wearing a shirt that says cat GPT on it. So we’ll see if we’re going to get into that or not.
Noam: I do love silly t-shirts. I just ordered a few more silly t-shirts yesterday. My partner is not very happy about that particular aspect of who I am. But, you know, it is what it is and you get what you get.
Steve: Right. You got to love all of you.
Noam: Yeah.
Steve: So that’s an interesting place to start. Let’s loop back. And so it’s to maybe a more normal discussion starter. We spoke for this podcast something like four years ago, early part of 2020. So, you know, I guess maybe a good place to start this conversation besides T-shirts and so on is what have you been up to professionally in the in the intervening years?
Noam: A lot has happened. I can tell you that quite a lot given it’s not that much of a long time period in the great scheme of things.
Steve: Yes. Mm hmm. Right.
Noam: When we chatted last, I was working at Wealthfront, a wonderful financial technology company, and I was head of UX research there at the time.
Steve: Right. Yes.
Noam: I left Wealthfront for a very particular opportunity within Twitter, now X, because I was very interested in contributing to the health, so to speak, of our public conversation. And I had an opportunity to join what was known at Twitter, now X, as the health research team. But we don’t mean health as in physical or mental health. We mean indeed health as in the health of the public conversation. In other companies, these types of teams are called integrity or trust and safety, et cetera. And we were dealing with everything to do with things like misinformation and disinformation, privacy, account security, and all sorts of other trust and safety related issues. Sadly, a few months, really, or less than a year after I joined, Elon Musk took over the company. And one of the first layoffs that happened at the company were a layoff of basically the entire research team. And so I left before the layoffs, but that was the situation there. And I’d love to talk more about what that means in terms of how we build technology, et cetera. We can jump into that.
From Twitter, now X, I moved to Meta, and I joined the team working on Facebook feed, which some people might view as the kind of brand page of Facebook or even the internet for some people. It’s a product used by billions of people daily. And it was a very interesting experience to work on both the front end of the feed and the back end as well, so to speak. So that was a very interesting experience. And in addition, I was also playing a role in what we were calling Facebook Research’s center of excellence or centers of excellence, where we were trying to improve our methodologies, our research thinking, our skills and knowledge, kind of working on how we work, which is very related to what we spoke of in the last podcast we did together a few years ago, when we talked all about research methodology, et cetera. So that was an interesting experience.
But in April of 2023, along with a couple of tens of thousands of other people, I was laid off from Meta, as was I think approximately half of the research team at Facebook at the time. And several weeks later, I joined Upwork, which is where I work now. Upwork, for those who don’t know, briefly is a marketplace for clients looking to hire people for all sorts of jobs and freelancers who are looking to work, primarily in kind of the knowledge worker space, I would say. Upwork also caters to enterprises who are looking to leverage Upwork as a way to, you know, augment their workforce and hire freelancers or people for full-time positions as well. And at Upwork, I’m a senior research manager. I focus on the kind of the core experiences within the product, which includes the marketplace for clients and freelancers, in addition to everything to do with payments and taxes and work management and trust and safety, which I’m very happy to still be involved in. It’s a topic I care a lot about.
Steve: For Twitter, because you were particularly interested in that health, that trust and safety aspect of it, I guess I want to ask why, what is it about that part of designing things for people to use that as a researcher or as a person that it’s something that you strongly connect to?
Noam: Yeah, I think we have a set of societal ills, let’s call it, very troubling societal ills that I think we need to address urgently and with great care and with great responsibility. And one of those societal ills is the evolution of public conversation, of how we interact with each other as people and how hateful and nasty and unkind we can be to each other. And how much information put out there online is either inaccurate or completely false.
This really came to be more salient in my mind during the 2016 elections to the US presidency. But I think it’s become even more salient ever since for multiple reasons, including the incredible and tragic rise in antisemitism in the world over recent years and all sorts of information running about out there on the interwebs that is, again, factually incorrect around all sorts of topics. Election related, related to certain geographical regions, to certain groups, et cetera. And so this is something I care deeply about, just given my personal background, just given what I’m observing in society. When we last had a conversation, I was at Wealthfront in the fintech space, and I recall a case happening, which really shocked all of us to our core with another company. I’m not going to name the company, but it’s another fintech company. A young person tried to use this other company to make certain types of investments and trades, but he was not well versed in how that world works, how those trades works, what options are and how to use them. And he believed that he had lost an incredible amount of money that he did not have and wasn’t able to lose. And it brought him to enough depths of despair that he ended up taking his own life. And that to me was just one story of many that made it incredibly clear that we need to be responsible and ethical in how we build technology products. And that few things could be more important than working on trust and safety. So yeah, it’s definitely an area I’m passionate about.
And we’re recording this a day after yet another senate hearing with all of the heads of different social media companies who were faced with difficult facts about the effects that they’ve had on society and on families who lost their loved ones and other incredibly tragic stories because of the way they built their platforms, because of things they ignored. And I think research can play an absolutely critical role in building trustworthy ethical experiences, responsible experiences that really matter in this world, probably more than anything else I could think of. So that’s the long answer to your question.
Steve: What kind of information can researchers provide that can — into situations like the ones that you’re describing?
Noam: What sort of research we should be doing or not be doing and at what level, at what altitude, if that company put more effort into the first of all, age gating the platform and ensuring that people have the knowledge and the skill to conduct certain trades, but beyond that, the usability of the platform. Going back to the basics, which we don’t do enough of, I would suggest, which is just making sure that the information one is seeing is clear, you know, and not open to interpretations that could have incredibly tragic consequences. Like thinking I lost $700,000, I think was the number when that was in fact not the case at all. So for me, it goes back to those basics. Research can inform all sorts of more nuanced reactions than the one we’re seeing.
Another thing that happened this week while we’re recording, which demonstrates what happens when you let go of your entire trust and safety team, including researchers, was that Taylor Swift, the incredible pop singer, artist extraordinaire, she was facing something incredibly tough to face online, whether you’re a celebrity or not, which was AI-generated nude images of her, fake images obviously. These were all generated by AI such that if you searched for her name, for Taylor Swift’s name, on particular social platforms, you would see those AI-generated images and perhaps believe, because they were very realistic, that these were in fact images of Taylor Swift when they were not. The solution this company came up with was to remove any search results for the terms Taylor Swift or Taylor or Swift or any combination of her first name and last name, which is moronic. I’m not sure what adjective to use. It’s incredibly aggressive, and I think as technologists we can do a lot better than cancel an entire search query because of that sort of thing happening. I think one thing for sure is that there’s no doubt there is a need for trust and safety professionals. There is a need for trust and safety researchers. We know how to inform the responsible and ethical building of these sorts of products and how to address these issues in much more nuanced and rational ways with much better outcomes. I mean, that seems pretty obvious, but it’s clearly not obvious to some of the people leading some of these companies. I hope that changes, and I’m very proud that at Upwork we do have these teams and we are working on these things. We care deeply about the trust and safety of the people we serve.
Steve: I mean, you’re describing a failure with Taylor Swift AI images that there’s, you know, I guess the jargon is bad actors. People are behaving in a way that’s harmful. And when that happens, when there’s a system that can be exploited or manipulated or used to cause harm, you know, I think you’re identifying like there’s a gap. The system can be used that way. But you’re saying also that without researchers, companies are not as well set up to respond to those malicious behaviors.
Noam: Yeah.
Steve: And I’m, I guess I’m just looking to have the dots connected for me a little bit more like, but you can see sort of the failure of the systems and the failure of the humans that are the malicious users. But in that scenario, or kind of analogous ones, how do researchers serve to either prevent or, you know, mitigate those kinds of malicious uses?
Noam: So I can give – there are a few answers here. One example would be that at X and other companies, some of the research we did and in some companies are still doing goes into supporting content moderators and support agents and other such people who are reviewing this type of content. And the research informs building tools so they can get to those problems faster and eliminate them and get rid of that content in more efficient and more effective ways. So for the time being, as you probably know, there are often humans in the loop here reviewing content. They’re using certain tools to do so. Those tools make them better at their job, and building those tools requires research. So that’s one example I would suggest.
Another example is that in certain companies that, again, care more about these trust and safety issues, research informs providing users with tools that enable them to control their experience. Whether it’s blocking certain people or removing certain things from their experience or a bunch of other things that we can do. But ultimately, some platforms choose to give people more agency and more control over their experience, and research has heavily informed those sorts of tools. And you end up with an experience that is catered to your needs and what you’re willing to see and what you’d prefer not to see. And I think a final example is that even though I think a lot of researchers, we think of ourselves as mostly informing the user-facing user experience. You know, the actual designs that people end up seeing when they use a product, for example, Facebook’s feed. Several researchers in our field work more on the back end of things and helping companies sharpen and calibrate their algorithms such that the content that shows up for users makes more sense. We had that at Twitter, now X. We had that at Meta. Most companies that have any sort of recommendation systems and search systems and other such systems, they’re doing a lot of research on what to showcase to users and what sorts of underlying taxonomies make sense and various tagging systems. All sorts of inputs and parameters that go into these models and adjust these models.
To give an even more specific example, in the realm of AI, we have all sorts of parameters, right? One of those parameters is called temperature, and when you adjust the temperature parameter, it sort of influences how creative versus how fixed by nature the algorithm responds to things, right? Like, how much it kind of thinks out of the box, so to speak, versus not. When you change the temperature of an AI-based tool, that of course influences how people experience it, right? And how they experience maybe how empathetic that experience feels or how aggressive it feels or how insulting it feels and so forth and so forth. And we need a lot of research going into these things to understand how tweaking all of these parameters affects how people perceive these tools, these technologies, these experiences that we’re building. So those are just some of the ways in which research, I think, can inform these topics.
Steve: I don’t know. You used the word health kind of early on here. There’s a quality to the experience that we have with these tools, these platforms, separate from bad actors and abuse and misinformation, disinformation. There are research questions to just set the tone or kind of create the baseline experience. It sounds like that’s what the — if you’re working on the feed at Facebook, if you’re thinking about that algorithm, you’re using research to just create a — ideally in the best situation, a healthy versus unhealthy experience. Just — I think there’s research that talks about, oh, when you compare yourself to others, if you see positive messages, you react this way. If you see negative messages, you react this way. So you’re making me realize that asking these kinds of questions around sort of the healthfulness of the experience, I think I locked in on sort of malicious behavior, bad actors, exploitation and so on. But I think I’m hearing from you that there’s just a base on it, like what’s it like to go on — you know, I mean, what’s it like to go on LinkedIn every day when people are being laid off or when people are trying to get your attention or when people are performing, as people do on all these platforms. There is an experience that research can help understand and inform fine-tuning of algorithms and sort of what’s shown to people and how in order to create the desired experience.
Noam: Absolutely. Trust and safety is an incredibly complex space. It’s very layered. To your point, you can create more trustworthy and safe experiences if you stop bad actors from even entering the experience in the first place. And again, ATX and I imagine other companies as well, part of what we did on the research side was inform things like account security.
And how do we help people secure their accounts and how do we make it harder for bad actors to open accounts even though their intentions are malicious? So you can create more trustworthy platforms by stopping bad actors at that stage. And then there are more lines of defense, which again, research can inform each and every one of those lines of defense to make sure that the ultimate, the end experience for each and every user is a healthy one, is a trustworthy one. And I really don’t think there’s any stage where research can’t have incredibly meaningful impact. And I just really, really hope that as we lean in even more to AI and other incredibly advanced and complex systems that to many of us are a weird and wonderful black box that we simply do not understand. I really hope that we increase our investment in trust and safety exponentially because if we don’t, I really think the results will be horrific. And it’s our responsibility as insight gathering functions, as researchers, whatever you want to call it, to take ownership of this, to advocate for this and to make sure we’re doing this in a way that matches the incredible evolution and development of these platforms. It’s just incredible to witness.
Steve: If we were to go into the future and write the history of, I guess I’ll just call it trust and safety user research, what era are we in right now for that as a practice or an adoption?
Noam: That’s a tough one. That’s a tough question. I think the only response I have for you now, but let’s talk again in four years or so, is that trust and safety in a sense, you mentioned bad actors earlier. Trust and safety in a sense is always this ongoing battle between forces of good, so to speak, and the forces of evil trying to catch up to each other and match each other’s capabilities and then beat the other side. With even better capabilities, I think what we’re witnessing now with AI-based systems is that the pace of innovation, the pace at which they are evolving and learning is shocking and hard to comprehend. It’s really, really hard to comprehend. Now, that’s not to say that it’s not going to take a long time before some of these systems are fully incorporated in our lives.
We’ve been talking about self-driving cars for a very long time, and they are absolutely out there right now in the streets of San Francisco and maybe Phoenix, Arizona, and maybe a few other cities doing their thing and learning how to do their thing. But I think it’s going to be quite a while before every single vehicle on the road is a self-driving car. But that said, these systems are just getting more and more complicated. I think our ability to understand them, it’s getting very difficult. We have to figure out what tools we need to develop in order to catch up, in order for the forces of good, so to speak, to match the forces of evil. And we also need to remember that everything these systems are learning, they’re learning from us. And sadly, the human history is riddled with terrible acts and a long list of biases and isms, racism and sexism and ageism and everything else. So these AI systems are sadly learning a lot of bad things from us and implementing them, so to speak. So again, we have a great responsibility to be better because a little bit similar to a child, AI systems are learning from what we are generating. So we kind of have to be a role model to AI and we have to make sure that we’re leveraging AI, maybe somewhat ironically, to deal with issues created by the incredible development of this technology. So I hope that sort of answers the question.
Steve: We know as researchers, right, any answer that doesn’t answer your question reveals the flaw in the question.vAnd my flaw is that I asked you to decouple user research for trust and safety from everything else. And I think you answered in a way that says, hey, this stuff is all connected. The problems, society at large, the technology, and the building of things are all connected and research is a player in that. So, yeah, you gave a bigger picture answer to my attempt to sort of segment things out. I think we’re going to come back to AI in a bit, but I wanted to ask you, in addition to sort of trust and safety that we’ve talked about over the four years and this issue of building responsibly that you’ve highlighted, are there other things that you have seen or observed about our field that you want to use this time to reflect on?
Noam: Yeah, absolutely. I think, as I mentioned, because this happened to me as well, we’ve seen a large number of tech layoffs and certainly for research teams, but not only, of course. We’ve seen reorgs happen, major reorgs. Because, I mean, reorgs are a reality in tech, everyone who’s worked in tech knows this, but we’ve seen some major reorganizations.
And in fact, we’ve seen entire research teams shut down, including the example I gave earlier of the team at Twitter, now X. And as part of that, we’ve seen some incredibly thought-provoking articles come out. And I’m sure you’ve read some of these. One of them from a former leader at Airbnb, Judd Antin. He was my skip level manager, wrote an article about how the UX research reckoning is here. Another incredibly interesting article was around the waves of research practice by Dave Hora. Jared Spool wrote an article about how strategic UX research is the next thing. And I think that what all of these articles had in common was some sort of discussion on the value that insight-gathering functions or research functions bring to the table. And you might not be surprised by this, but I have a hot take for you on this that I would be happy to discuss.
Steve: Bring it on. Hot take
Noam: Hot take time.
Steve: I’m ready.
Noam: Are you ready for this? So here’s the thing. If we stick to the UX research reckoning framing, I’m a bit of a stickler for words. So the relevant, or I believe that the relevant definition for reckoning that Judd meant to reference is the avenging or punishing of past mistakes or misdeeds. So basically, as UX researchers, we made some mistakes, we made some misdeeds, and now we are being punished for it by being laid off. And again, the broader point in that article, I think, is what’s the value we bring as researchers? And I am here to say that although I agree that we’ve made mistakes, this has, everything that’s happened, has very little, if anything, to do with value. To do with the value that we bring. And I think it has everything to do with valuation, which is a very different thing. And if I take Meta as an example, Meta was suffering from a tough time as a company, spending a whole lot of money on AR, VR, and other capabilities. The stock was at one of its lowest points in recent years, if not the history of the company. And so Mark Zuckerberg announced a year of efficiency. And part of his idea of efficiency was to lay off about half of the research organization. And we have to ask ourselves, is that because researchers did not bring value to the organization? And again, I would suggest not. I would suggest that these days I’ve had work, and in every company I’ve been part of, I’ve seen some incredible value brought forward by researchers. Insights that can make a huge difference to everything from the user experience to the strategy to use Jared Spool’s and others terms.
But there’s a couple of problems. The first problem is a problem of attribution. How can you calculate the return on investment of research? How do you know and how can you record and document which decisions and which things were influenced by research and which weren’t? If I’m an engineer or designer and I’m working within my Jira or whatever platform or linear or whatever platform you’re using to manage your software development, then I have some sort of ticket. I have some sort of task. I write 10 lines of code. Everyone knows those 10 lines of code are mine or mine and other people. Everyone knows what those lines of code translate into in the experience. And so the ownership of what that experience looks like from design to engineering is clear because it’s clear who made the Figma and it’s clear who wrote the code. And everything is incredibly accurately documented. When it comes to research, when it comes to knowledge, you know, research is circulated in all sorts of ways, right? From Slack channels to presentations to a variety of meetings and one-on-one get-togethers with cross-functional partners. And in all of those meetings and all of those interactions, research is coming through in some way. But it’s incredibly fuzzy and unclear how that translates into impacts on the products. That doesn’t mean research doesn’t have value. That means it’s hard to measure the value.
And then one more thing that I think is going on, which you probably know very well as one of the most knowledgeable people on the topic of interviews that I know, is what happens when I’m responding to your question? In this case, maybe you have some questions about what insights have we learned? What happens as I’m giving you a response? What are you doing? Make a guess.
Steve: I’m thinking about my next question.
Noam: You are thinking about your next question. It’s so hard to avoid that tendency. And I think in many cases, product managers, product leaders and other cross-functional partners of research, they’re taking in the research, but they’re just thinking about their next question. And to be fair, I think that one more thing that’s going on here is that we as researchers do not understand the feeling of being held accountable for certain metrics. And for millions, if not billions of dollars in revenue, that can be moved one way or the other by the quality of what we choose to build and what we choose not to build. And the roadmaps we have, the strategy we have, etc. Usually those are product leaders who are accountable for that. And we’re not. And so the pressure is on them. And so as they take in our insights, they can’t help but just think their own thoughts and think about their vision and maybe ignore certain things that we share. And then business leaders, ultimately, what do they care about? Again, valuation. The stock. That’s just how it works, which is why I said in the beginning that I don’t think this is about value at all. I think it’s about valuation. I think business leaders are optimizing their business for their valuation, for their stock price. They’re not laying off researchers because we didn’t deliver value or because we weren’t strategic enough. They’re laying off researchers and many other people because that’s one way of a few ways to become more efficient, to look good in front of your shareholders. It’s not such a complicated game. You know, we’re doing this interview a day after a particular company started offering dividends to its shareholders, and that had a very expected effect in the market on that stock. It just went up quite a little bit higher. That’s how the game works. Those are the dynamics of the market.
And so we’re in the situation where I’m not saying we haven’t made mistakes again. I think Judd, for example, absolutely had a point when he discussed the different levels of research and the fact that we’re making a mistake by looking at usability as some sort of basic, tactical type of research that only junior researchers should do and that we shouldn’t be focused on. And that we should only be looking at higher levels and higher altitudes of research. I couldn’t agree more. I absolutely agree with Judd on that. But this basic premise of research not delivering value I think is incredibly problematic. And I don’t think it’s correct. I don’t think we need to move into some third or fourth or whatever wave of research. I just don’t see that personally. I think many of us have already been in wave one and wave two and wave three of research. We’ve already been doing strategic research. We’ve already been affecting the business level, the product level, the design level. We’ve already been conducting all sorts of research from usability to incredibly foundational, generative research. And I think we’re being very, very hard on ourselves. And I think we need to cut ourselves a little bit of slack. Just a little bit.
Steve: I mean, I’m all about being kind to ourselves and not blaming ourselves for things that are beyond our control. We’re all susceptible to that and it’s hard to kind of watch that going on collectively. But when you’re in a situation where there is, I don’t know, a misalignment of values, like what, you know, like you said, value versus valuation. When that misalignment, that’s my word, not yours, when that exists, we can cut ourselves slack, but that’s not going to change that gap. I don’t want to say to you like, well, here’s, you know, you just outlined a systemic, deeply rooted, the nature of capitalism, it goes all the way up. How do we fix that? I guess I don’t, I think that’s not a fair question, although, you know, take a shot if you have a hot take there. Are there mindset changes or incremental steps or, you know, things that you’ve seen research teams do that acknowledge to some extent the difference between we’re not breaking value to their concern is about something else and how do we kind of meet them where they’re at?
Noam: So look, Steve, that’s an incredibly fair question. And I do want to be crisp about the fact that, yes, we need to be doing something. Something needs to change even if I view the problem differently. But before I get to that, just to reiterate, we as researchers know very well that it’s absolutely critical to identify the problem and to identify the correct problem at the right level. So before I get to what we should do, I just want to highlight the fact that in my view the issue here is that some of us have misidentified the problem, in my opinion. And we need to be tackling the actual problem.
And just to get to that and to pivot to kind of the second topic that we did cover last time and I want to cover again today. We did talk about in our original conversation about research methods and how we do research. And I do think, even though I believe we’ve brought a lot of value to the organizations that we work in as insight gathering functions, I do absolutely believe that given the broad evolution of the landscape we operate in, we do need to rethink how we operate. Not because we haven’t delivered value, but because the ways in which we can deliver value are rapidly changing. And I think we can now sort of extend ourselves.
And I was very influenced by a book titled Multipliers, not sure if you’ve read it. But the basic idea of it is that there are employees within any company who are multipliers in the sense that they don’t just do great work, they make everyone else’s work even better. They level up everyone around them and they create these situations where they define incredible opportunities and they liberate people around them to get to those opportunities and to make the most out of them. They create a certain climate, which is a comfortable climate for innovation, but at the same time an intense climate where a lot of incredible things can happen. Where I’m getting at is that, and this is not surprising probably to the people listening to this, is that the era of AI is upon us and I think it’s incredibly important to acknowledge the ways in which we can extend our work and ourselves with AI tools. So I know that my mind has moved a little bit from methods, so to speak, to leveraging AI to use similar methods but at a scale that we’ve never experienced before and we’ve never been able to offer before to our partners.
Yeah, I mean, I think there are certain paradigms in our industry that are changing and perhaps AI is even eradicating those paradigms and rendering them useless. I mean, if it’s okay, one recent example I have is that we had this paradigm that we need to make a tough choice. We’ve talked about this, you and I, a little bit. We have to make a tough choice between gathering qualitative data at small scales, which can often be okay, by the way, unless you’re developing a very complex product or unless you want to make sure that trust and safety is in the center of everything you do and then maybe you need a little bit more scale and you just couldn’t get it because you didn’t have the people to reach that scale of interviews or qualitative research. Or, of course, the other choice you could make was to gather quantitative data at any scale you like as long as you can afford it, namely by sending out surveys to hundreds or thousands of people. The issue is survey data is shallow data or thin data or whatever you want to call it, whereas I believe it was Sam Ladner who coined the term “thick data” for qualitative data. And sometimes you need that thick data and you need it at a scale that we were never able to reach before. And AI enables you to do that.
I’ve personally witnessed tools, one of them being Genway, which are completely revolutionizing the way we conduct research. I’ve seen existing research tools, Sprig would be a good example, Lookback, there’s so many incredible tools that have incorporated AI into their workflows. And they are making paradigms like the one I mentioned, this choice between thick and thin data, they’re making them irrelevant, absolutely irrelevant. Which is very interesting to me. And it ties to this idea of multipliers, this idea in this book I love. Because AI research tools, like the ones I mentioned and so many more that we could talk about all day, they enable us, in a sense, to be multipliers. They liberate us, in a sense, to do a lot more than we could ever do before. And hopefully that translates into us enabling our cross-functional partners and the teams we work in to deliver their best thinking and their best work as well. So that’s, I think, where our field is going in a nutshell.
Steve: Can you describe with maybe a little bit of specificity what a work process or set of work tasks that a researcher might go through where AI tools like the ones you’re describing, like how is that, yeah, what are they doing, what’s kind of coming to them and, you know, what does that process look like that’s AI enabled?
Noam: I can give a couple of examples. The first example, if I think of a tool like Genway, an interview tool, is that interviewing is tough, as you know well. You’ve written what I consider, and many people in our industry consider, kind of the Bible of interviewing people. No offense to the actual Bible. And as someone who’s written one of the primary guides to how to interview people, I think you appreciate more than others how complex being an interviewer can be. It’s something that you can learn over years and years of training and mentorship and still not nail some pretty critical aspects of interviewing. For example, asking the right, the best, the ideal follow-up question, and actually listening to what’s being told to you actively, rather than thinking about that follow-up question all the time, because listening is what enables you to ask a good follow-up question. Systems like these can train on an unlimited number of past interviews and an unlimited number of texts like your book, and learn from all of that how to conduct the best possible interviews, right? And these types of abilities to learn and then apply that learning in an interview situation, I believe it’s fair to say it would be technically impossible for any researcher to achieve that level of learning, certainly in a matter of hours or days or months at the most or weeks, rather than years in the case of a researcher.
You know, one of my hot takes, I hope the audience doesn’t kill me for this, is that David Letterman interviewed people for many, many years. And I personally think David Letterman is a horrible interviewer. I never understood why he asked the questions that he did, and everything about his interview style is very, very odd to me. But putting that aside, interviewing is a very complex skill. None of us can really ever witness how other people do it. And we all have to spend years of practice learning how to become better interviewers, which is a deceptively difficult skill to build. And these AI tools are coming in and, at least in theory, can learn all of that shockingly quickly. That’s one example, and I’m very curious to see how the research community responds to these types of tools and uses that, and what issues they do find in the quality of these types of interviews and how they can be improved.
The second example is that I recall from even my undergrad psychology studies, not to mention my graduate studies, that our ability to hold information in our brain is quite limited. And so even when we’re synthesizing five interviews, not to mention 500, because sometimes you need 500, it’s very, very challenging. If you do five 30-minute in-depth interviews with people, organizing your thoughts and synthesizing those interviews has never been a trivial task. And I think there are a large number of biases and other issues and strange heuristics that we use to synthesize information that might not lead to the optimal outcome, an outcome that’s as objective and accurate, an accurate representation of the entirety of those interviews and how they interact with each other as we would want them to be. One particular task that generative AI and AI in general is very good at is summarizing and synthesizing information. And especially as we collect more information, that becomes a lot more relevant and even critical, I’d suggest.
When we entered the big data era, we needed to develop a bunch of tools. You know, so many companies came out of that era building tools that enabled us to analyze and very easily visualize in beautiful dashboards what those data are telling us. Now, we can also start collecting qualitative data at unimaginable scales. And not just qualitative or quantitative data, because I think that distinction is going to matter less and less as time goes by, but more importantly, we will be and become so much closer to the people we serve, our users, our customers. I think we’ve talked about this in the previous podcast, but I think we talk about diary studies and how diary studies used to be physically sent to people’s mailboxes, right? And so you as a researcher had to plan your studies, send out an actual diary, have people log their entries into it, and then they would have to send it back, and then you would have to very manually look into those entries. And obviously that takes a very long time. These days, and especially with the support of AI tools, you can be in touch with the people you serve all the time, as much as you want and as much as they want, and you can both collect data and synthesize data and even communicate those insights at a pace that’s hard to even fathom, for me at least. But it’s very immediate. Can you say very immediate or can you not modify the word immediate? Is something either immediate or not? Okay, I don’t know. I’m just afraid of my mother and what she might say here about my grammatical choices. But anyway, yeah, yeah. But I think that’s what matters the most.
Steve: We’ll ask her to fast forward over this part.
Noam: People’s schedules don’t really matter anymore because they can choose to interact with AI, for example, whenever they want to. And it can be in context in real time. And then AI can immediately synthesize those learnings. And it can immediately improve the way it collects insights based on that interaction and all previous interactions. I was thinking about this a lot in this framing of multipliers. Like, who is the multiplier in this context?
Steve: What does this hold for researchers? Like there’s research, which I think you’re describing a really audacious vision for how, what research will be and I think speaks to the point about valuation versus value, but researchers, which currently refers to humans, what do you, what’s your vision for that or your anticipation for that?
Noam: Is it the AI? Is it the researchers? Like, who is multiplying whom? But what I do think is that, well, first of all, I’m a techno-optimist or whatever you want to call it. Even with everything that’s happened, even with all of the tragedies and the negative aspects of technology that we’ve discussed in this conversation and others, I am still at heart a techno-optimist. And so my deep belief is that certainly for the foreseeable future, if not beyond, AI will become a valuable extension of ourselves and our work. And I do believe that even if we don’t have to deliver more value necessarily than we already are, even if our value just goes underappreciated and there’s nothing terribly wrong with how we’ve approached things, I still think that AI will augment our work, will amplify our work, will enable us to really invite the teams we work with to do their best work ever.
Because AI will help us see opportunities for research that we haven’t seen. It will help us settle a bunch of debates that maybe we’ve struggled to settle before. It will help us to connect with more users, more customers, more clients, whatever you call them, from all over the world in a way which vastly improves how equitably and how inclusively we build technology products, which is something that we’ve struggled with traditionally, if we’re being honest here.
A very clear example of that is that an AI can speak a bunch of different languages and connect with people across all time zones and languages, and even mimic certain characteristics of a person so they feel more comfortable in that context. So for multiple reasons, I feel like if we’re at all concerned about getting buy-in from our partners, if we’re concerned about the value we bring, the impact we have, I definitely think AI tools can really improve our chances of getting to where we want to be. And I think it’s going to be a very long time, if ever, before these tools replace us as researchers. You know, the reason I chose to be a researcher, the reason I chose to be a psychologist is because of the incredible complexity of the human mind.
You know, the people listening to this can’t see this, but for my friends who are physicists, for example, so I’m holding up my phone right now in front of the camera, and if I drop my phone onto my desk or onto my floor, it’s a very easy calculation for physicists to say how quickly will the phone hit the desk, and what energy will there be there when the phone hits the desk, and what’s the chances that the phone will break given the speed of its fall. Physics is a beautiful thing, but it’s also a fairly reliable scientific practice.
You know, there are rules in physics, and they’re fairly clear. And even though physicists sometimes, on occasion, like to look down upon people like me with a PhD in psychology and a background in psychology, I think that in many ways, the field of psychology and other fields that deal with the human experience, with the human mind, they are so much fuzzier and so much more complex. And I’m saying this because I think many, many other professions will be replaced by some form of AI before researchers ever are. And that’s because of this complexity, this fuzziness that’s hard to capture. I think in many ways, our field is incredibly technical, but in other ways, our field is not technical at all.
There’s a lot of art to it, and there are all sorts of different aspects to it. It’s a lot easier for an AI to generate a piece of code or to generate a contract or to read a mammogram or an MRI and identify something, rather than talking to another human being and understanding them deeply. I think that’s a lot more complicated. So I’m not too concerned about the research fields, but we’ll wait and see, I guess.
Steve: So as we kind of head towards wrapping up, you know, since I’ve known of you and known you, I’ve always seen you doing different things, I guess, to be involved with the community of user research and what’s that look like for you now?
Noam: I, like many people in our profession, have definitely been on a journey. And if I’m being honest with myself and the audience, it’s been a very challenging few years, certainly for me. And I know for many others out there, whether it’s COVID and layoffs and a bunch of other personal events and things in life that just happen to us. And that’s part of the reason why I decided, and I know quite a few other people in our community decided to do this, to pursue coaching, among other things. I took a certification in coaching. I think even with my background in psychology, I felt like there was so much more to learn in this realm. I think that it’s always been important for me to support people in our community, and I wanted to do that even better.
And so one of the things I’m doing these days, to a limited extent, is some coaching, not just for UX researchers or UX professionals, but people in tech in general. And then I have some thoughts for the near future around sharing some of that coaching and ideas in other ways. In addition to continuing to teach in all sorts of ways, I’m still teaching at a bunch of different institutions and planning to restart some of my teaching on Maven, which is a wonderful platform for learning all sorts of things. I think the general trend in my life and career right now, and maybe this will resonate with people, is that definitely a challenging few years feel like maybe now coming out of it a little bit, ready to take on certain other challenges in addition to my role at Upwork, etc. And I know that I really want to be there to support our community in particular as we all go through rather challenging times.
So I just invite anyone who wants to get in touch to message me on LinkedIn or email me or get in touch in any way that works for you, and I’d be happy to chat and help. And then I also thought about, and we’ll see where this goes, but as you can tell, these topics of AI and where our field is going and how it’s evolving, that’s a lot to me and I’m thinking about it constantly and want to be part of this evolution, if not revolution, in how we work. And so I’d love to have conversations similar to this, whether out in public or privately around these topics to continue to understand them. And I’m just looking forward to seeing what is next for our industry. I feel like when we spoke a few years ago, I think we had a solid sense of what’s to come. And I think in many ways we discussed things that did end up manifesting in some way or another. But in this conversation, Steve, I don’t know what we’ll be talking about in four years. If you give me the opportunity to talk to you again. And I can’t decide with myself if that’s exciting or incredibly anxiety-provoking. So I don’t know. Why not both? Or if I’m going to be an optimist or say I’m an optimist, then I’ll choose to be an optimist and say, maybe that’s exciting.
Maybe it’s exciting that I really don’t know what’s coming down the line. But I do know that I want to thank you again so much for taking the time to do this. It’s always such a pleasure to talk to you. So thank you for giving me the opportunity.
Steve: That’s my line, man. I’m saying thank you.
Noam: Well, for me, it’s a special treat. Maybe I can share also with whoever’s listening to this that we did get a chance to meet in person finally. Not that long ago. And that was even more of a treat. And I really do hope that our community of research can get together more often moving forward and meet up and discuss all of these issues.
Steve: These are some really encouraging, I think, provocative things to think about and some really positive and encouraging sentiments for everyone. And there will be show notes that go with this podcast as always and so the stuff that Noam you’ve mentioned and you know ways to get in touch with you, we’ll put that all in there so people can connect with you if, if by some chance they aren’t already connected with you. So yeah, I’ll flip it back as well and say thank you for taking the time and for thinking so deeply about this and sharing with everybody. It was lovely to have the chance to revisit with you and kind of catch up on some of these topics after a few years and. I look forward to four years from now doing this again if not sooner.
Noam: Can’t wait. Adding it to my calendar right now.
Steve: All right.
Noam: Cheers, Steve. Have a good rest of the day.
Steve: Cheers.
Well, that’s another episode in the can. Thanks for listening. Tell everyone about Dollars to Donuts and give us a review on Apple Podcasts. You can find Dollars to Donuts in most of the places that you find podcasts. Or visit portigal.com/podcast to get all the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 36. Noam Segal returns first appeared on Portigal Consulting.
This episode of Dollars to Donuts features a return visit from Noam Segal, now a Senior Research Manager at Upwork.
AI will help us see opportunities for research that we haven’t seen. It will help us settle a bunch of debates that maybe we’ve struggled to settle before. It will help us to connect with more users, more customers, more clients, whatever you call them, from all over the world in a way which vastly improves how equitably and how inclusively we build technology products, which is something that we’ve struggled with traditionally, if we’re being honest here. – Noam Segal
Show Links* Dental Hygienist Explains Ultrasonic Scaling * Interviewing Users, second edition * Steve Portigal on the Content Strategy Insights podcast * Noam Segal on Dollars to Donuts, 2020 * Noam on LinkedIn * Upwork * CatGPT shirt * Young trader dies by suicide after thinking he racked up big losses on Robinhood * Meta, TikTok and other social media CEOs testify in Senate hearing on child exploitation * X is blocking Taylor Swift searches… barely * Trust and safety * Temperature in Generative AI * The UX Research Reckoning is Here (Judd Antin) * The Waves of Research Practice (Dave Hora) * Strategic UX Research is the next big thing (Jared Spool) * Sam Ladner * Big Data needs Thick Data (Tricia Wang) * Multipliers: How the Best Leaders Make Everyone Smarter * Genway
Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I went to the dentist recently for my regular teeth cleaning. I was in the chair while the hygienist was working away. This obviously wasn’t the best situation to ask a question, but I had a moment of curiosity and I found a chance between implements in my mouth. I should say that at this dentist, their cleaning process, first go over your teeth with something called an ultrasonic scaler. I had assumed this was just like an industrial strength water pick, or like a tightly focused pressure washer for the mouth. After that, they follow up with a metal scraping pick. So during the metal scraping pick portion, I asked the hygienist, “Does the water soften it?” I was wondering if the first stage softens up whatever gets removed by this mechanical pick. Somehow my weird question prompted her to give me a 101 lesson on how teeth cleaning works, what is being cleaned and how the tools are used to accomplish that. Anyway, she starts off by telling me that the water is just to cool the cleaning head. The water isn’t doing the cleaning. There’s a vibrating cleaning head that does that work. I was very excited to learn this because I had the entirely wrong mental model. I had assumed that this device was just water and I hadn’t ever perceived any mechanical tip. Of course, I’ve never seen what this device looks like, other than when it’s coming right at my face when I’m the patient. And I had made all these assumptions based on what I experienced from being in that role.
It was a lovely reminder about how we build mental models based on incomplete information, based on our role or interaction and how powerful those mental models are. And of course, this was also a reminder of the power of asking questions, where even this simple question in non-ideal circumstances led to a lot of information that really changed how I understood a process that I was involved in.
It was a great reminder about one aspect of why I do this work and some of the process that makes it interesting and insightful. Speaking of interesting and insightful, we’ll get to my guest, Noam Siegel, in a few minutes, but I wanted to make sure you know that I recently released a second edition of my book, “Interviewing Users.” It’s bigger and better. Two new chapters, a lot of updated content, new examples, new guest essays, and more. It’s the result of ten more years of me working as a researcher and teaching other people, as well as the changes that have happened in that time.
As part of the “book tour,” I’ve had a lot of great conversations about user research and interviewing users, and I want to share an excerpt from my discussion with Larry Swanson that was part of his podcast, Content Strategy Insights. Let’s go to that now.
Larry Swanson: It also reminds me, as you’re talking about that, it’s like you show up at a place like in the old days, you drive up and you’re in the car with a team, and that’s a good reminder that this is like a business activity. In fact, you open the book with a chapter about business and the business intent of your interviews, and I also like that you close the book with a chapter on impact, which I assume is about the measurement and the assessment of that satisfying that business intent. Was that bookending intentional, or am I just reading into that?
Steve: This is where I just laugh confidently and say, “Oh, of course, you saw my organizing scheme.” I hadn’t thought about it as bookending, which that’s a little bit of nice reflecting back. In some ways, I think I was just sort of following a chronology, like why are we doing this, how do we do it, and then what happens with that? So no, but sure.
Larry: Yeah, sorry, I didn’t mean to project on that. But anyhow, that’s sort of the — maybe just focusing on the business part of it, because I think that’s something that’s come to the fore in the content design world, and particularly the last couple years. I think it might have to do with the sort of economic environment we’re in, but also even before that, there were people talking about increasing concern with the ROI of our work and alignment with business values, and maybe we’re focusing too much on the customer and not balancing that. But how do you balance or kind of plant your business intent in your head as you go into an interviewing project?
Steve: I think it kind of — maybe it’s like a sine wave where it kind of comes in and out. We were just talking about transitioning into talking to Marnie, a hypothetical person, for 30 minutes. I really want people’s business intent to be absent during that interview, so that’s maybe the lower part of a curve. But leading up to that, who are we going to talk to, what are we going to talk to them about, who’s going to come with us? That’s very much rooted in — I don’t know why I made up this metaphor of the sine wave, but we’re very highly indexed on the business aspect of it.
We designed this project to address some context that we see in the business, either a request or an opportunity that we proactively identify. So we think about what decisions have to be made, what knowledge gaps are there, what’s being produced, and what will we need to help inform decisions between different paths kind of coming up.
I talk in the book about a business opportunity or a business question and a research question. So what do we as an organization — what decisions or tasks are kind of coming up for us? So what do we have to do? We want to launch a new X. We’re revising our queue. We need to make sure that people doing these and these things have this kind of information. That’s about us. Then from that, you can produce a research question. We need to learn from other people, our users, our customers, people downstream from them, whatever that is. We need to learn from them this information so that we can then make an intelligent response to this business challenge that we’re faced with. So all the planning, all the logistics, all the tactics, what method are we going to use, what sample are we going to create, what questions are we going to ask, what collateral are we going to produce to evaluate or to use as stimulus or prompting?All that is all coming from what is the business need and how we can go at it. Yes, there still is a sideways. So then we set that aside to talk to Marnie, to talk to everybody. We really embrace them. we have all this data. We have to make sense of this data. And then here, I think we sort of straddle a little bit because you’re going to answer the questions you started out with. I think if you do a reasonable job, you’re going to have a point of view about all the things that you wanted to learn about. But you always learn something that you didn’t know that you didn’t know beforehand.
And I think this goes to the impact piece. This goes to sort of the business thing that’s behind all this. What do you do with what we didn’t know that we didn’t know? I want there to be this universal truth like, oh, if you just show people the real opportunity, then they’ll embrace it. And then everybody makes a billion dollars and the product is successful. I think that principle from improv is of yes and. I think we have to meet our brief. We’re asked to have a perspective on something. Part of the politics or the compassion way of having impact is to not leave our teammates and stakeholders in the lurch.
So we have these questions. We have answers to these questions. And also, we feel like there’s some other questions that we should have been asking. We we want to challenge how we framed this business question to begin with. We see there’s new opportunities. We see there’s insights here that other teams outside the scope of this initiative can benefit from.
There’s all sorts of other stuff that you get. And I think it behooves us to be kind about how we bring that up, because no one necessarily wants a project or a thing to think about that they didn’t ask for. So how do you sort of find the learning ready moment or create that moment or find the advocate that can utilize the more that you learn that can have even more kind of impact on the business? That’s not a single moment. That’s an ongoing effort. Part of the dynamic that you have for the rest of the organization.
Again, that was me in conversation with Larry Swanson on the Content Strategy Insights podcast. Check out the whole episode. And if you haven’t got the second edition of interviewing users yet, I encourage you to check it out. If you use the offer code donuts, that’s D O N U T S, you can get a 10 percent discount from Rosenfeld Media. You can also check out portigal.com/services to read more about the consulting work that I do for teams and organizations.
But now let’s get to my conversation with Noam Siegel. He’s a research manager at Upwork, and he’s returning as a guest after four years. You can check out the original episode for a lot more about Noam’s background. Well, Noam, thank you for coming back to the podcast. It’s great to chat with you again.
Noam Segal: It’s absolutely my pleasure, Steve. Great to see you and great to be here.
Steve: Yes, if you are listening, we can see each other, but you can’t see us.
So that’s the magic of technology, although Noam is wearing a shirt that says cat GPT on it. So we’ll see if we’re going to get into that or not.
Noam: I do love silly t-shirts. I just ordered a few more silly t-shirts yesterday. My partner is not very happy about that particular aspect of who I am. But, you know, it is what it is and you get what you get.
Steve: Right. You got to love all of you.
Noam: Yeah.
Steve: So that’s an interesting place to start. Let’s loop back. And so it’s to maybe a more normal discussion starter. We spoke for this podcast something like four years ago, early part of 2020. So, you know, I guess maybe a good place to start this conversation besides T-shirts and so on is what have you been up to professionally in the in the intervening years?
Noam: A lot has happened. I can tell you that quite a lot given it’s not that much of a long time period in the great scheme of things.
Steve: Yes. Mm hmm. Right.
Noam: When we chatted last, I was working at Wealthfront, a wonderful financial technology company, and I was head of UX research there at the time.
Steve: Right. Yes.
Noam: I left Wealthfront for a very particular opportunity within Twitter, now X, because I was very interested in contributing to the health, so to speak, of our public conversation. And I had an opportunity to join what was known at Twitter, now X, as the health research team. But we don’t mean health as in physical or mental health. We mean indeed health as in the health of the public conversation. In other companies, these types of teams are called integrity or trust and safety, et cetera. And we were dealing with everything to do with things like misinformation and disinformation, privacy, account security, and all sorts of other trust and safety related issues. Sadly, a few months, really, or less than a year after I joined, Elon Musk took over the company. And one of the first layoffs that happened at the company were a layoff of basically the entire research team. And so I left before the layoffs, but that was the situation there. And I’d love to talk more about what that means in terms of how we build technology, et cetera. We can jump into that.
From Twitter, now X, I moved to Meta, and I joined the team working on Facebook feed, which some people might view as the kind of brand page of Facebook or even the internet for some people. It’s a product used by billions of people daily. And it was a very interesting experience to work on both the front end of the feed and the back end as well, so to speak. So that was a very interesting experience. And in addition, I was also playing a role in what we were calling Facebook Research’s center of excellence or centers of excellence, where we were trying to improve our methodologies, our research thinking, our skills and knowledge, kind of working on how we work, which is very related to what we spoke of in the last podcast we did together a few years ago, when we talked all about research methodology, et cetera. So that was an interesting experience.
But in April of 2023, along with a couple of tens of thousands of other people, I was laid off from Meta, as was I think approximately half of the research team at Facebook at the time. And several weeks later, I joined Upwork, which is where I work now. Upwork, for those who don’t know, briefly is a marketplace for clients looking to hire people for all sorts of jobs and freelancers who are looking to work, primarily in kind of the knowledge worker space, I would say. Upwork also caters to enterprises who are looking to leverage Upwork as a way to, you know, augment their workforce and hire freelancers or people for full-time positions as well. And at Upwork, I’m a senior research manager. I focus on the kind of the core experiences within the product, which includes the marketplace for clients and freelancers, in addition to everything to do with payments and taxes and work management and trust and safety, which I’m very happy to still be involved in. It’s a topic I care a lot about.
Steve: For Twitter, because you were particularly interested in that health, that trust and safety aspect of it, I guess I want to ask why, what is it about that part of designing things for people to use that as a researcher or as a person that it’s something that you strongly connect to?
Noam: Yeah, I think we have a set of societal ills, let’s call it, very troubling societal ills that I think we need to address urgently and with great care and with great responsibility. And one of those societal ills is the evolution of public conversation, of how we interact with each other as people and how hateful and nasty and unkind we can be to each other. And how much information put out there online is either inaccurate or completely false.
This really came to be more salient in my mind during the 2016 elections to the US presidency. But I think it’s become even more salient ever since for multiple reasons, including the incredible and tragic rise in antisemitism in the world over recent years and all sorts of information running about out there on the interwebs that is, again, factually incorrect around all sorts of topics. Election related, related to certain geographical regions, to certain groups, et cetera. And so this is something I care deeply about, just given my personal background, just given what I’m observing in society. When we last had a conversation, I was at Wealthfront in the fintech space, and I recall a case happening, which really shocked all of us to our core with another company. I’m not going to name the company, but it’s another fintech company. A young person tried to use this other company to make certain types of investments and trades, but he was not well versed in how that world works, how those trades works, what options are and how to use them. And he believed that he had lost an incredible amount of money that he did not have and wasn’t able to lose. And it brought him to enough depths of despair that he ended up taking his own life. And that to me was just one story of many that made it incredibly clear that we need to be responsible and ethical in how we build technology products. And that few things could be more important than working on trust and safety. So yeah, it’s definitely an area I’m passionate about.
And we’re recording this a day after yet another senate hearing with all of the heads of different social media companies who were faced with difficult facts about the effects that they’ve had on society and on families who lost their loved ones and other incredibly tragic stories because of the way they built their platforms, because of things they ignored. And I think research can play an absolutely critical role in building trustworthy ethical experiences, responsible experiences that really matter in this world, probably more than anything else I could think of. So that’s the long answer to your question.
Steve: What kind of information can researchers provide that can — into situations like the ones that you’re describing?
Noam: What sort of research we should be doing or not be doing and at what level, at what altitude, if that company put more effort into the first of all, age gating the platform and ensuring that people have the knowledge and the skill to conduct certain trades, but beyond that, the usability of the platform. Going back to the basics, which we don’t do enough of, I would suggest, which is just making sure that the information one is seeing is clear, you know, and not open to interpretations that could have incredibly tragic consequences. Like thinking I lost $700,000, I think was the number when that was in fact not the case at all. So for me, it goes back to those basics. Research can inform all sorts of more nuanced reactions than the one we’re seeing.
Another thing that happened this week while we’re recording, which demonstrates what happens when you let go of your entire trust and safety team, including researchers, was that Taylor Swift, the incredible pop singer, artist extraordinaire, she was facing something incredibly tough to face online, whether you’re a celebrity or not, which was AI-generated nude images of her, fake images obviously. These were all generated by AI such that if you searched for her name, for Taylor Swift’s name, on particular social platforms, you would see those AI-generated images and perhaps believe, because they were very realistic, that these were in fact images of Taylor Swift when they were not. The solution this company came up with was to remove any search results for the terms Taylor Swift or Taylor or Swift or any combination of her first name and last name, which is moronic. I’m not sure what adjective to use. It’s incredibly aggressive, and I think as technologists we can do a lot better than cancel an entire search query because of that sort of thing happening. I think one thing for sure is that there’s no doubt there is a need for trust and safety professionals. There is a need for trust and safety researchers. We know how to inform the responsible and ethical building of these sorts of products and how to address these issues in much more nuanced and rational ways with much better outcomes. I mean, that seems pretty obvious, but it’s clearly not obvious to some of the people leading some of these companies. I hope that changes, and I’m very proud that at Upwork we do have these teams and we are working on these things. We care deeply about the trust and safety of the people we serve.
Steve: I mean, you’re describing a failure with Taylor Swift AI images that there’s, you know, I guess the jargon is bad actors. People are behaving in a way that’s harmful. And when that happens, when there’s a system that can be exploited or manipulated or used to cause harm, you know, I think you’re identifying like there’s a gap. The system can be used that way. But you’re saying also that without researchers, companies are not as well set up to respond to those malicious behaviors.
Noam: Yeah.
Steve: And I’m, I guess I’m just looking to have the dots connected for me a little bit more like, but you can see sort of the failure of the systems and the failure of the humans that are the malicious users. But in that scenario, or kind of analogous ones, how do researchers serve to either prevent or, you know, mitigate those kinds of malicious uses?
Noam: So I can give – there are a few answers here. One example would be that at X and other companies, some of the research we did and in some companies are still doing goes into supporting content moderators and support agents and other such people who are reviewing this type of content. And the research informs building tools so they can get to those problems faster and eliminate them and get rid of that content in more efficient and more effective ways. So for the time being, as you probably know, there are often humans in the loop here reviewing content. They’re using certain tools to do so. Those tools make them better at their job, and building those tools requires research. So that’s one example I would suggest.
Another example is that in certain companies that, again, care more about these trust and safety issues, research informs providing users with tools that enable them to control their experience. Whether it’s blocking certain people or removing certain things from their experience or a bunch of other things that we can do. But ultimately, some platforms choose to give people more agency and more control over their experience, and research has heavily informed those sorts of tools. And you end up with an experience that is catered to your needs and what you’re willing to see and what you’d prefer not to see. And I think a final example is that even though I think a lot of researchers, we think of ourselves as mostly informing the user-facing user experience. You know, the actual designs that people end up seeing when they use a product, for example, Facebook’s feed. Several researchers in our field work more on the back end of things and helping companies sharpen and calibrate their algorithms such that the content that shows up for users makes more sense. We had that at Twitter, now X. We had that at Meta. Most companies that have any sort of recommendation systems and search systems and other such systems, they’re doing a lot of research on what to showcase to users and what sorts of underlying taxonomies make sense and various tagging systems. All sorts of inputs and parameters that go into these models and adjust these models.
To give an even more specific example, in the realm of AI, we have all sorts of parameters, right? One of those parameters is called temperature, and when you adjust the temperature parameter, it sort of influences how creative versus how fixed by nature the algorithm responds to things, right? Like, how much it kind of thinks out of the box, so to speak, versus not. When you change the temperature of an AI-based tool, that of course influences how people experience it, right? And how they experience maybe how empathetic that experience feels or how aggressive it feels or how insulting it feels and so forth and so forth. And we need a lot of research going into these things to understand how tweaking all of these parameters affects how people perceive these tools, these technologies, these experiences that we’re building. So those are just some of the ways in which research, I think, can inform these topics.
Steve: I don’t know. You used the word health kind of early on here. There’s a quality to the experience that we have with these tools, these platforms, separate from bad actors and abuse and misinformation, disinformation. There are research questions to just set the tone or kind of create the baseline experience. It sounds like that’s what the — if you’re working on the feed at Facebook, if you’re thinking about that algorithm, you’re using research to just create a — ideally in the best situation, a healthy versus unhealthy experience. Just — I think there’s research that talks about, oh, when you compare yourself to others, if you see positive messages, you react this way. If you see negative messages, you react this way. So you’re making me realize that asking these kinds of questions around sort of the healthfulness of the experience, I think I locked in on sort of malicious behavior, bad actors, exploitation and so on. But I think I’m hearing from you that there’s just a base on it, like what’s it like to go on — you know, I mean, what’s it like to go on LinkedIn every day when people are being laid off or when people are trying to get your attention or when people are performing, as people do on all these platforms. There is an experience that research can help understand and inform fine-tuning of algorithms and sort of what’s shown to people and how in order to create the desired experience.
Noam: Absolutely. Trust and safety is an incredibly complex space. It’s very layered. To your point, you can create more trustworthy and safe experiences if you stop bad actors from even entering the experience in the first place. And again, ATX and I imagine other companies as well, part of what we did on the research side was inform things like account security.
And how do we help people secure their accounts and how do we make it harder for bad actors to open accounts even though their intentions are malicious? So you can create more trustworthy platforms by stopping bad actors at that stage. And then there are more lines of defense, which again, research can inform each and every one of those lines of defense to make sure that the ultimate, the end experience for each and every user is a healthy one, is a trustworthy one. And I really don’t think there’s any stage where research can’t have incredibly meaningful impact. And I just really, really hope that as we lean in even more to AI and other incredibly advanced and complex systems that to many of us are a weird and wonderful black box that we simply do not understand. I really hope that we increase our investment in trust and safety exponentially because if we don’t, I really think the results will be horrific. And it’s our responsibility as insight gathering functions, as researchers, whatever you want to call it, to take ownership of this, to advocate for this and to make sure we’re doing this in a way that matches the incredible evolution and development of these platforms. It’s just incredible to witness.
Steve: If we were to go into the future and write the history of, I guess I’ll just call it trust and safety user research, what era are we in right now for that as a practice or an adoption?
Noam: That’s a tough one. That’s a tough question. I think the only response I have for you now, but let’s talk again in four years or so, is that trust and safety in a sense, you mentioned bad actors earlier. Trust and safety in a sense is always this ongoing battle between forces of good, so to speak, and the forces of evil trying to catch up to each other and match each other’s capabilities and then beat the other side. With even better capabilities, I think what we’re witnessing now with AI-based systems is that the pace of innovation, the pace at which they are evolving and learning is shocking and hard to comprehend. It’s really, really hard to comprehend. Now, that’s not to say that it’s not going to take a long time before some of these systems are fully incorporated in our lives.
We’ve been talking about self-driving cars for a very long time, and they are absolutely out there right now in the streets of San Francisco and maybe Phoenix, Arizona, and maybe a few other cities doing their thing and learning how to do their thing. But I think it’s going to be quite a while before every single vehicle on the road is a self-driving car. But that said, these systems are just getting more and more complicated. I think our ability to understand them, it’s getting very difficult. We have to figure out what tools we need to develop in order to catch up, in order for the forces of good, so to speak, to match the forces of evil. And we also need to remember that everything these systems are learning, they’re learning from us. And sadly, the human history is riddled with terrible acts and a long list of biases and isms, racism and sexism and ageism and everything else. So these AI systems are sadly learning a lot of bad things from us and implementing them, so to speak. So again, we have a great responsibility to be better because a little bit similar to a child, AI systems are learning from what we are generating. So we kind of have to be a role model to AI and we have to make sure that we’re leveraging AI, maybe somewhat ironically, to deal with issues created by the incredible development of this technology. So I hope that sort of answers the question.
Steve: We know as researchers, right, any answer that doesn’t answer your question reveals the flaw in the question.vAnd my flaw is that I asked you to decouple user research for trust and safety from everything else. And I think you answered in a way that says, hey, this stuff is all connected. The problems, society at large, the technology, and the building of things are all connected and research is a player in that. So, yeah, you gave a bigger picture answer to my attempt to sort of segment things out. I think we’re going to come back to AI in a bit, but I wanted to ask you, in addition to sort of trust and safety that we’ve talked about over the four years and this issue of building responsibly that you’ve highlighted, are there other things that you have seen or observed about our field that you want to use this time to reflect on?
Noam: Yeah, absolutely. I think, as I mentioned, because this happened to me as well, we’ve seen a large number of tech layoffs and certainly for research teams, but not only, of course. We’ve seen reorgs happen, major reorgs. Because, I mean, reorgs are a reality in tech, everyone who’s worked in tech knows this, but we’ve seen some major reorganizations.
And in fact, we’ve seen entire research teams shut down, including the example I gave earlier of the team at Twitter, now X. And as part of that, we’ve seen some incredibly thought-provoking articles come out. And I’m sure you’ve read some of these. One of them from a former leader at Airbnb, Judd Antin. He was my skip level manager, wrote an article about how the UX research reckoning is here. Another incredibly interesting article was around the waves of research practice by Dave Hora. Jared Spool wrote an article about how strategic UX research is the next thing. And I think that what all of these articles had in common was some sort of discussion on the value that insight-gathering functions or research functions bring to the table. And you might not be surprised by this, but I have a hot take for you on this that I would be happy to discuss.
Steve: Bring it on. Hot take
Noam: Hot take time.
Steve: I’m ready.
Noam: Are you ready for this? So here’s the thing. If we stick to the UX research reckoning framing, I’m a bit of a stickler for words. So the relevant, or I believe that the relevant definition for reckoning that Judd meant to reference is the avenging or punishing of past mistakes or misdeeds. So basically, as UX researchers, we made some mistakes, we made some misdeeds, and now we are being punished for it by being laid off. And again, the broader point in that article, I think, is what’s the value we bring as researchers? And I am here to say that although I agree that we’ve made mistakes, this has, everything that’s happened, has very little, if anything, to do with value. To do with the value that we bring. And I think it has everything to do with valuation, which is a very different thing. And if I take Meta as an example, Meta was suffering from a tough time as a company, spending a whole lot of money on AR, VR, and other capabilities. The stock was at one of its lowest points in recent years, if not the history of the company. And so Mark Zuckerberg announced a year of efficiency. And part of his idea of efficiency was to lay off about half of the research organization. And we have to ask ourselves, is that because researchers did not bring value to the organization? And again, I would suggest not. I would suggest that these days I’ve had work, and in every company I’ve been part of, I’ve seen some incredible value brought forward by researchers. Insights that can make a huge difference to everything from the user experience to the strategy to use Jared Spool’s and others terms.
But there’s a couple of problems. The first problem is a problem of attribution. How can you calculate the return on investment of research? How do you know and how can you record and document which decisions and which things were influenced by research and which weren’t? If I’m an engineer or designer and I’m working within my Jira or whatever platform or linear or whatever platform you’re using to manage your software development, then I have some sort of ticket. I have some sort of task. I write 10 lines of code. Everyone knows those 10 lines of code are mine or mine and other people. Everyone knows what those lines of code translate into in the experience. And so the ownership of what that experience looks like from design to engineering is clear because it’s clear who made the Figma and it’s clear who wrote the code. And everything is incredibly accurately documented. When it comes to research, when it comes to knowledge, you know, research is circulated in all sorts of ways, right? From Slack channels to presentations to a variety of meetings and one-on-one get-togethers with cross-functional partners. And in all of those meetings and all of those interactions, research is coming through in some way. But it’s incredibly fuzzy and unclear how that translates into impacts on the products. That doesn’t mean research doesn’t have value. That means it’s hard to measure the value.
And then one more thing that I think is going on, which you probably know very well as one of the most knowledgeable people on the topic of interviews that I know, is what happens when I’m responding to your question? In this case, maybe you have some questions about what insights have we learned? What happens as I’m giving you a response? What are you doing? Make a guess.
Steve: I’m thinking about my next question.
Noam: You are thinking about your next question. It’s so hard to avoid that tendency. And I think in many cases, product managers, product leaders and other cross-functional partners of research, they’re taking in the research, but they’re just thinking about their next question. And to be fair, I think that one more thing that’s going on here is that we as researchers do not understand the feeling of being held accountable for certain metrics. And for millions, if not billions of dollars in revenue, that can be moved one way or the other by the quality of what we choose to build and what we choose not to build. And the roadmaps we have, the strategy we have, etc. Usually those are product leaders who are accountable for that. And we’re not. And so the pressure is on them. And so as they take in our insights, they can’t help but just think their own thoughts and think about their vision and maybe ignore certain things that we share. And then business leaders, ultimately, what do they care about? Again, valuation. The stock. That’s just how it works, which is why I said in the beginning that I don’t think this is about value at all. I think it’s about valuation. I think business leaders are optimizing their business for their valuation, for their stock price. They’re not laying off researchers because we didn’t deliver value or because we weren’t strategic enough. They’re laying off researchers and many other people because that’s one way of a few ways to become more efficient, to look good in front of your shareholders. It’s not such a complicated game. You know, we’re doing this interview a day after a particular company started offering dividends to its shareholders, and that had a very expected effect in the market on that stock. It just went up quite a little bit higher. That’s how the game works. Those are the dynamics of the market.
And so we’re in the situation where I’m not saying we haven’t made mistakes again. I think Judd, for example, absolutely had a point when he discussed the different levels of research and the fact that we’re making a mistake by looking at usability as some sort of basic, tactical type of research that only junior researchers should do and that we shouldn’t be focused on. And that we should only be looking at higher levels and higher altitudes of research. I couldn’t agree more. I absolutely agree with Judd on that. But this basic premise of research not delivering value I think is incredibly problematic. And I don’t think it’s correct. I don’t think we need to move into some third or fourth or whatever wave of research. I just don’t see that personally. I think many of us have already been in wave one and wave two and wave three of research. We’ve already been doing strategic research. We’ve already been affecting the business level, the product level, the design level. We’ve already been conducting all sorts of research from usability to incredibly foundational, generative research. And I think we’re being very, very hard on ourselves. And I think we need to cut ourselves a little bit of slack. Just a little bit.
Steve: I mean, I’m all about being kind to ourselves and not blaming ourselves for things that are beyond our control. We’re all susceptible to that and it’s hard to kind of watch that going on collectively. But when you’re in a situation where there is, I don’t know, a misalignment of values, like what, you know, like you said, value versus valuation. When that misalignment, that’s my word, not yours, when that exists, we can cut ourselves slack, but that’s not going to change that gap. I don’t want to say to you like, well, here’s, you know, you just outlined a systemic, deeply rooted, the nature of capitalism, it goes all the way up. How do we fix that? I guess I don’t, I think that’s not a fair question, although, you know, take a shot if you have a hot take there. Are there mindset changes or incremental steps or, you know, things that you’ve seen research teams do that acknowledge to some extent the difference between we’re not breaking value to their concern is about something else and how do we kind of meet them where they’re at?
Noam: So look, Steve, that’s an incredibly fair question. And I do want to be crisp about the fact that, yes, we need to be doing something. Something needs to change even if I view the problem differently. But before I get to that, just to reiterate, we as researchers know very well that it’s absolutely critical to identify the problem and to identify the correct problem at the right level. So before I get to what we should do, I just want to highlight the fact that in my view the issue here is that some of us have misidentified the problem, in my opinion. And we need to be tackling the actual problem.
And just to get to that and to pivot to kind of the second topic that we did cover last time and I want to cover again today. We did talk about in our original conversation about research methods and how we do research. And I do think, even though I believe we’ve brought a lot of value to the organizations that we work in as insight gathering functions, I do absolutely believe that given the broad evolution of the landscape we operate in, we do need to rethink how we operate. Not because we haven’t delivered value, but because the ways in which we can deliver value are rapidly changing. And I think we can now sort of extend ourselves.
And I was very influenced by a book titled Multipliers, not sure if you’ve read it. But the basic idea of it is that there are employees within any company who are multipliers in the sense that they don’t just do great work, they make everyone else’s work even better. They level up everyone around them and they create these situations where they define incredible opportunities and they liberate people around them to get to those opportunities and to make the most out of them. They create a certain climate, which is a comfortable climate for innovation, but at the same time an intense climate where a lot of incredible things can happen. Where I’m getting at is that, and this is not surprising probably to the people listening to this, is that the era of AI is upon us and I think it’s incredibly important to acknowledge the ways in which we can extend our work and ourselves with AI tools. So I know that my mind has moved a little bit from methods, so to speak, to leveraging AI to use similar methods but at a scale that we’ve never experienced before and we’ve never been able to offer before to our partners.
Yeah, I mean, I think there are certain paradigms in our industry that are changing and perhaps AI is even eradicating those paradigms and rendering them useless. I mean, if it’s okay, one recent example I have is that we had this paradigm that we need to make a tough choice. We’ve talked about this, you and I, a little bit. We have to make a tough choice between gathering qualitative data at small scales, which can often be okay, by the way, unless you’re developing a very complex product or unless you want to make sure that trust and safety is in the center of everything you do and then maybe you need a little bit more scale and you just couldn’t get it because you didn’t have the people to reach that scale of interviews or qualitative research. Or, of course, the other choice you could make was to gather quantitative data at any scale you like as long as you can afford it, namely by sending out surveys to hundreds or thousands of people. The issue is survey data is shallow data or thin data or whatever you want to call it, whereas I believe it was Sam Ladner who coined the term “thick data” for qualitative data. And sometimes you need that thick data and you need it at a scale that we were never able to reach before. And AI enables you to do that.
I’ve personally witnessed tools, one of them being Genway, which are completely revolutionizing the way we conduct research. I’ve seen existing research tools, Sprig would be a good example, Lookback, there’s so many incredible tools that have incorporated AI into their workflows. And they are making paradigms like the one I mentioned, this choice between thick and thin data, they’re making them irrelevant, absolutely irrelevant. Which is very interesting to me. And it ties to this idea of multipliers, this idea in this book I love. Because AI research tools, like the ones I mentioned and so many more that we could talk about all day, they enable us, in a sense, to be multipliers. They liberate us, in a sense, to do a lot more than we could ever do before. And hopefully that translates into us enabling our cross-functional partners and the teams we work in to deliver their best thinking and their best work as well. So that’s, I think, where our field is going in a nutshell.
Steve: Can you describe with maybe a little bit of specificity what a work process or set of work tasks that a researcher might go through where AI tools like the ones you’re describing, like how is that, yeah, what are they doing, what’s kind of coming to them and, you know, what does that process look like that’s AI enabled?
Noam: I can give a couple of examples. The first example, if I think of a tool like Genway, an interview tool, is that interviewing is tough, as you know well. You’ve written what I consider, and many people in our industry consider, kind of the Bible of interviewing people. No offense to the actual Bible. And as someone who’s written one of the primary guides to how to interview people, I think you appreciate more than others how complex being an interviewer can be. It’s something that you can learn over years and years of training and mentorship and still not nail some pretty critical aspects of interviewing. For example, asking the right, the best, the ideal follow-up question, and actually listening to what’s being told to you actively, rather than thinking about that follow-up question all the time, because listening is what enables you to ask a good follow-up question. Systems like these can train on an unlimited number of past interviews and an unlimited number of texts like your book, and learn from all of that how to conduct the best possible interviews, right? And these types of abilities to learn and then apply that learning in an interview situation, I believe it’s fair to say it would be technically impossible for any researcher to achieve that level of learning, certainly in a matter of hours or days or months at the most or weeks, rather than years in the case of a researcher.
You know, one of my hot takes, I hope the audience doesn’t kill me for this, is that David Letterman interviewed people for many, many years. And I personally think David Letterman is a horrible interviewer. I never understood why he asked the questions that he did, and everything about his interview style is very, very odd to me. But putting that aside, interviewing is a very complex skill. None of us can really ever witness how other people do it. And we all have to spend years of practice learning how to become better interviewers, which is a deceptively difficult skill to build. And these AI tools are coming in and, at least in theory, can learn all of that shockingly quickly. That’s one example, and I’m very curious to see how the research community responds to these types of tools and uses that, and what issues they do find in the quality of these types of interviews and how they can be improved.
The second example is that I recall from even my undergrad psychology studies, not to mention my graduate studies, that our ability to hold information in our brain is quite limited. And so even when we’re synthesizing five interviews, not to mention 500, because sometimes you need 500, it’s very, very challenging. If you do five 30-minute in-depth interviews with people, organizing your thoughts and synthesizing those interviews has never been a trivial task. And I think there are a large number of biases and other issues and strange heuristics that we use to synthesize information that might not lead to the optimal outcome, an outcome that’s as objective and accurate, an accurate representation of the entirety of those interviews and how they interact with each other as we would want them to be. One particular task that generative AI and AI in general is very good at is summarizing and synthesizing information. And especially as we collect more information, that becomes a lot more relevant and even critical, I’d suggest.
When we entered the big data era, we needed to develop a bunch of tools. You know, so many companies came out of that era building tools that enabled us to analyze and very easily visualize in beautiful dashboards what those data are telling us. Now, we can also start collecting qualitative data at unimaginable scales. And not just qualitative or quantitative data, because I think that distinction is going to matter less and less as time goes by, but more importantly, we will be and become so much closer to the people we serve, our users, our customers. I think we’ve talked about this in the previous podcast, but I think we talk about diary studies and how diary studies used to be physically sent to people’s mailboxes, right? And so you as a researcher had to plan your studies, send out an actual diary, have people log their entries into it, and then they would have to send it back, and then you would have to very manually look into those entries. And obviously that takes a very long time. These days, and especially with the support of AI tools, you can be in touch with the people you serve all the time, as much as you want and as much as they want, and you can both collect data and synthesize data and even communicate those insights at a pace that’s hard to even fathom, for me at least. But it’s very immediate. Can you say very immediate or can you not modify the word immediate? Is something either immediate or not? Okay, I don’t know. I’m just afraid of my mother and what she might say here about my grammatical choices. But anyway, yeah, yeah. But I think that’s what matters the most.
Steve: We’ll ask her to fast forward over this part.
Noam: People’s schedules don’t really matter anymore because they can choose to interact with AI, for example, whenever they want to. And it can be in context in real time. And then AI can immediately synthesize those learnings. And it can immediately improve the way it collects insights based on that interaction and all previous interactions. I was thinking about this a lot in this framing of multipliers. Like, who is the multiplier in this context?
Steve: What does this hold for researchers? Like there’s research, which I think you’re describing a really audacious vision for how, what research will be and I think speaks to the point about valuation versus value, but researchers, which currently refers to humans, what do you, what’s your vision for that or your anticipation for that?
Noam: Is it the AI? Is it the researchers? Like, who is multiplying whom? But what I do think is that, well, first of all, I’m a techno-optimist or whatever you want to call it. Even with everything that’s happened, even with all of the tragedies and the negative aspects of technology that we’ve discussed in this conversation and others, I am still at heart a techno-optimist. And so my deep belief is that certainly for the foreseeable future, if not beyond, AI will become a valuable extension of ourselves and our work. And I do believe that even if we don’t have to deliver more value necessarily than we already are, even if our value just goes underappreciated and there’s nothing terribly wrong with how we’ve approached things, I still think that AI will augment our work, will amplify our work, will enable us to really invite the teams we work with to do their best work ever.
Because AI will help us see opportunities for research that we haven’t seen. It will help us settle a bunch of debates that maybe we’ve struggled to settle before. It will help us to connect with more users, more customers, more clients, whatever you call them, from all over the world in a way which vastly improves how equitably and how inclusively we build technology products, which is something that we’ve struggled with traditionally, if we’re being honest here.
A very clear example of that is that an AI can speak a bunch of different languages and connect with people across all time zones and languages, and even mimic certain characteristics of a person so they feel more comfortable in that context. So for multiple reasons, I feel like if we’re at all concerned about getting buy-in from our partners, if we’re concerned about the value we bring, the impact we have, I definitely think AI tools can really improve our chances of getting to where we want to be. And I think it’s going to be a very long time, if ever, before these tools replace us as researchers. You know, the reason I chose to be a researcher, the reason I chose to be a psychologist is because of the incredible complexity of the human mind.
You know, the people listening to this can’t see this, but for my friends who are physicists, for example, so I’m holding up my phone right now in front of the camera, and if I drop my phone onto my desk or onto my floor, it’s a very easy calculation for physicists to say how quickly will the phone hit the desk, and what energy will there be there when the phone hits the desk, and what’s the chances that the phone will break given the speed of its fall. Physics is a beautiful thing, but it’s also a fairly reliable scientific practice.
You know, there are rules in physics, and they’re fairly clear. And even though physicists sometimes, on occasion, like to look down upon people like me with a PhD in psychology and a background in psychology, I think that in many ways, the field of psychology and other fields that deal with the human experience, with the human mind, they are so much fuzzier and so much more complex. And I’m saying this because I think many, many other professions will be replaced by some form of AI before researchers ever are. And that’s because of this complexity, this fuzziness that’s hard to capture. I think in many ways, our field is incredibly technical, but in other ways, our field is not technical at all.
There’s a lot of art to it, and there are all sorts of different aspects to it. It’s a lot easier for an AI to generate a piece of code or to generate a contract or to read a mammogram or an MRI and identify something, rather than talking to another human being and understanding them deeply. I think that’s a lot more complicated. So I’m not too concerned about the research fields, but we’ll wait and see, I guess.
Steve: So as we kind of head towards wrapping up, you know, since I’ve known of you and known you, I’ve always seen you doing different things, I guess, to be involved with the community of user research and what’s that look like for you now?
Noam: I, like many people in our profession, have definitely been on a journey. And if I’m being honest with myself and the audience, it’s been a very challenging few years, certainly for me. And I know for many others out there, whether it’s COVID and layoffs and a bunch of other personal events and things in life that just happen to us. And that’s part of the reason why I decided, and I know quite a few other people in our community decided to do this, to pursue coaching, among other things. I took a certification in coaching. I think even with my background in psychology, I felt like there was so much more to learn in this realm. I think that it’s always been important for me to support people in our community, and I wanted to do that even better.
And so one of the things I’m doing these days, to a limited extent, is some coaching, not just for UX researchers or UX professionals, but people in tech in general. And then I have some thoughts for the near future around sharing some of that coaching and ideas in other ways. In addition to continuing to teach in all sorts of ways, I’m still teaching at a bunch of different institutions and planning to restart some of my teaching on Maven, which is a wonderful platform for learning all sorts of things. I think the general trend in my life and career right now, and maybe this will resonate with people, is that definitely a challenging few years feel like maybe now coming out of it a little bit, ready to take on certain other challenges in addition to my role at Upwork, etc. And I know that I really want to be there to support our community in particular as we all go through rather challenging times.
So I just invite anyone who wants to get in touch to message me on LinkedIn or email me or get in touch in any way that works for you, and I’d be happy to chat and help. And then I also thought about, and we’ll see where this goes, but as you can tell, these topics of AI and where our field is going and how it’s evolving, that’s a lot to me and I’m thinking about it constantly and want to be part of this evolution, if not revolution, in how we work. And so I’d love to have conversations similar to this, whether out in public or privately around these topics to continue to understand them. And I’m just looking forward to seeing what is next for our industry. I feel like when we spoke a few years ago, I think we had a solid sense of what’s to come. And I think in many ways we discussed things that did end up manifesting in some way or another. But in this conversation, Steve, I don’t know what we’ll be talking about in four years. If you give me the opportunity to talk to you again. And I can’t decide with myself if that’s exciting or incredibly anxiety-provoking. So I don’t know. Why not both? Or if I’m going to be an optimist or say I’m an optimist, then I’ll choose to be an optimist and say, maybe that’s exciting.
Maybe it’s exciting that I really don’t know what’s coming down the line. But I do know that I want to thank you again so much for taking the time to do this. It’s always such a pleasure to talk to you. So thank you for giving me the opportunity.
Steve: That’s my line, man. I’m saying thank you.
Noam: Well, for me, it’s a special treat. Maybe I can share also with whoever’s listening to this that we did get a chance to meet in person finally. Not that long ago. And that was even more of a treat. And I really do hope that our community of research can get together more often moving forward and meet up and discuss all of these issues.
Steve: These are some really encouraging, I think, provocative things to think about and some really positive and encouraging sentiments for everyone. And there will be show notes that go with this podcast as always and so the stuff that Noam you’ve mentioned and you know ways to get in touch with you, we’ll put that all in there so people can connect with you if, if by some chance they aren’t already connected with you. So yeah, I’ll flip it back as well and say thank you for taking the time and for thinking so deeply about this and sharing with everybody. It was lovely to have the chance to revisit with you and kind of catch up on some of these topics after a few years and. I look forward to four years from now doing this again if not sooner.
Noam: Can’t wait. Adding it to my calendar right now.
Steve: All right.
Noam: Cheers, Steve. Have a good rest of the day.
Steve: Cheers.
Well, that’s another episode in the can. Thanks for listening. Tell everyone about Dollars to Donuts and give us a review on Apple Podcasts. You can find Dollars to Donuts in most of the places that you find podcasts. Or visit portigal.com/podcast to get all the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 36. Noam Segal returns first appeared on Portigal Consulting.
Today we’ve got a quick program note about new episodes of Dollars to Donuts, an announcement about my new book, and an interview with Steve Portigal.
Show Links* Interviewing Users (2nd Edition) * Steve on The Rosenfeld Review Podcast * “How-to with John Wilson” on HBO
IntroductionSteve Portigal: Hi, everyone, it’s Steve Portigal. The podcast has been sitting quietly for a little while now, I guess, but I’m back today with two quick announcements and a bonus.
First of all, new episodes are coming. I don’t know exactly when but the wheels are in motion. I’m looking forward to some great conversations about leading user research, and I’m excited to share those with you.
Second, I have a new book just about to come out. I wrote Interviewing Users in 2013, and it’s become a classic text about user research. Now it’s 2023, 10 years later, and lots of things have changed. So I’ve updated it! The book – Interviewing Users, second edition, will be published on October 17, 2023, and is available NOW for pre-order from Rosenfeld Media, at a 15% discount.
The bonus is my conversation with Lou Rosenfeld, where we talked broadly about user research, as well as the second edition of Interviewing Users. This interview originally appeared on the Rosenfeld Review Podcast. So welcomed by the sound of the trumpeting elephant, we’ll go there now, and I’ll see you back here before too long with more episodes of Dollars to Donuts.
The post New episodes! New book! first appeared on Portigal Consulting.
Today we’ve got a quick program note about new episodes of Dollars to Donuts, an announcement about my new book, and an interview with Steve Portigal.
Show Links* Interviewing Users (2nd Edition) * Steve on The Rosenfeld Review Podcast * “How-to with John Wilson” on HBO
IntroductionSteve Portigal: Hi, everyone, it’s Steve Portigal. The podcast has been sitting quietly for a little while now, I guess, but I’m back today with two quick announcements and a bonus.
First of all, new episodes are coming. I don’t know exactly when but the wheels are in motion. I’m looking forward to some great conversations about leading user research, and I’m excited to share those with you.
Second, I have a new book just about to come out. I wrote Interviewing Users in 2013, and it’s become a classic text about user research. Now it’s 2023, 10 years later, and lots of things have changed. So I’ve updated it! The book – Interviewing Users, second edition, will be published on October 17, 2023, and is available NOW for pre-order from Rosenfeld Media, at a 15% discount.
The bonus is my conversation with Lou Rosenfeld, where we talked broadly about user research, as well as the second edition of Interviewing Users. This interview originally appeared on the Rosenfeld Review Podcast. So welcomed by the sound of the trumpeting elephant, we’ll go there now, and I’ll see you back here before too long with more episodes of Dollars to Donuts.
The post New episodes! New book! first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Danielle Smith, the Senior Director of Experience Research & Accessibility at Express Scripts.
Something that I’ve really changed the way I thought about since I’ve been at Express Scripts — we are in the healthcare ecosystem. So the experiences we deliver, if they are not of quality, they do have serious repercussions on people’s lives and people’s time. We are ethically bound to measure the user experience from different perspectives. Before something launches. We have prototypes or concepts or ideas, we do our due diligence in terms of user experience research, to make sure that the thing that we’re putting out on the world doesn’t just happen to people. – Danielle Smith
Show Links* Artwork hung in our living room * Danielle on LinkedIn * Danielle on Twitter * Express Scripts * Net Promoter Score (NPS) * DeBakey High School for Health Professions * Industrial/Organizational Psychology at Rice * Human Factors and HCI at Rice * Human Health Factors at Johnson Space Center
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Many years after we moved into our house, we finally hung up our art. Sure, we had hung up individual pieces, something that was already in a frame, say. But it was always piecemeal, a nail here, a hook there. And we continued to accumulate meaningful pieces from travels or family events. And we continued to occasionally pull out the box (a moving box, made for a mirror, I believe) and spread out all the various bits and pieces and just generally fantasize about having them up.
My goal, however, was to have a plan, an intentional way of placing the different posters, prints, photos throughout our home. Every time we would try that, I would get overwhelmed and give up. I tried taking small bites: after seeing homes with big frames and medium photographs, we chose a few photographs from our travels, blew them up, ordered specially cut mat boards, and frames. We mapped out where in our living room these would go; we essentially carved off part of the home and planned the photographs that would go there. I hoped that this would simplify the challenge of where to put the remaining posters, but we found ourselves stuck, still.
Eventually, we opened up that box and made some hard decisions about what to hang and what to set aside, and then – before too much time could go by – arranged to get everything framed. We were inching closer, but sitting with our posters and prints, all framed, we still couldn’t figure out what to hang where and so (this shouldn’t be surprising) I got overwhelmed and gave up.
But buried in that frustration and surrender was a recognition of what skills I’m missing – an ability to reorganize visual information spatially in a few different ways, a set of starting principles for grouping, placing, and so on. Surely someone must have this expertise and be offering it as a service?
It turns out that, yes, there are professional picture-hangers. We called one and booked an appointment. A few weeks later, on the scheduled day, at the expected time, the doorbell rang. We answered the door and before we could finish greeting them, two men were in our foyer, one of them having magically unfolded and gently placed a table (please assume it transformed from a flat briefcase shape to a table with a soft whoop noise).
The guy with the table handled production. The other one handled creative (and the clients). We showed him all of our framed pieces and talked a very little about what they meant, and we showed him around our house, pointing out areas we were interested in and the few pieces we had already hung. Meanwhile he was riffing constantly, throwing out ideas, getting energized, and delegating to the production guy, who began attaching extra mounting hooks and hanging wire to all of our store bought frames. After a short time, we backed off, and we watched them work.
The “creative” began moving frames into different rooms, laying them on the floor and trying different combinations. Like a problem-solving algorithm, the solution began to appear, bit by bit. The floors throughout our house began to fill with clusters and arrangements of different prints, both thematic and visual groupings. We were called in for frequent consultations as the plan emerged. Eventually there was a plan for where everything was going to go. This was the piece we could not have done ourselves and in a short time, they had done it.
Then came the rest of the production. They began hanging up everything. That meant figuring out where each item went – exactly, putting in nails in exactly the right spot, using a level, all the details. Especially because so many pictures were clustered, something being slightly off would really show, so perfect execution was key. This was also something we could have not pulled off ourselves.
The final results were astonishing. Rather than hanging things in a grid, with the top edges aligned and a consistent space between each, they put together a number of clusters where the posters emanated from a central point in an almost-spiral flow. And they chose how to place different prints within that cluster in order to create a kinetic sense, such as having a poster with a bird along the left hand side, with the bird facing to the right, so that the content of the images supported the physical placement on the wall. This was not something we could have even imagined, let alone executed.
It is immensely gratifying to be in the presence of a highly-skilled individual. When those skills are being deployed for your benefit, it adds another layer of delight. I believe that delight is further enhanced when we ourselves have tried and failed. This story is a reminder to me to seek out these magically-talented individuals and take advantage of what they have to offer, whenever I can.
And it’s my goal to be a magically talented individual for the people I work with, someone who brings in a level of skill that isn’t achievable without my help. I strive to conduct user research with that kind of finesse, I hope that when I coach and train teams in doing research, I’m helping them see what that magical level of skill looks like and move them forward on their own path towards achieving that.
I would love to hear from you about what you are working on and how my expertise can support you in moving your effort forward. Please keep me in mind.
Now let’s go to my interview with Danielle Smith. She’s the Senior Director of Experience Research & Accessibility at Express Scripts.
Steve: Well, Danielle, welcome to Dollars to Donuts. Thanks so much for being on the podcast.
Danielle Smith: Thanks for having me.
Steve: Why don’t we start with an introduction from you to say a little bit about what you do where you work.
Danielle: My name is Danielle Smith. I am Senior Director of experience research and accessibility at Express script. And Express Scripts is actually Cigna company. But we are a pharmacy benefit provider. So we help with getting your prescription drugs. If you have insurance coverage, we are the companies that help with that. And my team measures the quality of the user experience from a few different perspectives. One includes user experience research. The other is our NPS program for our digitally engaged customers, and our data science team and some digital metric reporting, oh resides with my team.
Steve: So I want to ask about your team. But I want to just back up slightly, who are the kinds of people that are having this digital experience with your company.
Danielle: So our digital experience is primarily geared towards people who have prescription drug coverage through their health insurance provider or directly if you’re on Medicare Part D, but you are interested in getting home delivery of your maintenance medications. Our website does have kind of general information. If you do have a health plan, you want to go in there and look up a medicine see how much it costs which pharmacy But most of the functionality on our digital tools for consumers really look at the home delivery of medicine,
Steve: and then use the phrase to kind of describe what your team does. It’s about measuring quality. Is that right?
Danielle: Yeah,
that’s sort of the way I think of it.
Steve: Can you pick apart that phrase a little bit? It’s an interesting one. And I’m curious how you think about
Danielle: something that I’ve really thought about or changed the way I thought about since I’ve been at Express Scripts, we are in the healthcare ecosystem. So the experiences we deliver if they are not, if they are poor, or if they are not of quality, they do have serious repercussions on people’s lives some people’s time. So when I think about my team, we are almost ethically bound to measure the user experience from different perspectives. Before something launches. We have prototypes or concepts or ideas, we do our due diligence in terms of user experience research, to make sure that the thing that we’re putting out on the world doesn’t just happen to people. We have some ideas It’s usability, its appropriateness. And we’ve really tried to test a test and air quotes, do research on everything before it hits the site. And some cases, of course, we can’t do that. And I’ll talk about that in a second. But we’ve tried to do, I would say far above 80%, of what hits the site goes through some level of user research related to that. And another aspect of what we do to ensure quality is to make sure it’s accessible. And most of our accessibility work has really geared towards visual impairments or different visual abilities, because we are the digital team. So my team serves as subject matter experts to help our developers aren’t marketers, and to deliver experiences that are usable by people that use screen readers or have low vision just right out of the gate. So we don’t have a separate experience for folks who may be having any sort of different visual abilities. And that was a big point of focus for the first few years of my role here was to just really get that going. But it’s part of our user experience, practice and it’s in our The research team because we do specifically do studies, blind and low vision users throughout the year, to make sure even though it’s something’s technically compliant, it is actually usable by folks that use a screen reader specifically. But following good design practice, and rolling in those things before you launch still does not guarantee good experiences. And so the other parts of my team work to measure how what we build interacts with reality, and to see if it does actually generate a good experience and has, you know, high quality like we want. So the NPS program I know NPS itself is a problematic metric and has its taters in the industry. And that’s fine. I’m not here to defend it. But what it does is gives it lets us speak in a voice that executives and understand and it gives us the leeway to have an open channel with our users. So we send out these monthly surveys for one part of the program. But another part of the program allows users to leave feedback directly in our mobile app on our website. Tell us what’s going on. So yeah, that gave us a score, report the score, executive lift score. But the words that they use when they tell us if we’re on the mark or off the mark have been priceless and making sure that things actually work as intended given reality. And then the other piece is behavioral analytics. So we run a B tests, we instrument the site and make sure that things are working the way we think they should work in terms of people flowing through different funnels or parts of the site and monitor the experience that way. So we work really closely with our product teams to help them understand their metrics, make sure they’re gathering metrics and help them use them and interpret them in the right ways. So when you use the word quality, what does that mean for you? So for me, I’m using quality to be synonymous with a good user experience and healthcare so complicated. I’m not going to be so arrogant as to say we’re going to delight people. I just want them to be able to get their task accomplished which is getting their drugs prescriptions delivered. or checking the status of their delivery. I want them to get their tasks accomplished with ease. And that’s what I mean by quality. Are we not getting in their way? Do we make them feel like they’re part of the process? And do they understand what’s going on? Because it’s complicated enough.
Steve: You also use the phrase, I hope I get this right. working as intended given reality.
Danielle: Oh, yeah. Reality when you’re talking about healthcare is something that we cannot ignore. And we can’t we can’t even formulate all of the scenarios a person might be in at least you can’t do that yet. And people’s realities may be like right now like, Okay, well, now my doctor’s office is closed, and I need a refill. How do I do that? It’s a pandemic, what’s going on. So we have to be making sure that we are listening to user feedback to make sure we update the website, and even communications and talk to our friends in the call center to make sure that we are ready to respond to our users. reality, smaller, like everyday situations happen. As you can imagine, if you’re sick, there are some conditions that give you kind of a transient low vision situation. So you use the website yesterday, I, my team wants to make sure you can use website tomorrow. Just because you’re on this medication or you have this condition should not take away your ability to use certain tools. We also have lots of families, family situations are fluid who has access to your account today, you might not want them to have access to your account tomorrow. And this is your health information. So we have to be always ready to listen and react when people’s contexts change, or their actual reality is way more complicated than we even thought of when we were designing the user interface.
Steve: Can you say a little more now about the word measure is kind of the key verb that you use to describe kind of overall what you all are doing.
Danielle: Sure, and I’m glad you picked up on that because I don’t want to give people the wrong impression that Don’t do discovery or you know the call, but I am of the belief that the work that we do is still measurement. We may not have a type metric, we might not have a number around it, but it is still the collecting of data to get at some underlying construct as best as we can. And I was recently reminded that I came from a grad program that was more quiet leaning. And it’s probably why I did grow my team, the way that it grows more, including analytics, as well as user research and having them live side by side. So it’s not just about things we can put a number one it is about understanding scenarios and understanding people and listening to how we can resonate with our users and more. So I don’t want to miss categorize what we do, but I think of measure in the broad sense of the word.
Steve: I mean, it’s a really powerful frame for what research does and there are lots of different frames for you know, if you have to say one word, what is it that we do and measure is One that I have heard all that often, but it’s very compelling, especially when you kind of explain, hey, there’s lots of different kinds of measurements. And even when you’re talking before about NPS, and part of its value as a way to open up a conversation with other people in the organization, when you say measure overall, I imagine that that is similar that by framing this around measurement, you are positioning yourself relative to your stakeholders and colleagues that this is the kind of guidance and information that you can bring.
Danielle: Yeah, and I talked about it that way I talked about what my team does is x data sounds very superhero like that. It’s like it’s data about the user experience. And just because I say data does not mean it’s all quantitative, and it’s taken a while for our partners and even some folks on my team to get on board with this idea. But we need to be able to have some sort of convergence on the user experience. And convergent validity is a goal of most research teams that I’ve been exposed to where you want to have some analytics show problems and user research give you the other side of it, maybe some survey data to give you some aspect of scale of some of those opinions. And by us all being on the same team, it gives us that ability to be fluid in our methods and speak to the business in a way that allows them to hear us. And sometimes that way is to use NPS or a web analytic metric to get our foot in the door and expand how they view the users by layering on the qualitative and sometimes it’s vice versa. We have different stakeholders that are more interested in in qualitative data and then we overlay some quants to help them understand scale and focus and or if it’s even able to get to that level of maturity given where we are in the process.
Steve: Accessibility is part of the group and for me it’s new to hear accessibility specifically called out, alongside other different roles. How did accessibility end up being part of what you’re focused on.
Danielle: Oh man, it’s hilarious he was just like a line in my job description analysis like that. Okay, and I thought what that meant was, this is literally my first week on the job I thought that would be similar to my accessibility role in other parts of my other jobs I’ve had before, which was its usability person I need to be aware accessibility exists, and I enrolled in some tests in my normal course of evaluation that help the accessibility team, check their boxes, about compliance. However, I did not do my due diligence about looking at the expressor sweb site to understand what that might in my job description actually meant. Just say like at previous jobs at Dell and larger companies have specific hardware interface requirements that come from the accessibility team that kind of get handed to the usability team that we just make sure as we go through development, make sure that they get rolled in, but what I did not notice and Express Scripts with that. Back when I joined the company in 2016. They had two different websites, there was a link on the top of the site that said accessible view. And it took you to a text only view of the site, and during my first week on the job My boss is explaining this to me and how, nobody really wanted that to be the strategy and the teacher. I was like, Oh, so we need to learn how to build accessible websites, she’s like yeah, like, oh, okay, well, let me figure out how to do that. So it was a whole different a different perspective on the same problem, but it gave me that opportunity for us to say that we’re going to build a website that works for folks of different abilities, without having a separate site that was personally offensive to me as a Black person to have something separate for people who are different, really annoyed me. So, I wanted to make that go away. And having more inclusive web experience. So that became part of our world, because at the time Express skip splits really open to having a much more modern digital experience, we came up with a plan to help spread that awareness spread that skillset across the organization, one person on my team really picked up the mantle and dove into accessibility, best practices, compliance. She’s not a lawyer but building relationships across the organization to have that compliance and a network, and then diving into doing research with totally different abilities like specifically people they use screen readers because we wanted to make sure that not only did it kind of pass all the checks, but that it, like I said, was usable, and bring that video and bring those usability results into the organization, so that they can understand what it meant to be inclusive in that way. And that just really fit with the way we do user research, because the whole point. A lot of the point of user research is to bring the voice of the user outside of our walls into our walls to help people inside the company, understand the operating environment of people outside of the company. And by having a focus on accessibility within user research allows us to apply those same principles same practices that folks were kind of used to, but with a different audience, and it just really worked well. And so, I can’t remember the date but it was at least two years ago, we sunset it that outside, we have one experience, and we have a group of folks within the organization that are really passionate about this developers help each other out. Under normal situations we would actually run usability testing on site at the American Council of the Blind. To get more engagement from the community, it actually turned into a natural extension of the usability team meeting over it’s showing my age but user experience team, because, as designers, as researchers, we are responsible for inclusivity and accessibility is one part of that, and in health care, especially cannot, an all good consciousness exclude that important of a population. And so we do have focus on that and we have it in the name of the group, we want to remind people that this is who we are, what we do, and now we have this culture in my work that I almost don’t have to, but I probably sit
Steve: right there, designers who are specifically focused on designing the accessible experiences.
Danielle: Now, every designer is responsible for it. So, we had, like I said, this me and my team helped to make sure that add new designers have gone through whatever training modules that are we have available designed for accessibility is rolled into our design language, and it is part of the core competency of every designer, the content team of the development team to just build it into everything we do. Can you talk a little bit about some of the ways that your team I know your team covers a lot of different functions, but how you work with other parts of the organization. So Express Scripts senior healthcare org it’s a little different than a lot of the technology companies I’ve worked with before the design team, lives in it. So there is one big group that handles all of the external and internal facing technologies, and that’s a little different from what I was used to, but it works out really well here we work very closely with our technical product owners and partnerships within the business.
Steve: How do you plan for what it is that people on your team are going to be working on.
Danielle: So we are aligned to what the technology or so, I’d say that once a year, we hire on our big buckets of work so we know we are going to work on the website, on the site of member experience for these of students who work on the website, we’re going to work on the mobile application, and we’re going to work on, maybe one or two other properties or different experiences. And so, once we’ve aligned on that across the product team, the design team, and the front end engineering team, then we each kind of go off to our own teams and look at how much work can we actually do. So, this is an ongoing area of maturity for us as a leader of people. It’s something I’ve had to learn to take different approaches to doing, and I noticed wasn’t really your question but I’m going to talk about anyway. Go for it. Easy research design content, kind of a big deal of design, we’re all in this field because we have this level of empathy for folks to believe we want to design things that make the world better. And what I admired as a leader of people it’s, who has to manage like resources and organizational requests and put together budgets plants and stuff, it’s hard to figure out how much work we actually can do when no one on the team wants to say no to anything. And, as a leader I’ve had to figure out different ways of training manager, it’s been very illuminating because I’m sure I was this person when I was like fresh out of school to, kind of, I can do it, I can do all this stuff, and then later I’ve my work quality suffers turns out I’m working on the weekends and at night. I might start to resent the work I’m doing, and if I just raised my hand and said hey I know I said I could do all these, all these things but turns out I can only do one of them, my life the good life. So, as a leader of people I try been working really hard closely with other leaders on my team, you know they’re people on the team to help understand how much work can be done in a given situation. To help evangelize a message of how much more impact we can do if we just focus on one or two things that we know we can achieve and say no politely, but start to learn to build the muscle of No, that’s been my own personal growth area as a leader, and as a individual I’m guilty of that too, like I said, but trying to plan the work and where to focus becomes easier as you learn as a UX professional to accept things that you can do, and be cognizant and aware of what you can’t do and communicate that to your, your leadership your peers, so that everyone knows what’s possible.
Steve: You reframe for me when you say become scribing No, as I can’t do it not know as I refuse to do it right, there’s a human limitation that we are always taking into account,
Danielle: right, there’s a human limitation to everything. And we know that our takes time. I mean, everyone’s work takes time, but if you’re going to do a good job. I mean, you know, as a consultant, it’s hard to tell a client No, but you’re not really telling them no or cheese you’re telling them. Well, I could do that, but it would suck.
If you want to do it well. It’s gonna take this long, or we can do, fewer things in a shorter amount of time, having that conversation is a skill that needs to be developed and it will make you much more successful in the long run.
Steve: Makes me think that there needs to always be a flat no but it can be an example you just model. You’re coming back with peers to trade off stickers to consequences. So you’re facilitating something.
Danielle: Yes, that’s my kind of, when I say my personal growth that that’s my, because then my growth is not just no good to say, Here’s why. I don’t think it’s a good idea, or I won’t be able to do these things here’s why. And this is what I mean time, right, this is how I think, engage other options to get this stuff, it’s not a plan now, and it’s not the Yeah, I’ll do it and I’m secretly working out.
Steve: When I asked you about accessibility you went all the way back to your first week on the job and what that was like and we’re talking a little more now about where things are at right now. I wonder, maybe just a stepping back question, if you could describe a little more of that arc, like what you came into how your job is evolved over the years that you’ve been there and up to date where we’re at now.
Danielle: When I came in. It was at the beginning of what we added at our technology transformation. And part of that was building a design team. So they were ugly at hand handful of folks, and I was brought in as a director to build a user research team. And that was a lot of executive conversations about the value of user research building an understanding about how we work. That’s a process where lots of quick and quick in your clips, does that mean it’s really quick, in healthcare, but doing some initial studies that demonstrate the utility of what we do, creating lots of new templates and getting feedback on presentation style so that we can communicate value and consistency and that didn’t take as long as I would have expected and it wasn’t as hard as I was expected. And by that I mean, I come from other organizations that had more or less established UX disciplines for some sort of experience if so, when I had been a consultant for a couple of years. So, what I was expecting coming in and like to have to do a lot of r&d, honestly to talk about how this is important and we needed it. I had to do very little that, and that was a real surprise. So lesson me it looks like it’s not able to super easy, it’s just that doing that work did not take up as much of my life as I thought it would. Because it was mostly an awareness problem, people just didn’t know that this existed, and once they heard about him can understand the work that we did, they became avid consumers. So it was a big we got a lot of fans. I did a couple of presentations to senior leadership, to this day, Express Scripts president we’re waiting in hallway because he back in those days it was a lot of conversations about what users are doing. And I also presented to clients a lot. So the way that our business works, like I mentioned the beginning is that we help with your prescriptions, if you have coverage with health insurance. So, our clients are really good health insurance plans are different businesses if they self fund their health insurance. And because this was a new function within the company, we wanted to make sure our clients understood that we were doing this and that, if I sent some of the survey, and they reached out to their health plan administrator, it wasn’t new news, so that was a different thing for me to have all these client conversations, and I still do that to an extent but now our sales team, kind of knows about what we do, so they can speak to it. I see pulled in a client conversations on a less frequent basis, but back then I was probably going to St. Louis, a couple times a month to show clients our disability lab and talk to them about what it is that we did for the first and probably two years after like the first couple of years, usability practice maturity, we had a couple of folks on my team to start to do analytics and data science lead and have taken over the NPS program to clean it up and systematize it, and now we have a winner I would say it’s healthy analytics practice where we can start to put things in place like a B testing and talking about how that is used in the organization and become consultants for different analysis questions. so we’ve gone from kind of not like having cabbie leaves little pieces of UX that there was a couple UX people bringing across the organization a couple of folks doing usability testing, they started building a website before I got hired, honestly, so they knew they wanted to do this they had executive buy in, but really, rolling it into product and having it not be optional check the box thing really started to happen over the last couple of years, and now let’s say almost gotten to be a management of demand like under this umbrella shield to keep my team from getting silex projects as we call it. So, and my job itself. I started off with a small team as a director. Now I’m the senior director some pretty well removed from doing research. Back then I think I managed a vendor on one project I was pretty well involved with a couple others I might have done a survey to myself, and now like I can’t, I joke all the time like I don’t know what you mean. I don’t know what it is the PowerPoint slides. I’ll barely do it. I just, I listen to what my team tells me, and I make a path for them that have impacted the organization. So my job has changed from building the competency, demonstrating that we have this competency and getting buy in to making sure that what we do the work that we do, is impactful, and I do that by creating barriers helping were a candidate, identify stakeholders and to fix some sort of weird problems in my own cover, to help in my team to understand how the business works, and vice versa. Okay, the business understand how different new parts of the business, understand how my team works and how they may or may not work with us.
Steve: What are sideways projects
Danielle: and sideways projects are, you are I some violence aged some big translate into like the lunch and learn or something, and someone in a different part of the business, let’s say, in our traditional IT department I say they they make dashboards for infrastructure monitoring, I make it so they heard of my team I heard it was their VR thing. They love this idea, they contact any person on my team, ask them if they can help them by running a usability test on their infrastructure monitoring thing, and that is a sideways project. I have to make sure that we don’t say yes to those kinds of things we don’t necessarily say no because they are under Resources they can point people to like I said like the sauce. But we don’t get involved with too deeply was just to preserve our family and focus, and it’s one of those tough ones. It’s not like things are unworthy. It’s just the human limitation, the team’s ability to do the work. If you’re doing that, then you’re not doing something else, because everybody is busy. If we do have spare cycles spare bandwidth, then we do consider those kinds of projects, but you shouldn’t be down.
Steve: So you’ve talked about the healthcare ecosystem, but I assume, when you call it an ecosystem that means you’re sort of outside. You’re not a care provider for example, but I’m wondering if you’re impacted by regulatory concerns as part of your role in that ecosystem.
Danielle: Yes, we are. And that was another thing that was surprised during my first week on the job. I have an academic background, doing user research, or research like this, and I’ve also been in industry for a good 1015 years. So, I understand, and expected that we would have MBAs that we need to put in place with our research participants, we would have to get informed consent, and we’d have to have certain ethical practices about letting people participate and back out and, you know, things that I come to expect is second nature. What I did not understand, was that healthcare is healthcare. There are quite a few laws that come into play, how you can talk about sensitive health information, and I had to get very friendly with all of our attorneys. And I will say all of them but quite a few. So my first week maybe, and that I got introduced to one of the attorneys that set in your mind. At the time, and he was like you did but let me invite people to come here and tell us what, and stuff like that they kicked out a series of meetings as me speaking with our attorney, are different, and they’re several different legal departments inside of a healthcare organization. I also learned, but he was feud with a few of them about what user research is and how we can use the data, and who we can and cannot talk to and what’s very important as a user researcher in healthcare, is that you are not soliciting information about somebody’s health condition, but it is the context of which they’re using our service. So we do have to could, and I learned this in this first couple of months, being on the job, and I now have to teach it to new researchers on the team put this person in place, but we do have to do very specific things to protect the data that I’ll share with us, because they are, as you can imagine, if you are in a usability session about using a prototype to refill a prescription, we have to make sure that none of that data is real, it’s a prototype, so you’re not pulling your health records to build this prototype, but as you get feedback he might start talking about what happened last time, and start talking about your specific prescription drugs, and my researchers have to redirect you. The reason why is because that is protected health information, and we have to be sensitive to that. And it’s just something I never really thought about coming from academia coming from industry, this idea that if we run like a common thing we do and I’m sure people still do, is in a gorilla hallway research where we just grabbed somebody, and it’s awesome sponsor of the project, give me unless you still a feedback or do an interview about how they might manage this certain situation at home. Well, you do that in healthcare, and you’re talking to an employee, and they might, as part of the usability evaluation or interview reveal some of their health conditions, and we have observers present. That might be a breach of their privacy. So, again, we have to be careful of who who observe sessions, who has access to recordings, and how we anonymize and research participant grids even when we send them out to observers we have to normalize, all of these things that I just didn’t even think of before I got you, and we I spent a lot of time with legal. Now the seas that broad term with legal for a couple of years to marginal boundary. So what we can and can’t do, so that we do, you know, he can do research more quickly, but in the beginning, it was kind of a curry let’s, let’s make sure we can recruit these people, let’s make sure that we are compensating them with fair market guidelines and not. And we have often documented the appropriate place because there’s also loss of power. It’s just an awareness of the regulatory environment we’re in how it impacts research that was such a surprise. Same thing applies for analytics, think, certain ways you cannot look at the data because it is not relevant to design, directly or because it uses your pH, your private health information.
Steve: Maybe we can switch gears slightly here. Great to hear you talk about, as you grow the team, what kinds of things you look for in user researchers,
Danielle: because I’ve grown the team in different ways. I look for a few different things, some things are the same. We do look for people that have some sort of demonstrated expertise, either in user research or analytics data science, you know, you can talk about your craft in cohesive coherent way and communication, again, is consistent, no matter where then where we do want to know that you can present data. One of those more intangible things that we look for and I’m can’t say that I have a perfect way of doing this is patience. One of the things that I think I’ve alluded to is that healthcare is we have a lot of rules to follow some things we cannot change very quickly, but we can work towards changing, and it may take a long time. And that can be challenging for some folks, it could be come from an environment that was very fast moving, or look at work worked at the placements where I was in charge of changing the UI so I could literally open a source file and change the words in the application, myself, and just publish it, and be done with it, and we have several rounds of review, to make sure that what we’re saying is legal for lack of a better term, summarize. So that’s something that I do look for is we don’t want to get folks in who just become really frustrated by how slow things move on the other side we also don’t want folks who are okay with slow things. So, it’s a delicate balance. We look for people who have the ability to kind of see the long view and make steps towards it, you know, push boundaries where you can but not just burn out. By doing that, other skills we look for. Honestly, it helps if you are methodologically agnostic from a user research perspective that means that you’re comfortable doing interviews, some ethnographic ethnographic methods and coursework, or something like that. Like you can do a few different methods may not be great at all of them that you don’t like I have met people who have specialized and we only do contextual interviews. That’s how we do. And this team would not be the place for someone like that. And the other thing that, again, that’s more difficult to select for guys some level of empowerment, were mentioned earlier, a little bit where you have the bounds of what you can do, and know what we’re trying to achieve and suggest studies or suggest data sets that may help with making decisions to get us there, and stead of waiting to be asked to do a thing, taking medicine, like being an order taker, like of course as collaboration where you sometimes get requests to do a study, but also having some initiative to be able to see something happening or see a problem, and suggest a way of getting more information to address
Steve: the problem, how someone demonstrates patience, but not too much patience, or empowerment.
Danielle: It’s all been in kind of conversations and interviews and just a description of your past work, and I know it’s very subjective and I, as I said I’m not really saying I’m great at it, and I love a better way, but we do try to ask questions about you know typical questions about like what was your involvement in projects. How did it start, how did it end. What happened with the findings and just listen to what people talk about. And that has helped a lot in knowing if you have a demonstrated history of taking initiative, or if you don’t. And if you don’t, it’s not necessarily the kind of disqualification, but we do try to ask questions to get out if you can, if you’re interested in taking initiative, and being empowered to do certain things. I have met people who are not interested. it’s like they do you really like getting requests for a specific thing and executing against it, and we try to get at in a conversation in different interview settings.
Steve: I want to go back, you talked about some of the kinds of places you’ve worked in, you talked about getting a PhD of wondering if you could talk about if there were some decision points. How did you decide to pursue your graduate degree, and when did you learn about user research.
Danielle: I’m sure my age, I wanted to be a doctor, and I was, and this is a bear with me. So when I was a kid I want to be a doctor, so I went to high school for health professions and he said I went down that path. I started down that path at a very early age, and when I was in high school, I met a psychologist, and it was a difficult time actually one of my classmates committed suicide. And so, the school district brought in, psychologists, and at that time I’ve been in hospitals doing rotations. As a teenager, so take that for what it means and what that is, but I saw this interaction with this, psychologists, much more interesting and helpful there when I was exposed to other other practices. So I shifted focus to becoming a psychologist in high school. That sounds like I have it all figured out. But anyway, so I did that in high school, and in undergrad, I worked as a clinical psychologists research assistant, and I didn’t like about clinical psychologists psychology was not very interesting, and even though they had interesting problems to set that experience was not, it was kind of boring I can’t think of a better way to say it, so I was in my kind of a research methods lab class, and the TA in that class asked if I want to be a research assistant for an applied cognitive psychology professor, and I said yes and started down the path of doing research on eyewitness testimony, and that was super interesting and Smit to two and a half years, my last two, two and a half years of my degree, doing that. My last year, to my senior year of college, I finally got to ask the grad students that I was working with how much cognitive psychologists make, and this was in the late 90s, and he told me like becoming like $35,000, as a professor, and as kind of freaked out because my mom did not do all this like work all these jobs, it could be through college and do this for me to come out and make about how much money she did so, like, I think it might have taken me a week to go back to my professor and tell him I was probably going to have to stop doing my research assistantship because I’m switching my major back to pre med and I was just going to become a doctor because I just needed to make more money than that. And he’s like, Whoa, let me set up some meetings with you with other parts of other professors in the department to tell you about other branches of psychology that are more profitable than becoming the professor of applied cog, and I was like, Okay, well you got a couple of weeks until I have to go to my counselor and change formation. And that’s how I found out about human factors and industrial organizational psych, and I also like I sent a man with a couple of professors to tell him about these other branches, I took a couple of courses, and apply for IO programs and human computer interaction and human factors for a fence for grad school, and I ended up going to rice that had combined IO Human Factors department, and really loved it. I took a couple classes in undergrad, like I said, and they didn’t really have a human factors class I just had read about it, so it’s kind of on the fence about IO psych but Bryce had a combined department, so it’s like okay like he figured out grad school, and took my first human factors class and it’s like, oh, this is my jam. This is my thing. Technology applied to humans, helping design technology to be more suitable to humans. Yes, super interesting I feel like I can help people and, and be able to pay my student loans. So, that’s how I ended up in the field or sticking with the field. And when I was in grad school I did a couple of internships, the one that really sealed the deal for me was, I worked in the habitability and Human Factors department at Johnson Space Center in Houston, as a contractor with Lockheed, but work in human factors department at NASA, Mike helped with disability evaluations of physical print instructions, websites, Space Station habitat layouts and all kinds of different stuff and I did a little bit of design too. So that was the thing that made that convinced me that I made the right choice because it was so interesting, and I could nerd out with all these people, like other key factors folks, and it was like meaningful work, so kind of ethic I ended up leaving NASA because I was taking too long graduate, and so I had to kind of finish my dissertation graduate early, and then around that time was like it was 2006 2007, and that’s when you started seeing kind of user experience jobs show up as call that versus Human Factors engineers or usability engineers and I ended up taking my kind of first post graduation job at Dell, actually I think I was a usability engineer for a while there. My job title change later to be a UX person. That’s how I ended up cheaper. So series of, kind of lucky events, and unlucky events but it’s not something I knew about early on in undergrad, but I do get a sense have a degree in physics.
Steve: What was the topic of your dissertation.
Danielle: Oh man, warning label design. So I was a risk perception and safety person, and my dissertation was about how to design warning labels for over the counter directives so how do you warn somebody to do something that they really really want to do you want to read it.
Steve: I mean, so from this period that you’re describing the different programs and the different, different disciplines that you are discovering and maybe these internships, do you feel like any events, maybe not in the literal sense but you feel like any of that is present in the work that you’re doing now.
Danielle: Definitely. I feel like be wrapped with my experience in terms of different methods, I had a lot of exposure early on to very complicated statistics took a lot of stats classes, my dissertation, it was a mix of ethnography interviews, and lots of regressions, so where I am now is a joining, all those things, so I familiar with offline experiences and familiar with online experiences, since the very extreme in our events in very, they need to design interactions that are flexible for different scenarios and different physical abilities, those are some things that I’ve learned at NASA, that come into play now. My experience doing like usability testing different styles of interviews and accelerate construction like all that stuff that I did in undergrad, our stuff we did at NASA at PayPal, like all the different environments I’ve been exposed to all come into play here, where I’m trying to help different types of researchers, understand work in an agile environment, even though it might not feel so natural. Like, understanding those principles and how we can consistently work to get better is all relevant. And one thing that I kind of consistently kick myself for is dropping the academic chair seconds, are the things that are learned in school that are still relevant today that I really dropped and try to distance myself from and early on in my career. Most cars for funerals, because there is this idea that, Oh, you don’t want to be to epidemic, you don’t want for folks who think that research takes too much time or you need to do all this math, so just drop all that, and I can I can take myself for that because I didn’t have to relearn it in this role. We learned a lot of the stats that I dropped and kind of the research design principles that I use a lot earlier, like a NASA and in grad school, because now we have so much quantitative data at our disposal, and a lot of us do come from a background like I do, where you did learn how to do a lot of statistics and think about the problem from a measurement perspective in terms of like chunking up a problem like okay this part should be qualitative. These are parts of the answer quantitatively these parts, we probably can ignore for a while until we get these first two chunks down, and that level of thinking has been invaluable in managing a team like this where we have to chunk up problems we have to try to answer things as much as he can, and it’s a useful skill but I do wish I could have kept can protect it more. Early in my career, now I get to bring effectiveness in really, really valuable.
Steve: I mean maybe just building a little a little bit new if you look at the profession, maybe it’s your own organization or maybe it’s just the collective profession of user research. Are there things that you would hope to see terms of evolution or growth in the practice of user research, maybe over the next few years.
Danielle: Yes, I do feel like there’s a lot of things that I would like to see, and I hadn’t really thought about this before so answers are not in any sort of priority or top of mind. I know that everything is Software as a Service, but there are some of us that world that’s highly regulated. So, for me tools and methods perspective, I really miss Moorea miss being able to run for having my team be able to run usability session to record things local people to have, like, analysis and work using local tools. I know it’s a crazy thing to kind of tap into a desire to see as a profession, we need a wide variety of tools and I feel us gravitating towards the same solutions that don’t really work very well for some of the hairier. The human existence, and I am going to go ahead and say that healthcare. If you think about the things that we have to design, we have their parts of our lives just people trying it, and trying to figure out your healthcare situation or trying to navigate some aspects of the government. It’s just really hard. And so those are opportunities for a user experience professional to be different, and happy. But another area, I’d like to see. And I believe my other other guests mentioned this too is a maturation and methods. We don’t see a lot of time on coming up with different approaches, and I don’t see a lot of meta analysis, and that term is to be loaded to but before you look at a breath, or even qualitative usability as we really like to see us advanced ways that we’ve looked at across time and across multiple iterations to pull out different insights and patterns that we should be looking into another is on the other side. I’ve really like to see a more user focus when it comes to data science, and this is not an express script statement but just in general, you see a lot of automation in place like bots, to help. Whatever you want, like, fill in the blank you think you’re on your I am in with a person and it’s about for you go to IBM and think you’re going to talk to a person. It’s hot, but like lots of applications of data science, and like user facing automation, we put in place to solve a problem. These are habits have been put in place to solve the problem business habits. So, in terms of user research it’s I feel like we can add a tremendous amount of value in that space to help bring that kind of work along in terms of solving real problems and real Tasks tab, and to do that you need to learn some of that. I will say, learn the methods that that world. Learn how it works at a high enough level to speak intelligently to user centered design in that context.
Steve: Those are great. Is there anything that we should have talked about today that we didn’t get to want to make sure we cover.
Danielle: I think we covered it but I’m away for starting to talk about the broad umbrella, that the experience research, the way that my, my team is made up way that I think is part of our ethical responsibilities, do you think that is come across in that conversation compensation but like I said earlier, we really want to make sure that serious. It’s not just happening to people that they are defined with their voices, considered as part of a solution, and that it’s continually refined. And I do feel like in some scenarios user experience professionals can be laser focused on a new for a few strains within the UI, this, that you. What does this mean in context of your users reality, and I know that can be kind of big and overblown, but you can have an approach that chunks better into reasonable solvable problems that you can then work with other folks in the organization to
Steve: I really like how you brought us full circle from where we started at the beginning to where we ended up. Thank you for the conversation today and for being on the podcast It’s been really excellent to get to speak to you.
Danielle: Oh, great! And I just realized that husband decided to go chop some wood while we were talking. But, okay. I think we’re having barbecue this weekend.
Steve: Great, well, thanks again!
Danielle: Thanks, Steve.
The post 35. Danielle Smith of Express Scripts first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Danielle Smith, the Senior Director of Experience Research & Accessibility at Express Scripts. Something that I’ve really changed the way I thought about since I’ve been at Express Scripts – we are in the healthcare ecosystem. So the experiences we deliver, if they are not […]
The post 35. Danielle Smith of Express Scripts first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Danielle Smith, the Senior Director of Experience Research & Accessibility at Express Scripts.
Something that I’ve really changed the way I thought about since I’ve been at Express Scripts — we are in the healthcare ecosystem. So the experiences we deliver, if they are not of quality, they do have serious repercussions on people’s lives and people’s time. We are ethically bound to measure the user experience from different perspectives. Before something launches. We have prototypes or concepts or ideas, we do our due diligence in terms of user experience research, to make sure that the thing that we’re putting out on the world doesn’t just happen to people. – Danielle Smith
Show Links* Artwork hung in our living room * Danielle on LinkedIn * Danielle on Twitter * Express Scripts * Net Promoter Score (NPS) * DeBakey High School for Health Professions * Industrial/Organizational Psychology at Rice * Human Factors and HCI at Rice * Human Health Factors at Johnson Space Center
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Many years after we moved into our house, we finally hung up our art. Sure, we had hung up individual pieces, something that was already in a frame, say. But it was always piecemeal, a nail here, a hook there. And we continued to accumulate meaningful pieces from travels or family events. And we continued to occasionally pull out the box (a moving box, made for a mirror, I believe) and spread out all the various bits and pieces and just generally fantasize about having them up.
My goal, however, was to have a plan, an intentional way of placing the different posters, prints, photos throughout our home. Every time we would try that, I would get overwhelmed and give up. I tried taking small bites: after seeing homes with big frames and medium photographs, we chose a few photographs from our travels, blew them up, ordered specially cut mat boards, and frames. We mapped out where in our living room these would go; we essentially carved off part of the home and planned the photographs that would go there. I hoped that this would simplify the challenge of where to put the remaining posters, but we found ourselves stuck, still.
Eventually, we opened up that box and made some hard decisions about what to hang and what to set aside, and then – before too much time could go by – arranged to get everything framed. We were inching closer, but sitting with our posters and prints, all framed, we still couldn’t figure out what to hang where and so (this shouldn’t be surprising) I got overwhelmed and gave up.
But buried in that frustration and surrender was a recognition of what skills I’m missing – an ability to reorganize visual information spatially in a few different ways, a set of starting principles for grouping, placing, and so on. Surely someone must have this expertise and be offering it as a service?
It turns out that, yes, there are professional picture-hangers. We called one and booked an appointment. A few weeks later, on the scheduled day, at the expected time, the doorbell rang. We answered the door and before we could finish greeting them, two men were in our foyer, one of them having magically unfolded and gently placed a table (please assume it transformed from a flat briefcase shape to a table with a soft whoop noise).
The guy with the table handled production. The other one handled creative (and the clients). We showed him all of our framed pieces and talked a very little about what they meant, and we showed him around our house, pointing out areas we were interested in and the few pieces we had already hung. Meanwhile he was riffing constantly, throwing out ideas, getting energized, and delegating to the production guy, who began attaching extra mounting hooks and hanging wire to all of our store bought frames. After a short time, we backed off, and we watched them work.
The “creative” began moving frames into different rooms, laying them on the floor and trying different combinations. Like a problem-solving algorithm, the solution began to appear, bit by bit. The floors throughout our house began to fill with clusters and arrangements of different prints, both thematic and visual groupings. We were called in for frequent consultations as the plan emerged. Eventually there was a plan for where everything was going to go. This was the piece we could not have done ourselves and in a short time, they had done it.
Then came the rest of the production. They began hanging up everything. That meant figuring out where each item went – exactly, putting in nails in exactly the right spot, using a level, all the details. Especially because so many pictures were clustered, something being slightly off would really show, so perfect execution was key. This was also something we could have not pulled off ourselves.
The final results were astonishing. Rather than hanging things in a grid, with the top edges aligned and a consistent space between each, they put together a number of clusters where the posters emanated from a central point in an almost-spiral flow. And they chose how to place different prints within that cluster in order to create a kinetic sense, such as having a poster with a bird along the left hand side, with the bird facing to the right, so that the content of the images supported the physical placement on the wall. This was not something we could have even imagined, let alone executed.
It is immensely gratifying to be in the presence of a highly-skilled individual. When those skills are being deployed for your benefit, it adds another layer of delight. I believe that delight is further enhanced when we ourselves have tried and failed. This story is a reminder to me to seek out these magically-talented individuals and take advantage of what they have to offer, whenever I can.
And it’s my goal to be a magically talented individual for the people I work with, someone who brings in a level of skill that isn’t achievable without my help. I strive to conduct user research with that kind of finesse, I hope that when I coach and train teams in doing research, I’m helping them see what that magical level of skill looks like and move them forward on their own path towards achieving that.
I would love to hear from you about what you are working on and how my expertise can support you in moving your effort forward. Please keep me in mind.
Now let’s go to my interview with Danielle Smith. She’s the Senior Director of Experience Research & Accessibility at Express Scripts.
Steve: Well, Danielle, welcome to Dollars to Donuts. Thanks so much for being on the podcast.
Danielle Smith: Thanks for having me.
Steve: Why don’t we start with an introduction from you to say a little bit about what you do where you work.
Danielle: My name is Danielle Smith. I am Senior Director of experience research and accessibility at Express script. And Express Scripts is actually Cigna company. But we are a pharmacy benefit provider. So we help with getting your prescription drugs. If you have insurance coverage, we are the companies that help with that. And my team measures the quality of the user experience from a few different perspectives. One includes user experience research. The other is our NPS program for our digitally engaged customers, and our data science team and some digital metric reporting, oh resides with my team.
Steve: So I want to ask about your team. But I want to just back up slightly, who are the kinds of people that are having this digital experience with your company.
Danielle: So our digital experience is primarily geared towards people who have prescription drug coverage through their health insurance provider or directly if you’re on Medicare Part D, but you are interested in getting home delivery of your maintenance medications. Our website does have kind of general information. If you do have a health plan, you want to go in there and look up a medicine see how much it costs which pharmacy But most of the functionality on our digital tools for consumers really look at the home delivery of medicine,
Steve: and then use the phrase to kind of describe what your team does. It’s about measuring quality. Is that right?
Danielle: Yeah,
that’s sort of the way I think of it.
Steve: Can you pick apart that phrase a little bit? It’s an interesting one. And I’m curious how you think about
Danielle: something that I’ve really thought about or changed the way I thought about since I’ve been at Express Scripts, we are in the healthcare ecosystem. So the experiences we deliver if they are not, if they are poor, or if they are not of quality, they do have serious repercussions on people’s lives some people’s time. So when I think about my team, we are almost ethically bound to measure the user experience from different perspectives. Before something launches. We have prototypes or concepts or ideas, we do our due diligence in terms of user experience research, to make sure that the thing that we’re putting out on the world doesn’t just happen to people. We have some ideas It’s usability, its appropriateness. And we’ve really tried to test a test and air quotes, do research on everything before it hits the site. And some cases, of course, we can’t do that. And I’ll talk about that in a second. But we’ve tried to do, I would say far above 80%, of what hits the site goes through some level of user research related to that. And another aspect of what we do to ensure quality is to make sure it’s accessible. And most of our accessibility work has really geared towards visual impairments or different visual abilities, because we are the digital team. So my team serves as subject matter experts to help our developers aren’t marketers, and to deliver experiences that are usable by people that use screen readers or have low vision just right out of the gate. So we don’t have a separate experience for folks who may be having any sort of different visual abilities. And that was a big point of focus for the first few years of my role here was to just really get that going. But it’s part of our user experience, practice and it’s in our The research team because we do specifically do studies, blind and low vision users throughout the year, to make sure even though it’s something’s technically compliant, it is actually usable by folks that use a screen reader specifically. But following good design practice, and rolling in those things before you launch still does not guarantee good experiences. And so the other parts of my team work to measure how what we build interacts with reality, and to see if it does actually generate a good experience and has, you know, high quality like we want. So the NPS program I know NPS itself is a problematic metric and has its taters in the industry. And that’s fine. I’m not here to defend it. But what it does is gives it lets us speak in a voice that executives and understand and it gives us the leeway to have an open channel with our users. So we send out these monthly surveys for one part of the program. But another part of the program allows users to leave feedback directly in our mobile app on our website. Tell us what’s going on. So yeah, that gave us a score, report the score, executive lift score. But the words that they use when they tell us if we’re on the mark or off the mark have been priceless and making sure that things actually work as intended given reality. And then the other piece is behavioral analytics. So we run a B tests, we instrument the site and make sure that things are working the way we think they should work in terms of people flowing through different funnels or parts of the site and monitor the experience that way. So we work really closely with our product teams to help them understand their metrics, make sure they’re gathering metrics and help them use them and interpret them in the right ways. So when you use the word quality, what does that mean for you? So for me, I’m using quality to be synonymous with a good user experience and healthcare so complicated. I’m not going to be so arrogant as to say we’re going to delight people. I just want them to be able to get their task accomplished which is getting their drugs prescriptions delivered. or checking the status of their delivery. I want them to get their tasks accomplished with ease. And that’s what I mean by quality. Are we not getting in their way? Do we make them feel like they’re part of the process? And do they understand what’s going on? Because it’s complicated enough.
Steve: You also use the phrase, I hope I get this right. working as intended given reality.
Danielle: Oh, yeah. Reality when you’re talking about healthcare is something that we cannot ignore. And we can’t we can’t even formulate all of the scenarios a person might be in at least you can’t do that yet. And people’s realities may be like right now like, Okay, well, now my doctor’s office is closed, and I need a refill. How do I do that? It’s a pandemic, what’s going on. So we have to be making sure that we are listening to user feedback to make sure we update the website, and even communications and talk to our friends in the call center to make sure that we are ready to respond to our users. reality, smaller, like everyday situations happen. As you can imagine, if you’re sick, there are some conditions that give you kind of a transient low vision situation. So you use the website yesterday, I, my team wants to make sure you can use website tomorrow. Just because you’re on this medication or you have this condition should not take away your ability to use certain tools. We also have lots of families, family situations are fluid who has access to your account today, you might not want them to have access to your account tomorrow. And this is your health information. So we have to be always ready to listen and react when people’s contexts change, or their actual reality is way more complicated than we even thought of when we were designing the user interface.
Steve: Can you say a little more now about the word measure is kind of the key verb that you use to describe kind of overall what you all are doing.
Danielle: Sure, and I’m glad you picked up on that because I don’t want to give people the wrong impression that Don’t do discovery or you know the call, but I am of the belief that the work that we do is still measurement. We may not have a type metric, we might not have a number around it, but it is still the collecting of data to get at some underlying construct as best as we can. And I was recently reminded that I came from a grad program that was more quiet leaning. And it’s probably why I did grow my team, the way that it grows more, including analytics, as well as user research and having them live side by side. So it’s not just about things we can put a number one it is about understanding scenarios and understanding people and listening to how we can resonate with our users and more. So I don’t want to miss categorize what we do, but I think of measure in the broad sense of the word.
Steve: I mean, it’s a really powerful frame for what research does and there are lots of different frames for you know, if you have to say one word, what is it that we do and measure is One that I have heard all that often, but it’s very compelling, especially when you kind of explain, hey, there’s lots of different kinds of measurements. And even when you’re talking before about NPS, and part of its value as a way to open up a conversation with other people in the organization, when you say measure overall, I imagine that that is similar that by framing this around measurement, you are positioning yourself relative to your stakeholders and colleagues that this is the kind of guidance and information that you can bring.
Danielle: Yeah, and I talked about it that way I talked about what my team does is x data sounds very superhero like that. It’s like it’s data about the user experience. And just because I say data does not mean it’s all quantitative, and it’s taken a while for our partners and even some folks on my team to get on board with this idea. But we need to be able to have some sort of convergence on the user experience. And convergent validity is a goal of most research teams that I’ve been exposed to where you want to have some analytics show problems and user research give you the other side of it, maybe some survey data to give you some aspect of scale of some of those opinions. And by us all being on the same team, it gives us that ability to be fluid in our methods and speak to the business in a way that allows them to hear us. And sometimes that way is to use NPS or a web analytic metric to get our foot in the door and expand how they view the users by layering on the qualitative and sometimes it’s vice versa. We have different stakeholders that are more interested in in qualitative data and then we overlay some quants to help them understand scale and focus and or if it’s even able to get to that level of maturity given where we are in the process.
Steve: Accessibility is part of the group and for me it’s new to hear accessibility specifically called out, alongside other different roles. How did accessibility end up being part of what you’re focused on.
Danielle: Oh man, it’s hilarious he was just like a line in my job description analysis like that. Okay, and I thought what that meant was, this is literally my first week on the job I thought that would be similar to my accessibility role in other parts of my other jobs I’ve had before, which was its usability person I need to be aware accessibility exists, and I enrolled in some tests in my normal course of evaluation that help the accessibility team, check their boxes, about compliance. However, I did not do my due diligence about looking at the expressor sweb site to understand what that might in my job description actually meant. Just say like at previous jobs at Dell and larger companies have specific hardware interface requirements that come from the accessibility team that kind of get handed to the usability team that we just make sure as we go through development, make sure that they get rolled in, but what I did not notice and Express Scripts with that. Back when I joined the company in 2016. They had two different websites, there was a link on the top of the site that said accessible view. And it took you to a text only view of the site, and during my first week on the job My boss is explaining this to me and how, nobody really wanted that to be the strategy and the teacher. I was like, Oh, so we need to learn how to build accessible websites, she’s like yeah, like, oh, okay, well, let me figure out how to do that. So it was a whole different a different perspective on the same problem, but it gave me that opportunity for us to say that we’re going to build a website that works for folks of different abilities, without having a separate site that was personally offensive to me as a Black person to have something separate for people who are different, really annoyed me. So, I wanted to make that go away. And having more inclusive web experience. So that became part of our world, because at the time Express skip splits really open to having a much more modern digital experience, we came up with a plan to help spread that awareness spread that skillset across the organization, one person on my team really picked up the mantle and dove into accessibility, best practices, compliance. She’s not a lawyer but building relationships across the organization to have that compliance and a network, and then diving into doing research with totally different abilities like specifically people they use screen readers because we wanted to make sure that not only did it kind of pass all the checks, but that it, like I said, was usable, and bring that video and bring those usability results into the organization, so that they can understand what it meant to be inclusive in that way. And that just really fit with the way we do user research, because the whole point. A lot of the point of user research is to bring the voice of the user outside of our walls into our walls to help people inside the company, understand the operating environment of people outside of the company. And by having a focus on accessibility within user research allows us to apply those same principles same practices that folks were kind of used to, but with a different audience, and it just really worked well. And so, I can’t remember the date but it was at least two years ago, we sunset it that outside, we have one experience, and we have a group of folks within the organization that are really passionate about this developers help each other out. Under normal situations we would actually run usability testing on site at the American Council of the Blind. To get more engagement from the community, it actually turned into a natural extension of the usability team meeting over it’s showing my age but user experience team, because, as designers, as researchers, we are responsible for inclusivity and accessibility is one part of that, and in health care, especially cannot, an all good consciousness exclude that important of a population. And so we do have focus on that and we have it in the name of the group, we want to remind people that this is who we are, what we do, and now we have this culture in my work that I almost don’t have to, but I probably sit
Steve: right there, designers who are specifically focused on designing the accessible experiences.
Danielle: Now, every designer is responsible for it. So, we had, like I said, this me and my team helped to make sure that add new designers have gone through whatever training modules that are we have available designed for accessibility is rolled into our design language, and it is part of the core competency of every designer, the content team of the development team to just build it into everything we do. Can you talk a little bit about some of the ways that your team I know your team covers a lot of different functions, but how you work with other parts of the organization. So Express Scripts senior healthcare org it’s a little different than a lot of the technology companies I’ve worked with before the design team, lives in it. So there is one big group that handles all of the external and internal facing technologies, and that’s a little different from what I was used to, but it works out really well here we work very closely with our technical product owners and partnerships within the business.
Steve: How do you plan for what it is that people on your team are going to be working on.
Danielle: So we are aligned to what the technology or so, I’d say that once a year, we hire on our big buckets of work so we know we are going to work on the website, on the site of member experience for these of students who work on the website, we’re going to work on the mobile application, and we’re going to work on, maybe one or two other properties or different experiences. And so, once we’ve aligned on that across the product team, the design team, and the front end engineering team, then we each kind of go off to our own teams and look at how much work can we actually do. So, this is an ongoing area of maturity for us as a leader of people. It’s something I’ve had to learn to take different approaches to doing, and I noticed wasn’t really your question but I’m going to talk about anyway. Go for it. Easy research design content, kind of a big deal of design, we’re all in this field because we have this level of empathy for folks to believe we want to design things that make the world better. And what I admired as a leader of people it’s, who has to manage like resources and organizational requests and put together budgets plants and stuff, it’s hard to figure out how much work we actually can do when no one on the team wants to say no to anything. And, as a leader I’ve had to figure out different ways of training manager, it’s been very illuminating because I’m sure I was this person when I was like fresh out of school to, kind of, I can do it, I can do all this stuff, and then later I’ve my work quality suffers turns out I’m working on the weekends and at night. I might start to resent the work I’m doing, and if I just raised my hand and said hey I know I said I could do all these, all these things but turns out I can only do one of them, my life the good life. So, as a leader of people I try been working really hard closely with other leaders on my team, you know they’re people on the team to help understand how much work can be done in a given situation. To help evangelize a message of how much more impact we can do if we just focus on one or two things that we know we can achieve and say no politely, but start to learn to build the muscle of No, that’s been my own personal growth area as a leader, and as a individual I’m guilty of that too, like I said, but trying to plan the work and where to focus becomes easier as you learn as a UX professional to accept things that you can do, and be cognizant and aware of what you can’t do and communicate that to your, your leadership your peers, so that everyone knows what’s possible.
Steve: You reframe for me when you say become scribing No, as I can’t do it not know as I refuse to do it right, there’s a human limitation that we are always taking into account,
Danielle: right, there’s a human limitation to everything. And we know that our takes time. I mean, everyone’s work takes time, but if you’re going to do a good job. I mean, you know, as a consultant, it’s hard to tell a client No, but you’re not really telling them no or cheese you’re telling them. Well, I could do that, but it would suck.
If you want to do it well. It’s gonna take this long, or we can do, fewer things in a shorter amount of time, having that conversation is a skill that needs to be developed and it will make you much more successful in the long run.
Steve: Makes me think that there needs to always be a flat no but it can be an example you just model. You’re coming back with peers to trade off stickers to consequences. So you’re facilitating something.
Danielle: Yes, that’s my kind of, when I say my personal growth that that’s my, because then my growth is not just no good to say, Here’s why. I don’t think it’s a good idea, or I won’t be able to do these things here’s why. And this is what I mean time, right, this is how I think, engage other options to get this stuff, it’s not a plan now, and it’s not the Yeah, I’ll do it and I’m secretly working out.
Steve: When I asked you about accessibility you went all the way back to your first week on the job and what that was like and we’re talking a little more now about where things are at right now. I wonder, maybe just a stepping back question, if you could describe a little more of that arc, like what you came into how your job is evolved over the years that you’ve been there and up to date where we’re at now.
Danielle: When I came in. It was at the beginning of what we added at our technology transformation. And part of that was building a design team. So they were ugly at hand handful of folks, and I was brought in as a director to build a user research team. And that was a lot of executive conversations about the value of user research building an understanding about how we work. That’s a process where lots of quick and quick in your clips, does that mean it’s really quick, in healthcare, but doing some initial studies that demonstrate the utility of what we do, creating lots of new templates and getting feedback on presentation style so that we can communicate value and consistency and that didn’t take as long as I would have expected and it wasn’t as hard as I was expected. And by that I mean, I come from other organizations that had more or less established UX disciplines for some sort of experience if so, when I had been a consultant for a couple of years. So, what I was expecting coming in and like to have to do a lot of r&d, honestly to talk about how this is important and we needed it. I had to do very little that, and that was a real surprise. So lesson me it looks like it’s not able to super easy, it’s just that doing that work did not take up as much of my life as I thought it would. Because it was mostly an awareness problem, people just didn’t know that this existed, and once they heard about him can understand the work that we did, they became avid consumers. So it was a big we got a lot of fans. I did a couple of presentations to senior leadership, to this day, Express Scripts president we’re waiting in hallway because he back in those days it was a lot of conversations about what users are doing. And I also presented to clients a lot. So the way that our business works, like I mentioned the beginning is that we help with your prescriptions, if you have coverage with health insurance. So, our clients are really good health insurance plans are different businesses if they self fund their health insurance. And because this was a new function within the company, we wanted to make sure our clients understood that we were doing this and that, if I sent some of the survey, and they reached out to their health plan administrator, it wasn’t new news, so that was a different thing for me to have all these client conversations, and I still do that to an extent but now our sales team, kind of knows about what we do, so they can speak to it. I see pulled in a client conversations on a less frequent basis, but back then I was probably going to St. Louis, a couple times a month to show clients our disability lab and talk to them about what it is that we did for the first and probably two years after like the first couple of years, usability practice maturity, we had a couple of folks on my team to start to do analytics and data science lead and have taken over the NPS program to clean it up and systematize it, and now we have a winner I would say it’s healthy analytics practice where we can start to put things in place like a B testing and talking about how that is used in the organization and become consultants for different analysis questions. so we’ve gone from kind of not like having cabbie leaves little pieces of UX that there was a couple UX people bringing across the organization a couple of folks doing usability testing, they started building a website before I got hired, honestly, so they knew they wanted to do this they had executive buy in, but really, rolling it into product and having it not be optional check the box thing really started to happen over the last couple of years, and now let’s say almost gotten to be a management of demand like under this umbrella shield to keep my team from getting silex projects as we call it. So, and my job itself. I started off with a small team as a director. Now I’m the senior director some pretty well removed from doing research. Back then I think I managed a vendor on one project I was pretty well involved with a couple others I might have done a survey to myself, and now like I can’t, I joke all the time like I don’t know what you mean. I don’t know what it is the PowerPoint slides. I’ll barely do it. I just, I listen to what my team tells me, and I make a path for them that have impacted the organization. So my job has changed from building the competency, demonstrating that we have this competency and getting buy in to making sure that what we do the work that we do, is impactful, and I do that by creating barriers helping were a candidate, identify stakeholders and to fix some sort of weird problems in my own cover, to help in my team to understand how the business works, and vice versa. Okay, the business understand how different new parts of the business, understand how my team works and how they may or may not work with us.
Steve: What are sideways projects
Danielle: and sideways projects are, you are I some violence aged some big translate into like the lunch and learn or something, and someone in a different part of the business, let’s say, in our traditional IT department I say they they make dashboards for infrastructure monitoring, I make it so they heard of my team I heard it was their VR thing. They love this idea, they contact any person on my team, ask them if they can help them by running a usability test on their infrastructure monitoring thing, and that is a sideways project. I have to make sure that we don’t say yes to those kinds of things we don’t necessarily say no because they are under Resources they can point people to like I said like the sauce. But we don’t get involved with too deeply was just to preserve our family and focus, and it’s one of those tough ones. It’s not like things are unworthy. It’s just the human limitation, the team’s ability to do the work. If you’re doing that, then you’re not doing something else, because everybody is busy. If we do have spare cycles spare bandwidth, then we do consider those kinds of projects, but you shouldn’t be down.
Steve: So you’ve talked about the healthcare ecosystem, but I assume, when you call it an ecosystem that means you’re sort of outside. You’re not a care provider for example, but I’m wondering if you’re impacted by regulatory concerns as part of your role in that ecosystem.
Danielle: Yes, we are. And that was another thing that was surprised during my first week on the job. I have an academic background, doing user research, or research like this, and I’ve also been in industry for a good 1015 years. So, I understand, and expected that we would have MBAs that we need to put in place with our research participants, we would have to get informed consent, and we’d have to have certain ethical practices about letting people participate and back out and, you know, things that I come to expect is second nature. What I did not understand, was that healthcare is healthcare. There are quite a few laws that come into play, how you can talk about sensitive health information, and I had to get very friendly with all of our attorneys. And I will say all of them but quite a few. So my first week maybe, and that I got introduced to one of the attorneys that set in your mind. At the time, and he was like you did but let me invite people to come here and tell us what, and stuff like that they kicked out a series of meetings as me speaking with our attorney, are different, and they’re several different legal departments inside of a healthcare organization. I also learned, but he was feud with a few of them about what user research is and how we can use the data, and who we can and cannot talk to and what’s very important as a user researcher in healthcare, is that you are not soliciting information about somebody’s health condition, but it is the context of which they’re using our service. So we do have to could, and I learned this in this first couple of months, being on the job, and I now have to teach it to new researchers on the team put this person in place, but we do have to do very specific things to protect the data that I’ll share with us, because they are, as you can imagine, if you are in a usability session about using a prototype to refill a prescription, we have to make sure that none of that data is real, it’s a prototype, so you’re not pulling your health records to build this prototype, but as you get feedback he might start talking about what happened last time, and start talking about your specific prescription drugs, and my researchers have to redirect you. The reason why is because that is protected health information, and we have to be sensitive to that. And it’s just something I never really thought about coming from academia coming from industry, this idea that if we run like a common thing we do and I’m sure people still do, is in a gorilla hallway research where we just grabbed somebody, and it’s awesome sponsor of the project, give me unless you still a feedback or do an interview about how they might manage this certain situation at home. Well, you do that in healthcare, and you’re talking to an employee, and they might, as part of the usability evaluation or interview reveal some of their health conditions, and we have observers present. That might be a breach of their privacy. So, again, we have to be careful of who who observe sessions, who has access to recordings, and how we anonymize and research participant grids even when we send them out to observers we have to normalize, all of these things that I just didn’t even think of before I got you, and we I spent a lot of time with legal. Now the seas that broad term with legal for a couple of years to marginal boundary. So what we can and can’t do, so that we do, you know, he can do research more quickly, but in the beginning, it was kind of a curry let’s, let’s make sure we can recruit these people, let’s make sure that we are compensating them with fair market guidelines and not. And we have often documented the appropriate place because there’s also loss of power. It’s just an awareness of the regulatory environment we’re in how it impacts research that was such a surprise. Same thing applies for analytics, think, certain ways you cannot look at the data because it is not relevant to design, directly or because it uses your pH, your private health information.
Steve: Maybe we can switch gears slightly here. Great to hear you talk about, as you grow the team, what kinds of things you look for in user researchers,
Danielle: because I’ve grown the team in different ways. I look for a few different things, some things are the same. We do look for people that have some sort of demonstrated expertise, either in user research or analytics data science, you know, you can talk about your craft in cohesive coherent way and communication, again, is consistent, no matter where then where we do want to know that you can present data. One of those more intangible things that we look for and I’m can’t say that I have a perfect way of doing this is patience. One of the things that I think I’ve alluded to is that healthcare is we have a lot of rules to follow some things we cannot change very quickly, but we can work towards changing, and it may take a long time. And that can be challenging for some folks, it could be come from an environment that was very fast moving, or look at work worked at the placements where I was in charge of changing the UI so I could literally open a source file and change the words in the application, myself, and just publish it, and be done with it, and we have several rounds of review, to make sure that what we’re saying is legal for lack of a better term, summarize. So that’s something that I do look for is we don’t want to get folks in who just become really frustrated by how slow things move on the other side we also don’t want folks who are okay with slow things. So, it’s a delicate balance. We look for people who have the ability to kind of see the long view and make steps towards it, you know, push boundaries where you can but not just burn out. By doing that, other skills we look for. Honestly, it helps if you are methodologically agnostic from a user research perspective that means that you’re comfortable doing interviews, some ethnographic ethnographic methods and coursework, or something like that. Like you can do a few different methods may not be great at all of them that you don’t like I have met people who have specialized and we only do contextual interviews. That’s how we do. And this team would not be the place for someone like that. And the other thing that, again, that’s more difficult to select for guys some level of empowerment, were mentioned earlier, a little bit where you have the bounds of what you can do, and know what we’re trying to achieve and suggest studies or suggest data sets that may help with making decisions to get us there, and stead of waiting to be asked to do a thing, taking medicine, like being an order taker, like of course as collaboration where you sometimes get requests to do a study, but also having some initiative to be able to see something happening or see a problem, and suggest a way of getting more information to address
Steve: the problem, how someone demonstrates patience, but not too much patience, or empowerment.
Danielle: It’s all been in kind of conversations and interviews and just a description of your past work, and I know it’s very subjective and I, as I said I’m not really saying I’m great at it, and I love a better way, but we do try to ask questions about you know typical questions about like what was your involvement in projects. How did it start, how did it end. What happened with the findings and just listen to what people talk about. And that has helped a lot in knowing if you have a demonstrated history of taking initiative, or if you don’t. And if you don’t, it’s not necessarily the kind of disqualification, but we do try to ask questions to get out if you can, if you’re interested in taking initiative, and being empowered to do certain things. I have met people who are not interested. it’s like they do you really like getting requests for a specific thing and executing against it, and we try to get at in a conversation in different interview settings.
Steve: I want to go back, you talked about some of the kinds of places you’ve worked in, you talked about getting a PhD of wondering if you could talk about if there were some decision points. How did you decide to pursue your graduate degree, and when did you learn about user research.
Danielle: I’m sure my age, I wanted to be a doctor, and I was, and this is a bear with me. So when I was a kid I want to be a doctor, so I went to high school for health professions and he said I went down that path. I started down that path at a very early age, and when I was in high school, I met a psychologist, and it was a difficult time actually one of my classmates committed suicide. And so, the school district brought in, psychologists, and at that time I’ve been in hospitals doing rotations. As a teenager, so take that for what it means and what that is, but I saw this interaction with this, psychologists, much more interesting and helpful there when I was exposed to other other practices. So I shifted focus to becoming a psychologist in high school. That sounds like I have it all figured out. But anyway, so I did that in high school, and in undergrad, I worked as a clinical psychologists research assistant, and I didn’t like about clinical psychologists psychology was not very interesting, and even though they had interesting problems to set that experience was not, it was kind of boring I can’t think of a better way to say it, so I was in my kind of a research methods lab class, and the TA in that class asked if I want to be a research assistant for an applied cognitive psychology professor, and I said yes and started down the path of doing research on eyewitness testimony, and that was super interesting and Smit to two and a half years, my last two, two and a half years of my degree, doing that. My last year, to my senior year of college, I finally got to ask the grad students that I was working with how much cognitive psychologists make, and this was in the late 90s, and he told me like becoming like $35,000, as a professor, and as kind of freaked out because my mom did not do all this like work all these jobs, it could be through college and do this for me to come out and make about how much money she did so, like, I think it might have taken me a week to go back to my professor and tell him I was probably going to have to stop doing my research assistantship because I’m switching my major back to pre med and I was just going to become a doctor because I just needed to make more money than that. And he’s like, Whoa, let me set up some meetings with you with other parts of other professors in the department to tell you about other branches of psychology that are more profitable than becoming the professor of applied cog, and I was like, Okay, well you got a couple of weeks until I have to go to my counselor and change formation. And that’s how I found out about human factors and industrial organizational psych, and I also like I sent a man with a couple of professors to tell him about these other branches, I took a couple of courses, and apply for IO programs and human computer interaction and human factors for a fence for grad school, and I ended up going to rice that had combined IO Human Factors department, and really loved it. I took a couple classes in undergrad, like I said, and they didn’t really have a human factors class I just had read about it, so it’s kind of on the fence about IO psych but Bryce had a combined department, so it’s like okay like he figured out grad school, and took my first human factors class and it’s like, oh, this is my jam. This is my thing. Technology applied to humans, helping design technology to be more suitable to humans. Yes, super interesting I feel like I can help people and, and be able to pay my student loans. So, that’s how I ended up in the field or sticking with the field. And when I was in grad school I did a couple of internships, the one that really sealed the deal for me was, I worked in the habitability and Human Factors department at Johnson Space Center in Houston, as a contractor with Lockheed, but work in human factors department at NASA, Mike helped with disability evaluations of physical print instructions, websites, Space Station habitat layouts and all kinds of different stuff and I did a little bit of design too. So that was the thing that made that convinced me that I made the right choice because it was so interesting, and I could nerd out with all these people, like other key factors folks, and it was like meaningful work, so kind of ethic I ended up leaving NASA because I was taking too long graduate, and so I had to kind of finish my dissertation graduate early, and then around that time was like it was 2006 2007, and that’s when you started seeing kind of user experience jobs show up as call that versus Human Factors engineers or usability engineers and I ended up taking my kind of first post graduation job at Dell, actually I think I was a usability engineer for a while there. My job title change later to be a UX person. That’s how I ended up cheaper. So series of, kind of lucky events, and unlucky events but it’s not something I knew about early on in undergrad, but I do get a sense have a degree in physics.
Steve: What was the topic of your dissertation.
Danielle: Oh man, warning label design. So I was a risk perception and safety person, and my dissertation was about how to design warning labels for over the counter directives so how do you warn somebody to do something that they really really want to do you want to read it.
Steve: I mean, so from this period that you’re describing the different programs and the different, different disciplines that you are discovering and maybe these internships, do you feel like any events, maybe not in the literal sense but you feel like any of that is present in the work that you’re doing now.
Danielle: Definitely. I feel like be wrapped with my experience in terms of different methods, I had a lot of exposure early on to very complicated statistics took a lot of stats classes, my dissertation, it was a mix of ethnography interviews, and lots of regressions, so where I am now is a joining, all those things, so I familiar with offline experiences and familiar with online experiences, since the very extreme in our events in very, they need to design interactions that are flexible for different scenarios and different physical abilities, those are some things that I’ve learned at NASA, that come into play now. My experience doing like usability testing different styles of interviews and accelerate construction like all that stuff that I did in undergrad, our stuff we did at NASA at PayPal, like all the different environments I’ve been exposed to all come into play here, where I’m trying to help different types of researchers, understand work in an agile environment, even though it might not feel so natural. Like, understanding those principles and how we can consistently work to get better is all relevant. And one thing that I kind of consistently kick myself for is dropping the academic chair seconds, are the things that are learned in school that are still relevant today that I really dropped and try to distance myself from and early on in my career. Most cars for funerals, because there is this idea that, Oh, you don’t want to be to epidemic, you don’t want for folks who think that research takes too much time or you need to do all this math, so just drop all that, and I can I can take myself for that because I didn’t have to relearn it in this role. We learned a lot of the stats that I dropped and kind of the research design principles that I use a lot earlier, like a NASA and in grad school, because now we have so much quantitative data at our disposal, and a lot of us do come from a background like I do, where you did learn how to do a lot of statistics and think about the problem from a measurement perspective in terms of like chunking up a problem like okay this part should be qualitative. These are parts of the answer quantitatively these parts, we probably can ignore for a while until we get these first two chunks down, and that level of thinking has been invaluable in managing a team like this where we have to chunk up problems we have to try to answer things as much as he can, and it’s a useful skill but I do wish I could have kept can protect it more. Early in my career, now I get to bring effectiveness in really, really valuable.
Steve: I mean maybe just building a little a little bit new if you look at the profession, maybe it’s your own organization or maybe it’s just the collective profession of user research. Are there things that you would hope to see terms of evolution or growth in the practice of user research, maybe over the next few years.
Danielle: Yes, I do feel like there’s a lot of things that I would like to see, and I hadn’t really thought about this before so answers are not in any sort of priority or top of mind. I know that everything is Software as a Service, but there are some of us that world that’s highly regulated. So, for me tools and methods perspective, I really miss Moorea miss being able to run for having my team be able to run usability session to record things local people to have, like, analysis and work using local tools. I know it’s a crazy thing to kind of tap into a desire to see as a profession, we need a wide variety of tools and I feel us gravitating towards the same solutions that don’t really work very well for some of the hairier. The human existence, and I am going to go ahead and say that healthcare. If you think about the things that we have to design, we have their parts of our lives just people trying it, and trying to figure out your healthcare situation or trying to navigate some aspects of the government. It’s just really hard. And so those are opportunities for a user experience professional to be different, and happy. But another area, I’d like to see. And I believe my other other guests mentioned this too is a maturation and methods. We don’t see a lot of time on coming up with different approaches, and I don’t see a lot of meta analysis, and that term is to be loaded to but before you look at a breath, or even qualitative usability as we really like to see us advanced ways that we’ve looked at across time and across multiple iterations to pull out different insights and patterns that we should be looking into another is on the other side. I’ve really like to see a more user focus when it comes to data science, and this is not an express script statement but just in general, you see a lot of automation in place like bots, to help. Whatever you want, like, fill in the blank you think you’re on your I am in with a person and it’s about for you go to IBM and think you’re going to talk to a person. It’s hot, but like lots of applications of data science, and like user facing automation, we put in place to solve a problem. These are habits have been put in place to solve the problem business habits. So, in terms of user research it’s I feel like we can add a tremendous amount of value in that space to help bring that kind of work along in terms of solving real problems and real Tasks tab, and to do that you need to learn some of that. I will say, learn the methods that that world. Learn how it works at a high enough level to speak intelligently to user centered design in that context.
Steve: Those are great. Is there anything that we should have talked about today that we didn’t get to want to make sure we cover.
Danielle: I think we covered it but I’m away for starting to talk about the broad umbrella, that the experience research, the way that my, my team is made up way that I think is part of our ethical responsibilities, do you think that is come across in that conversation compensation but like I said earlier, we really want to make sure that serious. It’s not just happening to people that they are defined with their voices, considered as part of a solution, and that it’s continually refined. And I do feel like in some scenarios user experience professionals can be laser focused on a new for a few strains within the UI, this, that you. What does this mean in context of your users reality, and I know that can be kind of big and overblown, but you can have an approach that chunks better into reasonable solvable problems that you can then work with other folks in the organization to
Steve: I really like how you brought us full circle from where we started at the beginning to where we ended up. Thank you for the conversation today and for being on the podcast It’s been really excellent to get to speak to you.
Danielle: Oh, great! And I just realized that husband decided to go chop some wood while we were talking. But, okay. I think we’re having barbecue this weekend.
Steve: Great, well, thanks again!
Danielle: Thanks, Steve.
The post 35. Danielle Smith of Express Scripts first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Amber Lindholm, the Head of User Research at Duo Security.
That’s the sign of a really good researcher – it can never be just about research for research’s sake, like this is a cool project, this is a neat thing, I really wanna go in-depth and understand perceptions of XYZ with these people, if you don’t have that ability to understand the organizational and business contexts and the types of decisions that are having to be made every day by the rest of the folks in your organization, your research isn’t going to have an impact. – Amber Lindholm
Show Links* John Mulaney Has Become the Muse of TikTok’s Makeup Artists * Amber on LinkedIn * Amber on Twitter * Duo Security * University of Illinois Graphic Design * Bauhaus education * Institute of Design * frog * projekt202 * Atlassian * Stride * Sally Carson, Head of Product Design * Mark Thompson-Kolar, Design Researcher * Annie Diu, Research Coordinator * Liz Donovan, Design Research Manager * Dug Song, Founder * Jon Oberheide, Founder * Chester Kustarz, Head of Engineering
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I read recently about a new genre of TikTok videos that featured people applying makeup, while lip syncing to standup comedy routines by John Mulaney. I believe that TikTok has it roots as a platform for lip-sync performances, and of course makeup tutorials and demonstrations are their own thing on the Internet. But how do we end up with the combination, and not just one but a whole series? How did this come about, why are people doing it, and are there other niche sub-genres or patterns that this relates to? Even if we are tempted to dismiss these behaviors as just a bunch of people being weird, we need to be doing the research to understand how, and why this is happening.
Sure, people at TikTok should know how people are using their service. But insight about this also is valuable to other platforms like YouTube and Twitter and Instagram. How could this information give you a new perspective on user behaviors if you work at Dropbox? Or if you work for Michelin, either their travel department or their tire division? How could Nationwide Insurance make use of this?
Our culture swerves and leaps and when these emergent behaviors poke their head up through into the mainstream, it’s an invitation to take note, and to be curious.
This is what clients hire me to do, whether it’s to be the one that leads the investigation, to unlock the motivations and desires of current and prospective customers, or to be someone who helps their team as they themselves dig into the hidden behaviors and objectives of users. I help companies look at their own organization, their culture, and their processes to help set the stage for this kind of discovery work to flourish and have impact.
If you’re a fan of this podcast, then remember you can support the podcast by supporting me and my practice. Please reach out to me at Portigal dot com and let’s find a way to work together.
Okay it’s time for my interview with Amber Lindholm, she’s the head of Design Research at Duo Security.
Steve: Well Amber, thank you for being on dollars to donuts. It’s really great to get to speak with you.
Amber Lindholm: Hey, Steve, it’s great to be here.
Steve: Hey, I guess I should have said, hey, hey, Amber. So I’m gonna ask you to introduce yourself.
Amber: So I’m Amber Lindholm. I’m currently the head of design research at Duo Security, which is part of Cisco. And my background is in design, both kind of traditional print design, moving into interaction design, and then design research, which I practiced before, you know, moving into more leadership positions.
Steve: What does do a security to
Amber: do security, we provide various security products that help protect our customers basically access to data. So our core product is a multi factor authentication product. So when you’re going to, let’s say log into, you know, a tool that you use at work, you know, after you do your password, you get some sort of maybe a push notification or something that verifies that To you, so we help protect organizations and their data as well as individuals.
Steve: So if I am a consumer and I go to my bank and they said, oh, we’re going to send you a text message to prove that you’re you before we can let you access your account, is that the kind of tools that you’re providing?
Amber: Yeah, that’s, that’s a great example a lot of folks are familiar with, with us through like financial institutions. And we basically have created a product that’s super easy for organizations to roll out and get folks enrolled in. But it does that where it’s it’s verifying through a second method that you are actually who you say you are,
Steve: eventually, let’s get to talk more about the work that you are doing to a security but since you said a little bit about some of what your background was it be great to just hear more about like you mentioned, print design. So what’s kind of the arc or the history for you from when you got into figuring out your profession to you know, how you move to where you are now.
Amber: So I started out as I was mentioning in graphic design, so I was trying I went to University of Illinois we did really kind of what I would call more classical, like Bauhaus, you know, educational style, learning all about formal typography and form and color. And so after school, I followed that path, I found a work with a PR agency at first and then in house at a nonprofit called Rotary International. And this was up in Chicago. And I spent my time you know, designing brochures, you know, reports, invitations, billboards, all kinds of print materials. And during that time, I just remember there was this particular project I was working on where we translated most of our materials into nine languages. So it was going out around the world. And I was creating these, they were kind of these little packets that had a CD that had files for the different rotary clubs to produce their own kind of marketing materials. And I worked with the translators to you know, translate the copy in the on the files and the file names, but after that, I Didn’t know when those things were sent out in the world if they were going to be used properly if they were going to meet their needs or expectations, and at that time, I started searching around and trying to think, you know, I feel like something’s missing. And I came across, you know, the terminology then was Human Centered Design. And it was just a revelation. For me, it seems like, you know, you can create these things, but if you don’t understand how people are going to use them and what they need, you’re really gonna miss the mark. So that was a catalyst for me to continue like my education. So I found a grad program at Institute of Design in Chicago where I went and I learned, you know, how to do research. They taught all about human factors, all kinds of approaches pulled from social sciences, very qualitative in nature, learned about design strategy, I learned more about interaction design, and that was really I really felt like I found my sweet spot. I loved the research side of it. I loved going out and talking to people. And trying to understand you know who they are and how they behave and why. So when I graduated actually, right before I graduated, I did a couple of things. So one of the things was I wanted to do a little bit of living abroad. So I went for a year to New Zealand and I worked inside the government there so did a project looking, it was more like a service design customer experience project, which was really great, because it neat to get that government experience came back and my husband and I moved to Austin, just sight unseen moved down here. And I’ve had a couple of different jobs here in Austin. I started out at frog and I was still mostly focused on design. I was doing interaction design, like mobile, tablet type design, but then I started adding in the research projects, and that again, really flourished there and love doing that work. Got to do some research in China and other places. And this was more contextual work like going in people’s homes, spending a lot of time with them to understand their world and A couple other things I did after that I went and worked in healthcare for a year trying to do similar kind of research understanding. For instance, I did a project about the new parent experience. And then I went back to consulting to a consultancy called Project 202. So spent, you know, quite a few years actually being more of a consultant role working across many industries, which was fun and started leading a team that project to research and Insights Team. So we would work on, you know, tons of projects, looking at really large customer journeys. And then after consulting for so long, I decided that I wanted to really understand the other side, so to go more in house to work at a product company. So I worked at Atlassian and led a design and research team for a product called stripe, which is a video and chat tool. And then I moved to do where I’m at now. And again, I lead the research team there love helping our team grow love the side of research that really has an impact and provides value. So that’s kind of my story
Steve: to talk about today. There’s two sort of vectors that I hear one is in house and consulting as different perspectives. But you’ve also talked about research and design. And it sounds like over the course of your career, your title at least has. Well, I guess you also said some of the work you’re doing might lean more heavily towards one or the other. It’s hard to ask the question in an open ended way as I want to. So I’ll just try the leading version. We think of research and design, like they’re different words. We call them different things. Sometimes they’re different kind of career paths. But one of the threads of your narrative that is compelling to me is that you’ve kind of worked with both those pieces of research and design. I guess I’m wondering when you think about for yourself at least and where you’re contributing where you’re adding value or how you put all these pieces together? What’s the relationship between the kind of the R word research and the D word design in your makeup and your skill set.
Amber: I think they really complement each other. I think it makes me as a designer, it made me a stronger designer to have that you know, research mind and as a researcher, it made me stronger because I can Really make sure that the research I was doing was framed around and presented in a way that provided information relevant to the folks having to make decisions. So I always think about that. It’s like when we’re trying to figure out what to research and what to prioritize. There’s always a whole set of questions. And there are decisions that are having to be made that some are more ambiguous questions where you really don’t have a lot of information. Some are really high risk questions. So you want to make sure you’re understanding those and providing that information because people are having to make big decisions and tactical decisions and thinking about how you frame that research and present it back to folks or include them in that process is really important. So I think understanding how designers work and also you know, now knowing how product companies work, understanding the questions that product managers are trying to answer engineers are trying to answer and helping them actually articulate those questions is something that I use strive to do. So I think they just really complement each other.
Steve: The way you describe how you want to make sure that the research you do connects and supports, I hear an echo of your brochure project where you were, you wanted that context to make sure that you knew that the work you’re doing was gonna have the outcomes and have the results. It sounds very similar, that kind of questioning of, Hey, who’s doing what and how do we make sure that we help them do what they’re going to do that that applies to your quote users as well as your quote, stakeholders or colleagues?
Amber: Yeah, absolutely. We spend a lot of time like when I’m working with the researchers on my team, and we’re looking at a new project, or we’re looking across the projects that we could support over quarter whatever it is, we’re looking exactly at that like where can we have the most impact? Where’s the greatest need? Sometimes we’re looking at is there a new project spinning up Is there a new product manager coming in is there you know, a team that’s just starting out that’s never gone through this process. before. We really like to prioritize those kind of things as well so that we help get them off on the right foot so that they do frame those questions really well, we help show them a good process to follow. And those are really, really fruitful projects.
Steve: What’s the split between proactive you identifying what research you want to do versus something that’s more reactive, where there’s requests coming in? Hey, can you help us do this?
Amber: That’s interesting to try to assess out. I think it’s so it’s a little bit hard, because so at duo we have, we’re part of the product design team. There’s about 30. Folks on that team, research team. I have four researchers on a team and a research coordinator who helps Do you know all of our recruiting and scheduling and all kinds of stuff. And so none of the researchers are dedicated to a particular product because duo does have multiple products. We kind of stay as a little bit of a hub, but we start to embed like I was saying, Okay, let’s kind of embed over here on this team. Let’s embed over this on this team for this particular project or this time span. And so the researchers are communicating with the designers, I’m communicating with the design managers and product managers to learn, you know, what their roadmap looks like or what kind of questions they have coming up. So I end up really driving the prioritization of the work we’re doing based on all those conversations, looking at the priorities of the business and you know, kind of larger trajectory of where the business is going and make the decisions of like, where we can actually prioritize our time and where we’re going to have the most impact. But we do also get requests. So we work with folks outside of the kind of r&d process as well. We’ve been working with our creative team on website design. We’ve been working with technical communications folks in our like knowledge community, so those people will come to us with requests. We hold office hours every Monday, so sometimes, we’ll just provide some support or advice or kind of review things and other times we’ll decide to take on a full project.
Steve: Can you maybe reward And a little bit and talk about how you came to do and what was research like there maybe what was the context that you that was established when you joined and what some of the progress and evolution has been since you’ve been leading the team?
Amber: So do I guess the design team was formed about five years ago. And so Sally Carson’s, our head of design, so she was hired to come in and, you know, build up this team. And really early on, she wanted to make sure that research was part of it. So one of the maybe the second or third hire was a researcher, and His name’s Mark Thompson Kohler. He’s still with the team. And he came in and at that point, because design was so new and research with some new, the focus was, you know, on really rapid, like usability testing, right. So like, every two weeks, we’re going to do testing. We’ve got people, you know, what do you want to test and so the company was a startup, it was a lot smaller. And the idea was just sort of this like rapid, more evaluative type research. And that went on for a while. And then The team, Mark developed a set of personas that got embedded across the entire dual organization. You know, as a team continue to grow, we’re still kind of like a one person research shop, do the way the design team has grown though designers are hired that also have an interest and skillset and research. So designers also do research. When I came in, another researcher had been hired. So there were two researchers, and there wasn’t what you would call like a research team yet. So when I came in, we decided, Okay, let’s sit down and really talk about you know, who we are, what’s our mission? What’s our value? What are the strengths that we have going for us and the gaps and how do we want to move this team forward? And we did you know, some kind of off site exercises and that kind of thing. And we started at basics. So there was a ton of knowledge. There was a ton of things had been done, and we’re like, let’s make sure like we have it right. So that first quarter was a lot of poetry. together in a place where people can find all of the research that’s been done, document some of the tools and templates we have so that other people, you know, on the team can can use this stuff. And then we wanted to make sure that instead of just focusing on the evaluative work, you know, we were allowing the designers to really pick up a lot of that and that we could move into more foundational kind of generative type research projects. And so that’s what we did after that first quarter, getting all those foundations in place doing some additional trainings with the design team, the researchers, then were able to focus on these more larger, more strategic type research projects. And then we added in research Ops, we hired a research coordinator to help us so that the researchers weren’t spending you know, half their time coordinating, recruiting, maintaining our database of participants and all that good stuff. So she came in last summer. Her name’s Annie fan, she’s been just a fantastic value to the whole team. And then I hired two more researchers this spring, and now everyone, you know, sort of has their own area that They’re working on right now. And it’s been great. It’s just been fantastic.
Steve: you’re describing some of that, I think is a pattern and a lot of organizations that give the designers the tools to do some of the evaluative work and you know, a shift in what the research team itself specializes in. What’s been the outcome for the designers or the design work that they’re doing as you’ve empowered them to do that evaluative work?
Amber: I think it’s great. I mean, I think rather than, you know, us being like, No, we can’t help you and know that, you know, those questions aren’t going to get answered, the designers do have the toolset available. And we, you know, depending on somebody’s comfort level, or you know how much they’ve done this in the past will provide more or less support, right. So, like I said, it’s kind of like reviewing the plans, reviewing the script, doing test runs, maybe we’ll moderate the session, and then they pick it up after that, or some folks, you know, they’ve done this many times, and they can run more fully, but they are the experts in sort of this area that they’re working on. And they have the belief With their product manager and their engineers, and they can take all of that, you know, directly back into the work that they’re doing. So I think it’s I think it’s great, you know, I always have some some hesitancy if it’s the subjectivity that might come out of somebody who’s testing, you know, their own work. So we do kind of talk about that, keep an eye out for that. But so far like, it’s it’s been great and allowed our researchers to, like I said, focus on not just like those larger research projects, but a researcher on my team was Donovan right now is helping kind of consult across six or seven teams for this larger project. And so she has really good line of sight around the different questions being asked and helping people, you know, make sure that they’re all kind of sharing the insights they’re learning across those teams. So it’s allowed us to really level up our scope of influence.
Steve: That’s a great explanation and I feel like there’s an implicit value judgments around what kind of research is cool and fun and worth our time and then We can offload some of these tasks to the lower value tasks to other people, then we can work on the cool stuff. Not that anyone has ever said that. But it feels like that’s lurking underneath this move to kind of, you know, empower people to do the value of work. But I think what I’m hearing from you is, in a lot of ways, those people are better positioned to do that work, because they’re very close to the problem. And they can kind of zoom in on how they can get the most actionable results, and do do so maybe more quickly. And that research, I think your point about having influence. So the work that you’re doing is is more around the organization and more strategic. And that’s kind of what you’re positioned to do.
Amber: Yeah. So besides, you know, helping bring those designers together, Liz is helping to align pm as well are kind of like share across the hims. And she’s running a set of like metrics workshops across those teams like a created as a standard sort of workshop to help us to find UX metrics look like For signals and define those metrics, and so she’s bringing those to each of those teams. So now again, like it’s a more kind of strategic view across those, but then we’re still able to weigh in on and influence the the type of research, the type of questions and even the synthesis of the research that’s happening across those teams.
Steve: What are some other things that you’re doing to whether it’s around influence or you know, kind of bringing the customers into the organization?
Amber: When we set out on our kind of mission and the things that we do, you know, obviously sort of like executing and running this kind of product focused or foundational focus research is really important. But sort of the other half of it that we wanted to make clear about was like our role and helping to get folks closer to the customer to bring that context to them, to keep the customer Top of Mind, and I do frankly, it’s pretty easy because the organization across the board is extremely customer centric. We’re not like begging for people to listen to us. The value of research is really understood. It’s, it’s asked for actually, like, people are really interested in hearing more about the context RPMs. You know, they’re fantastic. They out there out there, they do tons of customer calls, but they’re still, as we’ve continued to grow. The risk that we identified a couple of things like early on, I mentioned that there was a set of personas that were created and these get referred to all the time across the board in big meetings, you know, big call hands and all of that. Salespeople reference them, our customer success, people reference them. But as people join the company, as we were kind of in hyper growth mode, the real deep understanding of those personas may not have persisted. Right. So we wanted to take a step back and say, you know, how do we, as a research team, help other folks have the type of context that we have. So Mark, who developed the personas, we have a program set up where, you know, as new hires come in, he presents something called personas one on one to make sure that those are really top of mind and understood and we do a deep dive session on one of the more key personas. He also does something called these fireside chats. So we just chat with different folks. So for instance, different IT administrators or help desk managers or helped us folks or CSOs, we’ll set up you know, an hour long chat to just learn about their day. And that’s open invitation for folks to learn. Listen and on. And then we also with the whole of our research sessions and things, they’re open for folks to come and listen to. So it’s very much accessible. We have a research Condor, everyone can see and they can pop in on different things. And lots of folks take advantage of that. And then I think another thing that we wanted to do was make sure that we had created additional sort of artifacts and frameworks for people to think about when they’re making decisions. So we took all this data that had been collected over across, you know, about four years of tests and interviews, and ran these large workshops, and Mark, you know, synthesize all that information into a set of design principles that are persona specific. So they’re not these like really super high level principles that are hard to make decisions with. They’re very applicable to making, you know, decisions on UI on making decisions on copy you might be creating. And so those principles are referenced by You know, designers, they’re used in critique sessions. They’re referenced by product managers as they’re talking about their roadmaps, folks like technical communications and things like that. So the personas and the principles, they’re not just an artifact that just kind of sits on the table, we bring them to the forefront and a lot of different ways.
Steve: Could you genericized one of the principles just to give a sense of the granularity of it?
Amber: Yeah, so for our IT administrators, for instance, one of the principles is around the ability to take action. And so anytime we’re making a decision about something in our UI or a flow, if there’s a dead end, that that person cannot actually, let’s say they’re seeing some kind of thing that may alert them to a security problem. If they cannot immediately take action on that item, then we’re not meeting that principle. It’s very important for folks to be able to move from information that they’re seeing directly into action. And then like for our end users, one of our principles is reduce friction to the extreme. So we’re not in the kind of product where, you know, you’re trying to create a sticky situation where someone’s in your product, you’re trying to actually minimize their interaction with your product, because what they’re actually trying to do is they’re, you know, get into their work application. So we think a lot about that, like, how do we reduce friction to that extreme?
Steve: So those are specific, as you said, and meant to be actionable. Yeah. And how many principles in a project like this, how many principles should one be trying to come up with?
Amber: What a question Steve? No, I think we narrowed it down to five principles for our two main personas and that I think is a good number like three is too few eights, too many kind of a thing. But five has been like the perfect number. And when we first rolled these out, we we did some fun things to where we created some really neat posters. We put out in all the offices, you know, we socialize them with people. We brought them around to meetings, we personified the principles even, which was really fun. But yeah, I think you got to, you can’t have too many.
Steve: How do you personify a principle?
Amber: So one of the designers on our team name, Andrea actually came up with this idea to, you know, get designers more familiar with all of these. And so each of the principles, one of us was given, and you had to basically just act like that principle. So we had to create a character. And in this sort of mock critique, we had to just care about that particular principle. So it’s actually ended up to be quite fun to do.
Steve: Sounds like a good improv game was very fun. Just going back to one of the points you made was that this is a an organizational culture that believes in research and she said asks for research. Do you have any perspective on for duo how that came to be?
Amber: I imagine it’s a couple of different factors. I mean, I think that the folks, Doug and Johnno, who co founded the company, that was like a big thing. Like they really cared about the end user experience. They cared about creating a company that was different than the other security companies out there at the time. Their whole mission is about democratizing security. So making it easy and effective was there from the beginning. And it wasn’t just, you know, something they were saying was something they really believed in, they saw how complicated you know, so many security products were and they wanted to make sure that when these products are out there, and it’s, you know, an IT administrator, it’s not somebody with, you know, a cybersecurity background or anything that these more generalist folks can protect their organization. So I do think a lot of it came from the founders and then the folks that they hired. So like our head of engineering, Chester, who we product design actually reports up through engineering. He’s completely supportive of design. You know, we hire engineers that have that kind of mindset also have like customer experience. So I think it just kind of permeates through the culture, because as you said, starts at the beginning and starts at the top. I think so. I mean, that’s that’s just my impression. I mean, I wasn’t there at the beginning. But I see that legacy. And I see that through the people that were hired on that continued to be in leadership.
Steve: I have so many interactions in so many different venues where somebody who maybe is at an individual contributor level sees that their culture is not like the one that you’re describing. And they’re being blocked in a lot of different ways. And the question is often, like, well, what can I do to succeed this way when these things are kind of arrayed against me? I don’t know if you’ve encountered that or heard that question. I personally, I find it challenging to give them super actionable advice when I feel like you’re describing you know, hey, it starts at the top that says how this company was founded. You know, they’ve grown and hired and supportive that and if you’re in a situation Well, that hasn’t happened. It’s an interesting challenge for people that aren’t embedded in those kinds of organizations to make those kinds of changes and advocate for that.
Amber: Yeah, I think especially like, to your point, if you’re not at a leadership level where you have a, if your scope of influence is just with your immediate project team, it’s gonna be really hard to sort of like, level that up through the organization.
Steve: You’ve named some of the people on your team and the kinds of contributions they’ve made. And I’m just wondering how you think about, but I think you said, you just you hire two people not that long ago to join the team. Yeah, yeah. Can you describe a little bit about your approach to finding people what that looks like in terms of saying, Hey, this is someone that would fit with our organization?
Amber: We have a probably what’s it fairly standard interview process, but I’ll just describe that and then sort of what kind of questions I’m asking and looking for, but I typically, you know, reach out to folks I have like an initial conversation with them. Just to understand what they’re looking for in an organization, or what their kind of career goals are just to make sure it aligns with, you know, the kind of goal we have open, then we do a longer interview. And we use, again, pretty standard, like behavioral interview questions to ask people to describe real scenario. So behavioral interviewing, it’s kind of like doing user research or like doing design research, where you’re really trying to get somebody to give you very specific real examples like to tell a story, because you want to understand what the situation was, how they thought about it, and how they reacted to it versus like a generalized answer. So we follow that I put together like a set of questions. And then once folks kind of go through that, they go into like a two hour we do a bit of a portfolio review. And then a final round interview is more like a four hour thing. And we have designers in there. We have PMS come in, sometimes engineering managers, so different folks who they might you know, interact with. than their day to day, participate in that more long kind of final interview. But some of the things that I’m always looking out for like, obviously, you know that they can do the work, but the way that they present the work is really important. So, you know, coming from a consulting background, I am design background, I do a lot of emphasis on how people share their work and how clear it is and how compelling it is. How do they tie that story together? How do they tell high level insights, but get into enough details to really support and paint the picture of why that insight matters? So that’s a big part of it. And then the behavioral interview questions I’m always looking at, can someone learn from a situation like can they I don’t want to say like, admit they’re wrong, but if they encountered something where it went wrong, how did they go about learning, growing and kind of improving for the next time? I’m also looking at people that are willing to ask questions, so I really love hearing the questions, I always leave a lot of time for folks to ask me questions. So do they have really good questions? Were the kind of topics they’re thinking about? Are they people who are going to come into an organization and be really curious and willing to ask people questions and not just kind of sit and be scared if they don’t know the answer, trying to think what else? We ask questions, you know, around working in diverse teams that’s really important to us, and just how they kind of collaborate and work with others.
Steve: Yeah, I want to ask about one of the things you mentioned, telling a story about an insight and you know, help it come to life. And you’ve worked in consulting. So you’ve experienced some version of this, I think, I mean, I, in some ways, I think I’ve grappled with this for much of my career, that there are things that happen over the course of a project that are galvanizing that just change everything, but to tell somebody else outside that organizations, that’s a harder story to tell. So I’m wondering, what are the best practices for that? How do I make a story about an experience I had In the past come to life for an audience that didn’t have the assumptions, the biases, the worries, the you know, all the sort of things that make it so impactful. How do I help somebody else understand that part of the insight?
Amber: Well, so what you just said is exactly what I would expect is that, you know, folks that know how to tell a story upfront, they give me enough context about the organization or the problem, or the assumptions that were there in the beginning, and what kind of impact their research had, and setting the stage of that context. It’s not just about the project, right, exactly what you’re saying, like, every single organization is unique. Every single set of people has different assumptions about things, the challenges that someone might be up against within that organization are going to be different. So a really good researcher is going to help you understand that context of why the thing that they’re telling you matters. I think that’s the sign of a really good researcher is that it can never just be about, you know, research for research sake like this is a cool project, this is a neat thing I really want to, you know, go in depth and understand perceptions of XYZ with these people. If you don’t have that ability to understand the organizational and business context and the types of decisions that are having to be made every day by the rest of the folks in your organization, your research isn’t going to have an impact. And so I really listen for that. And interviews are people able to help me understand how they made an impact and how they understand that larger ecosystem that they’re working in?
Steve: So back to the archetype, I was describing the person that approaches me and I think all of us with here’s why I’m unable to have impact. Here’s how my culture says doesn’t support me. If that person comes in to speak to your team and tell stories about the research they’ve done. How do they highlight the impact of their research when there’s forces arrayed against them to have that impact?
Amber: Yeah, I mean, I think that is a really challenging and I have had interviews where folks have revealed some of those more negative challenges, I would say like, it can sometimes feel like a person puts up all the barriers and just those like I couldn’t really do anything versus somebody saying like, this was the situation. This is how I tried to take my work and have influence at the level I could influence right. So you know, maybe it’s getting the one person to shift their perspective and get them to like, listen in on the interview, I don’t know, whatever that small measure of change or influences I think focusing on that is better than just saying, you know, there’s all these constraints and I couldn’t do anything.
Steve: That’s good. I’m gonna use all these because we are we’re always doing storytelling and I think it just to kind of reflect on how we can most effectively communicate overcoming challenges, having success, success at different levels, all that context. Yeah. It’s a really good way of putting it. What are you seeing in terms of where people are coming from to research,
Amber: maybe it’s helpful to describe like the folks on my team just because that’s, you know, who I’m interacting with. So one of them was actually in journalism for most of his career before transitioning into user experience research. One of them was working in academia for a while doing all kinds of different unrelated, you know, research around philosophy and and other things and then transitioned into technology actually, being a technology director for again, publication before transitioning into user research. One was a designer before making a transition. And one of them was actually doing market research for consumer packaged goods companies. So all of them had career changes before coming into the research side.
Steve: She’s interesting. Would you include yourself under that umbrella?
Amber: Yeah, absolutely.
Steve: It’s just a fascinating artifact of what makes up our field. Mm hmm.
Amber: Yeah. And my so I have an intern coming this summer. And she, again was more she’s been in the writing field. So she’s been a professor like an English professor and now working on her master’s to transition into research. So yeah, a lot of I think folks who come from like that, writing journalism, storytelling background, it’s a really neat transition to see. Do you
Steve: have any theories as to what is it about research that pulls people in from these different backgrounds?
Amber: I think in that case, like, there’s a easy connection there where journalism folks, they are trying to uncover information. They’re trying to understand people, they’re trying to tell stories. So that makes sense. I think other folks like the woman who was working more in technology, and then the woman who was working more in design and same with me is that you find yourself drawn to the beginning of the process where you’re really trying to understand the problem or you come into a project where you’re delivering something where you didn’t under And the people or the context, and you’ve gone through that yourself, and you want to, you’re really drawn to like, how do I make sure this doesn’t happen again? Or how do I learn more about this? And then people kind of stumble into it sometimes. You know,
Steve: I’ve been identifying as a researcher for a long time. And I always felt like yeah, I stumbled into this. But that was more it felt like that was more common because the professional wasn’t defined. We didn’t have, let’s say, the program, instead of design that that you went through, like, I don’t think that existed. If it did, no one knew about it. So I feel like oh, I came from this sort of prehistoric era where field was nascent, and so we had to kind of stumble into it. That’s how it was being formed. And it’s just fascinating to be many years later, decades later, and programs like yours and so many more, and yet, you’re characterizing a bunch of people with amazing skills and oriented towards different work and kind of transitioning into this. Yes, still what’s going on. I don’t know if that may be how careers happen. Regardless of what discipline or business you’re in, but I guess I always thought it was gonna change. And it’s just so curious that we still have this happening now in this field,
Amber: at least from what I can see, like from the, you know, folks on my team and other folks I’ve interviewed recently, I think having that other kind of background or perspective, as gives them an upper hand in some ways, because they bring a whole nother set of skills and perspective into the work they’re doing. I’ve just found, yeah, just the way that people can craft those stories or the experiences that they can share, help build like a bigger picture of just different perspectives, kind of bring them to this work and help them have kind of a point of view around certain things that is very interesting,
Steve: if you’re saying this explicitly, but I’m thinking as part of a team when you have a range of other contexts or other skill sets that people bring in even more so even better, because you have some just diversity of frameworks and superpowers.
Amber: Yeah, absolutely. It’s, it’s awesome because the other team can build on you Other, they have these different experiences. And right now I’m actually like the next kind of skill set I’m hoping to bring in is somebody who has pretty solid background and mixed methods. So bringing in more of the quantitative, somebody who’s done a lot more quantitative work, but still really understands the qualitative approach as well, because I think that’s an area I don’t have a lot of expertise in on quantitative research. And so I’m hoping to continue to add to the team in that way.
Steve: So just a slight topic shift here and pulling in some of the things that you’ve talked about when you’re kind of giving the story of different roles that you’d had you described at one point moving into, I think it was a project 202 like moving into leading a team, and then that being, I think, a defining characteristic of the work you’d been doing then and since then, research, leadership is new ish, relatively speaking, it was kind of the impetus for me in this podcast, like, Hey, here’s kind of a new role that we just didn’t have. Again, if you go back a number of years, do you have, you know, a perspective on what research leadership Looks like is it different than other kinds of leadership?
Amber: Yeah, I don’t know that it’s different than other kinds of leadership. I think there’s all the similar aspects, right? You’re trying to sort of set a vision of you know, who you are and what value you provide. You’re working with folks who I love being a manager, I love understanding people’s goals and their career paths and the things that they’re trying to do, and figuring out how do we match that with the needs of the organization? How do we help someone sort of grow and get the opportunities that they want, at the same time that we’re providing, you know, the value that we need to provide to a company. And so that’s one of my favorite things. And then thinking about, I guess, as a leader, you need to really, again, understand the business understand and keep in contact with a lot of different folks know how to grow your team, how to think about, I guess one of the big decisions with research that might be a little bit different, but is similar to things like data To science or QA, it’s like you can have a centralized team or you can have a distributed team. There’s something about how you organize yourself and how you work with parts of the organization that the leader really has to have a vision around that. And the kind of, again, I would call it like principles, but like, what are the values or principles for how your team is going to operate? So like, one of the things we talk about a lot is we don’t want anyone to have any barriers to interact with us. So we don’t require you know, somebody to go through a forum or put in a request or anything like that. We do office hours, you can ping us on slack. You can, you know, there’s all these different ways that you can interact with us. So I think kind of setting those values for how your group operates is really important.
Steve: Given that you’re someone who’s done research many points in your career, does that influence or help define the way that you approach being a manager for research?
Amber: That’s a good question. I mean, I think just trying to About I don’t know if this is a good example. But when we were a little bit over a year ago, like helping figure out what we wanted to do our goals for the next year, this was actually driven by one of the researchers, Liz, she created a survey and went and did you know, one on one interviews with everybody on the team? So we actually kind of researched the situation that was happening in order to determine let’s say, do we need to provide more education around, you know, writing a discussion guide or something like that. So we kind of used our own approach of research to understand our own situation, come up with ideas and prioritize them. I guess that that skill of listening and talking to people and being able to analyze a lot of information and synthesize it is a skill set that I find valuable every day. The ability to kind of like take that synthesis and make a plan, organize it and prioritize things has been very handy as a leader.
Steve: So there’s an example of superpowers that researchers have that you have given your career that influence support your management work. Do you have other superpowers? Or things in your background that maybe that we haven’t talked about that you find yourself drawing from?
Amber: It’s a good question. I don’t know if it’s from my background, or just like, my personality or what exactly I just say, like being able to simplify complexity into really clear communications, whether that’s written or you know, a slide deck or whatever it is, is something that I’ve honed over time, right? It’s it’s something that I especially like, being a consultant, and then you know, research like how do you how do you take all this information and how do you put something in front of somebody that they’re going to be able to remember or respond to, or, you know, kind of take in? So thinking about that. And then I think to just like, researchers have to have to have a plan. Right, like you’re going in to do research. You have to, there’s a lot of like logistical things. There’s a lot of strategic things too. And I think being able to both think high level and also think, you know, tactically are really important. So, okay, we need to get you know, in in six months, we need to understand the answer to this huge question. How are we going to get there? I think researchers, you know, have to figure out how they’re going to get to that end goal. designs the same way, right, you have to kind of break things down into manageable chunks and kind of make this plan. And I think that’s, that’s a really good skill, like, a lot of people, I would say a lot of people but I think some people have, you know, visions of something they want to get to, but don’t have a way to kind of break it down and figure out how to move to that vision. And I think researchers and designers are skilled in that.
Steve: It just makes me wonder, you know, if we sort of go see researchers and they’re off work lives, like what kinds of life skills what do we see or what kinds of applications so some of these researcher strengths would we see in the non work parts people’s lives. Because I think you’re right that we’ve talked about things like storytelling and complexity and planning and just to those things manifest, or do we shut those things off? Or are there other things that kind of come to the fore when we’re quote on the job?
Amber: I mean, I guess the other thing I see from a lot of researchers and I love is just their curiosity in other areas. So what obscure thing, are they off kind of looking at reading about learning about that they then, you know, take something from and bring it back into their work. Like, I always find that fascinating. I think that’s what I loved about school, you know, grad school, undergrad, where you’re learning about all these different subjects and all of a sudden you find that something about acupuncture and the philosophy like applies to a design project that you’re doing. So I think, you know, having those curiosities and finding those connections are really important.
Steve: So there’s anything else that you think we should talk about that ask you about?
Amber: I can’t think of anything to be honest. I think that was really fun to to share all those different things. I’m trying to think if there’s anything else,
Steve: I’ll throw another one at you then. Okay. When you think about whether it’s the field or you and your team at duo, what do you sort of look forward to in the future? Where do you think things are heading? Maybe if we say five years, do you have any vision or anticipation for where we might be at that point?
Amber: Wow, five years is a long time. I don’t know, I find those questions really, really challenging, I guess, thinking about where my team at is, that is one thing or just the field in general. You know, I do see you know, there’s so there’s so many evolutions around different aspects of technology that are happening right now that I don’t think we quite know how they’re going to play out, you know, around like, augmented reality and voice assisted technology and all these things that I don’t think we’ve quite embedded in how we do our work. Like I don’t know if there’s going to be some Different ways of us interacting with people. I mean, just even thinking about the current climate right now, you know, we’re trying to figure out how to gain contextual information when we cannot be in context with people. So is there a way to get that kind of closeness or realness with people in a way that is different than being on site with someone or different than, you know, like a diary study tool or something like that? I don’t know what that might look like. Cleaning. That’s one thing. I think I’m also, you know, there’s a lot of folks who practice like mixed methods, that’s still an area that I feel like we still have quite a separation between qualitative and quantitative specialties and not quite figuring out how to bridge that gap. So that’s something that I want to watch for in the future, like how do we really take advantage of lots of data that we have access to that could help us make decisions and learn to think what else? I don’t know. I don’t know if those are very good answers, but
Steve: they’re great. answers because my question was a terrible one. It said, what’s the future going to be? And you said, Well, here’s what I’m going to be looking for, which is a good researcher response. Yeah. here’s, here’s the signals I’ll be paying attention to.
Amber: It’s interesting. I really, I just, I don’t know. I mean, I think research is so needed, I hope that we evolve where we are not. There’s always there’s kind of like, you know, UX research. And then like my team, we call it design research. There’s research that’s, you know, in the service of sort of like product UI versus like research that’s, you know, much more looking at, you know, the customer journey and the customer lifecycle more holistically. And there’s a lot of areas that have been developed in other countries, I would say like in service design aspects in like the UK and Australia, New Zealand that just aren’t quite here in the US. So that’s something that, you know, I hope that researchers are included more or we we get that influence in other areas like public works. Like government departments that really impact people’s lives. Like I think there’s a whole spectrum of things that researchers could have a big impact on that we’re just not. There’s just those areas just aren’t really built up here yet.
Steve: So it’s it’s a demand issue, not a supply issue. Yeah,
Amber: for sure. Yeah.
Steve: This has been very interesting and lots to think about. I just really appreciate you sharing so much about, you know, your own path and all the great work that you’ve been doing it to security and thanks again for being on the podcast.
Amber: Thank you, Steve. I enjoyed the conversation as always, and guys record to next time we get to catch up.
The post 34. Amber Lindholm of Duo Security first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Amber Lindholm, the Head of User Research at Duo Security.
That’s the sign of a really good researcher – it can never be just about research for research’s sake, like this is a cool project, this is a neat thing, I really wanna go in-depth and understand perceptions of XYZ with these people, if you don’t have that ability to understand the organizational and business contexts and the types of decisions that are having to be made every day by the rest of the folks in your organization, your research isn’t going to have an impact. – Amber Lindholm
Show Links* John Mulaney Has Become the Muse of TikTok’s Makeup Artists * Amber on LinkedIn * Amber on Twitter * Duo Security * University of Illinois Graphic Design * Bauhaus education * Institute of Design * frog * projekt202 * Atlassian * Stride * Sally Carson, Head of Product Design * Mark Thompson-Kolar, Design Researcher * Annie Diu, Research Coordinator * Liz Donovan, Design Research Manager * Dug Song, Founder * Jon Oberheide, Founder * Chester Kustarz, Head of Engineering
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I read recently about a new genre of TikTok videos that featured people applying makeup, while lip syncing to standup comedy routines by John Mulaney. I believe that TikTok has it roots as a platform for lip-sync performances, and of course makeup tutorials and demonstrations are their own thing on the Internet. But how do we end up with the combination, and not just one but a whole series? How did this come about, why are people doing it, and are there other niche sub-genres or patterns that this relates to? Even if we are tempted to dismiss these behaviors as just a bunch of people being weird, we need to be doing the research to understand how, and why this is happening.
Sure, people at TikTok should know how people are using their service. But insight about this also is valuable to other platforms like YouTube and Twitter and Instagram. How could this information give you a new perspective on user behaviors if you work at Dropbox? Or if you work for Michelin, either their travel department or their tire division? How could Nationwide Insurance make use of this?
Our culture swerves and leaps and when these emergent behaviors poke their head up through into the mainstream, it’s an invitation to take note, and to be curious.
This is what clients hire me to do, whether it’s to be the one that leads the investigation, to unlock the motivations and desires of current and prospective customers, or to be someone who helps their team as they themselves dig into the hidden behaviors and objectives of users. I help companies look at their own organization, their culture, and their processes to help set the stage for this kind of discovery work to flourish and have impact.
If you’re a fan of this podcast, then remember you can support the podcast by supporting me and my practice. Please reach out to me at Portigal dot com and let’s find a way to work together.
Okay it’s time for my interview with Amber Lindholm, she’s the head of Design Research at Duo Security.
Steve: Well Amber, thank you for being on dollars to donuts. It’s really great to get to speak with you.
Amber Lindholm: Hey, Steve, it’s great to be here.
Steve: Hey, I guess I should have said, hey, hey, Amber. So I’m gonna ask you to introduce yourself.
Amber: So I’m Amber Lindholm. I’m currently the head of design research at Duo Security, which is part of Cisco. And my background is in design, both kind of traditional print design, moving into interaction design, and then design research, which I practiced before, you know, moving into more leadership positions.
Steve: What does do a security to
Amber: do security, we provide various security products that help protect our customers basically access to data. So our core product is a multi factor authentication product. So when you’re going to, let’s say log into, you know, a tool that you use at work, you know, after you do your password, you get some sort of maybe a push notification or something that verifies that To you, so we help protect organizations and their data as well as individuals.
Steve: So if I am a consumer and I go to my bank and they said, oh, we’re going to send you a text message to prove that you’re you before we can let you access your account, is that the kind of tools that you’re providing?
Amber: Yeah, that’s, that’s a great example a lot of folks are familiar with, with us through like financial institutions. And we basically have created a product that’s super easy for organizations to roll out and get folks enrolled in. But it does that where it’s it’s verifying through a second method that you are actually who you say you are,
Steve: eventually, let’s get to talk more about the work that you are doing to a security but since you said a little bit about some of what your background was it be great to just hear more about like you mentioned, print design. So what’s kind of the arc or the history for you from when you got into figuring out your profession to you know, how you move to where you are now.
Amber: So I started out as I was mentioning in graphic design, so I was trying I went to University of Illinois we did really kind of what I would call more classical, like Bauhaus, you know, educational style, learning all about formal typography and form and color. And so after school, I followed that path, I found a work with a PR agency at first and then in house at a nonprofit called Rotary International. And this was up in Chicago. And I spent my time you know, designing brochures, you know, reports, invitations, billboards, all kinds of print materials. And during that time, I just remember there was this particular project I was working on where we translated most of our materials into nine languages. So it was going out around the world. And I was creating these, they were kind of these little packets that had a CD that had files for the different rotary clubs to produce their own kind of marketing materials. And I worked with the translators to you know, translate the copy in the on the files and the file names, but after that, I Didn’t know when those things were sent out in the world if they were going to be used properly if they were going to meet their needs or expectations, and at that time, I started searching around and trying to think, you know, I feel like something’s missing. And I came across, you know, the terminology then was Human Centered Design. And it was just a revelation. For me, it seems like, you know, you can create these things, but if you don’t understand how people are going to use them and what they need, you’re really gonna miss the mark. So that was a catalyst for me to continue like my education. So I found a grad program at Institute of Design in Chicago where I went and I learned, you know, how to do research. They taught all about human factors, all kinds of approaches pulled from social sciences, very qualitative in nature, learned about design strategy, I learned more about interaction design, and that was really I really felt like I found my sweet spot. I loved the research side of it. I loved going out and talking to people. And trying to understand you know who they are and how they behave and why. So when I graduated actually, right before I graduated, I did a couple of things. So one of the things was I wanted to do a little bit of living abroad. So I went for a year to New Zealand and I worked inside the government there so did a project looking, it was more like a service design customer experience project, which was really great, because it neat to get that government experience came back and my husband and I moved to Austin, just sight unseen moved down here. And I’ve had a couple of different jobs here in Austin. I started out at frog and I was still mostly focused on design. I was doing interaction design, like mobile, tablet type design, but then I started adding in the research projects, and that again, really flourished there and love doing that work. Got to do some research in China and other places. And this was more contextual work like going in people’s homes, spending a lot of time with them to understand their world and A couple other things I did after that I went and worked in healthcare for a year trying to do similar kind of research understanding. For instance, I did a project about the new parent experience. And then I went back to consulting to a consultancy called Project 202. So spent, you know, quite a few years actually being more of a consultant role working across many industries, which was fun and started leading a team that project to research and Insights Team. So we would work on, you know, tons of projects, looking at really large customer journeys. And then after consulting for so long, I decided that I wanted to really understand the other side, so to go more in house to work at a product company. So I worked at Atlassian and led a design and research team for a product called stripe, which is a video and chat tool. And then I moved to do where I’m at now. And again, I lead the research team there love helping our team grow love the side of research that really has an impact and provides value. So that’s kind of my story
Steve: to talk about today. There’s two sort of vectors that I hear one is in house and consulting as different perspectives. But you’ve also talked about research and design. And it sounds like over the course of your career, your title at least has. Well, I guess you also said some of the work you’re doing might lean more heavily towards one or the other. It’s hard to ask the question in an open ended way as I want to. So I’ll just try the leading version. We think of research and design, like they’re different words. We call them different things. Sometimes they’re different kind of career paths. But one of the threads of your narrative that is compelling to me is that you’ve kind of worked with both those pieces of research and design. I guess I’m wondering when you think about for yourself at least and where you’re contributing where you’re adding value or how you put all these pieces together? What’s the relationship between the kind of the R word research and the D word design in your makeup and your skill set.
Amber: I think they really complement each other. I think it makes me as a designer, it made me a stronger designer to have that you know, research mind and as a researcher, it made me stronger because I can Really make sure that the research I was doing was framed around and presented in a way that provided information relevant to the folks having to make decisions. So I always think about that. It’s like when we’re trying to figure out what to research and what to prioritize. There’s always a whole set of questions. And there are decisions that are having to be made that some are more ambiguous questions where you really don’t have a lot of information. Some are really high risk questions. So you want to make sure you’re understanding those and providing that information because people are having to make big decisions and tactical decisions and thinking about how you frame that research and present it back to folks or include them in that process is really important. So I think understanding how designers work and also you know, now knowing how product companies work, understanding the questions that product managers are trying to answer engineers are trying to answer and helping them actually articulate those questions is something that I use strive to do. So I think they just really complement each other.
Steve: The way you describe how you want to make sure that the research you do connects and supports, I hear an echo of your brochure project where you were, you wanted that context to make sure that you knew that the work you’re doing was gonna have the outcomes and have the results. It sounds very similar, that kind of questioning of, Hey, who’s doing what and how do we make sure that we help them do what they’re going to do that that applies to your quote users as well as your quote, stakeholders or colleagues?
Amber: Yeah, absolutely. We spend a lot of time like when I’m working with the researchers on my team, and we’re looking at a new project, or we’re looking across the projects that we could support over quarter whatever it is, we’re looking exactly at that like where can we have the most impact? Where’s the greatest need? Sometimes we’re looking at is there a new project spinning up Is there a new product manager coming in is there you know, a team that’s just starting out that’s never gone through this process. before. We really like to prioritize those kind of things as well so that we help get them off on the right foot so that they do frame those questions really well, we help show them a good process to follow. And those are really, really fruitful projects.
Steve: What’s the split between proactive you identifying what research you want to do versus something that’s more reactive, where there’s requests coming in? Hey, can you help us do this?
Amber: That’s interesting to try to assess out. I think it’s so it’s a little bit hard, because so at duo we have, we’re part of the product design team. There’s about 30. Folks on that team, research team. I have four researchers on a team and a research coordinator who helps Do you know all of our recruiting and scheduling and all kinds of stuff. And so none of the researchers are dedicated to a particular product because duo does have multiple products. We kind of stay as a little bit of a hub, but we start to embed like I was saying, Okay, let’s kind of embed over here on this team. Let’s embed over this on this team for this particular project or this time span. And so the researchers are communicating with the designers, I’m communicating with the design managers and product managers to learn, you know, what their roadmap looks like or what kind of questions they have coming up. So I end up really driving the prioritization of the work we’re doing based on all those conversations, looking at the priorities of the business and you know, kind of larger trajectory of where the business is going and make the decisions of like, where we can actually prioritize our time and where we’re going to have the most impact. But we do also get requests. So we work with folks outside of the kind of r&d process as well. We’ve been working with our creative team on website design. We’ve been working with technical communications folks in our like knowledge community, so those people will come to us with requests. We hold office hours every Monday, so sometimes, we’ll just provide some support or advice or kind of review things and other times we’ll decide to take on a full project.
Steve: Can you maybe reward And a little bit and talk about how you came to do and what was research like there maybe what was the context that you that was established when you joined and what some of the progress and evolution has been since you’ve been leading the team?
Amber: So do I guess the design team was formed about five years ago. And so Sally Carson’s, our head of design, so she was hired to come in and, you know, build up this team. And really early on, she wanted to make sure that research was part of it. So one of the maybe the second or third hire was a researcher, and His name’s Mark Thompson Kohler. He’s still with the team. And he came in and at that point, because design was so new and research with some new, the focus was, you know, on really rapid, like usability testing, right. So like, every two weeks, we’re going to do testing. We’ve got people, you know, what do you want to test and so the company was a startup, it was a lot smaller. And the idea was just sort of this like rapid, more evaluative type research. And that went on for a while. And then The team, Mark developed a set of personas that got embedded across the entire dual organization. You know, as a team continue to grow, we’re still kind of like a one person research shop, do the way the design team has grown though designers are hired that also have an interest and skillset and research. So designers also do research. When I came in, another researcher had been hired. So there were two researchers, and there wasn’t what you would call like a research team yet. So when I came in, we decided, Okay, let’s sit down and really talk about you know, who we are, what’s our mission? What’s our value? What are the strengths that we have going for us and the gaps and how do we want to move this team forward? And we did you know, some kind of off site exercises and that kind of thing. And we started at basics. So there was a ton of knowledge. There was a ton of things had been done, and we’re like, let’s make sure like we have it right. So that first quarter was a lot of poetry. together in a place where people can find all of the research that’s been done, document some of the tools and templates we have so that other people, you know, on the team can can use this stuff. And then we wanted to make sure that instead of just focusing on the evaluative work, you know, we were allowing the designers to really pick up a lot of that and that we could move into more foundational kind of generative type research projects. And so that’s what we did after that first quarter, getting all those foundations in place doing some additional trainings with the design team, the researchers, then were able to focus on these more larger, more strategic type research projects. And then we added in research Ops, we hired a research coordinator to help us so that the researchers weren’t spending you know, half their time coordinating, recruiting, maintaining our database of participants and all that good stuff. So she came in last summer. Her name’s Annie fan, she’s been just a fantastic value to the whole team. And then I hired two more researchers this spring, and now everyone, you know, sort of has their own area that They’re working on right now. And it’s been great. It’s just been fantastic.
Steve: you’re describing some of that, I think is a pattern and a lot of organizations that give the designers the tools to do some of the evaluative work and you know, a shift in what the research team itself specializes in. What’s been the outcome for the designers or the design work that they’re doing as you’ve empowered them to do that evaluative work?
Amber: I think it’s great. I mean, I think rather than, you know, us being like, No, we can’t help you and know that, you know, those questions aren’t going to get answered, the designers do have the toolset available. And we, you know, depending on somebody’s comfort level, or you know how much they’ve done this in the past will provide more or less support, right. So, like I said, it’s kind of like reviewing the plans, reviewing the script, doing test runs, maybe we’ll moderate the session, and then they pick it up after that, or some folks, you know, they’ve done this many times, and they can run more fully, but they are the experts in sort of this area that they’re working on. And they have the belief With their product manager and their engineers, and they can take all of that, you know, directly back into the work that they’re doing. So I think it’s I think it’s great, you know, I always have some some hesitancy if it’s the subjectivity that might come out of somebody who’s testing, you know, their own work. So we do kind of talk about that, keep an eye out for that. But so far like, it’s it’s been great and allowed our researchers to, like I said, focus on not just like those larger research projects, but a researcher on my team was Donovan right now is helping kind of consult across six or seven teams for this larger project. And so she has really good line of sight around the different questions being asked and helping people, you know, make sure that they’re all kind of sharing the insights they’re learning across those teams. So it’s allowed us to really level up our scope of influence.
Steve: That’s a great explanation and I feel like there’s an implicit value judgments around what kind of research is cool and fun and worth our time and then We can offload some of these tasks to the lower value tasks to other people, then we can work on the cool stuff. Not that anyone has ever said that. But it feels like that’s lurking underneath this move to kind of, you know, empower people to do the value of work. But I think what I’m hearing from you is, in a lot of ways, those people are better positioned to do that work, because they’re very close to the problem. And they can kind of zoom in on how they can get the most actionable results, and do do so maybe more quickly. And that research, I think your point about having influence. So the work that you’re doing is is more around the organization and more strategic. And that’s kind of what you’re positioned to do.
Amber: Yeah. So besides, you know, helping bring those designers together, Liz is helping to align pm as well are kind of like share across the hims. And she’s running a set of like metrics workshops across those teams like a created as a standard sort of workshop to help us to find UX metrics look like For signals and define those metrics, and so she’s bringing those to each of those teams. So now again, like it’s a more kind of strategic view across those, but then we’re still able to weigh in on and influence the the type of research, the type of questions and even the synthesis of the research that’s happening across those teams.
Steve: What are some other things that you’re doing to whether it’s around influence or you know, kind of bringing the customers into the organization?
Amber: When we set out on our kind of mission and the things that we do, you know, obviously sort of like executing and running this kind of product focused or foundational focus research is really important. But sort of the other half of it that we wanted to make clear about was like our role and helping to get folks closer to the customer to bring that context to them, to keep the customer Top of Mind, and I do frankly, it’s pretty easy because the organization across the board is extremely customer centric. We’re not like begging for people to listen to us. The value of research is really understood. It’s, it’s asked for actually, like, people are really interested in hearing more about the context RPMs. You know, they’re fantastic. They out there out there, they do tons of customer calls, but they’re still, as we’ve continued to grow. The risk that we identified a couple of things like early on, I mentioned that there was a set of personas that were created and these get referred to all the time across the board in big meetings, you know, big call hands and all of that. Salespeople reference them, our customer success, people reference them. But as people join the company, as we were kind of in hyper growth mode, the real deep understanding of those personas may not have persisted. Right. So we wanted to take a step back and say, you know, how do we, as a research team, help other folks have the type of context that we have. So Mark, who developed the personas, we have a program set up where, you know, as new hires come in, he presents something called personas one on one to make sure that those are really top of mind and understood and we do a deep dive session on one of the more key personas. He also does something called these fireside chats. So we just chat with different folks. So for instance, different IT administrators or help desk managers or helped us folks or CSOs, we’ll set up you know, an hour long chat to just learn about their day. And that’s open invitation for folks to learn. Listen and on. And then we also with the whole of our research sessions and things, they’re open for folks to come and listen to. So it’s very much accessible. We have a research Condor, everyone can see and they can pop in on different things. And lots of folks take advantage of that. And then I think another thing that we wanted to do was make sure that we had created additional sort of artifacts and frameworks for people to think about when they’re making decisions. So we took all this data that had been collected over across, you know, about four years of tests and interviews, and ran these large workshops, and Mark, you know, synthesize all that information into a set of design principles that are persona specific. So they’re not these like really super high level principles that are hard to make decisions with. They’re very applicable to making, you know, decisions on UI on making decisions on copy you might be creating. And so those principles are referenced by You know, designers, they’re used in critique sessions. They’re referenced by product managers as they’re talking about their roadmaps, folks like technical communications and things like that. So the personas and the principles, they’re not just an artifact that just kind of sits on the table, we bring them to the forefront and a lot of different ways.
Steve: Could you genericized one of the principles just to give a sense of the granularity of it?
Amber: Yeah, so for our IT administrators, for instance, one of the principles is around the ability to take action. And so anytime we’re making a decision about something in our UI or a flow, if there’s a dead end, that that person cannot actually, let’s say they’re seeing some kind of thing that may alert them to a security problem. If they cannot immediately take action on that item, then we’re not meeting that principle. It’s very important for folks to be able to move from information that they’re seeing directly into action. And then like for our end users, one of our principles is reduce friction to the extreme. So we’re not in the kind of product where, you know, you’re trying to create a sticky situation where someone’s in your product, you’re trying to actually minimize their interaction with your product, because what they’re actually trying to do is they’re, you know, get into their work application. So we think a lot about that, like, how do we reduce friction to that extreme?
Steve: So those are specific, as you said, and meant to be actionable. Yeah. And how many principles in a project like this, how many principles should one be trying to come up with?
Amber: What a question Steve? No, I think we narrowed it down to five principles for our two main personas and that I think is a good number like three is too few eights, too many kind of a thing. But five has been like the perfect number. And when we first rolled these out, we we did some fun things to where we created some really neat posters. We put out in all the offices, you know, we socialize them with people. We brought them around to meetings, we personified the principles even, which was really fun. But yeah, I think you got to, you can’t have too many.
Steve: How do you personify a principle?
Amber: So one of the designers on our team name, Andrea actually came up with this idea to, you know, get designers more familiar with all of these. And so each of the principles, one of us was given, and you had to basically just act like that principle. So we had to create a character. And in this sort of mock critique, we had to just care about that particular principle. So it’s actually ended up to be quite fun to do.
Steve: Sounds like a good improv game was very fun. Just going back to one of the points you made was that this is a an organizational culture that believes in research and she said asks for research. Do you have any perspective on for duo how that came to be?
Amber: I imagine it’s a couple of different factors. I mean, I think that the folks, Doug and Johnno, who co founded the company, that was like a big thing. Like they really cared about the end user experience. They cared about creating a company that was different than the other security companies out there at the time. Their whole mission is about democratizing security. So making it easy and effective was there from the beginning. And it wasn’t just, you know, something they were saying was something they really believed in, they saw how complicated you know, so many security products were and they wanted to make sure that when these products are out there, and it’s, you know, an IT administrator, it’s not somebody with, you know, a cybersecurity background or anything that these more generalist folks can protect their organization. So I do think a lot of it came from the founders and then the folks that they hired. So like our head of engineering, Chester, who we product design actually reports up through engineering. He’s completely supportive of design. You know, we hire engineers that have that kind of mindset also have like customer experience. So I think it just kind of permeates through the culture, because as you said, starts at the beginning and starts at the top. I think so. I mean, that’s that’s just my impression. I mean, I wasn’t there at the beginning. But I see that legacy. And I see that through the people that were hired on that continued to be in leadership.
Steve: I have so many interactions in so many different venues where somebody who maybe is at an individual contributor level sees that their culture is not like the one that you’re describing. And they’re being blocked in a lot of different ways. And the question is often, like, well, what can I do to succeed this way when these things are kind of arrayed against me? I don’t know if you’ve encountered that or heard that question. I personally, I find it challenging to give them super actionable advice when I feel like you’re describing you know, hey, it starts at the top that says how this company was founded. You know, they’ve grown and hired and supportive that and if you’re in a situation Well, that hasn’t happened. It’s an interesting challenge for people that aren’t embedded in those kinds of organizations to make those kinds of changes and advocate for that.
Amber: Yeah, I think especially like, to your point, if you’re not at a leadership level where you have a, if your scope of influence is just with your immediate project team, it’s gonna be really hard to sort of like, level that up through the organization.
Steve: You’ve named some of the people on your team and the kinds of contributions they’ve made. And I’m just wondering how you think about, but I think you said, you just you hire two people not that long ago to join the team. Yeah, yeah. Can you describe a little bit about your approach to finding people what that looks like in terms of saying, Hey, this is someone that would fit with our organization?
Amber: We have a probably what’s it fairly standard interview process, but I’ll just describe that and then sort of what kind of questions I’m asking and looking for, but I typically, you know, reach out to folks I have like an initial conversation with them. Just to understand what they’re looking for in an organization, or what their kind of career goals are just to make sure it aligns with, you know, the kind of goal we have open, then we do a longer interview. And we use, again, pretty standard, like behavioral interview questions to ask people to describe real scenario. So behavioral interviewing, it’s kind of like doing user research or like doing design research, where you’re really trying to get somebody to give you very specific real examples like to tell a story, because you want to understand what the situation was, how they thought about it, and how they reacted to it versus like a generalized answer. So we follow that I put together like a set of questions. And then once folks kind of go through that, they go into like a two hour we do a bit of a portfolio review. And then a final round interview is more like a four hour thing. And we have designers in there. We have PMS come in, sometimes engineering managers, so different folks who they might you know, interact with. than their day to day, participate in that more long kind of final interview. But some of the things that I’m always looking out for like, obviously, you know that they can do the work, but the way that they present the work is really important. So, you know, coming from a consulting background, I am design background, I do a lot of emphasis on how people share their work and how clear it is and how compelling it is. How do they tie that story together? How do they tell high level insights, but get into enough details to really support and paint the picture of why that insight matters? So that’s a big part of it. And then the behavioral interview questions I’m always looking at, can someone learn from a situation like can they I don’t want to say like, admit they’re wrong, but if they encountered something where it went wrong, how did they go about learning, growing and kind of improving for the next time? I’m also looking at people that are willing to ask questions, so I really love hearing the questions, I always leave a lot of time for folks to ask me questions. So do they have really good questions? Were the kind of topics they’re thinking about? Are they people who are going to come into an organization and be really curious and willing to ask people questions and not just kind of sit and be scared if they don’t know the answer, trying to think what else? We ask questions, you know, around working in diverse teams that’s really important to us, and just how they kind of collaborate and work with others.
Steve: Yeah, I want to ask about one of the things you mentioned, telling a story about an insight and you know, help it come to life. And you’ve worked in consulting. So you’ve experienced some version of this, I think, I mean, I, in some ways, I think I’ve grappled with this for much of my career, that there are things that happen over the course of a project that are galvanizing that just change everything, but to tell somebody else outside that organizations, that’s a harder story to tell. So I’m wondering, what are the best practices for that? How do I make a story about an experience I had In the past come to life for an audience that didn’t have the assumptions, the biases, the worries, the you know, all the sort of things that make it so impactful. How do I help somebody else understand that part of the insight?
Amber: Well, so what you just said is exactly what I would expect is that, you know, folks that know how to tell a story upfront, they give me enough context about the organization or the problem, or the assumptions that were there in the beginning, and what kind of impact their research had, and setting the stage of that context. It’s not just about the project, right, exactly what you’re saying, like, every single organization is unique. Every single set of people has different assumptions about things, the challenges that someone might be up against within that organization are going to be different. So a really good researcher is going to help you understand that context of why the thing that they’re telling you matters. I think that’s the sign of a really good researcher is that it can never just be about, you know, research for research sake like this is a cool project, this is a neat thing I really want to, you know, go in depth and understand perceptions of XYZ with these people. If you don’t have that ability to understand the organizational and business context and the types of decisions that are having to be made every day by the rest of the folks in your organization, your research isn’t going to have an impact. And so I really listen for that. And interviews are people able to help me understand how they made an impact and how they understand that larger ecosystem that they’re working in?
Steve: So back to the archetype, I was describing the person that approaches me and I think all of us with here’s why I’m unable to have impact. Here’s how my culture says doesn’t support me. If that person comes in to speak to your team and tell stories about the research they’ve done. How do they highlight the impact of their research when there’s forces arrayed against them to have that impact?
Amber: Yeah, I mean, I think that is a really challenging and I have had interviews where folks have revealed some of those more negative challenges, I would say like, it can sometimes feel like a person puts up all the barriers and just those like I couldn’t really do anything versus somebody saying like, this was the situation. This is how I tried to take my work and have influence at the level I could influence right. So you know, maybe it’s getting the one person to shift their perspective and get them to like, listen in on the interview, I don’t know, whatever that small measure of change or influences I think focusing on that is better than just saying, you know, there’s all these constraints and I couldn’t do anything.
Steve: That’s good. I’m gonna use all these because we are we’re always doing storytelling and I think it just to kind of reflect on how we can most effectively communicate overcoming challenges, having success, success at different levels, all that context. Yeah. It’s a really good way of putting it. What are you seeing in terms of where people are coming from to research,
Amber: maybe it’s helpful to describe like the folks on my team just because that’s, you know, who I’m interacting with. So one of them was actually in journalism for most of his career before transitioning into user experience research. One of them was working in academia for a while doing all kinds of different unrelated, you know, research around philosophy and and other things and then transitioned into technology actually, being a technology director for again, publication before transitioning into user research. One was a designer before making a transition. And one of them was actually doing market research for consumer packaged goods companies. So all of them had career changes before coming into the research side.
Steve: She’s interesting. Would you include yourself under that umbrella?
Amber: Yeah, absolutely.
Steve: It’s just a fascinating artifact of what makes up our field. Mm hmm.
Amber: Yeah. And my so I have an intern coming this summer. And she, again was more she’s been in the writing field. So she’s been a professor like an English professor and now working on her master’s to transition into research. So yeah, a lot of I think folks who come from like that, writing journalism, storytelling background, it’s a really neat transition to see. Do you
Steve: have any theories as to what is it about research that pulls people in from these different backgrounds?
Amber: I think in that case, like, there’s a easy connection there where journalism folks, they are trying to uncover information. They’re trying to understand people, they’re trying to tell stories. So that makes sense. I think other folks like the woman who was working more in technology, and then the woman who was working more in design and same with me is that you find yourself drawn to the beginning of the process where you’re really trying to understand the problem or you come into a project where you’re delivering something where you didn’t under And the people or the context, and you’ve gone through that yourself, and you want to, you’re really drawn to like, how do I make sure this doesn’t happen again? Or how do I learn more about this? And then people kind of stumble into it sometimes. You know,
Steve: I’ve been identifying as a researcher for a long time. And I always felt like yeah, I stumbled into this. But that was more it felt like that was more common because the professional wasn’t defined. We didn’t have, let’s say, the program, instead of design that that you went through, like, I don’t think that existed. If it did, no one knew about it. So I feel like oh, I came from this sort of prehistoric era where field was nascent, and so we had to kind of stumble into it. That’s how it was being formed. And it’s just fascinating to be many years later, decades later, and programs like yours and so many more, and yet, you’re characterizing a bunch of people with amazing skills and oriented towards different work and kind of transitioning into this. Yes, still what’s going on. I don’t know if that may be how careers happen. Regardless of what discipline or business you’re in, but I guess I always thought it was gonna change. And it’s just so curious that we still have this happening now in this field,
Amber: at least from what I can see, like from the, you know, folks on my team and other folks I’ve interviewed recently, I think having that other kind of background or perspective, as gives them an upper hand in some ways, because they bring a whole nother set of skills and perspective into the work they’re doing. I’ve just found, yeah, just the way that people can craft those stories or the experiences that they can share, help build like a bigger picture of just different perspectives, kind of bring them to this work and help them have kind of a point of view around certain things that is very interesting,
Steve: if you’re saying this explicitly, but I’m thinking as part of a team when you have a range of other contexts or other skill sets that people bring in even more so even better, because you have some just diversity of frameworks and superpowers.
Amber: Yeah, absolutely. It’s, it’s awesome because the other team can build on you Other, they have these different experiences. And right now I’m actually like the next kind of skill set I’m hoping to bring in is somebody who has pretty solid background and mixed methods. So bringing in more of the quantitative, somebody who’s done a lot more quantitative work, but still really understands the qualitative approach as well, because I think that’s an area I don’t have a lot of expertise in on quantitative research. And so I’m hoping to continue to add to the team in that way.
Steve: So just a slight topic shift here and pulling in some of the things that you’ve talked about when you’re kind of giving the story of different roles that you’d had you described at one point moving into, I think it was a project 202 like moving into leading a team, and then that being, I think, a defining characteristic of the work you’d been doing then and since then, research, leadership is new ish, relatively speaking, it was kind of the impetus for me in this podcast, like, Hey, here’s kind of a new role that we just didn’t have. Again, if you go back a number of years, do you have, you know, a perspective on what research leadership Looks like is it different than other kinds of leadership?
Amber: Yeah, I don’t know that it’s different than other kinds of leadership. I think there’s all the similar aspects, right? You’re trying to sort of set a vision of you know, who you are and what value you provide. You’re working with folks who I love being a manager, I love understanding people’s goals and their career paths and the things that they’re trying to do, and figuring out how do we match that with the needs of the organization? How do we help someone sort of grow and get the opportunities that they want, at the same time that we’re providing, you know, the value that we need to provide to a company. And so that’s one of my favorite things. And then thinking about, I guess, as a leader, you need to really, again, understand the business understand and keep in contact with a lot of different folks know how to grow your team, how to think about, I guess one of the big decisions with research that might be a little bit different, but is similar to things like data To science or QA, it’s like you can have a centralized team or you can have a distributed team. There’s something about how you organize yourself and how you work with parts of the organization that the leader really has to have a vision around that. And the kind of, again, I would call it like principles, but like, what are the values or principles for how your team is going to operate? So like, one of the things we talk about a lot is we don’t want anyone to have any barriers to interact with us. So we don’t require you know, somebody to go through a forum or put in a request or anything like that. We do office hours, you can ping us on slack. You can, you know, there’s all these different ways that you can interact with us. So I think kind of setting those values for how your group operates is really important.
Steve: Given that you’re someone who’s done research many points in your career, does that influence or help define the way that you approach being a manager for research?
Amber: That’s a good question. I mean, I think just trying to About I don’t know if this is a good example. But when we were a little bit over a year ago, like helping figure out what we wanted to do our goals for the next year, this was actually driven by one of the researchers, Liz, she created a survey and went and did you know, one on one interviews with everybody on the team? So we actually kind of researched the situation that was happening in order to determine let’s say, do we need to provide more education around, you know, writing a discussion guide or something like that. So we kind of used our own approach of research to understand our own situation, come up with ideas and prioritize them. I guess that that skill of listening and talking to people and being able to analyze a lot of information and synthesize it is a skill set that I find valuable every day. The ability to kind of like take that synthesis and make a plan, organize it and prioritize things has been very handy as a leader.
Steve: So there’s an example of superpowers that researchers have that you have given your career that influence support your management work. Do you have other superpowers? Or things in your background that maybe that we haven’t talked about that you find yourself drawing from?
Amber: It’s a good question. I don’t know if it’s from my background, or just like, my personality or what exactly I just say, like being able to simplify complexity into really clear communications, whether that’s written or you know, a slide deck or whatever it is, is something that I’ve honed over time, right? It’s it’s something that I especially like, being a consultant, and then you know, research like how do you how do you take all this information and how do you put something in front of somebody that they’re going to be able to remember or respond to, or, you know, kind of take in? So thinking about that. And then I think to just like, researchers have to have to have a plan. Right, like you’re going in to do research. You have to, there’s a lot of like logistical things. There’s a lot of strategic things too. And I think being able to both think high level and also think, you know, tactically are really important. So, okay, we need to get you know, in in six months, we need to understand the answer to this huge question. How are we going to get there? I think researchers, you know, have to figure out how they’re going to get to that end goal. designs the same way, right, you have to kind of break things down into manageable chunks and kind of make this plan. And I think that’s, that’s a really good skill, like, a lot of people, I would say a lot of people but I think some people have, you know, visions of something they want to get to, but don’t have a way to kind of break it down and figure out how to move to that vision. And I think researchers and designers are skilled in that.
Steve: It just makes me wonder, you know, if we sort of go see researchers and they’re off work lives, like what kinds of life skills what do we see or what kinds of applications so some of these researcher strengths would we see in the non work parts people’s lives. Because I think you’re right that we’ve talked about things like storytelling and complexity and planning and just to those things manifest, or do we shut those things off? Or are there other things that kind of come to the fore when we’re quote on the job?
Amber: I mean, I guess the other thing I see from a lot of researchers and I love is just their curiosity in other areas. So what obscure thing, are they off kind of looking at reading about learning about that they then, you know, take something from and bring it back into their work. Like, I always find that fascinating. I think that’s what I loved about school, you know, grad school, undergrad, where you’re learning about all these different subjects and all of a sudden you find that something about acupuncture and the philosophy like applies to a design project that you’re doing. So I think, you know, having those curiosities and finding those connections are really important.
Steve: So there’s anything else that you think we should talk about that ask you about?
Amber: I can’t think of anything to be honest. I think that was really fun to to share all those different things. I’m trying to think if there’s anything else,
Steve: I’ll throw another one at you then. Okay. When you think about whether it’s the field or you and your team at duo, what do you sort of look forward to in the future? Where do you think things are heading? Maybe if we say five years, do you have any vision or anticipation for where we might be at that point?
Amber: Wow, five years is a long time. I don’t know, I find those questions really, really challenging, I guess, thinking about where my team at is, that is one thing or just the field in general. You know, I do see you know, there’s so there’s so many evolutions around different aspects of technology that are happening right now that I don’t think we quite know how they’re going to play out, you know, around like, augmented reality and voice assisted technology and all these things that I don’t think we’ve quite embedded in how we do our work. Like I don’t know if there’s going to be some Different ways of us interacting with people. I mean, just even thinking about the current climate right now, you know, we’re trying to figure out how to gain contextual information when we cannot be in context with people. So is there a way to get that kind of closeness or realness with people in a way that is different than being on site with someone or different than, you know, like a diary study tool or something like that? I don’t know what that might look like. Cleaning. That’s one thing. I think I’m also, you know, there’s a lot of folks who practice like mixed methods, that’s still an area that I feel like we still have quite a separation between qualitative and quantitative specialties and not quite figuring out how to bridge that gap. So that’s something that I want to watch for in the future, like how do we really take advantage of lots of data that we have access to that could help us make decisions and learn to think what else? I don’t know. I don’t know if those are very good answers, but
Steve: they’re great. answers because my question was a terrible one. It said, what’s the future going to be? And you said, Well, here’s what I’m going to be looking for, which is a good researcher response. Yeah. here’s, here’s the signals I’ll be paying attention to.
Amber: It’s interesting. I really, I just, I don’t know. I mean, I think research is so needed, I hope that we evolve where we are not. There’s always there’s kind of like, you know, UX research. And then like my team, we call it design research. There’s research that’s, you know, in the service of sort of like product UI versus like research that’s, you know, much more looking at, you know, the customer journey and the customer lifecycle more holistically. And there’s a lot of areas that have been developed in other countries, I would say like in service design aspects in like the UK and Australia, New Zealand that just aren’t quite here in the US. So that’s something that, you know, I hope that researchers are included more or we we get that influence in other areas like public works. Like government departments that really impact people’s lives. Like I think there’s a whole spectrum of things that researchers could have a big impact on that we’re just not. There’s just those areas just aren’t really built up here yet.
Steve: So it’s it’s a demand issue, not a supply issue. Yeah,
Amber: for sure. Yeah.
Steve: This has been very interesting and lots to think about. I just really appreciate you sharing so much about, you know, your own path and all the great work that you’ve been doing it to security and thanks again for being on the podcast.
Amber: Thank you, Steve. I enjoyed the conversation as always, and guys record to next time we get to catch up.
The post 34. Amber Lindholm of Duo Security first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Amber Lindholm, the Head of User Research at Duo Security. That’s the sign of a really good researcher – it can never be just about research for research’s sake, like this is a cool project, this is a neat thing, I really wanna go in-depth […]
The post 34. Amber Lindholm of Duo Security first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Julia Nelson, the Director Of Research at MOO. All researchers say to some degree that they don’t necessarily have a traditional background when they come into the research field. But I think there’s a lot of strength in welcoming people with different perspectives onto your […]
The post 33. Julia Nelson of MOO first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Julia Nelson, the Director Of Research at MOO.
All researchers say to some degree that they don’t necessarily have a traditional background when they come into the research field. But I think there’s a lot of strength in welcoming people with different perspectives onto your team, so someone who used to be a designer or someone who comes from a more academic background or someone who comes from a completely different application of qualitative research, there’s an element of resilience and perspective that that lends to a team which is the sum is greater than the parts, and that’s something that is crucial to seek out on a research team. – Julia Nelson
Show Links* Julia on LinkedIn * Julia on Twitter * MOO * Stanford d.school * Cambridge Judge Business School
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Over the past few weeks many of us have spent a lot of time on Zoom, on FaceTime, or Google Hangout, or whatever, for work, for meetups, for catching up with family and friends, for celebrations and holidays, and for other newly urgent reasons. I’m not referring to relatively passive consumption of all the “new” experiences, from film festivals to talks, museums, fundraising comedy festivals, musical performances all in addition to the television and Netflix and Hulu, but rather these active conversations when you are participating, where you are seen, and heard.
On one hand, we have turned to this alternative form because we must, we feel an imperative to connect with others, to support each other while also drawing strength from each other’s mediated glitching presence, and in the crises, this is the only way. And maybe there’s even a bit of the trend at work here, because this is just what we’re doing now.
Perhaps you’ve heard the term Zoom fatigue, especially acute for those who are expected to follow a work schedule like the one from the before times, all online, and then find themselves using their off-work hours in the very same mode. Because it’s hard. I mean, really hard.
It’s hard when people who can’t stop talking for hours when hanging out on a back porch find themselves staring at each other through a screen and just don’t know what to say, and don’t have a clue why that is.
It’s hard when members of a group have different levels of familiarity with the norms the technology demands, such as knowing to mute yourself so that the video doesn’t switch over to you when you rustle papers, even though someone else is talking.
It’s hard when convenors of our online meetings don’t know about those norms either, and don’t know the additional facilitator labor required to ensure compliance so that one person can’t accidentally stomp all over the fragile emergent communal vibe.
And on and on.
I went to a professional meetup that included a fascinating recap of many of the technologies over the decades that have tried to connect people remotely over video so that they can collaborate. And yet the meeting began with the familiar fumbling aloud in search of the sharing screen button, the host squinting away from the camera, at a second monitor, navigating the intricacies of the interface while we waited patiently but increasingly felt disconnected instead of connected.
I went to a community meeting where someone gave a spiritual musical performance full of commitment and enthusiasm. I imagine that many of the attendees felt this was a beautiful gift, but honestly I was terribly self-conscious, knowing that I was on-screen myself and had to manage my own performance, my own reaction to something very intimate that I just was not prepared for.
I joined a family celebration where one house had such terrible latency (of which they were entirely unaware) that the conversational disconnect spiraled repeatedly into something highly comical and highly frustrating.
The flawed assumption that we can simply use this technology as a replacement for the ways we used to connect is more clearly revealed as time passes. As with anything, rather than replicating experiences, the technology transforms experiences. Right now we the users are mostly in problem-solving mode, and the problems we are solving are mostly implicit. The problem is the technology, but also it’s the user interface, it’s the intended use case versus the emergent use cases, it’s the social constraints and norms, and a whole lot more.
And this is what researchers do. We don’t come with answers, but we come with ways to look at complex problems that involve people, systems, and other people. My litany of complaints is an invitation to both incremental and innovative changes, but those changes have to be based on a deep understanding of the issues, and that’s the work.
And this is one of the things I do for clients, well, really, it’s what I do WITH clients. We unpack the hidden aspects of their products, current and future, in order to prioritize their next steps to delivering the kinds of experiences that meet their goals.
And this is something I can do for you and your team. I’d appreciate you reaching out to me to find ways that we can work together.
And now, my interview with Julia Nelson, the Director of Research at MOO.
Julia, it’s really excellent to have you on Dollars to Donuts. Thanks so much for being here.
Julia Nelson: Thank you for having me. It’s an honor to be here.
Steve: So let’s start. Yeah, talk about the work that you do and maybe talk about MOO, which is sorry, I don’t get to say very.
Julia: Absolutely. I know it’s an unusual name. Um, so mu is an online printed design company. We’ve been around since 2004. based in London, that’s where the company was started. But we have several offices in the US and we operate globally. And we’re mostly known for business cards, but our real passion is product design, particularly physical product design. So we’ve been expanding over the last few years into new things like notebooks and postcards and flyers and other things. things you can do with paper. We are a bit unusual, I think, in the sense that we in two ways in terms of how we’re structured as a company, so one is that we’re completely vertically integrated. So we control all of our manufacturing. And in fact, that is a core part of some of the, the software that we build and maintain is our own sort of infrastructure for communicating with our manufacturing operations. And we’re also a very flat organization in terms of how we’re physically structured so the headquarters in London has tech and product teams, but also most of our online marketing all of our in house industrial design team. We also have our own creative team who do all their photography, photography, videography, and customer services in house which is great because it’s super easy to walk over and drop in on a customer call anytime of the day. So I’m director of research. I’ve been at mu for about three years and four months now, I started as a head of in January of 2017. My team is quite small. Over the years, we’ve fluctuated between myself and one other researcher up to about four people in total. And I think we’ve had a bit of an interesting arc as a research team. So when I joined the company, we were more of a user research function. We reported into the design organization, which sat within the product team. And we were primarily focused on evaluative user testing on the website, but over my tenure there we’ve explicitly shifted the function to be much more of almost a research and insights function. So we not only handle user research website, but also cover design research from physical products. So in forming innovation around the things that our industrial design team is creating, and then we have sort of taken over a small portfolio of market research as well. So everything from claims testing to market sizing comes within our purview.
Steve: How did that change come about?
Julia: Um, I think that’s a really interesting question. I think it’s a combination of two things. One was sort of an internal desire within my team to make sure that we were doing more generative, more strategic work that had more of a role in setting product strategy within the company and eventually company’s strategy. So it was something we explicitly work towards, in the ways that we work with our Stakeholders internally was pushing towards that more generative research. But also we got really lucky, we had some sponsors within the company who really understood the potential of what our research function could be and really championed us with senior management. So we lucked out in that sense. But I think the combination of those two things and sort of over time being able to show people how you could leverage research insight earlier in the process earlier in the design process, or even more broadly into business decision making. It’s a it’s been part of just demonstrating the value of what we can provide and how we can provide it in different ways across the company.
Steve: All right. So you’re talking about kind of two key factors here. One is the team themselves identifying, hey, we want to work you know, If we have more than we can bring here, we want to Yes, work in this way. And then sponsors, people who, you know, were in influential decision making positions, who saw, as you said, saw the potential here.
Julia: Absolutely. Yeah. And I think it’s not just about potential, but I think I’ve been really lucky. I’ve worked with both people who have intimate knowledge of research and have worked with researchers before. And I’ve also reported into AI reporting to someone who isn’t a researcher but is really open to sort of having a dialogue about research and the value that research can bring and then scouts for opportunities for us to move into doing more strategic work that is more about core business strategy, about product strategy, etc. So I think it’s, it’s possible to have allies who may not fully understand what you do but understand enough to kind of be on the lookout for ways to leverage what your team can offer to the company.
Steve: Right. Well, when you say, I understand what you do, it makes me think about how, I mean maybe this is true of every group of professionals that have a shared practice, whether they’re accountants or engineers or you know, farmers, but I don’t know if this is unique for researchers, we like to talk about the how and not like the, the process and not the outcomes. Hmm. I don’t know if that’s if you agree with that characterization.
Julia: I do think we love I think researchers are people who love details and who enjoy the mechanics of the work. You know, I think most people I know who get into research love the process, because that’s part of what fulfills them. So we naturally gravitate towards talking to talking to the process. And I think, I think when you’re talking to people when you’re talking to state holder’s from, you know, whether it’s people in different departments or senior managers, I think you picked up on something right, which is the language of outcomes? What is it going to deliver? How is it going to help us? What choices that going to enable us to make? How is it going to reduce risk speaking in the language of business tends to resonate more? Right? It’s the How will this be applicable to this challenge or problem or decision that I have to make, as, you know, the VP of marketing or the VP of product, etc.
Steve: And this comes up, I think, a lot in you know, when researchers get together they talk about, you know, having impact and having influence. Mm hmm. What you’re saying is, you know, probably the number one opportunity for researchers to do better, but I want to go back to one of the things he pointed out, I think, which is the is the consequences of doing that. Well, I guess I don’t, I don’t want to imply causality. causality to say that but you’re describing Where people may not know what you do, but they know what kinds of outcomes you can provide. And they’re on the lookout for those opportunities. So it’s not we should get those folks to do some research. It’s those folks can provide this kind of information that we need right here.
Julia: Absolutely. I think that’s exactly the right way to think about it. And you’ve articulated it more clearly than I have before. Yeah, it’s the knowing what, what’s this? I don’t want to say tangible because research can be quite an intangible thing, but it helps articulate that outcome that it’s going to give them that way. It’s going to enable their work. So even if they don’t understand the mechanics of how the research will get done And I think that’s almost easier for some people to some people, for other stakeholders to run with, you know, even if they can’t explain how the research will be done or the method or the you know, sample size or any of that. Being able to say this will help us do x within the business I think is incredibly powerful and persuasive not only for them, but for them to make the case on your behalf to others. So I often when I talk about the value of research, I often talk about risk and decision making. Rather than what specifically we might be doing.
Steve: I mean, I so I agree, and then I feel anxious when I
Julia: Ah, okay.
Steve: And so some of this is just I can make statements like as researchers, we and I may just be talking about my, what my bad habits are. But I don’t think we can just dig into this, sir, the framing here because I feel like I or researchers, you know, we do talk about the details of the process as a way to I mean, sometimes we call it we call this quality or sometimes we call it, you know, yeah, rigor. Mm hmm. Because there are so as a researcher, you know how to make process decisions that will get to the right. The right outcomes. Yes. And, and I think we’re often given requests that include process details.
Julia: Yeah.
Steve: Here’s the kind of people we want you to talk to, here’s how many, here’s how quickly, here’s what we want you to do with them. And so you can find yourself pulled into a process discussion. You know, if we don’t do this, then we’re obviously setting people up to be give us favorable responses this way. You know, that’s the guidance that we’re asked to provide.
Julia: Absolutely.
Steve: So, what’s my question for you? Is and maybe you’re just Describing a more more mature state where not having those conversations where the, you know, the request is for outcomes, and the responsibility for process decision falls to you.
Julia: Yeah, I think we’ve been really lucky in that sense. I think that’s been one of our triumphs over the years that I’ve been there is really consistent. I didn’t mean to negate the need to speak to process. But I think we’ve managed to, in some ways, educate our audience that what we really need to be asking ourselves is what do we want to know? And what are we going to do with that information? And we’re happy to talk your ears off about process but we’re going to go away and make a decision about the options for that process that are rigorous enough to give us confidence and in the quality of our output. And there’s always trade offs, right? They’re always time pressures, there’s always resource pressures. So our job as researchers is to help Deliver the insights that you need to address the questions that you want to answer. But also to figure out the how of how to do it and how to how to getting off track here a little bit. Let me restart on my answer there. I’m sorry, said that not to negate processed in it. And I think I’m speaking to more of the advocacy around research. So I think arming people to speak to the outcomes that we can provide, to make the case for the role that we can play within the organization is a different thing from speaking of what are the goals of the research process, or goals of research and the process behind it? So I didn’t say that to negate the importance of the process aspect. But yeah, I think that’s been one of our successes. Over my time there is we actually have gotten people to to let go of process a little bit more We certainly get pressure around time that sort of everyone always needs their insights yesterday. And we have to manage that as researchers, but I think I had a just illustrate it. I think that’s one of our successes within the culture. So I was in a meeting where another researcher was talking a group of stakeholders through a sort of brainstorming around, you know, the research objectives and the research questions and then spoke a bit to potential process. And I’m saying if you see the word ethnography, I get really excited. So I started geeking out and every single stakeholder in that room, who is not a researcher turned around and said, Julia, you lecture us about not prescribing process, you’re not allowed to prescribe process here. So we’ve kind of managed to at least train them a little bit to let go of trying to design the process for us. And to trust that we’re going to walk them through how and why we chose the approach. If we need to go for something lean Where we’re making decisions about how to take a leaner approach, etc. And I think that’s been a bit of a cultural win on our part. I hope that answered the question.
Steve: Yes. And I note that you don’t keep them out of the loop around. Yes,
Julia: absolutely not. Yeah, the more they’re in the process, the better. Not only, you know, just keeping we walk them through the research design, but we try to engage our stakeholders as much as possible in the process, I think is every every research, you know, here, I’m gonna generalize community, but there’s so much power and having those stakeholders in the room. So as much as possible, we try to encourage them to get involved.
Steve: Right, well, so I think you’re, you’re identifying it’s not a contradiction, but it’s an interesting challenge. And maybe this is why we’re not it’s not accountancy or so. other profession? Because I don’t know, because you’re very clear, this isn’t a black box. Right? It’s not we want these outcomes. Okay, we’ll go get them for you know, and yes to collaborate and have them participate helps the results for all the reasons that we know. But you’ve maybe, you know, as you talk about this education process has clarified where the responsibilities are where the what your value add is here, what your unique skill set is, is not is in being able to plan how to best meet their goals.
Julia: Absolutely. Absolutely.
Steve: So this is I mean, I feel like we’re hearing the, you know, the evolution towards a really strong positioning, which is still collaborative and still engaging but is relieving them of the responsibility relieving stakeholders to the responsible for things that just make their job harder, and that are actually, you know, within your, within your skill set.
Julia: Yeah. And it’s not been instant, it’s definitely come over time. It’s definitely a trust that we had to build. And also, you know, a lot of the stakeholders who we work with in different parts of the company, we’ve now been working with them for one or two or three years. So we’ve managed to, they’ve seen enough of the process and they’ve seen enough of our decision making, and we’ve walked them through a lot of that to I think, have a level of, of confidence that we’re going to define something that is both that is rigorous and will deliver what they need to know. So it’s an earned trust. It’s not an instant trust. And people are still anxious about process. I think we try and take time to explain why we’re taking the approach that we’re taking. Why we’re not considering something else, or why We feel this is the best option. So I think really speaking to that, why have we made these choices is really helpful in building their confidence around our decision making around how we design the work that we do
Steve: this this dynamic, you know, the, the trust that you’re earning, and then how you, you know, take ownership of getting towards outcomes, but you engage them in the way that you see variations. I mean, in our conversation so far, you’ve described like, a range of types of business that the organization has, and yeah, different and different work that you’re doing towards those. So, you know, how might it be different if you’re working on the industrial design of a, of a notebook versus maybe something more strategic versus something on the websites, etc, etc. Does that lead to variations in what we’re talking about?
Julia: Absolutely. I mean, I think I think there’s different cultures within a company, right? And there’s different points at which people start in their understanding of design and of design or user research. So I think we struggle. I wouldn’t call it struggle. I think we have a natural sort of common language with folks in UX design and product to have a specific experience around user research and the role that it can play in product development. Whereas the language that I speak with industrial designers is sort of, we have a similar understanding of process from in terms of the framework of design thinking and the need for sort of deep customer empathy to inform your ideation and prototyping etc. But a lot of industrial designers don’t necessarily come from a human centered design background. So even within our industrial design team, they have different levels. have experience doing research with users of a product or buyers of a product from a market research perspective, so, you know, with there’s different sort of pockets of education that need to happen there about how do we translate what we’re understanding into choices about form and the tactical experience of a product. Which are different conversations that I would be having elsewhere, partially because of subject matter, but also because they’re starting with a different understanding of some of the basics of Human Centered Design, etc, and how it plays out in practice. Whereas on the market research side, you know, they have, there’s parts of the company that are much more used to market research surveys, focus groups, etc, which isn’t necessarily what we do. Although it’s, you know, it’s parallel in some ways. Many of the techniques that we use are also used in more market research. But you know, their understanding of what research is and how it can be used and what do you get, what’s the insight you get out of it is it is entirely different from the industrial designers and the product design portion of the company. So I think we have different conversations and also different challenges with those different cultures within the company in terms of how we communicate and advocate for what we do. And also speak to the limitations of what we have what we can deliver.
Steve: So I’m thinking about, you know, maybe two dimensions here one is time, and you’ve kind of talked about, you know, the Earned trust over time. And the other is maybe the horizontal axis of in my in the chart, I’m waving my hands to illustrate. Yeah, I mean, one way, maybe I’ve screwed it up because time should be horizontal and the, the, the different parts of the organization that are at have different cultures and sort of think differently about it. And so you approach each of them differently. And this is all evolving over time. Yep. So, in the moments in any particular moments where you have to you and your team have to bring something different to bear maybe more patience or simplification or more speed and more camaraderie or kind of shared language, as you said,
Julia: Yeah.
Steve: How do you do you have a way of keeping that all in balance or, you know, the word that comes to mind for me, is, uh, is around patience. But that’s, that’s my, that’s overlay on to what your experiences mean. How do you When, when, when things go when there’s probably an optimum situation here, these people are easiest to work with and they have been? Yeah, you know, once you’ve tasted Did that and sort of, you know, achieved something, you know, like a micro Nirvana for the researchers role. You know, how do you remember? How do you create for yourself the long view? I mean, you’ve been doing this for a few years in this organization? How do you kind of put the individual experiences in context of that long view that you’ve been working towards?
Julia: Oh, that’s a tricky question. Tell me more about what you mean, my long view.
Steve: The, you’re describing a change over time, when you talk about trust, yeah, means at a point in time a, it’s going there’s gonna be more stress or concern in the party, collaborating. And you might be thinking, yeah, you know, I’m here right now with this team, and I Want, again somewhere else with this team in the future? And I mean, you’re kind of saying when you say earn trust, yeah, that makes me think there’s not a shortcut you have to work and kind of get better and get better and get better.
Julia: Yeah.
Well, I think sometimes it’s just about a lack of experience with research. It’s sometimes hard to visualize what something can tell you before you’ve, you’ve been through it. So try to think about how we’ve tackled this. I mean, I think we’re lucky in that one victory makes you more confident about the next victory. But I think I think what helps me is putting it in the perspective of the field as a whole and I talked to a lot of designers about this. Which is that in some ways design, while also very old is a very young function for many companies. And similarly, although user research or design research is an established field and is establishing many companies, it’s not as ubiquitous as like, as you said accountancy or HR are easy to understand in a way because they’re sort of ingrained in the institutions we work in. Whereas research and design are I want to say younger, but maybe more amorphous for people. So I think thinking about the fact that this is a field that is still new for many companies, that is still new for many people in the workplace, who maybe haven’t interacted with researchers before. And then it’s also rapidly evolving. I think that helps me get a bit of perspective on you know, when I may be feel challenged, working With a set of stakeholders is to just put it in that context of, we have different levels of experience with a, with a, with this arena or this discipline. And that’s, that helps me step back to, you know, I have to start with the baby steps, I have to start with ensuring we have shared assumptions and shared language and a shared understanding of what user research or design research can be and can do. So I think, I think that for some reason, that helps me with the frustration to step up to the bigger picture. And to put it in context to the wider discipline as something that is not as embedded or institutionalized as an HR function. And just to be a bit Kinder about you know, it’s it’s going to take time to educate people and for people to wrap their minds around it.
Steve: Right I think your point about the hit the youth or You know, duration of the field versus in the particular company? Yeah. Right. And, you know, movie said 2004, I think, right? Yes, it’s a 16 year old company, which is, maybe for a lot of people, an old company in a lot of it. Hmm. So you could be a researcher in a company where your career as a researcher exceeds the duration that company has been in existence.
Julia: Exactly.
Steve: And maybe the professional life of the people that you’re working with? Yeah, cuz we have people of all ages working in companies. So if for five years, your colleague may only have been out of school for two years. Yep. So yeah, I think that that’s a really good point to kind of contextualize where everybody is at.
Julia: Yeah. And it goes both ways, right. I think on the marketing side, there’s a much more Establish discipline around market research, which shapes certain expectations around the nature of the qualitative research that we do, which aren’t, you know, people come in with preconceived notions of what research looks like because of experience with that discipline. So it’s not always that they don’t have experience with the field, you know, they might be younger, but it’s out to that they might come from maybe a more traditional background or have less exposure to a more agile way of working or to, to the discipline as a whole.
Steve: So what are ways for, you know, people in the come up in the design research or user research tradition and people in the market research tradition? What are ways for them to collaborate and coexist effectively?
Julia: Hmm, I think it’s about knowing about how you fit together. So it comes a bit back to methodology and to understand that difference between sort of opinion data market research and more of the user and the design experience and how they can complement each other rather than conflict. So I think having an intimate understanding, and I’m still working on this, I’m not a market researcher by training. So my growth over the past couple years has been immersing myself more in that field and trying to understand their disciplinary heritage and also understand methodologically. What can those methods bring to the table? And where does that end? And where does design slash user research begin? I think being able to speak to that and to have conversations with people about that fit, whether they’re market researchers or they’re people within your company who kind of need both, or maybe need one and not the other. I think getting really clear on how you explain that to people that complementarity and where one begins the other ends is really helpful to communicating how they fit and when one is appropriate when the other is approved. Corporate how they can complement each other, and etc.
Steve: So we’re talking, I think, I think you’re talking about processes or methods, right? Yes. complementarity?
Julia: Um, yes. Well, knowing what each of them deliver. So, you know, my market research is going to give me insight on things like, potentially, you know, stated behavior and on opinions and that sort of thing. And what is it that that can help me understand, and what are the limitations of that work? And then how can I leverage that alongside something that’s coming from more of a design research perspective? How can we leverage the strength of both of those disciplines to work together? So So yes, from a methodology perspective, but more than Uber level?
Steve: How that how those two things fit? And then how does that look from a team perspective? Have you have the activities of market research? And yeah, I mean, you can talk more about kind of what your team looks like. But is there a person who I where the market research hat and I, where’s the user research hat? Because that’s, that’s, I think, key to making, what you’re talking about happen is, is the individuals and how they can guess, you know, complement each other.
Julia: Yeah. So the current structure of my team is, is a sort of a mix of generalists and specialists. So we are, we are because we’re very small. We are a centralized research function. And also because we not only service our product teams and crews, but also we work across the company. So myself and our other core researcher, at the moment are more generalists. We work across the board on a variety of different projects. I spend about 50% of my time as an individual contributor as well as manager. And then we have a couple people who are disciplinary specialists who are who are part time and who help sort of lead or advise. They not only do the work, but they provide that methodological expertise. So we have a part time market researcher we work with out in New York, and a very experienced design researcher for physical products who we work with out of Boston. And so that kind of odd some of the the experience and the methodological, it complements sort of the more generalist skills that myself and the other senior design researcher bring to the team. They can provide that injection of disciplines specific perspective and experience.
Steve: You’ve used generalists and specialists. Hmm. This came up in a discussion I was in recently Someone was asking as a researcher, should they be a generalist or specialist and it just made me wonder, like, what do we mean by that? So you Great question. What what are those terms mean for you?
Julia: Ah, for me, um I think it’s a combination of I’m thinking about the two people on my team. So it can be either skillset or experience. So, my market researcher has a specific set of methodological skills and also skills around survey design that she brings to the table which strengthen that element of our practice. Whereas our physical product researcher has a lot of experience thinking about sort of the physical medium of design research. So he brings perspective on what we’re looking for. From a from an industrial design perspective, what we need to understand how we need to explore some of the physical dimensions of those products in ways that are different than what we would do online. So that’s a bit more less methodological and more experiential expertise that he brings to the table. So I think it can be in either of those two dimensions. I mean, I think over the course of a career, you inevitably become a bit of a generalist in the sense of at least I find that that researchers are always hungry to learn new things and to explore new methods and to understand how they can utilize them in their work. So I think I think we all have a little bit of a gelatin generalist tendency within research.
Steve: So I’m going to ask the magic wand question. Okay. Apologies for asking a researcher the magic one question. With a genie appeared and offered to wave a magic wand and said you could add You could add two people to your team. How would you start to characterize what you would look for?
Julia: generalist or specialist? Do you want me to speak more generally to how I approach hiring or more team structure?
Steve: Well, let’s start with how you approach hiring.
Julia: Hmm. Okay. How do I approach hiring? Um, I think I, and this might be a personal preference. So I think that the key things that I look for when I’m hiring someone are, I focus a lot on research design. So do people understand can they elicit the objectives of a project can they design, a process a project that will address those objections? That’s something really, really critical to me in terms of rigor is making sure that people are flexible and adept at really thinking through that aspect to the process and defining the objectives of research, developing a design that’s, that’s rigorous and and effective and delivering the kinds of insights that we want. That we need to know. And I look for but a lot of it beyond that. So certainly there’s core skills around interviewing and observing and stuff like that, which I think are were difficult to assess in most traditional hiring processes. But you get a sense of that through how someone interviews and how they interact, the questions that they ask in the hiring process and through their portfolio, but I also tend to look for A couple of intangible qualities. I look for a growth mindset. Because research is such an experiential field in some ways, certainly. It there’s a lot more formalized education around user research, then there was more than then there has been in the past, but a lot of what you learn, you learn on the job. So I think having a hunger to learn a hunger to experiment, a desire for constant improvement. I think that really set someone up for success as a researcher because it’s not ability to learn something that the desire to try new things that allows you to expand your skill set. So that to me is a really strong signal of someone who I want to have on my team because of the potential for growth that they bring to the table. I also look for people who…I think this comes back to it, but who but who turned failure into learning so that this is a bit of growth mindset as well, but who can show me that they’ve taken something, applied it and improve their practice. And then I think somebody who can adapt who’s flexible when it comes to methodology, who can think through practical challenges, like what happens if I have half the time and half the money? And, and can adapt their approach to to those constraints? And then stakeholder management is the last one is this someone who has a has dealt with challenging stakeholders, even if it’s not in a research context, but who has sort of the emotional intelligence to think through how do I meet this person where they are How do I build a relationship with this person? How do I? How do I educate and advocate with them for how research can play a role in the work that they do day to day? So I look for kind of those three things growth, mindset, ability to learn from failure, stakeholder management and adaptability.
Steve: So for those last those last four, how’re our people communicating that to you? How does How do you find that from someone? Hmm.
Julia: It’s a lot of behavioral interviewing, it’s a lot of talking about past experiences and how they’ve dealt with past situations. Even if again, it’s not necessarily in a research, research traditional research role. Some of it we get to through, we do do task interviews, but they’re always I dislike giving people take home tasks, because I think that’s a bit unfair. We do do a thinking exercise with them, where we talk about how they might approach different challenges in the context of designing a project. So it’s a lot of behavioral questions, and a lot of just seeing their thought process about how they might approach a problem that’s presented to them.
Steve: So the other piece you said we might explore is the composition of your team.
Julia : Sure, at the moment, it’s extremely tiny. So the composition and the structure of my team has changed over time when we started and we were more embedded within the product function. We still weren’t large enough to necessarily be embedded in crews. There were two of us for for 12 crews. So we had to take almost by default, a bit more of a centralized approach. But we did try and sort of service different sets of crews. So in that instance, we were set up more around the product structure. So with different people looking after crews who kind of work on similar aspects of our website and our creation experience, etc. So they can build that depth of knowledge around that particular element of the product in the website. As we’ve sort of shifted within the organization, we actually moved out of the product function under the VP of strategy for a while. And then under the VP of commercial planning and strategy, we’ve evolved our structure to adapt to broader changes within the business. So mu has been working hard to align the company a lot more cross functionally around some of the key outcomes that we’re trying to achieve. So now we’re structured around there’s three major outcomes the company is pursuing and there’s a researcher and Is strip at the strategic level within each of those outcomes, which have steering groups that look after the outcome. So with that person is sort of responsible for overseeing for being engaged in the thinking and the decision making of the of the strategic group that’s guiding that outcome within the business. Being able to identify research needs, being able to work to shape that research and define projects and then engage with me and with a broader team to figure out how we deliver on that. So we’ve moved into something that is a little bit as a company, we’ve transformed in a way that has sort of transformed the nature of the team. So yeah, so we’ve got those three people embedded in each of those outcomes or two, and we’re seeking a third and then the specialist support kind of sits across all three of those. So for our part timers who work on through quantitative market research and our physical product researcher, they they service the needs or they work with A researcher who’s sort of leading the work for each of those outcomes to deliver what we need to.
Steve: So the person that’s embedded in one of these outcome groups, they’re involved in planning, assessing what research needs to be done and planning for that, is that the only or the primary way that what research is gonna happen gets determined is that is that what’s happening? I would
Julia : say the majority, but not all of it. So we also get asked to do research by our Executive Committee. So there are sort of business level strategic projects, which are fewer and further between, but they will make specific requests advice. And then we also for sort of day to day evaluative research, usability testing, things that need to be done and more of the executional side of design. We work extremely closely with our UX design team, to sort of empower them to do some of the research and also to oversee the work that they do or to support them on that work, depending on the needs of the designer and their experience. With UX research there, we help them drive that sort of more tactical aspect of work, or we step in to do that where there’s, there’s not someone available to do that.
Steve: I want to go back to something you’ve said, I don’t know, maybe 15 seconds ago, when you were talking about understanding the point of view of people who maybe have less experience or just different levels of familiarity with research, you made the observation that despite its you, as a field, you commented that research is rapidly changing. And I just want to go way back to that and say, in what ways do you see it rapidly changing?
Julia : I think there’s a few different forces of change that I think about well, so independent of your context, and what’s happening. I think there is an aspect of as a researcher, you’re always evolving and changing within your own practices of researcher. I’ve often said to people that I don’t think anyone comes into this field as a full stack researcher. You don’t know all the methods. You have different levels of experience with different aspects of research. You know, you might be an excellent interviewer, but project management is a challenge. So I think even within our own practices, we are always evolving as researchers. And I also think it attracts people who love that process of learning and evolving their practices they go. I think the other ways that I see things changing are, some of them are organizational. You know, going back to what we were saying earlier about people’s understanding of design and research or innovation, there’s a cultural evolution that happens within companies with their experience with those fields. So I think as you mature within an organization that also feels like a source of change. And then I think, you know, technology wise, we’re moving into new technologies and to voice into a high into these new areas and bit sorry, not Bitcoin. I’m completely blanking on what the Bitcoin technology is. I’m so sorry, blockchain blockchain. So understanding these new contexts and these new interactions is another source. A change in evolution in our discipline, learning to how our work applies or can can enable those things, I think is another continuous education element to what we do.
Steve: You had this great line that, you know, no one comes into the field full stack researcher. Yeah, leads me to ask you, was there a point at which you learned that this was a thing? This was a field or a practice?
Julia : Yes. And I also embarrassingly sort of was very up close to its emergence without really realizing it was there. So I did my undergrad at Stanford, and I remember cycling past the D school probably 1000 times and I think I went in there exactly once. my undergrad. So right at the height of when, you know, the D school was emerging as a voice and design thinking, I sort of missed the birth where the emergence of that is a mainstream discipline. I was there and I missed it. But I was sort of aware of design thinking and innovation and design research as a discipline, I think tangentially from that point. And I spent about 15 years in the Bay Area. So I had a couple friends who were one was a design researcher at IDEO. And the other was a user researcher, who is not quite senior in Seattle. So I think in some ways, I got exposed to the idea that this field existed through just those friendships and knowing a little bit about what they did. And I remember feeling that tweak of interest when talking to them about what they did and what they enjoyed about it. But I didn’t so I always had it kind of in the back of my head is something that I might potentially be interested in professionally, but it didn’t really click for me as something that I wanted to make part of my career until 10 years into another career. So I did my undergrad work in anthropology, specializing and looking at the intersection between culture and environment. So I was always interested in people and how people interacted with their physical environment and sort of used the natural world around them. And then I went into sustainability for 10 years working first as a consultant. And then in house in the corporate social responsibility and sustainability were my work was largely around doing extensive qualitative fieldwork. So I specialized in community engagement and community development around mining oil and gas and renewable energy projects. So I remember so I was about 10 years into this career. And I remember having a moment when I was working in house for one of the largest Canadian gold miners in the world. And my job was to go to different sites and engage in field work with the communities around these mine sites and really understand the impacts and the effectiveness of how the company was managing social impacts on these communities. And I just remember having this thought we were talking about local procurement and using procurement to encourage economic development around mine sites. And I just thought we could be doing so much more. We’re doing a lot of stuff that’s about managing risk, and mitigating impacts. But we’re not talking about opportunity. And we’re not talking about innovation. So I had a bit of an aha moment around that and just thought, I really want to be doing something that’s more on that innovation side of the picture. So I went back to school in my early 30s, and I decided to go get an MBA because I knew I wanted to do something around strategy and business and innovation and I wasn’t quite sure what it was and ended up at Cambridge, which is why I’m in the UK today. But I took a number of classes from professors there who specialized in service design and sort of innovation more broadly. And I really enjoyed that and that felt like the thing that I was looking for, and we did a few small designs. research projects in that context and not that really gave me a sense for, oh, this is a thing that I can do. I’m enjoying it. it leverages all this past experience that I have doing qualitative research in a different context. I think this is something that I want to pursue. And that was what led me to move. Well, I ended up finding something that was the right fit for taking a dive into the field.
Steve: In addition to the experience with field work that you pulled forward, are there other things from other points in your career or your education that you see yourself adapting or leaning on in the work that you’re doing now?
Julia : Absolutely. I think a lot of the business elements of what I did the training that being a consultant gave me around speaking to strategy and speaking to a business audience is something I leverage all the time. I think all of my managerial experience comes from managing teams and managing reports in that context. And a lot of the experience and guidance that I got from other people during that time is something I use a lot in managing my own team. Now, I did a lot of intensive facilitation work when I was a consultant, large cross functional, multi party, sort of international Working Group type of stuff, which I think gave me a lot of confidence around running meetings and facilitating decision making, and a bit of that conflict management elements. So all of that comes to bear in my stakeholder management. What else? I think all of that I think it gives me a bit of perspective on some of the organizational and business challenges that we face as researchers sort of like having a like a spare in your toolbox. You can draw on this different wealth of experience in a completely different fields, but that ends up being surprisingly applicable in what you do.
Steve: Is there anything different about managing researchers Specifically.
Julia : Hmm, I think I think we have some inherent organizational challenges. I think knowing what the long term career path is for researchers seems to be challenging for many of my peers, right? Talk to in the industry, like there’s the managerial path. So moving into managing researchers, but what is a career look like for someone who is an individual contributor? What is that arc of growth look like over 10 1520 years? How do we help people figure out what that looks like for them? Sometimes they hear from my peers that there’s a sense that what we can do as researchers is capped, that there’s a limit.
Steve: So you are having these conversations with your peers about what’s the career path for researchers and it sounds like this is a question mark in the field right now.
Julia : I think it does feel a little bit that way. Is that is that something you feel like you’ve heard from others more broadly?
Steve: You know, I hear phrases like career ladders. Yeah. And so on, and that these are starting to get defined for Yes, I haven’t looked at them. Mm hmm. And I don’t know that I have this sort of expertise to assess them. Mm hmm. You know, and I think about your point about the field rapidly changing, or I think about my own career where the field that I’m in now is not the field that I came into. Yes. Field 10 years ago is not the field. But I don’t know what the what the period of that transformation like. But yeah, if you say, well, what’s what is an individual contributor going to be doing in 10 years? Well, in what field? Will they be working is? Yes. Like this one? Yes. Yeah. I find that challenging. And so so I think your point is like that’s, that’s a challenge to manage a team manage individuals on your team be successful because the path for them is hard to plan for?
Julia : Yeah, I think so. And I think I think it’s also quite individual. And this might be because of the nature of research as a distance. To plan so if you were working in say, operations or in factory work or as a lawyer, there’s a very defined sort of ladder that you might go up if you’re a lawyer, you know, you start as an associate and you work your way up to partner, Managing Director, you know, that’s very well understood in a very clear sort of hierarchical path. I think for researchers, there’s, there’s many, the career path might look a little bit less linear. There’s certainly like a hierarchy of experience and potentially responsibility if you’re on the managerial side. But I think something that I find both sort of freeing and challenging when matching researchers is it’s very individual what you find fulfilling and motivating, I think to a greater degree in the field in our field rather than another fields because of the breadth of what we do. You know, you can grow in your skills constantly. You can grow into new skills, you can develop experience in new areas you can try out new methodologies. But because there isn’t a defined career path to sort of hang your aspirations on, it becomes much more individual to figure out what motivates you as a researcher, are you motivated by learning how to manage others? Are you motivated by the advocacy element of what we do? Are you motivated with by the joy of learning new methodologies that you can apply to your work? Are you more excited about being able to work in different products or different aspects of the business? So it ends up being? I think it’s both joyful and also challenging when managing a team of researchers to support that diversity in people’s aspirations and to create opportunities for their growth that are going to look different from person to person. So to make this more concrete, I have managed researchers who Junior researchers who’ve realized that they’re much more interested in the ideation and maybe a little bit of the design element of what we’re doing and so for them focused On ideation skills is the thing that motivates them on a day to day basis. I’ve had researchers who’ve gone into UX design because they realized they were passionate about not only getting the insights, but the translation into a solution. And I’ve had others for whom you know, what motivates them as being an expert. So what they want to do is be given opportunities to master something new. So I think learning to support those ambitions and to carve out opportunities that support those different ways of growing is a bit more challenging in our function than maybe in more traditional fields,
Steve: in maps really well to what you said about looking for a growth mindset in meeting candidates and you listed a number of different ways that that candidate might have shown in the past or indicate currently that they are interested in growing and I think this is something that’s, that’s extended even further, you listed even more ways that someone could grow and I’m sure nowhere near an exhaustive list. Yeah. You know, when you describe this, and makes me want to create this really awful matrix, which is like some grid of all the expertise is that a team? We’ve been given the challenges and so on, so that it’s balanced and you don’t? Yes. And you have like a bit of everything. Yeah. And I say awful because you made a really important point that this is very individualized. So yes, you can’t put people in those little slots of that grid. So your your team ends up looking more organic, or there’s a different model that comes out of helping individuals follow that, yes. seems too hard to plan for or control or undesirable to control.
Julia : Yeah, but I think there that’s also where the magic of how you build a team comes in, right? We know while there’s sort of, to me, there’s more of an individual, maybe career pathing that that tends to happen. It’s also about identifying what are the complimentary skills that each member brings to that team and how do you bring in people who bring who filled I don’t To say gaps, because they’re not necessarily gaps, but who add that additional, whether it’s expertise or skill set or experience that complements the other people within the team. So I think that’s where you’re not just helping guide people down their individual career paths, but you’re also playing conductor for the team as a whole and figuring out how do you bring together those different strengths and and leverage them as a group so that you take advantage of how they can complement each other and how they can fill out your practice as a whole.
Steve: Is there anything that we should talk about today that we haven’t got to?
Julia : The ideal question, I think, I feel very passionately about encouraging people who don’t necessarily have traditional backgrounds to jump into this fields. I think all researchers say to some degree that they don’t necessarily have a traditional background when they when they come into research, but I think there’s a lot of strength in it. welcoming people with different perspectives on to your team. So someone who used to be a designer or someone who maybe comes from a more academic background or someone who comes from a completely different application of qualitative research, I think there is an element of resilience and perspective that that lens to a team, which is sort of the sum is greater than the parts. And I think that’s something that is crucial to seek out on a research team because I think it’s strengthens you and strengthens your practice as a team.
Steve: Certainly, I see a lot of I’m trying to transition into into UX research. And what comes after that usually is how do I do that? Yeah, you’re pointing out that it behooves the team to welcome those people. Is there more like how does how does the field create that welcome?
Julia : How does the field create that welcome, I think there’s a lot more we can do to create opportunities for people to get their feet wet. So I’ve started to see a lot more Associate researcher, we’ve created sort of a more junior level within our career level of associate researcher, something that makes it easier for someone to to get experience within the field we’ve taken on interns or an intern, which I think is is a nice door into the discipline. So I think creating some of those formal opportunities, but I think being willing to mentor as well facilitating connections for for people who are just trying to get into the field, even if it’s just for informational interviews, or maybe informal projects is something else we can definitely do. And I think also, as someone who came in from the outside, I think it’s possible to I think we underestimate how much it’s worth really sitting down with someone who has been a week. When you’re on the outside, it’s a bit more difficult to understand what this wall looks like what the day to day looks like. What are the skill sets. So when you’re trying to think about what are my transport skills that bring value to this field that will help me transition into design or into research. I think just being able to talk to someone who is a researcher who works on the day to day and to have a conversation with them about how your skills fit into this picture is completely invaluable. So I think making yourself available to people who are really trying to understand what this work is about how they can leverage the skills and experience they already have, making it concrete for them and helping them figure out how to articulate that value is a big part of giving them the confidence to take the leap and an understanding of how they can they can bring those skills over the fence into research.
Steve: That’s great if some very concrete advice I think for people in the field and looking to get into the field. I think that’s a great place for us to to wrap up here. It’s been really interesting and enjoyable to speak with you Julie. I really appreciate you sharing so much with us.
The post 33. Julia Nelson of MOO first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Julia Nelson, the Director Of Research at MOO.
All researchers say to some degree that they don’t necessarily have a traditional background when they come into the research field. But I think there’s a lot of strength in welcoming people with different perspectives onto your team, so someone who used to be a designer or someone who comes from a more academic background or someone who comes from a completely different application of qualitative research, there’s an element of resilience and perspective that that lends to a team which is the sum is greater than the parts, and that’s something that is crucial to seek out on a research team. – Julia Nelson
Show Links* Julia on LinkedIn * Julia on Twitter * MOO * Stanford d.school * Cambridge Judge Business School
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Over the past few weeks many of us have spent a lot of time on Zoom, on FaceTime, or Google Hangout, or whatever, for work, for meetups, for catching up with family and friends, for celebrations and holidays, and for other newly urgent reasons. I’m not referring to relatively passive consumption of all the “new” experiences, from film festivals to talks, museums, fundraising comedy festivals, musical performances all in addition to the television and Netflix and Hulu, but rather these active conversations when you are participating, where you are seen, and heard.
On one hand, we have turned to this alternative form because we must, we feel an imperative to connect with others, to support each other while also drawing strength from each other’s mediated glitching presence, and in the crises, this is the only way. And maybe there’s even a bit of the trend at work here, because this is just what we’re doing now.
Perhaps you’ve heard the term Zoom fatigue, especially acute for those who are expected to follow a work schedule like the one from the before times, all online, and then find themselves using their off-work hours in the very same mode. Because it’s hard. I mean, really hard.
It’s hard when people who can’t stop talking for hours when hanging out on a back porch find themselves staring at each other through a screen and just don’t know what to say, and don’t have a clue why that is.
It’s hard when members of a group have different levels of familiarity with the norms the technology demands, such as knowing to mute yourself so that the video doesn’t switch over to you when you rustle papers, even though someone else is talking.
It’s hard when convenors of our online meetings don’t know about those norms either, and don’t know the additional facilitator labor required to ensure compliance so that one person can’t accidentally stomp all over the fragile emergent communal vibe.
And on and on.
I went to a professional meetup that included a fascinating recap of many of the technologies over the decades that have tried to connect people remotely over video so that they can collaborate. And yet the meeting began with the familiar fumbling aloud in search of the sharing screen button, the host squinting away from the camera, at a second monitor, navigating the intricacies of the interface while we waited patiently but increasingly felt disconnected instead of connected.
I went to a community meeting where someone gave a spiritual musical performance full of commitment and enthusiasm. I imagine that many of the attendees felt this was a beautiful gift, but honestly I was terribly self-conscious, knowing that I was on-screen myself and had to manage my own performance, my own reaction to something very intimate that I just was not prepared for.
I joined a family celebration where one house had such terrible latency (of which they were entirely unaware) that the conversational disconnect spiraled repeatedly into something highly comical and highly frustrating.
The flawed assumption that we can simply use this technology as a replacement for the ways we used to connect is more clearly revealed as time passes. As with anything, rather than replicating experiences, the technology transforms experiences. Right now we the users are mostly in problem-solving mode, and the problems we are solving are mostly implicit. The problem is the technology, but also it’s the user interface, it’s the intended use case versus the emergent use cases, it’s the social constraints and norms, and a whole lot more.
And this is what researchers do. We don’t come with answers, but we come with ways to look at complex problems that involve people, systems, and other people. My litany of complaints is an invitation to both incremental and innovative changes, but those changes have to be based on a deep understanding of the issues, and that’s the work.
And this is one of the things I do for clients, well, really, it’s what I do WITH clients. We unpack the hidden aspects of their products, current and future, in order to prioritize their next steps to delivering the kinds of experiences that meet their goals.
And this is something I can do for you and your team. I’d appreciate you reaching out to me to find ways that we can work together.
And now, my interview with Julia Nelson, the Director of Research at MOO.
Julia, it’s really excellent to have you on Dollars to Donuts. Thanks so much for being here.
Julia Nelson: Thank you for having me. It’s an honor to be here.
Steve: So let’s start. Yeah, talk about the work that you do and maybe talk about MOO, which is sorry, I don’t get to say very.
Julia: Absolutely. I know it’s an unusual name. Um, so mu is an online printed design company. We’ve been around since 2004. based in London, that’s where the company was started. But we have several offices in the US and we operate globally. And we’re mostly known for business cards, but our real passion is product design, particularly physical product design. So we’ve been expanding over the last few years into new things like notebooks and postcards and flyers and other things. things you can do with paper. We are a bit unusual, I think, in the sense that we in two ways in terms of how we’re structured as a company, so one is that we’re completely vertically integrated. So we control all of our manufacturing. And in fact, that is a core part of some of the, the software that we build and maintain is our own sort of infrastructure for communicating with our manufacturing operations. And we’re also a very flat organization in terms of how we’re physically structured so the headquarters in London has tech and product teams, but also most of our online marketing all of our in house industrial design team. We also have our own creative team who do all their photography, photography, videography, and customer services in house which is great because it’s super easy to walk over and drop in on a customer call anytime of the day. So I’m director of research. I’ve been at mu for about three years and four months now, I started as a head of in January of 2017. My team is quite small. Over the years, we’ve fluctuated between myself and one other researcher up to about four people in total. And I think we’ve had a bit of an interesting arc as a research team. So when I joined the company, we were more of a user research function. We reported into the design organization, which sat within the product team. And we were primarily focused on evaluative user testing on the website, but over my tenure there we’ve explicitly shifted the function to be much more of almost a research and insights function. So we not only handle user research website, but also cover design research from physical products. So in forming innovation around the things that our industrial design team is creating, and then we have sort of taken over a small portfolio of market research as well. So everything from claims testing to market sizing comes within our purview.
Steve: How did that change come about?
Julia: Um, I think that’s a really interesting question. I think it’s a combination of two things. One was sort of an internal desire within my team to make sure that we were doing more generative, more strategic work that had more of a role in setting product strategy within the company and eventually company’s strategy. So it was something we explicitly work towards, in the ways that we work with our Stakeholders internally was pushing towards that more generative research. But also we got really lucky, we had some sponsors within the company who really understood the potential of what our research function could be and really championed us with senior management. So we lucked out in that sense. But I think the combination of those two things and sort of over time being able to show people how you could leverage research insight earlier in the process earlier in the design process, or even more broadly into business decision making. It’s a it’s been part of just demonstrating the value of what we can provide and how we can provide it in different ways across the company.
Steve: All right. So you’re talking about kind of two key factors here. One is the team themselves identifying, hey, we want to work you know, If we have more than we can bring here, we want to Yes, work in this way. And then sponsors, people who, you know, were in influential decision making positions, who saw, as you said, saw the potential here.
Julia: Absolutely. Yeah. And I think it’s not just about potential, but I think I’ve been really lucky. I’ve worked with both people who have intimate knowledge of research and have worked with researchers before. And I’ve also reported into AI reporting to someone who isn’t a researcher but is really open to sort of having a dialogue about research and the value that research can bring and then scouts for opportunities for us to move into doing more strategic work that is more about core business strategy, about product strategy, etc. So I think it’s, it’s possible to have allies who may not fully understand what you do but understand enough to kind of be on the lookout for ways to leverage what your team can offer to the company.
Steve: Right. Well, when you say, I understand what you do, it makes me think about how, I mean maybe this is true of every group of professionals that have a shared practice, whether they’re accountants or engineers or you know, farmers, but I don’t know if this is unique for researchers, we like to talk about the how and not like the, the process and not the outcomes. Hmm. I don’t know if that’s if you agree with that characterization.
Julia: I do think we love I think researchers are people who love details and who enjoy the mechanics of the work. You know, I think most people I know who get into research love the process, because that’s part of what fulfills them. So we naturally gravitate towards talking to talking to the process. And I think, I think when you’re talking to people when you’re talking to state holder’s from, you know, whether it’s people in different departments or senior managers, I think you picked up on something right, which is the language of outcomes? What is it going to deliver? How is it going to help us? What choices that going to enable us to make? How is it going to reduce risk speaking in the language of business tends to resonate more? Right? It’s the How will this be applicable to this challenge or problem or decision that I have to make, as, you know, the VP of marketing or the VP of product, etc.
Steve: And this comes up, I think, a lot in you know, when researchers get together they talk about, you know, having impact and having influence. Mm hmm. What you’re saying is, you know, probably the number one opportunity for researchers to do better, but I want to go back to one of the things he pointed out, I think, which is the is the consequences of doing that. Well, I guess I don’t, I don’t want to imply causality. causality to say that but you’re describing Where people may not know what you do, but they know what kinds of outcomes you can provide. And they’re on the lookout for those opportunities. So it’s not we should get those folks to do some research. It’s those folks can provide this kind of information that we need right here.
Julia: Absolutely. I think that’s exactly the right way to think about it. And you’ve articulated it more clearly than I have before. Yeah, it’s the knowing what, what’s this? I don’t want to say tangible because research can be quite an intangible thing, but it helps articulate that outcome that it’s going to give them that way. It’s going to enable their work. So even if they don’t understand the mechanics of how the research will get done And I think that’s almost easier for some people to some people, for other stakeholders to run with, you know, even if they can’t explain how the research will be done or the method or the you know, sample size or any of that. Being able to say this will help us do x within the business I think is incredibly powerful and persuasive not only for them, but for them to make the case on your behalf to others. So I often when I talk about the value of research, I often talk about risk and decision making. Rather than what specifically we might be doing.
Steve: I mean, I so I agree, and then I feel anxious when I
Julia: Ah, okay.
Steve: And so some of this is just I can make statements like as researchers, we and I may just be talking about my, what my bad habits are. But I don’t think we can just dig into this, sir, the framing here because I feel like I or researchers, you know, we do talk about the details of the process as a way to I mean, sometimes we call it we call this quality or sometimes we call it, you know, yeah, rigor. Mm hmm. Because there are so as a researcher, you know how to make process decisions that will get to the right. The right outcomes. Yes. And, and I think we’re often given requests that include process details.
Julia: Yeah.
Steve: Here’s the kind of people we want you to talk to, here’s how many, here’s how quickly, here’s what we want you to do with them. And so you can find yourself pulled into a process discussion. You know, if we don’t do this, then we’re obviously setting people up to be give us favorable responses this way. You know, that’s the guidance that we’re asked to provide.
Julia: Absolutely.
Steve: So, what’s my question for you? Is and maybe you’re just Describing a more more mature state where not having those conversations where the, you know, the request is for outcomes, and the responsibility for process decision falls to you.
Julia: Yeah, I think we’ve been really lucky in that sense. I think that’s been one of our triumphs over the years that I’ve been there is really consistent. I didn’t mean to negate the need to speak to process. But I think we’ve managed to, in some ways, educate our audience that what we really need to be asking ourselves is what do we want to know? And what are we going to do with that information? And we’re happy to talk your ears off about process but we’re going to go away and make a decision about the options for that process that are rigorous enough to give us confidence and in the quality of our output. And there’s always trade offs, right? They’re always time pressures, there’s always resource pressures. So our job as researchers is to help Deliver the insights that you need to address the questions that you want to answer. But also to figure out the how of how to do it and how to how to getting off track here a little bit. Let me restart on my answer there. I’m sorry, said that not to negate processed in it. And I think I’m speaking to more of the advocacy around research. So I think arming people to speak to the outcomes that we can provide, to make the case for the role that we can play within the organization is a different thing from speaking of what are the goals of the research process, or goals of research and the process behind it? So I didn’t say that to negate the importance of the process aspect. But yeah, I think that’s been one of our successes. Over my time there is we actually have gotten people to to let go of process a little bit more We certainly get pressure around time that sort of everyone always needs their insights yesterday. And we have to manage that as researchers, but I think I had a just illustrate it. I think that’s one of our successes within the culture. So I was in a meeting where another researcher was talking a group of stakeholders through a sort of brainstorming around, you know, the research objectives and the research questions and then spoke a bit to potential process. And I’m saying if you see the word ethnography, I get really excited. So I started geeking out and every single stakeholder in that room, who is not a researcher turned around and said, Julia, you lecture us about not prescribing process, you’re not allowed to prescribe process here. So we’ve kind of managed to at least train them a little bit to let go of trying to design the process for us. And to trust that we’re going to walk them through how and why we chose the approach. If we need to go for something lean Where we’re making decisions about how to take a leaner approach, etc. And I think that’s been a bit of a cultural win on our part. I hope that answered the question.
Steve: Yes. And I note that you don’t keep them out of the loop around. Yes,
Julia: absolutely not. Yeah, the more they’re in the process, the better. Not only, you know, just keeping we walk them through the research design, but we try to engage our stakeholders as much as possible in the process, I think is every every research, you know, here, I’m gonna generalize community, but there’s so much power and having those stakeholders in the room. So as much as possible, we try to encourage them to get involved.
Steve: Right, well, so I think you’re, you’re identifying it’s not a contradiction, but it’s an interesting challenge. And maybe this is why we’re not it’s not accountancy or so. other profession? Because I don’t know, because you’re very clear, this isn’t a black box. Right? It’s not we want these outcomes. Okay, we’ll go get them for you know, and yes to collaborate and have them participate helps the results for all the reasons that we know. But you’ve maybe, you know, as you talk about this education process has clarified where the responsibilities are where the what your value add is here, what your unique skill set is, is not is in being able to plan how to best meet their goals.
Julia: Absolutely. Absolutely.
Steve: So this is I mean, I feel like we’re hearing the, you know, the evolution towards a really strong positioning, which is still collaborative and still engaging but is relieving them of the responsibility relieving stakeholders to the responsible for things that just make their job harder, and that are actually, you know, within your, within your skill set.
Julia: Yeah. And it’s not been instant, it’s definitely come over time. It’s definitely a trust that we had to build. And also, you know, a lot of the stakeholders who we work with in different parts of the company, we’ve now been working with them for one or two or three years. So we’ve managed to, they’ve seen enough of the process and they’ve seen enough of our decision making, and we’ve walked them through a lot of that to I think, have a level of, of confidence that we’re going to define something that is both that is rigorous and will deliver what they need to know. So it’s an earned trust. It’s not an instant trust. And people are still anxious about process. I think we try and take time to explain why we’re taking the approach that we’re taking. Why we’re not considering something else, or why We feel this is the best option. So I think really speaking to that, why have we made these choices is really helpful in building their confidence around our decision making around how we design the work that we do
Steve: this this dynamic, you know, the, the trust that you’re earning, and then how you, you know, take ownership of getting towards outcomes, but you engage them in the way that you see variations. I mean, in our conversation so far, you’ve described like, a range of types of business that the organization has, and yeah, different and different work that you’re doing towards those. So, you know, how might it be different if you’re working on the industrial design of a, of a notebook versus maybe something more strategic versus something on the websites, etc, etc. Does that lead to variations in what we’re talking about?
Julia: Absolutely. I mean, I think I think there’s different cultures within a company, right? And there’s different points at which people start in their understanding of design and of design or user research. So I think we struggle. I wouldn’t call it struggle. I think we have a natural sort of common language with folks in UX design and product to have a specific experience around user research and the role that it can play in product development. Whereas the language that I speak with industrial designers is sort of, we have a similar understanding of process from in terms of the framework of design thinking and the need for sort of deep customer empathy to inform your ideation and prototyping etc. But a lot of industrial designers don’t necessarily come from a human centered design background. So even within our industrial design team, they have different levels. have experience doing research with users of a product or buyers of a product from a market research perspective, so, you know, with there’s different sort of pockets of education that need to happen there about how do we translate what we’re understanding into choices about form and the tactical experience of a product. Which are different conversations that I would be having elsewhere, partially because of subject matter, but also because they’re starting with a different understanding of some of the basics of Human Centered Design, etc, and how it plays out in practice. Whereas on the market research side, you know, they have, there’s parts of the company that are much more used to market research surveys, focus groups, etc, which isn’t necessarily what we do. Although it’s, you know, it’s parallel in some ways. Many of the techniques that we use are also used in more market research. But you know, their understanding of what research is and how it can be used and what do you get, what’s the insight you get out of it is it is entirely different from the industrial designers and the product design portion of the company. So I think we have different conversations and also different challenges with those different cultures within the company in terms of how we communicate and advocate for what we do. And also speak to the limitations of what we have what we can deliver.
Steve: So I’m thinking about, you know, maybe two dimensions here one is time, and you’ve kind of talked about, you know, the Earned trust over time. And the other is maybe the horizontal axis of in my in the chart, I’m waving my hands to illustrate. Yeah, I mean, one way, maybe I’ve screwed it up because time should be horizontal and the, the, the different parts of the organization that are at have different cultures and sort of think differently about it. And so you approach each of them differently. And this is all evolving over time. Yep. So, in the moments in any particular moments where you have to you and your team have to bring something different to bear maybe more patience or simplification or more speed and more camaraderie or kind of shared language, as you said,
Julia: Yeah.
Steve: How do you do you have a way of keeping that all in balance or, you know, the word that comes to mind for me, is, uh, is around patience. But that’s, that’s my, that’s overlay on to what your experiences mean. How do you When, when, when things go when there’s probably an optimum situation here, these people are easiest to work with and they have been? Yeah, you know, once you’ve tasted Did that and sort of, you know, achieved something, you know, like a micro Nirvana for the researchers role. You know, how do you remember? How do you create for yourself the long view? I mean, you’ve been doing this for a few years in this organization? How do you kind of put the individual experiences in context of that long view that you’ve been working towards?
Julia: Oh, that’s a tricky question. Tell me more about what you mean, my long view.
Steve: The, you’re describing a change over time, when you talk about trust, yeah, means at a point in time a, it’s going there’s gonna be more stress or concern in the party, collaborating. And you might be thinking, yeah, you know, I’m here right now with this team, and I Want, again somewhere else with this team in the future? And I mean, you’re kind of saying when you say earn trust, yeah, that makes me think there’s not a shortcut you have to work and kind of get better and get better and get better.
Julia: Yeah.
Well, I think sometimes it’s just about a lack of experience with research. It’s sometimes hard to visualize what something can tell you before you’ve, you’ve been through it. So try to think about how we’ve tackled this. I mean, I think we’re lucky in that one victory makes you more confident about the next victory. But I think I think what helps me is putting it in the perspective of the field as a whole and I talked to a lot of designers about this. Which is that in some ways design, while also very old is a very young function for many companies. And similarly, although user research or design research is an established field and is establishing many companies, it’s not as ubiquitous as like, as you said accountancy or HR are easy to understand in a way because they’re sort of ingrained in the institutions we work in. Whereas research and design are I want to say younger, but maybe more amorphous for people. So I think thinking about the fact that this is a field that is still new for many companies, that is still new for many people in the workplace, who maybe haven’t interacted with researchers before. And then it’s also rapidly evolving. I think that helps me get a bit of perspective on you know, when I may be feel challenged, working With a set of stakeholders is to just put it in that context of, we have different levels of experience with a, with a, with this arena or this discipline. And that’s, that helps me step back to, you know, I have to start with the baby steps, I have to start with ensuring we have shared assumptions and shared language and a shared understanding of what user research or design research can be and can do. So I think, I think that for some reason, that helps me with the frustration to step up to the bigger picture. And to put it in context to the wider discipline as something that is not as embedded or institutionalized as an HR function. And just to be a bit Kinder about you know, it’s it’s going to take time to educate people and for people to wrap their minds around it.
Steve: Right I think your point about the hit the youth or You know, duration of the field versus in the particular company? Yeah. Right. And, you know, movie said 2004, I think, right? Yes, it’s a 16 year old company, which is, maybe for a lot of people, an old company in a lot of it. Hmm. So you could be a researcher in a company where your career as a researcher exceeds the duration that company has been in existence.
Julia: Exactly.
Steve: And maybe the professional life of the people that you’re working with? Yeah, cuz we have people of all ages working in companies. So if for five years, your colleague may only have been out of school for two years. Yep. So yeah, I think that that’s a really good point to kind of contextualize where everybody is at.
Julia: Yeah. And it goes both ways, right. I think on the marketing side, there’s a much more Establish discipline around market research, which shapes certain expectations around the nature of the qualitative research that we do, which aren’t, you know, people come in with preconceived notions of what research looks like because of experience with that discipline. So it’s not always that they don’t have experience with the field, you know, they might be younger, but it’s out to that they might come from maybe a more traditional background or have less exposure to a more agile way of working or to, to the discipline as a whole.
Steve: So what are ways for, you know, people in the come up in the design research or user research tradition and people in the market research tradition? What are ways for them to collaborate and coexist effectively?
Julia: Hmm, I think it’s about knowing about how you fit together. So it comes a bit back to methodology and to understand that difference between sort of opinion data market research and more of the user and the design experience and how they can complement each other rather than conflict. So I think having an intimate understanding, and I’m still working on this, I’m not a market researcher by training. So my growth over the past couple years has been immersing myself more in that field and trying to understand their disciplinary heritage and also understand methodologically. What can those methods bring to the table? And where does that end? And where does design slash user research begin? I think being able to speak to that and to have conversations with people about that fit, whether they’re market researchers or they’re people within your company who kind of need both, or maybe need one and not the other. I think getting really clear on how you explain that to people that complementarity and where one begins the other ends is really helpful to communicating how they fit and when one is appropriate when the other is approved. Corporate how they can complement each other, and etc.
Steve: So we’re talking, I think, I think you’re talking about processes or methods, right? Yes. complementarity?
Julia: Um, yes. Well, knowing what each of them deliver. So, you know, my market research is going to give me insight on things like, potentially, you know, stated behavior and on opinions and that sort of thing. And what is it that that can help me understand, and what are the limitations of that work? And then how can I leverage that alongside something that’s coming from more of a design research perspective? How can we leverage the strength of both of those disciplines to work together? So So yes, from a methodology perspective, but more than Uber level?
Steve: How that how those two things fit? And then how does that look from a team perspective? Have you have the activities of market research? And yeah, I mean, you can talk more about kind of what your team looks like. But is there a person who I where the market research hat and I, where’s the user research hat? Because that’s, that’s, I think, key to making, what you’re talking about happen is, is the individuals and how they can guess, you know, complement each other.
Julia: Yeah. So the current structure of my team is, is a sort of a mix of generalists and specialists. So we are, we are because we’re very small. We are a centralized research function. And also because we not only service our product teams and crews, but also we work across the company. So myself and our other core researcher, at the moment are more generalists. We work across the board on a variety of different projects. I spend about 50% of my time as an individual contributor as well as manager. And then we have a couple people who are disciplinary specialists who are who are part time and who help sort of lead or advise. They not only do the work, but they provide that methodological expertise. So we have a part time market researcher we work with out in New York, and a very experienced design researcher for physical products who we work with out of Boston. And so that kind of odd some of the the experience and the methodological, it complements sort of the more generalist skills that myself and the other senior design researcher bring to the team. They can provide that injection of disciplines specific perspective and experience.
Steve: You’ve used generalists and specialists. Hmm. This came up in a discussion I was in recently Someone was asking as a researcher, should they be a generalist or specialist and it just made me wonder, like, what do we mean by that? So you Great question. What what are those terms mean for you?
Julia: Ah, for me, um I think it’s a combination of I’m thinking about the two people on my team. So it can be either skillset or experience. So, my market researcher has a specific set of methodological skills and also skills around survey design that she brings to the table which strengthen that element of our practice. Whereas our physical product researcher has a lot of experience thinking about sort of the physical medium of design research. So he brings perspective on what we’re looking for. From a from an industrial design perspective, what we need to understand how we need to explore some of the physical dimensions of those products in ways that are different than what we would do online. So that’s a bit more less methodological and more experiential expertise that he brings to the table. So I think it can be in either of those two dimensions. I mean, I think over the course of a career, you inevitably become a bit of a generalist in the sense of at least I find that that researchers are always hungry to learn new things and to explore new methods and to understand how they can utilize them in their work. So I think I think we all have a little bit of a gelatin generalist tendency within research.
Steve: So I’m going to ask the magic wand question. Okay. Apologies for asking a researcher the magic one question. With a genie appeared and offered to wave a magic wand and said you could add You could add two people to your team. How would you start to characterize what you would look for?
Julia: generalist or specialist? Do you want me to speak more generally to how I approach hiring or more team structure?
Steve: Well, let’s start with how you approach hiring.
Julia: Hmm. Okay. How do I approach hiring? Um, I think I, and this might be a personal preference. So I think that the key things that I look for when I’m hiring someone are, I focus a lot on research design. So do people understand can they elicit the objectives of a project can they design, a process a project that will address those objections? That’s something really, really critical to me in terms of rigor is making sure that people are flexible and adept at really thinking through that aspect to the process and defining the objectives of research, developing a design that’s, that’s rigorous and and effective and delivering the kinds of insights that we want. That we need to know. And I look for but a lot of it beyond that. So certainly there’s core skills around interviewing and observing and stuff like that, which I think are were difficult to assess in most traditional hiring processes. But you get a sense of that through how someone interviews and how they interact, the questions that they ask in the hiring process and through their portfolio, but I also tend to look for A couple of intangible qualities. I look for a growth mindset. Because research is such an experiential field in some ways, certainly. It there’s a lot more formalized education around user research, then there was more than then there has been in the past, but a lot of what you learn, you learn on the job. So I think having a hunger to learn a hunger to experiment, a desire for constant improvement. I think that really set someone up for success as a researcher because it’s not ability to learn something that the desire to try new things that allows you to expand your skill set. So that to me is a really strong signal of someone who I want to have on my team because of the potential for growth that they bring to the table. I also look for people who…I think this comes back to it, but who but who turned failure into learning so that this is a bit of growth mindset as well, but who can show me that they’ve taken something, applied it and improve their practice. And then I think somebody who can adapt who’s flexible when it comes to methodology, who can think through practical challenges, like what happens if I have half the time and half the money? And, and can adapt their approach to to those constraints? And then stakeholder management is the last one is this someone who has a has dealt with challenging stakeholders, even if it’s not in a research context, but who has sort of the emotional intelligence to think through how do I meet this person where they are How do I build a relationship with this person? How do I? How do I educate and advocate with them for how research can play a role in the work that they do day to day? So I look for kind of those three things growth, mindset, ability to learn from failure, stakeholder management and adaptability.
Steve: So for those last those last four, how’re our people communicating that to you? How does How do you find that from someone? Hmm.
Julia: It’s a lot of behavioral interviewing, it’s a lot of talking about past experiences and how they’ve dealt with past situations. Even if again, it’s not necessarily in a research, research traditional research role. Some of it we get to through, we do do task interviews, but they’re always I dislike giving people take home tasks, because I think that’s a bit unfair. We do do a thinking exercise with them, where we talk about how they might approach different challenges in the context of designing a project. So it’s a lot of behavioral questions, and a lot of just seeing their thought process about how they might approach a problem that’s presented to them.
Steve: So the other piece you said we might explore is the composition of your team.
Julia : Sure, at the moment, it’s extremely tiny. So the composition and the structure of my team has changed over time when we started and we were more embedded within the product function. We still weren’t large enough to necessarily be embedded in crews. There were two of us for for 12 crews. So we had to take almost by default, a bit more of a centralized approach. But we did try and sort of service different sets of crews. So in that instance, we were set up more around the product structure. So with different people looking after crews who kind of work on similar aspects of our website and our creation experience, etc. So they can build that depth of knowledge around that particular element of the product in the website. As we’ve sort of shifted within the organization, we actually moved out of the product function under the VP of strategy for a while. And then under the VP of commercial planning and strategy, we’ve evolved our structure to adapt to broader changes within the business. So mu has been working hard to align the company a lot more cross functionally around some of the key outcomes that we’re trying to achieve. So now we’re structured around there’s three major outcomes the company is pursuing and there’s a researcher and Is strip at the strategic level within each of those outcomes, which have steering groups that look after the outcome. So with that person is sort of responsible for overseeing for being engaged in the thinking and the decision making of the of the strategic group that’s guiding that outcome within the business. Being able to identify research needs, being able to work to shape that research and define projects and then engage with me and with a broader team to figure out how we deliver on that. So we’ve moved into something that is a little bit as a company, we’ve transformed in a way that has sort of transformed the nature of the team. So yeah, so we’ve got those three people embedded in each of those outcomes or two, and we’re seeking a third and then the specialist support kind of sits across all three of those. So for our part timers who work on through quantitative market research and our physical product researcher, they they service the needs or they work with A researcher who’s sort of leading the work for each of those outcomes to deliver what we need to.
Steve: So the person that’s embedded in one of these outcome groups, they’re involved in planning, assessing what research needs to be done and planning for that, is that the only or the primary way that what research is gonna happen gets determined is that is that what’s happening? I would
Julia : say the majority, but not all of it. So we also get asked to do research by our Executive Committee. So there are sort of business level strategic projects, which are fewer and further between, but they will make specific requests advice. And then we also for sort of day to day evaluative research, usability testing, things that need to be done and more of the executional side of design. We work extremely closely with our UX design team, to sort of empower them to do some of the research and also to oversee the work that they do or to support them on that work, depending on the needs of the designer and their experience. With UX research there, we help them drive that sort of more tactical aspect of work, or we step in to do that where there’s, there’s not someone available to do that.
Steve: I want to go back to something you’ve said, I don’t know, maybe 15 seconds ago, when you were talking about understanding the point of view of people who maybe have less experience or just different levels of familiarity with research, you made the observation that despite its you, as a field, you commented that research is rapidly changing. And I just want to go way back to that and say, in what ways do you see it rapidly changing?
Julia : I think there’s a few different forces of change that I think about well, so independent of your context, and what’s happening. I think there is an aspect of as a researcher, you’re always evolving and changing within your own practices of researcher. I’ve often said to people that I don’t think anyone comes into this field as a full stack researcher. You don’t know all the methods. You have different levels of experience with different aspects of research. You know, you might be an excellent interviewer, but project management is a challenge. So I think even within our own practices, we are always evolving as researchers. And I also think it attracts people who love that process of learning and evolving their practices they go. I think the other ways that I see things changing are, some of them are organizational. You know, going back to what we were saying earlier about people’s understanding of design and research or innovation, there’s a cultural evolution that happens within companies with their experience with those fields. So I think as you mature within an organization that also feels like a source of change. And then I think, you know, technology wise, we’re moving into new technologies and to voice into a high into these new areas and bit sorry, not Bitcoin. I’m completely blanking on what the Bitcoin technology is. I’m so sorry, blockchain blockchain. So understanding these new contexts and these new interactions is another source. A change in evolution in our discipline, learning to how our work applies or can can enable those things, I think is another continuous education element to what we do.
Steve: You had this great line that, you know, no one comes into the field full stack researcher. Yeah, leads me to ask you, was there a point at which you learned that this was a thing? This was a field or a practice?
Julia : Yes. And I also embarrassingly sort of was very up close to its emergence without really realizing it was there. So I did my undergrad at Stanford, and I remember cycling past the D school probably 1000 times and I think I went in there exactly once. my undergrad. So right at the height of when, you know, the D school was emerging as a voice and design thinking, I sort of missed the birth where the emergence of that is a mainstream discipline. I was there and I missed it. But I was sort of aware of design thinking and innovation and design research as a discipline, I think tangentially from that point. And I spent about 15 years in the Bay Area. So I had a couple friends who were one was a design researcher at IDEO. And the other was a user researcher, who is not quite senior in Seattle. So I think in some ways, I got exposed to the idea that this field existed through just those friendships and knowing a little bit about what they did. And I remember feeling that tweak of interest when talking to them about what they did and what they enjoyed about it. But I didn’t so I always had it kind of in the back of my head is something that I might potentially be interested in professionally, but it didn’t really click for me as something that I wanted to make part of my career until 10 years into another career. So I did my undergrad work in anthropology, specializing and looking at the intersection between culture and environment. So I was always interested in people and how people interacted with their physical environment and sort of used the natural world around them. And then I went into sustainability for 10 years working first as a consultant. And then in house in the corporate social responsibility and sustainability were my work was largely around doing extensive qualitative fieldwork. So I specialized in community engagement and community development around mining oil and gas and renewable energy projects. So I remember so I was about 10 years into this career. And I remember having a moment when I was working in house for one of the largest Canadian gold miners in the world. And my job was to go to different sites and engage in field work with the communities around these mine sites and really understand the impacts and the effectiveness of how the company was managing social impacts on these communities. And I just remember having this thought we were talking about local procurement and using procurement to encourage economic development around mine sites. And I just thought we could be doing so much more. We’re doing a lot of stuff that’s about managing risk, and mitigating impacts. But we’re not talking about opportunity. And we’re not talking about innovation. So I had a bit of an aha moment around that and just thought, I really want to be doing something that’s more on that innovation side of the picture. So I went back to school in my early 30s, and I decided to go get an MBA because I knew I wanted to do something around strategy and business and innovation and I wasn’t quite sure what it was and ended up at Cambridge, which is why I’m in the UK today. But I took a number of classes from professors there who specialized in service design and sort of innovation more broadly. And I really enjoyed that and that felt like the thing that I was looking for, and we did a few small designs. research projects in that context and not that really gave me a sense for, oh, this is a thing that I can do. I’m enjoying it. it leverages all this past experience that I have doing qualitative research in a different context. I think this is something that I want to pursue. And that was what led me to move. Well, I ended up finding something that was the right fit for taking a dive into the field.
Steve: In addition to the experience with field work that you pulled forward, are there other things from other points in your career or your education that you see yourself adapting or leaning on in the work that you’re doing now?
Julia : Absolutely. I think a lot of the business elements of what I did the training that being a consultant gave me around speaking to strategy and speaking to a business audience is something I leverage all the time. I think all of my managerial experience comes from managing teams and managing reports in that context. And a lot of the experience and guidance that I got from other people during that time is something I use a lot in managing my own team. Now, I did a lot of intensive facilitation work when I was a consultant, large cross functional, multi party, sort of international Working Group type of stuff, which I think gave me a lot of confidence around running meetings and facilitating decision making, and a bit of that conflict management elements. So all of that comes to bear in my stakeholder management. What else? I think all of that I think it gives me a bit of perspective on some of the organizational and business challenges that we face as researchers sort of like having a like a spare in your toolbox. You can draw on this different wealth of experience in a completely different fields, but that ends up being surprisingly applicable in what you do.
Steve: Is there anything different about managing researchers Specifically.
Julia : Hmm, I think I think we have some inherent organizational challenges. I think knowing what the long term career path is for researchers seems to be challenging for many of my peers, right? Talk to in the industry, like there’s the managerial path. So moving into managing researchers, but what is a career look like for someone who is an individual contributor? What is that arc of growth look like over 10 1520 years? How do we help people figure out what that looks like for them? Sometimes they hear from my peers that there’s a sense that what we can do as researchers is capped, that there’s a limit.
Steve: So you are having these conversations with your peers about what’s the career path for researchers and it sounds like this is a question mark in the field right now.
Julia : I think it does feel a little bit that way. Is that is that something you feel like you’ve heard from others more broadly?
Steve: You know, I hear phrases like career ladders. Yeah. And so on, and that these are starting to get defined for Yes, I haven’t looked at them. Mm hmm. And I don’t know that I have this sort of expertise to assess them. Mm hmm. You know, and I think about your point about the field rapidly changing, or I think about my own career where the field that I’m in now is not the field that I came into. Yes. Field 10 years ago is not the field. But I don’t know what the what the period of that transformation like. But yeah, if you say, well, what’s what is an individual contributor going to be doing in 10 years? Well, in what field? Will they be working is? Yes. Like this one? Yes. Yeah. I find that challenging. And so so I think your point is like that’s, that’s a challenge to manage a team manage individuals on your team be successful because the path for them is hard to plan for?
Julia : Yeah, I think so. And I think I think it’s also quite individual. And this might be because of the nature of research as a distance. To plan so if you were working in say, operations or in factory work or as a lawyer, there’s a very defined sort of ladder that you might go up if you’re a lawyer, you know, you start as an associate and you work your way up to partner, Managing Director, you know, that’s very well understood in a very clear sort of hierarchical path. I think for researchers, there’s, there’s many, the career path might look a little bit less linear. There’s certainly like a hierarchy of experience and potentially responsibility if you’re on the managerial side. But I think something that I find both sort of freeing and challenging when matching researchers is it’s very individual what you find fulfilling and motivating, I think to a greater degree in the field in our field rather than another fields because of the breadth of what we do. You know, you can grow in your skills constantly. You can grow into new skills, you can develop experience in new areas you can try out new methodologies. But because there isn’t a defined career path to sort of hang your aspirations on, it becomes much more individual to figure out what motivates you as a researcher, are you motivated by learning how to manage others? Are you motivated by the advocacy element of what we do? Are you motivated with by the joy of learning new methodologies that you can apply to your work? Are you more excited about being able to work in different products or different aspects of the business? So it ends up being? I think it’s both joyful and also challenging when managing a team of researchers to support that diversity in people’s aspirations and to create opportunities for their growth that are going to look different from person to person. So to make this more concrete, I have managed researchers who Junior researchers who’ve realized that they’re much more interested in the ideation and maybe a little bit of the design element of what we’re doing and so for them focused On ideation skills is the thing that motivates them on a day to day basis. I’ve had researchers who’ve gone into UX design because they realized they were passionate about not only getting the insights, but the translation into a solution. And I’ve had others for whom you know, what motivates them as being an expert. So what they want to do is be given opportunities to master something new. So I think learning to support those ambitions and to carve out opportunities that support those different ways of growing is a bit more challenging in our function than maybe in more traditional fields,
Steve: in maps really well to what you said about looking for a growth mindset in meeting candidates and you listed a number of different ways that that candidate might have shown in the past or indicate currently that they are interested in growing and I think this is something that’s, that’s extended even further, you listed even more ways that someone could grow and I’m sure nowhere near an exhaustive list. Yeah. You know, when you describe this, and makes me want to create this really awful matrix, which is like some grid of all the expertise is that a team? We’ve been given the challenges and so on, so that it’s balanced and you don’t? Yes. And you have like a bit of everything. Yeah. And I say awful because you made a really important point that this is very individualized. So yes, you can’t put people in those little slots of that grid. So your your team ends up looking more organic, or there’s a different model that comes out of helping individuals follow that, yes. seems too hard to plan for or control or undesirable to control.
Julia : Yeah, but I think there that’s also where the magic of how you build a team comes in, right? We know while there’s sort of, to me, there’s more of an individual, maybe career pathing that that tends to happen. It’s also about identifying what are the complimentary skills that each member brings to that team and how do you bring in people who bring who filled I don’t To say gaps, because they’re not necessarily gaps, but who add that additional, whether it’s expertise or skill set or experience that complements the other people within the team. So I think that’s where you’re not just helping guide people down their individual career paths, but you’re also playing conductor for the team as a whole and figuring out how do you bring together those different strengths and and leverage them as a group so that you take advantage of how they can complement each other and how they can fill out your practice as a whole.
Steve: Is there anything that we should talk about today that we haven’t got to?
Julia : The ideal question, I think, I feel very passionately about encouraging people who don’t necessarily have traditional backgrounds to jump into this fields. I think all researchers say to some degree that they don’t necessarily have a traditional background when they when they come into research, but I think there’s a lot of strength in it. welcoming people with different perspectives on to your team. So someone who used to be a designer or someone who maybe comes from a more academic background or someone who comes from a completely different application of qualitative research, I think there is an element of resilience and perspective that that lens to a team, which is sort of the sum is greater than the parts. And I think that’s something that is crucial to seek out on a research team because I think it’s strengthens you and strengthens your practice as a team.
Steve: Certainly, I see a lot of I’m trying to transition into into UX research. And what comes after that usually is how do I do that? Yeah, you’re pointing out that it behooves the team to welcome those people. Is there more like how does how does the field create that welcome?
Julia : How does the field create that welcome, I think there’s a lot more we can do to create opportunities for people to get their feet wet. So I’ve started to see a lot more Associate researcher, we’ve created sort of a more junior level within our career level of associate researcher, something that makes it easier for someone to to get experience within the field we’ve taken on interns or an intern, which I think is is a nice door into the discipline. So I think creating some of those formal opportunities, but I think being willing to mentor as well facilitating connections for for people who are just trying to get into the field, even if it’s just for informational interviews, or maybe informal projects is something else we can definitely do. And I think also, as someone who came in from the outside, I think it’s possible to I think we underestimate how much it’s worth really sitting down with someone who has been a week. When you’re on the outside, it’s a bit more difficult to understand what this wall looks like what the day to day looks like. What are the skill sets. So when you’re trying to think about what are my transport skills that bring value to this field that will help me transition into design or into research. I think just being able to talk to someone who is a researcher who works on the day to day and to have a conversation with them about how your skills fit into this picture is completely invaluable. So I think making yourself available to people who are really trying to understand what this work is about how they can leverage the skills and experience they already have, making it concrete for them and helping them figure out how to articulate that value is a big part of giving them the confidence to take the leap and an understanding of how they can they can bring those skills over the fence into research.
Steve: That’s great if some very concrete advice I think for people in the field and looking to get into the field. I think that’s a great place for us to to wrap up here. It’s been really interesting and enjoyable to speak with you Julie. I really appreciate you sharing so much with us.
The post 33. Julia Nelson of MOO first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Chris Kovel, the Head Of Research at First Abu Dhabi Bank (FAB).
I look at needs as proximate needs and ultimate needs. An ultimate need is why the product exists in the first place. And then the proximate need is the experience of using that product, right? So if you take for example, a hamburger, that’s a product and the ultimate need is the hunger that it satisfies, right? We as humans need to eat things and hamburgers are one of one of those things that we can eat. That’s the universal need that we that it solves. But then there’s also the needs of actually getting it, getting to the to the place, getting to the restaurant, then there’s the needs of having a good experience in line, being able to read the menu, being able to take it to go if I wanted to. So there’s these nested needs within the greater need of why the product exists. Both are important. But I think that product teams don’t always take both into account. – Chris Kovel
Show Links* Chris on LinkedIn * First Abu Dhabi Bank * Gavin Payne, Head of Innovation LAB * Stanford d.school * John Arnold and Design Thinking * Lawrence Krauss * Medtronic * Jay Reader * Needfinding by Dev Patnaik * Books by David Kelley * Books by Tom Kelley
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
We numbered eight, a cadre of 10-year old boys, posted around the kitchen table and ancillary horizontal surfaces, awaiting the culinary culmination of this birthday party, when pop and chips would give way to the chocolate birthday cake. The flaming dessert made its appearance and we warbled “Happy Birthday” to the celebrant, mischievously goading him about his simian appearance and odor. And with that, the cake was ours – paper plates and metal forks, droplets of melted wax that we flicked off the frosting, fragments of decorative icing. So, we set to our primal task, inhaling sugar, chocolate, and oh yeah did I mention sugar?
Noticing something about my headlong progress through my slice, my friend paused and looked up from his own cake, curiously asking me, “So…you don’t save the best for last?” This was a new concept to me and I stared blankly, crumbs leaking out the corners of my mouth. His mother pops by to affirm, “Yes, Stephen, we save the best part for last!”
Beyond surprised, I was enlightened. Of course, some parts of the cake had more value than others. A forkful of plain ol’ cake wasn’t as good as cake and frosting which itself wasn’t as good as that mouthful of corner megafrosting. But the big news was that I had an option to eat to an end goal. I could eat differently – portioning, partitioning, and planning – suffering through not-the-best bite, so that down the road a slice I would have the absolute best bite. That would become my reward, that was something to work towards. Gratification delayed was gratification improved.
In a fundamental way, this exchange broke me. Decades later I am still struggling to effect repairs. I absorbed a significant lesson – that even a tiny in-the-moment decision can have consequences, risks, and downsides. This week, even, I noted how, with hardly any conscious thought, I was actively managing how much salsa to put on a tortilla chip, weighing that if I took too much from the little container, I could wind up chomping on dry, salsa-less chips by the end of the meal.
As eaters, we face moments for optimizing (or at least risk-mitigating) constantly. Food designers don’t seem to consider this aspect in their apportioning. Whether it’s Lunchables, or an airplane snack box, a meze platter, or a charcuterie plate, there’s an intended combination (nominally meat or cheese or spread on bread or cracker), by design. But the quantities are rarely calibrated so that there’s a cracker for every morsel of cheese, etc. Of course, we can eat things in a variety of ways, and an extra apple slice isn’t really a problem. But, a solitary dollop of marmalade is absolutely a fail. And so, when designers don’t consider this, it falls to us, the concerned eater, to plan, calculate, adapt. For some of us, that isn’t an enjoyable part of the experience. Indeed, the affordances, the aesthetics in a bundle of base/toppings is that we are consuming a set of choices, a purposeful selection of quantities of elements. So the mismatch is just that much worse when the cues tell us to expect coherence.
Going beyond what may be dismissed as my compulsiveness, this is really about being present. It’s hard to be present with your food, or indeed be present at all, if you consider what may happen in the future, worrying about potential microannoyances. Navigating the pleasure of being present with the pressures of what lies ahead is the burden – and freedom – of being an adult, of constantly having to make choices. At my most hopeful, I might reconsider learning about the-best-for-last less as fracture to my equanimity and more as an important and inevitable loss of innocence.
Here’s another zero-budget episode to share with you. And I want to remind you that you can support Dollars to Donuts by supporting my small business. Here are some of the ways I work with companies.
The first is User Research – I work well in situations where the team knows some things about a set of users, often they each know something different that they strongly believe, and have no way to get from their anecdotal familiarity to an actual framework to make decisions. “Oh, we’ve talked to our customers many times, but we’ve never heard THOSE things from them…” This is often research about people, rather than “a product”
The second is a Player Coach where I’m working with “People who do Research” or maybe there’s a junior research person on a project, where they can use some of my expertise but they are going to own the project from start to finish, so I come in to advise some parts of the process, roll my sleeves up for other parts of the process, depending on what the team needs.
There’s Team Coaching where I’m an available resource with office hours or other format so that people can come with questions, seek feedback, etc. I usually not “doing” anything, just there to talk and review documents
Training is for people who are moving to do more research, this is a half-day to two-day curriculum on planning research, interviewing, analyzing data
A Master Class is for a research team that has a particular area of interest, this is a customized combo of lecture, exercises, and facilitation
And finally, Alignment Facilitation where I’m bringing together a group of people to identify and sort through issues, often around optimizing and improving process
If I can help your team, please get in touch.
All right, let’s get to my interview with Chris Kovel who heads UX Research at First Abu Dhabi Bank.
Well, Chris, thanks so much for being on dollars to doughnuts. It’s great to get a chance to speak with you.
Chris Kovel: Pleasure to be here, Steve.
Steve: Chris, thanks so much for being on Dollars to Donuts. It’s great to have the chance to speak with you. Why don’t we start By Why don’t you introduce yourself?
Chris: Yeah. Hi, I’m Chris. I head up UX research at First Abu Dhabi Bank.
Steve: How long have you been in that role?
Chris: So, First Abu Dhabi Bank is a is the biggest bank in, in the region. So I work in Dubai, and I’ve been here for about 14 months.
Steve: And did you join the organization to take on this role?
Chris: Yes. So within the organization, there is an innovation lab. So fabric first Abu Dhabi bank Research and Innovation Center. And in that position, I head up the research and that position was created and I decided to take the plunge and go out to do By and it’s been a great experience.
Steve: Do you have I mean, this is an unfair question, but you may have some context since this happened before you got there. But do you know what led to the identification of the need for this role creation of the role, you know, the, the conditions that were established that led you to start talking with the organization and join?
Chris: So yeah, the history of the lab or the fabric, It started about two years ago. So, my boss, the head of research for fab guy named Gavin, he was brought on to really try to help with the digital transformation. First of Adobe banks, a really big bank, like, like I said, biggest bank in the region. And in terms of technology in terms of digitizing the products and the services. That was really the mission of the innovation lab or fabric as we call ourselves now. And in terms of research, Gavin kind of comes from a software engineering and design background, and knew that research was going to be very important in terms of creating products and digitizing products. And the other kind of core mission other than digital transformation with that fabric has is kind of spreading a culture of innovation within the bank. And that means empowering other departments, not just the innovation lab to be innovative, to think kind of like designers to be problem kind of obsessed, and all that and kind of requires research and hopefully we can get into kind of how that how that happens.
Steve: Well, that’s no time like the present. So how does that happen?
Chris: Yeah, so there’s a lot of different ways to kind of come at it. At least from my perspective, in terms of kind of developing products, you know, I think when research we like to follow design thinking principles comes out of the D school, most famously in Stanford. And but it traces all the way back to a guy named john Arnold, who create who came out with a kind of a book pamphlet, I think, in the 50s, or 50s, or 60s, called creative engineering. And it’s just a way of kind of problem thinking about problems, right. And, and, and needs, right. So when you think about problems when you think about needs from a customer point of view, that will allow you to kind of frame issues and then solve them through products and services. Most of the time when research kind of gets embedded in my opinion into this design thinking framework either up front and towards the end of the product likes, lifecycle. So I kind of think of it as bookends. And up front or kind of upstream is a lot of times when you’re trying to find a need and then downstream is a lot of times just smoothing out the experience or smoothing out the product through testing we do both at the Innovation Lab. But I’m very passionate about the upstream kind of research because it’s really where these the tricky problems are. You know, trying to understand human motivation is not a not a simple thing sometimes
Steve: From your perspective, or maybe from all the folks in fabric, you know, the value of understanding human motivation, in terms of how it applies. To the kinds of products and services or decisions that have to be made in a financial institution, maybe that’s clear to you. Does, how does the rest of the organization see that relationship between, hey, we need to understand human motivation? Because here’s what we’re going to do as a bank.
Chris: Yeah, yeah, it’s a learning experience. To answer that, that question because that was something that I kind of have found, wherever I’ve worked throughout my career as other people who kind of don’t, you know, live, you know, live the UX kind of world of the user research world or really being obsessed about the customer and making the product kind of fit the customer’s needs. is a difficult one. I mean, it’s about education. It’s about culture, you know at fabric. We have four core components within the You know, the lab. So one is research. The other one or one is research. One is accelerate, which is kind of more about more design. One is FinTech. We can get into that if you’d like. And then the last one, and arguably the most important one is culture. And that was that it’s just an education piece, right? It’s getting people to come and walk through the lab and show them how and the pods that we do user testing. And with, to kind of do roadshows to kind of get the word out there. We’ve done hackathons where we bring people together for kind of, you know, interactive brainstorming. We do talks and kind of fireside chats about design thinking and why they why people should care. So it’s, it’s difficult, right? Because culture is kind of a amorphous, intangible thing and to change it. It’s hard to change it if you can’t even put your finger on it sometimes. So it’s a work in progress. But at the end of the day, it’s it really is just about people and about just talking with them and kind of getting the word out there. And, you know, in the beginning, the bank kind of, you know, we had some Slow, slow days to begin with, but slowly but surely after that we started, especially within research after we start delivering value. And they kind of see the power of answering questions and have been kind of evidence based, then I think that there’s two ways I think it’s the education and then it’s also delivering high quality real projects.
Steve: One of the things that, you know, we talked about in researches is in addition to, you know, doing quality work is, is having people is having influences having people sort of be willing to hear that information that maybe challenges their beliefs, and then ideally, take some action based on that. And so, you know, you’re talking about driving this change, what have you seen on that side of things in terms of, you know, I mean, believing engaging with taking new perspectives on and then, you know, having that influence the decisions that are being made.
Chris: Yeah, so, a couple things on that point. So first off, you know, it’s, it’s never, you know, most of the time, I probably argue, arguably all of the time when a group comes to us to get some research. It’s usually a question that is pretty polarizing. You know, it’s research and also in many ways, I found is kind of a way for people to kind of settle internal conflict or settle internal debate. So, you know, these the results that we provide really matter and they really kind of, you know, take it to heart, but that’s a good thing. But the other kind of bad thing sometimes is that when you’re in the room, and you’re kind of, you know, for example, if you’re testing an application, and you had peep designers and product managers kind of working really, really hard to making things good. And in many cases, the research is the questions that we ask kind of the data that’s presented in many cases is really negative and sometimes and not negative. But that it’s kind of supposed to be that’s why that’s why the research is there. And to kind of manage that manage that in the room is something that it’s a it’s a learning experience, but You know, we’re kind of, we’re kind of human we get we get defensive, or the people who are in the room are human, and they get defensive. When, when the research is being is being presented sometimes so that’s just something you have to be kind of conscious of, you know, I tell that the younger Junior guys, that’s, you know, there’s a certain way to kind of couch the reports and the and the data. And when kind of infighting begins in the room, you kind of have to just take a breath, and really just look back at the rely on the methods. And that’s why, in my opinion, kind of the methods and the way you present and the quality of the research is so important because if it’s done well and it’s executed properly, then the data is the data and you can kind of just kind of, you know, sleep well at night, even if it’s caused some internal strain So, so that’s kind of kind of the first point The other one about taking them taking it forward, right is a kind of an interesting one, it’s me and me and Gavin, my, my boss, we kind of go back and forth about this a lot. And in my kind of philosophy is that the data is the data. Sometimes they want us to recommend, and sometimes they don’t. But in terms of solutioning, that’s kind of their I mean, we’re the researchers, right? And they kind of other product owners. And if the if the research and the project is scoped up in a way that’s pretty narrow, and they just have a couple of questions when we kind of provide that that information if they kind of because of politics, or because of infighting or because of, you know, other bulwarks. If they don’t take it forward, then it’s kind of out of out of our hands, which is kind of arguably good and bad. I don’t know how to how do you see it?
Steve: Yeah, I mean, I think you’re hitting a really This is a personal hot button of mine. You know, how do we measure our own success? I’ll tell you I had a kind of a galvanizing experience a number of years ago with a friend of mine who had worked with a number of design agencies. And we were just we were, we decided to meet and share our capabilities presentations as a way of just learning about each other and learning from each other. These were public, public documents that we could share comfortably. And, you know, I had all these stories were research, it uncovered something really insightful and potentially impactful to these business areas that my clients were working on. And I kept saying, you know, unfortunately, this didn’t happen or you know, they weren’t ready or they decided not to and you He kind of stopped me and said, like, Steve, you’re not in the business of them shipping a product, you’re in the business of highlighting opportunities. And that, and that, that was really grounding for me. I think, as I, you know, I want my success stories to be success stories and not failure stories, but we can’t control everything. You know, and I think it’s, it’s a little, it’s a little distressing to hear researchers tag their own success to a thing that’s ultimately out of their control.
Chris: Yes, yeah. That’s Yeah, it’s interesting. I, I agree. I agree with all that. Everything that you said there. And I think it’s kind of if you look at kind of the different levels of the macro and the micro, it’s like, I think if you’re, if you think about it, in the macro level, right. So the product that I’m researching isn’t going to be a success or failure. Then that’s, that’s definitely It can be kind of disheartening, right? If you do all this really good work and then the, the product kind of dies or it goes away or it takes a different direction, a direction that the research isn’t what you know, kind of away from the research findings, but if you look at it in the macro and just kind of, you know, look at each project as each project and as long as you know, you hit this hit ticked all the boxes in the scope, and you kind of went out and you know, you did the, you know, the research the way, the way it should, should have been done the way that you wanted to wanted it to go and then reported it and then you can kind of I think rest easy if you just kind of look at it with kind of net very narrow with kind of almost with blinders on but it’s hard sometimes. Right.
Steve: And I, you know, it changes and I think you’re kind of alluding to this. A little It changes based on, you know, how emotionally invested we are as individuals. And, you know, I work as a consultant. So, I love my clients and I’m super engaged, but at the end of the project, you know, they have to move forward without me so I have a, I have a way to detach my success or failure or, you know, how identify my own success or failure because project ends, even if a relationship continues. There’s a point where the that’s, that’s over, you know, you are part of an organization even though fabric is, you know, a center. It’s in the name. I wonder if I mean, I wonder if different researchers just based on sort of where they sit in the organization or you know, how they feel about their own ownership of that product or their own ownership of that team. I wonder if their own assessment of what success looks like very for them.
Chris: Yeah. I wonder if for me if I’m kind of okay with just delivering and then and then kind of moving on to the next thing and kind of judging the success project by project, not kind of the overall life cycle. I’m wondering if I it was kind of imprinted on my brain kind of from the early days when I first got into doing research. So I, I worked originally or fresh out of college in Manhattan, at a human factors engineering firm, very small, kind of boutique firm. And we would work on interesting and cut with big, big clients like Google and Amgen, and others, Nike, but it was literally like the brief would come in and you know, a month or two and we would do the research and then we wouldn’t hear anything because of you know, confidentiality. situations and what have you. So it was very much just like here’s the here’s the findings, here’s the data, and then on to the next thing, and I got very comfortable with that. So it might just be I don’t know, maybe how you get your start. But I wonder how if there are teams out there where researchers are kind of, you know, embedded into the whole product from kind of soup to nuts from cradle to grave, and if they prefer that, or if they prefer for me I love one of the reasons why I love research is because one week you can be working on digital wallets in the next week on a personal loan app. And you know, I love I love that for the versatility and the diversity of working on many different types of projects and kind of moving on to the next one.
Steve: Yeah, I’m thinking of several things here at once. Now. I mean, it’s interesting. You know what? Yeah, you make a really good point that, you know, sort of how you were raised up, how did you get your start and what sort of model formed for you versus, you know, the experiences we had versus someone that’s, you know, part of a team and maybe sees themselves as being tied to the success of that product. And, you know, I wonder, and again, I’m super biased because of the, you know, how my, how my approach was formed and how I operate now. But I wonder about, like, what’s good for research, and I feel like the truth to power role that we play, you know, is essential, and, and being self contained or self sufficient, like we’re going to do great work, and then we’re going to help you as much as we possibly can but that next part is up to you. That seems that mindset to me seems good for truth to power, you know, role but, you know, what I’ve come to admire with people who are embedded is you know, and maybe, you know, fabric is different than my business, you are part of a larger organization. He talked about culture, but, you know, the being embedded and working long term with people kind of day to day sitting with them. You those researchers, they really know those people, they, they can influence. I mean, literally from the inside, right? They’re part of that team. So I don’t know what you’re seeing with that fabric as you build relationships.
Chris: Well, for me, I think I’m kind of torn right because I can see the value and kind of embedding researcher through from the beginning of the product to kind of The end to when shipping or you know, when it’s becoming live. But I think there’s also value in kind of the outsider kind of role that sometimes we play as being brought into a new project or on a new product team. And kind of not having all those internal biases and kind of the curse of knowledge, right, that research kind of, you know, it’s just another it’s another set of user eyes, new fresh eyes of seeing, seeing the product and kind of looking at it from different point of views, different point of views. I don’t know I think that there if you really kind of take a step back and kind of look at a product or a service lifecycle, and no two are the same, but I really like kind of following the I mentioned it before the D school design thinking kind of philosophy and if you look at that The circles there’s five different steps or five different kind of parts to the design thinking model, kind of empathy is usually up front, and then testing is at the end. And I think having researchers in involved in the process for those two things is essential. I feel like it’s a lot of times, you know, we people and businesses and, you know, marketers or product people look at research as kind of a nice to have, and I really do kind of think that it is at least in those two areas upfront and then at the end, it really is essential. It’s, it’s a must have.
Steve: And what’s what is the research doing for the process or for the team or for you know, what’s the what’s the impact of research in those two different stages. of the process.
Chris: Yeah. So I think the easy one is the testings kind of down downstream, right? So when a product kind of comes into the business from and starts having, you know, getting some traction and people start becoming, you know, working on it. You know, usually a prototype is made, it depends on what kind of product for us doing UX research, most of the time, it’s digital, it’s, but, you know, that’s not to say that it’s always the case, it could be a physical product, it could be an experience, it could be a service of some, some, some sort. And usually, after a prototype is made, then the researchers come in and that’s when that person prototype is kind of smoothed out. So usability test tests are kind of bread and butter for UX researchers. But in many cases, I think upfront, you know, the research hasn’t, you know, the teams don’t bring it in. The teams don’t bring researchers into the process to begin with. And I think that there’s a real impact and a real value that away for, you know, product teams to kind of set themselves apart, and making sure that the product that they’re creating has been tailored to a specific need. I mean, I kind of see it’s, you know, some simple terms, that a product is just a sad product is basically satisfying a need and the way that you find needs, and it’s a very tricky, tricky kind of business, but the way that you find needs in my, in my opinion is through empathy and through going out and kind of applying the methods that we as researchers use to kind of, you know, make things a lot clearer and distill down the essentials in terms of what the problem areas are, what the need areas are no jobs to be done. framework, I think gets this right to where, you know, it’s kind of I was just watching a webinar, I think, and I really like this, this phraseology that they use they, they were thinking about milkshakes. Right? And like, why milkshakes were being sold more in the mornings, during, during the workweek than any other time. And the way that I remember that the way it was framed was, you know, why do people what are they hiring, a milkshake for? And I really like that kind of reconceptualization about that. Thinking about products and services and what do you Why are you know, hiring your computer? or Why are you hiring your water bottle? And I’m thinking about it that way I think really can kind of generate interesting new ideas and that’s kind of the heart of innovation in my opinion.
Steve: When you at fabric, have these initial conversations with teams at the bank. What are they? What are they looking for? Like? Do they have a product? Are they looking for needs? How are they kind of framing that initial interaction with you?
Chris: Yeah, yeah. So most of the time they come they have a product that’s kind of either out there and kind of in the wild or, or an MVP or a prototype and they want us to kind of help them bring in users have them you know, put that put that product in front of them and kind of observe their behavior and kind of get back to them with the findings about whether they the users kind of could understand whether there are any kind of issues with the content issues with the lexicon, this these kind of things. And so, you know, we work with the digital team, we also we work with other departments within, within, within fab, on kind of lack of productivity. And so, a couple of months after I started, I started interacting and working with the head of marketing. And we kind of created a market research branch off of fabric. So now, any type of market research initiatives come through us and we, we test ads we do kind of more discovery type work about products, you know, for example, there was one credit card. So fab has 22 different credit cards if you can believe that. And the one that was performing the worst was a card called gems card and we were able to bring in people who were using the card and kind of getting their sense about why they’re not using it more. Why kind of other confused about the point system or you know, kind of just getting them to kind of talking and Having an informal kind of discussion about problem areas and things that we can improve and kind of the whys as to the underperformance. So yeah, was that Did that answer your question?
Steve: Sure. And it raised another one. So that that says we’re going somewhere we want to go. So I guess that raises a question for me around the definition. So I think you’re describing that as a market research project. How do you define or delineate what’s you know, what’s UX research? What’s market research? Yeah, think about it.
Chris: It’s interesting. I think of it all as kind of the same thing. I mean, this is now kind of being in the industry for as long as I have been. I’ve seen I’ve kind of seen different titles or worked with the from corporations, you know, things like human factors, things like user research, UX, kind of design designers. And I think that at the end of the day, it’s really all kind of coming, coming from the same place or I guess the it’s all looking towards the same thing when it comes to all these, you know, you know, human factors or UX or you know, market research. Just at the end of the day is trying to answer questions with data from your actual your customers. And it’s trying to get at the need, I think it’s trying to find needs and try and discover needs, and then trying to help the people who make the products or make the services, make them better, make them more, not just making you know, Making people that’s not just, you know, listening to people about their problems of, you know, with, with a product or their problems with their day to day life, but really trying to understand the motivation and understand their feelings and their desires and kind of satisfying them with services and, and products.
Steve: So I want to ask a different question, following something you said before you talked about the sort of four areas and where culture was like a really important one. Yeah. And so you, I mean, from where I sit in California, I’m curious about culture in your organization, because not only are you know, in a specific industry, which, you know, based on my experience, like banking, has a certain culture, sort of historically around and you know, maybe that’s very much an American perspective. But, you know, a regulatory a regulated industry with certain history means there’s a certain approach to decisions and a certain approach to, quote, risk and so on. But you’re in a very different region of the world than I am. And I’m so and that makes me wonder, you know, what observations do you have about, you know, the regional business culture? And how that how that responds to innovation? And the changes that innovation is asking for, as well as financial banking kind of services culture as a factor that characterizes the culture that you’re working in.
Chris: Yeah, so I think culture is changing the culture is one of our core missions with one of fabrics core missions. So it’s a work in progress. You know, I think fabric and then fabric versus fab. I think within fabric, we kind of have in our own culture, it’s a little bit different than kind of the corporate, you know, buttoned up tie wearing jacket wearing kind of traditional stereotype of the bankers. I don’t see too much of that, that stereotype within the bank, you know, meeting with people in different departments. Um, but yeah, I think work I’ve worked in the corporate setting a couple times. And I think kind of, you know, there is there is a difference. There is a division between kind of the traditional, you know, normal kind of roles that you would think about when you think business versus kind of the UX or user research designer type people. And it says it’s, it’s something that yeah, it’s I feel like there should be more in integration, I would, I would like to see more integration, you know, in the future and to just from a business point of view, I think it’s good business to kind of have people like us at the table. And in many cases, it’s there. You know, we’re not we’re just not there yet. So, and it’s something I think that organizations that, you know, struggle with about how to kind of you hear a lot of people, you know, a lot of people in the industry, I don’t know if you get this as well, but talk about when they’re working in a big corporation like to evangelize, you know, that user research or just kind of UX in general, has to be evangelized, right. And kind of the, it’s kind of a little bit that you know, first off, it’s not great that it even needs to be evangelized. And then the second part is like How do you actually what’s the best way to, to evangelize? Yeah, we talked a little bit about, you know, it’s educating, I think that the way the approach that we have is kind of through education, you know, events, these kind of kind of cultural events and then and then just producing, you know, kick ass work.
Steve: Can you What do you think would be different if, you know, you’re at fabric within fab, but you were, this was a company in Manhattan, or this was a company in London. Do you think there’s any regional shifts that you would see?
Chris: Yeah, it’s an interesting question, um, regional shifts. So you know, it’s funny, I when I first moved out here, I was expecting it to be a lot more of a culture shock than I was like, then it actually was so. Yeah, I’m sure there would probably be some differences without a doubt, less sand. But I, I think I think for the most part though, it’s you know, it would there wouldn’t be much it’s funny Dubai actually has I think it’s 8080 or 85% expats right so there are very few locals and nationals UAE nationals. So the bank is comprised of a lot of people from a lot from all over the world, right so it’s a very diverse very multinational community. Not a not too many people from the States but a lot of people from England, a lot of people from Australia. India is there’s a high percentage so yeah, it’s kind of a melting pot, and it’s a really cool and interesting environment to work. But I think at the end of the day, yeah, it’s People are kind of people. So
Steve: do you have any sense of, you know, the business community whether it’s you know, outside your organization, but in the region you know, what the awareness or adoption of, you know, user experience user experience research, you know, elsewhere outside of fab.
Chris: Yeah. So it’s something that’s that is, I think taking off and growing within the within the region. I know for us personally, what we want to do is fab not only serves the UAE, but it also serves other countries in the MENA region like Egypt and like Saudi Arabia, and others, and I know that we as fabric have kind of goals to take certain elements of our of our toolkit in our service Kind of export them into, into these regions, right? So research being one of them, I think because of its kind of universality. And its way for, you know, you can kind of use it in many different contexts. So, so that’s definitely exciting. But, you know, it’s funny, I think user research and UX in general, is kind of a very niche, you know, industry. So even within the United States, you know, a lot of people don’t really know about it, whereas in this region, it’s almost like I think there’s a lot of people who don’t know about it about it as well. But I think that there is receptivity, they’re much more receptive to kind of embrace these kind of new and maybe kind of non traditional ways of approaching work. And I think that’s really exciting.
Steve: Yeah, sounds very positive. And we’ve talked in and around this a little bit, as you’ve talked about some of the ways that, you know, that fabric is working with teams, you know, throughout the bank and so on. When you start working with these teams, what kinds of disciplines or functions or roles do you most? Or is there a typical, you know, set of rules that you are collaborating with most directly?
Chris: Yeah, I would say I mean marketing for one because, you know, fabric and marketing, there’s kind of this dotted line between fabric, fabric reach research team and then the market research that we do. So that’s definitely a division that we work with very closely. We started working with product a lot. We’re doing some kind of foundational fundamental research project that we’re really, really excited about, on personal loans that hopefully, it can kind of spread and we can do this this kind of research, research stream with all different products that really start from the ground, ground floor and get those needs and kind of assess and evaluate the current state of the products and then kind of kind of move forward involved in evolving from there. We work with the digital Team A lot, a lot of times that the guys will come and request meetings for, you know, to do kind of benchmarking testing, like I said before, a lot of usability testing, touchpoint testing, sometimes card sorts, every now and again, some kind of upfront research about kind of ideas and content concept testing and this type of thing. But for the most part, it’s kind of its product, it’s the product testing. Let’s see this. Yeah, I mean, we’ve been working with a lot of different departments, which has been great. I think we had 25 or 28 different research projects completed last year. vast majority of them was from ppg, which is personal Banking Group. We have corporate banking, we haven’t been working with corporate all that much. But that might be changing in the next couple. next couple weeks, weeks and months ahead. But yeah, I think that’s weird. So there’s also a team that that that head that basically is single product team with it’s called pay it and it’s a it’s a digital wallet and we’ve been doing a lot of different types of ethnography work, you know, competitive landscape research, and you know, customer interviews and these types of things. So yeah, that’s been exciting too, because I think that the digital e wallet space is a new one. And I think it can be. It can it can there’s a lot of value in kind of thinking differently and really trying to figure out the needs of people and why kind of a digital wallet exists, or why they exist, and then kind of helping them shape and smooth the experience.
Steve: And are there situations where fabric is proactive, say, as opposed to responding to requests from a from one of these teams initiating some effort, some program from within fabric does that happen?
Chris: I think at the beginning, once we started kind of ramping up, we were so busy that we were just focusing on the products that I’m sorry, the projects that were that were kind of coming to us and There were times when we had to kind of deny or you know, tell people sorry, we’re kind of we’ve hit capacity, we have no more bandwidth. But I think with this kind of Corona situation, we’ve had a time to kind of level set and start thinking a lot more about, you know, spinning out projects internally. The prot, the personal loan project that I just that I mentioned in my last my last response to your last question, was something that we kind of were developing ourselves and when the product came to us, they were just like, What’s on your mind? What what’s kind of some, some new stuff that you got? What’s kind of what do you guys thinking about how, you know, we kind of want to kind of work with you like what can you kind of offer us you know, there wants to want to have this relationship and that was something that was internal, an internal idea or an internal model, turtle project that we kind of then took off the shelf and applied it to, to product when they kind of came knocking. So that was kind of kind of cool. I you know, it was to give a little bit of a background, it’s a new and innovative, new and innovative way to think about personas. And what we really wanted to do was we wanted to I mean, I think most people within the industry when they hear persona, it, they kind of, you know, they’re, I think it’s a very ugly, ugly words in some people’s minds. You know, but I think that there’s, there’s, there’s use and there’s value to them, but I, I see them as being kind of a little limiting or limited, right. So the idea for this project that we’re working on with product is kind of to do the personas the traditional way, which is where you go out and you get together. Either through focus groups or in depth interviews, a bunch of a bunch of users, a bunch of customers within that specific product. You talk to them about their lifes, you talk to them, you would get you understand, you know, their demographics, you understand their kind of, you know, their motivations, why they’re the other products, they’re using, you know, kind of a comprehensive 360 degree view. And from all these interviews and focus groups, you know, personas kind of emerge out of them. So the idea is, is to kind of take that traditional method and kind of mash it together with a method that’s kind of been now being developed over the past couple of years with using machine learning and AI by feeding all of you know, customer data into And an algorithm and have the algorithm then create the personas, you know, mathematically, and then if there’s some way for us to kind of merge those two, where you get kind of the, the best of both worlds of approach, because you know, in many cases, the traditional UX. Personas through research are kind of limiting. I mean, you can ask a lot more questions than you can with a datasheet, you can get a lot more kind of subjective internal motivate motivations, you can kind of capture those but the power of generalizing that is not the same as the power that the machine learning model has. And if there’s a way for us to kind of, you know, understand that those two are, there’s a way for us to kind of push those two methods together. I think that would be a really huge impact for the bank. So it’s kind of a combination of that persona piece with this idea of service design, which, in many, in my, in my opinion is very, very much a just another method of, of user of user research, where you go, and you talk to people, this is for personal loans, you talk to people about their experience, you know, taking out a personal loan, so usually people who, within the last, you know, three to six months have taken out a personal loan with fab. And you kind of map out the whole journey from kind of conceptualizing, to kind of get you know, that kind of motive motivation moment of like, I’m gonna need a personal loan, why they need one, and then going through the experience of, you know, coming to fab going, setting, you know, signing all the documents, and then, you know, lastly making the payments and kind of having that map and looking for gaps looking for pain, areas of pain, things that can be improved upon. And if we kind of have those two models, and we do those that piece of work for every product in the bank, we think that, you know, hopefully, we can really change the trajectory of the products and make them more about kind of user needs and user solving user problems then, then kind of how they’ve been set up so far.
Steve: That’s an amazing approach. I haven’t heard of anything like that before. You’re talking about? Yeah, just like you said about market researcher kind of blurring the boundaries or eliminating the boundaries between things that were sort of separate islands. And here you’re pulling in two very different kinds of things to create. One, one better approach. It’s very exciting to hear Yeah, it’s
Chris: it’s funny you say about marketing because I remember when I was meeting with so the merchant team came to us came to fabric. And we did just kind of a normal, you know, textbook, piece of user research on some credit card designs and then also asking them some questions about what would get them to sign up for a credit card. And, you know, they were really happy with the results. And then they were like, okay, we want to kind of grow the team, we want to kind of have you hire some dedicated researchers and kind of have this this dotted line between marketing and between UX and I was kind of like, wait, I, I don’t I don’t know like, I don’t know marketing. This is not my world. Like I don’t know if you have the right guy, but the more I kind of learned, and the more I talked to the marketing people and the kind of the decision makers, I learned that you know, there really is not bright lines between product and marketing, and, you know, and research and just kind of thinking holistically about how products Come about thinking about, you know holistically about the economics, you know, why? Why do products succeed? What makes what are they? For what, you know, what makes a good one? What makes a bad one? You know, at the end of the day, it all starts with understanding your customers. And that’s what we as researchers, do, we try to understand all the different facets of people and what would make them buy and what even what is it what is a product doing? What purpose does it serve? When you start there, then I think the marketing and the product and all of the those other parts of the business kind of just take care of themselves, it becomes much more easier to navigate and make decisions based on this fundamental understanding of the people that you’re serving.
Steve: And right now that thing that activity is But we call it research. It’s, and you know, people that do our job are the ones that do that. Yeah. And, I don’t know, I feel like in some ways, it’s a historical accidents like you when you describe, you know, this is what the process should be. And this is, you know, in order to get to these ideal results, and here’s the activities that kind of started off you know, the way that research is as a discipline has done evangelizing has done has, you know, invited other people to participate and sort of said, Hey, you know, everyone needs to do this. I mean, if you mentioned the D school right there D school is like classic for saying, Hey, we’re going to train everybody to do this not people with you know, the researcher title. This is everybody’s job.
Chris: I mean, I think it it’s a it’s a way of reopening Kind of reimagining kind of what businesses are doing, and why they exist and what their purposes are, right? I mean, it’s Yes, I think at the end of the day, it’s like, it has to do with monetary gains, and, you know, stakeholders, you know, you know, making sure your stock is going up and these types of these kind of economic questions, but, you know, I think, you know, when you say things like product, we talk about products, we talked about services, we talked about needs, and they can be really defined kind of in very broad strokes. And when you when you define them in broad strokes, then then the power and the use and the kind of impact and the relevance of understanding your customers, which is what as you’re saying is what we do, becomes, I think Paramount, right, because that’s the way that you can separate yourself from the competition. It’s not just about innovating for innovating sake, right, it’s innovating to me is to just find a need and solve it in a new way. And if you can do that, I think you’re, you’re poised to have to have success. The problem, of course, is some of the bureaucracy, some of the politics that kind of happens in you know, in the different layers within the business. And also the kind of tricky thing of, you know, researching humans, it’s, I love this quote, it was a physicist, guy named Lawrence Krauss said, I do physics because it’s easy. Right? Well, it’s clearly it’s not easy, right? But I think what he was getting at is when you’re when you’re researching humans, they’re tricky. They’re complicated. It’s a little bit messy, right. He kind of love He was his point was the elegance of physics. Right? You can describe parts of the world in very elegant mathematical ways. And it will always behave that way. Whereas people were unpredictable. And kind of getting at the root causes and those root needs and those root desires is tricky. But if you get them then then i think i think everything falls into place.
Steve: I would love to change directions. And, you know, hear a little about what your trajectory has been, you know, how you found research and you know, what kinds of roles you’ve had that kind of, you know, we all know about Manhattan a little bit, but yeah, describe more of that story.
Chris: Sure. So I kind of I kind of fell it fell into this accidentally, actually. And yeah, I love this question asking Others in the industry this one because it seems like you know that there isn’t a clear path into kind of the world of UX in the world of user research. Which maybe would be maybe that’s the case for all industries. I don’t know. I have a hunch that this one is a little bit different though. People get into it through kind of untraditional ways. So yeah, I was. I was going to grad school in New York. And I was getting interested in fascinated with science and I started working in a behavioral science lab. And I graduated, and I moved to Brooklyn and I didn’t have a job. And I was kind of running out of money and I was and I wanted to do research. I really liked the idea of kind of trying that out as a full time kind of career and the first job I interviewed ATS was a, like I was saying a small boutique firm on the Upper East Side of Manhattan that did Human Factors engineering research. And I saw the research paper the human factors for I had no idea. And it was like kind of a pretty low level, entry level position. And I went interviewed, I got the job. And I kind of just fell in love with it like, and I never really looked back since and kind of this idea of merging science and research and data with design was something to me that I never really thought about and, and kind of, you know, you know, for me when I when I thought when I think of design, I always would think of kind of just someone sitting in a room, you know, you know, twisting their moustache and coming up with ideas. I didn’t realize that you can actually kind of use science a US kind of methods, research methods to influence how products Shape products, right? Which is really what a lot of times what human factors engineering does. It, it’s, you know, if you if you looked at the D school model, it was the kind of lives in the testing area for, for a lot of for most of what it does. So I did that and I kind of after six months of working there, it was, like I said, a small a small firm, the my direct reports left kind of suddenly and the next in line. So, she reported to the CEO, a guy named Charles Morrow who has been in the industry for 45 years. And then so that left a big wide open role that he you know, said that now it’s mine and I was kind of again in moment of crisis like I don’t know if I can if I can handle this right like I’m only six months in, it was a very much like a sink or swim moment. And now here this, you know, person who really didn’t go to school for engineering and research is now kind of managing these huge projects for Amgen and Google I was I yeah, it was definitely harrowing, to say the least. But I kind of kind of made through it. And then a couple of years after I decided to move to the Midwest and I took a job working for more, more human factors, stuff working for a company called Medtronic that makes medical devices I worked in the neuromodulation. Department, working on a couple of different products kind of flew around the world, around the country, mostly talking to doctors, kind of getting their feedback and trying to help The shape certain medical devices. And then I kind of switched gears and got into the world of UX and UX research, actually that same mice the same directed my direct report of my first direct report, Emily, lady named Emily actually suggested we kept in touch after she left and she suggested that I start looking into this, this thing called UX research, it’s very similar uses the same methods of kind of the methods of that we employed for human factors employees. And I kind of thought that it was Yeah, it was I was getting a little bit bored of the physical products a little bit bored with the medical devices. You know, tech was kind of booming and it was kind of an exciting place to work. I thought that it’d be cool to kind of open up my tools. And start working on screens. I did a little bit of screens, kind of within the human factors, but definitely not as much. And I started then working in in a few different innovation labs, I worked for a company in Minnesota to call I’m sure you’ve heard of it Best Buy, how many people here have heard of it, and that was my last. My last role was Best Buy was before I moved out, moved out here to Dubai. To do to help them kind of creates UX research department within an embedded into the bank, which has been lots of fun, I think.
Steve: So outside of fabric and the work you’re doing for the Banco? Are there other things that you’re working on in your profession?
Chris: Yeah, so, um, since the past year, year and a half, I’ve been kind of kicking around an idea concept of trying to think about, you know, things like UX and jobs to be done theory and market research and human factors, all these different kind of disciplines that seem to be kind of putting a lot of a lot of weight on understanding the customer bringing the customer into the design process. Being customer obsessed. And it kind of kind of all kind of filled. I was thinking about writing a book about you UX research more technical, then I was working in Best Buy. And I kind of struck up a friendship with a guy named Jay reader who was a graduate from the Stanford D school and he kind of introduced me I broadwalk brought him the idea of the book and he kind of introduced me to different material, different kind of literature material that the D school was kind of putting out in their philosophy which I had heard of design thinking but I hadn’t really dove too deep in it. So we had a lot of interesting conversations about you know, design thinking and you know, books like need finding by dead panic, and the books by, by the Kelly brothers kind of I started kind of getting interested in that and, and design thinking I felt was very cool approach to kind of what user researchers were doing already, right when we do user research. We’re trying to make better products and we’re trying to come design help, you know, collect data to help designers and to help product people shape products and make and make really better experiences. And, you know, I found I found that, you know, it was really hard for that will that you know, a way that when you look at the design thinking framework, in many cases research is not so much talked about when you get kind of down into the details they do. Talk about how empathy which is kind of the first pillar the first step requires kind of a journalistic kind of anthropological mindset kind of going out and observing people and asking them questions and watching how they use the product and seeing the context and all of these kind of kind of more research stuff. And I thought that yeah, that this was a really cool model, but I thought that it kind of needed a little bit. It was kind of maybe outdated, a little bit. In many cases, it was it was used for physical products. And now I think, you know, because the world is becoming digitized. So there’s so many now, so many services and products that are now on a screen. And I think that kind of user research, especially for digital products, you know, I think that looking at them as a product I think in at the end of the day, I think some people when they look at, you know, a journey for example, have, you know a bunch of screens to get you to sign up for a loan that that is actually a product if you define if you can, you can actually define it that way. And when you start looking and seeing all the different products, you know that almost if anything, is customer facing and you know, you can use it or, you know, Facebook as a product or Candy Crush as a product, all these things, you can look at them through this kind of new lens as Okay, this is a product, this is a service, then the kind of the design, the D school design thinking kind of applies. And the problem I think one of the problems with the D school is that it just doesn’t go far enough into the methods and into the kind of deep dive that user researchers kind of do the kind of, you know, the ways that we kind of understand problems and the knowledge that we build through our methods is something that can Really, I think benefit the upfront beginning empathy part. So if people, and this was kind of the kind of the theory or the thesis of the book is that if you do your due diligence, and you do your research upfront, before, before, before you even have an idea about what the product is, that will actually help, you know, go waiting out into that kind of unknown area, and kind of relying and believing in research. That, that, that that’s the way to kind of kickstart these products, and then they these projects. And that’s kind of the kind of the idea.
Steve: So where were you at in that process? And do you have a title like, Is there a place we can find out more?
Chris: Yeah, so it’s still pretty new. I’m kind of it’s all outlined right now. That kind of the writing is began, but very slowly, so it’s the tentative title is the universal lens. And yeah, yeah, hopefully I, you know, I can kind of get it get it to completion I kind of have some more time on my hands now with everything that’s happened with kind of the work slowing down because of, you know, Corona and, and all this and social distancing. So hopefully by next year we can I can have something to share, you know, with the world so we’ll see.
Steve: What is the universal lens?
Chris: Yeah, so the, the universal lens is kind of the metaphor is you know, looking at the world and as I was rambling on, you know, before, if you look at the world as everything is a kind of a anything could be a product or service and kind of The way that that products are just basically solutions to needs, then that is kind of the universal lens that this kind of this kind of framework is applicable to whether you’re designing a medical device or your you know, you have a shoe running shoe or a bank app or a chair, it’s all kind of at the end of the day, understanding user needs. Then from based on what those needs are solving them with a service or solving them with a product that will that will at the end of the day, hopefully make that that product successful. You know, we talked a lot about experience. And I like to use the term smoothing the experience. But a and many, many times when you know we as researchers work on these projects, the Product themselves didn’t really seem like it was vetted. It seemed like it kind of like a you ask the question, what is the product for? And, you know, I think that there’s kind of I look at needs as, you know, proximate needs and almost, you know, kind of like. So proximate needs an ultimate needs and an ultimate need is kind of why the product exists in the first place. And then the proximate need is kind of the experience of using that product, right? So if you take for example, a hamburger, that’s a product and the ultimate need is the hunger that it satisfies, right? We as humans need to eat things and hamburgers are one of one of those things that we can eat. And that’s the kind of the universal need that we that it solves. But then there’s also the needs of you know, the needs of actually, you know, getting it, you know, getting to the to the place getting to the restaurant, then there’s the needs of having a good experience in line, being able to read the menu being able to take it to go if I wanted to. So there’s this kind of kind of nested needs within the kind of greater need of why the product exists. Both are important. But I think that product teams don’t always take both into account.
Steve: Well, Chris, thanks so much for being on the podcast. It’s great to hear what you’re working on what you’re thinking about, and just some of these bigger picture ideas, I think are really very interesting. So I’m thanks for taking the time.
Chris: Thanks, Steve. It was a pleasure.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 32. Chris Kovel of First Abu Dhabi Bank first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Chris Kovel, the Head Of Research at First Abu Dhabi Bank (FAB). I look at needs as proximate needs and ultimate needs. An ultimate need is why the product exists in the first place. And then the proximate need is the experience of using that […]
The post 32. Chris Kovel of First Abu Dhabi Bank first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Chris Kovel, the Head Of Research at First Abu Dhabi Bank (FAB).
I look at needs as proximate needs and ultimate needs. An ultimate need is why the product exists in the first place. And then the proximate need is the experience of using that product, right? So if you take for example, a hamburger, that’s a product and the ultimate need is the hunger that it satisfies, right? We as humans need to eat things and hamburgers are one of one of those things that we can eat. That’s the universal need that we that it solves. But then there’s also the needs of actually getting it, getting to the to the place, getting to the restaurant, then there’s the needs of having a good experience in line, being able to read the menu, being able to take it to go if I wanted to. So there’s these nested needs within the greater need of why the product exists. Both are important. But I think that product teams don’t always take both into account. – Chris Kovel
Show Links* Chris on LinkedIn * First Abu Dhabi Bank * Gavin Payne, Head of Innovation LAB * Stanford d.school * John Arnold and Design Thinking * Lawrence Krauss * Medtronic * Jay Reader * Needfinding by Dev Patnaik * Books by David Kelley * Books by Tom Kelley
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
We numbered eight, a cadre of 10-year old boys, posted around the kitchen table and ancillary horizontal surfaces, awaiting the culinary culmination of this birthday party, when pop and chips would give way to the chocolate birthday cake. The flaming dessert made its appearance and we warbled “Happy Birthday” to the celebrant, mischievously goading him about his simian appearance and odor. And with that, the cake was ours – paper plates and metal forks, droplets of melted wax that we flicked off the frosting, fragments of decorative icing. So, we set to our primal task, inhaling sugar, chocolate, and oh yeah did I mention sugar?
Noticing something about my headlong progress through my slice, my friend paused and looked up from his own cake, curiously asking me, “So…you don’t save the best for last?” This was a new concept to me and I stared blankly, crumbs leaking out the corners of my mouth. His mother pops by to affirm, “Yes, Stephen, we save the best part for last!”
Beyond surprised, I was enlightened. Of course, some parts of the cake had more value than others. A forkful of plain ol’ cake wasn’t as good as cake and frosting which itself wasn’t as good as that mouthful of corner megafrosting. But the big news was that I had an option to eat to an end goal. I could eat differently – portioning, partitioning, and planning – suffering through not-the-best bite, so that down the road a slice I would have the absolute best bite. That would become my reward, that was something to work towards. Gratification delayed was gratification improved.
In a fundamental way, this exchange broke me. Decades later I am still struggling to effect repairs. I absorbed a significant lesson – that even a tiny in-the-moment decision can have consequences, risks, and downsides. This week, even, I noted how, with hardly any conscious thought, I was actively managing how much salsa to put on a tortilla chip, weighing that if I took too much from the little container, I could wind up chomping on dry, salsa-less chips by the end of the meal.
As eaters, we face moments for optimizing (or at least risk-mitigating) constantly. Food designers don’t seem to consider this aspect in their apportioning. Whether it’s Lunchables, or an airplane snack box, a meze platter, or a charcuterie plate, there’s an intended combination (nominally meat or cheese or spread on bread or cracker), by design. But the quantities are rarely calibrated so that there’s a cracker for every morsel of cheese, etc. Of course, we can eat things in a variety of ways, and an extra apple slice isn’t really a problem. But, a solitary dollop of marmalade is absolutely a fail. And so, when designers don’t consider this, it falls to us, the concerned eater, to plan, calculate, adapt. For some of us, that isn’t an enjoyable part of the experience. Indeed, the affordances, the aesthetics in a bundle of base/toppings is that we are consuming a set of choices, a purposeful selection of quantities of elements. So the mismatch is just that much worse when the cues tell us to expect coherence.
Going beyond what may be dismissed as my compulsiveness, this is really about being present. It’s hard to be present with your food, or indeed be present at all, if you consider what may happen in the future, worrying about potential microannoyances. Navigating the pleasure of being present with the pressures of what lies ahead is the burden – and freedom – of being an adult, of constantly having to make choices. At my most hopeful, I might reconsider learning about the-best-for-last less as fracture to my equanimity and more as an important and inevitable loss of innocence.
Here’s another zero-budget episode to share with you. And I want to remind you that you can support Dollars to Donuts by supporting my small business. Here are some of the ways I work with companies.
The first is User Research – I work well in situations where the team knows some things about a set of users, often they each know something different that they strongly believe, and have no way to get from their anecdotal familiarity to an actual framework to make decisions. “Oh, we’ve talked to our customers many times, but we’ve never heard THOSE things from them…” This is often research about people, rather than “a product”
The second is a Player Coach where I’m working with “People who do Research” or maybe there’s a junior research person on a project, where they can use some of my expertise but they are going to own the project from start to finish, so I come in to advise some parts of the process, roll my sleeves up for other parts of the process, depending on what the team needs.
There’s Team Coaching where I’m an available resource with office hours or other format so that people can come with questions, seek feedback, etc. I usually not “doing” anything, just there to talk and review documents
Training is for people who are moving to do more research, this is a half-day to two-day curriculum on planning research, interviewing, analyzing data
A Master Class is for a research team that has a particular area of interest, this is a customized combo of lecture, exercises, and facilitation
And finally, Alignment Facilitation where I’m bringing together a group of people to identify and sort through issues, often around optimizing and improving process
If I can help your team, please get in touch.
All right, let’s get to my interview with Chris Kovel who heads UX Research at First Abu Dhabi Bank.
Well, Chris, thanks so much for being on dollars to doughnuts. It’s great to get a chance to speak with you.
Chris Kovel: Pleasure to be here, Steve.
Steve: Chris, thanks so much for being on Dollars to Donuts. It’s great to have the chance to speak with you. Why don’t we start By Why don’t you introduce yourself?
Chris: Yeah. Hi, I’m Chris. I head up UX research at First Abu Dhabi Bank.
Steve: How long have you been in that role?
Chris: So, First Abu Dhabi Bank is a is the biggest bank in, in the region. So I work in Dubai, and I’ve been here for about 14 months.
Steve: And did you join the organization to take on this role?
Chris: Yes. So within the organization, there is an innovation lab. So fabric first Abu Dhabi bank Research and Innovation Center. And in that position, I head up the research and that position was created and I decided to take the plunge and go out to do By and it’s been a great experience.
Steve: Do you have I mean, this is an unfair question, but you may have some context since this happened before you got there. But do you know what led to the identification of the need for this role creation of the role, you know, the, the conditions that were established that led you to start talking with the organization and join?
Chris: So yeah, the history of the lab or the fabric, It started about two years ago. So, my boss, the head of research for fab guy named Gavin, he was brought on to really try to help with the digital transformation. First of Adobe banks, a really big bank, like, like I said, biggest bank in the region. And in terms of technology in terms of digitizing the products and the services. That was really the mission of the innovation lab or fabric as we call ourselves now. And in terms of research, Gavin kind of comes from a software engineering and design background, and knew that research was going to be very important in terms of creating products and digitizing products. And the other kind of core mission other than digital transformation with that fabric has is kind of spreading a culture of innovation within the bank. And that means empowering other departments, not just the innovation lab to be innovative, to think kind of like designers to be problem kind of obsessed, and all that and kind of requires research and hopefully we can get into kind of how that how that happens.
Steve: Well, that’s no time like the present. So how does that happen?
Chris: Yeah, so there’s a lot of different ways to kind of come at it. At least from my perspective, in terms of kind of developing products, you know, I think when research we like to follow design thinking principles comes out of the D school, most famously in Stanford. And but it traces all the way back to a guy named john Arnold, who create who came out with a kind of a book pamphlet, I think, in the 50s, or 50s, or 60s, called creative engineering. And it’s just a way of kind of problem thinking about problems, right. And, and, and needs, right. So when you think about problems when you think about needs from a customer point of view, that will allow you to kind of frame issues and then solve them through products and services. Most of the time when research kind of gets embedded in my opinion into this design thinking framework either up front and towards the end of the product likes, lifecycle. So I kind of think of it as bookends. And up front or kind of upstream is a lot of times when you’re trying to find a need and then downstream is a lot of times just smoothing out the experience or smoothing out the product through testing we do both at the Innovation Lab. But I’m very passionate about the upstream kind of research because it’s really where these the tricky problems are. You know, trying to understand human motivation is not a not a simple thing sometimes
Steve: From your perspective, or maybe from all the folks in fabric, you know, the value of understanding human motivation, in terms of how it applies. To the kinds of products and services or decisions that have to be made in a financial institution, maybe that’s clear to you. Does, how does the rest of the organization see that relationship between, hey, we need to understand human motivation? Because here’s what we’re going to do as a bank.
Chris: Yeah, yeah, it’s a learning experience. To answer that, that question because that was something that I kind of have found, wherever I’ve worked throughout my career as other people who kind of don’t, you know, live, you know, live the UX kind of world of the user research world or really being obsessed about the customer and making the product kind of fit the customer’s needs. is a difficult one. I mean, it’s about education. It’s about culture, you know at fabric. We have four core components within the You know, the lab. So one is research. The other one or one is research. One is accelerate, which is kind of more about more design. One is FinTech. We can get into that if you’d like. And then the last one, and arguably the most important one is culture. And that was that it’s just an education piece, right? It’s getting people to come and walk through the lab and show them how and the pods that we do user testing. And with, to kind of do roadshows to kind of get the word out there. We’ve done hackathons where we bring people together for kind of, you know, interactive brainstorming. We do talks and kind of fireside chats about design thinking and why they why people should care. So it’s, it’s difficult, right? Because culture is kind of a amorphous, intangible thing and to change it. It’s hard to change it if you can’t even put your finger on it sometimes. So it’s a work in progress. But at the end of the day, it’s it really is just about people and about just talking with them and kind of getting the word out there. And, you know, in the beginning, the bank kind of, you know, we had some Slow, slow days to begin with, but slowly but surely after that we started, especially within research after we start delivering value. And they kind of see the power of answering questions and have been kind of evidence based, then I think that there’s two ways I think it’s the education and then it’s also delivering high quality real projects.
Steve: One of the things that, you know, we talked about in researches is in addition to, you know, doing quality work is, is having people is having influences having people sort of be willing to hear that information that maybe challenges their beliefs, and then ideally, take some action based on that. And so, you know, you’re talking about driving this change, what have you seen on that side of things in terms of, you know, I mean, believing engaging with taking new perspectives on and then, you know, having that influence the decisions that are being made.
Chris: Yeah, so, a couple things on that point. So first off, you know, it’s, it’s never, you know, most of the time, I probably argue, arguably all of the time when a group comes to us to get some research. It’s usually a question that is pretty polarizing. You know, it’s research and also in many ways, I found is kind of a way for people to kind of settle internal conflict or settle internal debate. So, you know, these the results that we provide really matter and they really kind of, you know, take it to heart, but that’s a good thing. But the other kind of bad thing sometimes is that when you’re in the room, and you’re kind of, you know, for example, if you’re testing an application, and you had peep designers and product managers kind of working really, really hard to making things good. And in many cases, the research is the questions that we ask kind of the data that’s presented in many cases is really negative and sometimes and not negative. But that it’s kind of supposed to be that’s why that’s why the research is there. And to kind of manage that manage that in the room is something that it’s a it’s a learning experience, but You know, we’re kind of, we’re kind of human we get we get defensive, or the people who are in the room are human, and they get defensive. When, when the research is being is being presented sometimes so that’s just something you have to be kind of conscious of, you know, I tell that the younger Junior guys, that’s, you know, there’s a certain way to kind of couch the reports and the and the data. And when kind of infighting begins in the room, you kind of have to just take a breath, and really just look back at the rely on the methods. And that’s why, in my opinion, kind of the methods and the way you present and the quality of the research is so important because if it’s done well and it’s executed properly, then the data is the data and you can kind of just kind of, you know, sleep well at night, even if it’s caused some internal strain So, so that’s kind of kind of the first point The other one about taking them taking it forward, right is a kind of an interesting one, it’s me and me and Gavin, my, my boss, we kind of go back and forth about this a lot. And in my kind of philosophy is that the data is the data. Sometimes they want us to recommend, and sometimes they don’t. But in terms of solutioning, that’s kind of their I mean, we’re the researchers, right? And they kind of other product owners. And if the if the research and the project is scoped up in a way that’s pretty narrow, and they just have a couple of questions when we kind of provide that that information if they kind of because of politics, or because of infighting or because of, you know, other bulwarks. If they don’t take it forward, then it’s kind of out of out of our hands, which is kind of arguably good and bad. I don’t know how to how do you see it?
Steve: Yeah, I mean, I think you’re hitting a really This is a personal hot button of mine. You know, how do we measure our own success? I’ll tell you I had a kind of a galvanizing experience a number of years ago with a friend of mine who had worked with a number of design agencies. And we were just we were, we decided to meet and share our capabilities presentations as a way of just learning about each other and learning from each other. These were public, public documents that we could share comfortably. And, you know, I had all these stories were research, it uncovered something really insightful and potentially impactful to these business areas that my clients were working on. And I kept saying, you know, unfortunately, this didn’t happen or you know, they weren’t ready or they decided not to and you He kind of stopped me and said, like, Steve, you’re not in the business of them shipping a product, you’re in the business of highlighting opportunities. And that, and that, that was really grounding for me. I think, as I, you know, I want my success stories to be success stories and not failure stories, but we can’t control everything. You know, and I think it’s, it’s a little, it’s a little distressing to hear researchers tag their own success to a thing that’s ultimately out of their control.
Chris: Yes, yeah. That’s Yeah, it’s interesting. I, I agree. I agree with all that. Everything that you said there. And I think it’s kind of if you look at kind of the different levels of the macro and the micro, it’s like, I think if you’re, if you think about it, in the macro level, right. So the product that I’m researching isn’t going to be a success or failure. Then that’s, that’s definitely It can be kind of disheartening, right? If you do all this really good work and then the, the product kind of dies or it goes away or it takes a different direction, a direction that the research isn’t what you know, kind of away from the research findings, but if you look at it in the macro and just kind of, you know, look at each project as each project and as long as you know, you hit this hit ticked all the boxes in the scope, and you kind of went out and you know, you did the, you know, the research the way, the way it should, should have been done the way that you wanted to wanted it to go and then reported it and then you can kind of I think rest easy if you just kind of look at it with kind of net very narrow with kind of almost with blinders on but it’s hard sometimes. Right.
Steve: And I, you know, it changes and I think you’re kind of alluding to this. A little It changes based on, you know, how emotionally invested we are as individuals. And, you know, I work as a consultant. So, I love my clients and I’m super engaged, but at the end of the project, you know, they have to move forward without me so I have a, I have a way to detach my success or failure or, you know, how identify my own success or failure because project ends, even if a relationship continues. There’s a point where the that’s, that’s over, you know, you are part of an organization even though fabric is, you know, a center. It’s in the name. I wonder if I mean, I wonder if different researchers just based on sort of where they sit in the organization or you know, how they feel about their own ownership of that product or their own ownership of that team. I wonder if their own assessment of what success looks like very for them.
Chris: Yeah. I wonder if for me if I’m kind of okay with just delivering and then and then kind of moving on to the next thing and kind of judging the success project by project, not kind of the overall life cycle. I’m wondering if I it was kind of imprinted on my brain kind of from the early days when I first got into doing research. So I, I worked originally or fresh out of college in Manhattan, at a human factors engineering firm, very small, kind of boutique firm. And we would work on interesting and cut with big, big clients like Google and Amgen, and others, Nike, but it was literally like the brief would come in and you know, a month or two and we would do the research and then we wouldn’t hear anything because of you know, confidentiality. situations and what have you. So it was very much just like here’s the here’s the findings, here’s the data, and then on to the next thing, and I got very comfortable with that. So it might just be I don’t know, maybe how you get your start. But I wonder how if there are teams out there where researchers are kind of, you know, embedded into the whole product from kind of soup to nuts from cradle to grave, and if they prefer that, or if they prefer for me I love one of the reasons why I love research is because one week you can be working on digital wallets in the next week on a personal loan app. And you know, I love I love that for the versatility and the diversity of working on many different types of projects and kind of moving on to the next one.
Steve: Yeah, I’m thinking of several things here at once. Now. I mean, it’s interesting. You know what? Yeah, you make a really good point that, you know, sort of how you were raised up, how did you get your start and what sort of model formed for you versus, you know, the experiences we had versus someone that’s, you know, part of a team and maybe sees themselves as being tied to the success of that product. And, you know, I wonder, and again, I’m super biased because of the, you know, how my, how my approach was formed and how I operate now. But I wonder about, like, what’s good for research, and I feel like the truth to power role that we play, you know, is essential, and, and being self contained or self sufficient, like we’re going to do great work, and then we’re going to help you as much as we possibly can but that next part is up to you. That seems that mindset to me seems good for truth to power, you know, role but, you know, what I’ve come to admire with people who are embedded is you know, and maybe, you know, fabric is different than my business, you are part of a larger organization. He talked about culture, but, you know, the being embedded and working long term with people kind of day to day sitting with them. You those researchers, they really know those people, they, they can influence. I mean, literally from the inside, right? They’re part of that team. So I don’t know what you’re seeing with that fabric as you build relationships.
Chris: Well, for me, I think I’m kind of torn right because I can see the value and kind of embedding researcher through from the beginning of the product to kind of The end to when shipping or you know, when it’s becoming live. But I think there’s also value in kind of the outsider kind of role that sometimes we play as being brought into a new project or on a new product team. And kind of not having all those internal biases and kind of the curse of knowledge, right, that research kind of, you know, it’s just another it’s another set of user eyes, new fresh eyes of seeing, seeing the product and kind of looking at it from different point of views, different point of views. I don’t know I think that there if you really kind of take a step back and kind of look at a product or a service lifecycle, and no two are the same, but I really like kind of following the I mentioned it before the D school design thinking kind of philosophy and if you look at that The circles there’s five different steps or five different kind of parts to the design thinking model, kind of empathy is usually up front, and then testing is at the end. And I think having researchers in involved in the process for those two things is essential. I feel like it’s a lot of times, you know, we people and businesses and, you know, marketers or product people look at research as kind of a nice to have, and I really do kind of think that it is at least in those two areas upfront and then at the end, it really is essential. It’s, it’s a must have.
Steve: And what’s what is the research doing for the process or for the team or for you know, what’s the what’s the impact of research in those two different stages. of the process.
Chris: Yeah. So I think the easy one is the testings kind of down downstream, right? So when a product kind of comes into the business from and starts having, you know, getting some traction and people start becoming, you know, working on it. You know, usually a prototype is made, it depends on what kind of product for us doing UX research, most of the time, it’s digital, it’s, but, you know, that’s not to say that it’s always the case, it could be a physical product, it could be an experience, it could be a service of some, some, some sort. And usually, after a prototype is made, then the researchers come in and that’s when that person prototype is kind of smoothed out. So usability test tests are kind of bread and butter for UX researchers. But in many cases, I think upfront, you know, the research hasn’t, you know, the teams don’t bring it in. The teams don’t bring researchers into the process to begin with. And I think that there’s a real impact and a real value that away for, you know, product teams to kind of set themselves apart, and making sure that the product that they’re creating has been tailored to a specific need. I mean, I kind of see it’s, you know, some simple terms, that a product is just a sad product is basically satisfying a need and the way that you find needs, and it’s a very tricky, tricky kind of business, but the way that you find needs in my, in my opinion is through empathy and through going out and kind of applying the methods that we as researchers use to kind of, you know, make things a lot clearer and distill down the essentials in terms of what the problem areas are, what the need areas are no jobs to be done. framework, I think gets this right to where, you know, it’s kind of I was just watching a webinar, I think, and I really like this, this phraseology that they use they, they were thinking about milkshakes. Right? And like, why milkshakes were being sold more in the mornings, during, during the workweek than any other time. And the way that I remember that the way it was framed was, you know, why do people what are they hiring, a milkshake for? And I really like that kind of reconceptualization about that. Thinking about products and services and what do you Why are you know, hiring your computer? or Why are you hiring your water bottle? And I’m thinking about it that way I think really can kind of generate interesting new ideas and that’s kind of the heart of innovation in my opinion.
Steve: When you at fabric, have these initial conversations with teams at the bank. What are they? What are they looking for? Like? Do they have a product? Are they looking for needs? How are they kind of framing that initial interaction with you?
Chris: Yeah, yeah. So most of the time they come they have a product that’s kind of either out there and kind of in the wild or, or an MVP or a prototype and they want us to kind of help them bring in users have them you know, put that put that product in front of them and kind of observe their behavior and kind of get back to them with the findings about whether they the users kind of could understand whether there are any kind of issues with the content issues with the lexicon, this these kind of things. And so, you know, we work with the digital team, we also we work with other departments within, within, within fab, on kind of lack of productivity. And so, a couple of months after I started, I started interacting and working with the head of marketing. And we kind of created a market research branch off of fabric. So now, any type of market research initiatives come through us and we, we test ads we do kind of more discovery type work about products, you know, for example, there was one credit card. So fab has 22 different credit cards if you can believe that. And the one that was performing the worst was a card called gems card and we were able to bring in people who were using the card and kind of getting their sense about why they’re not using it more. Why kind of other confused about the point system or you know, kind of just getting them to kind of talking and Having an informal kind of discussion about problem areas and things that we can improve and kind of the whys as to the underperformance. So yeah, was that Did that answer your question?
Steve: Sure. And it raised another one. So that that says we’re going somewhere we want to go. So I guess that raises a question for me around the definition. So I think you’re describing that as a market research project. How do you define or delineate what’s you know, what’s UX research? What’s market research? Yeah, think about it.
Chris: It’s interesting. I think of it all as kind of the same thing. I mean, this is now kind of being in the industry for as long as I have been. I’ve seen I’ve kind of seen different titles or worked with the from corporations, you know, things like human factors, things like user research, UX, kind of design designers. And I think that at the end of the day, it’s really all kind of coming, coming from the same place or I guess the it’s all looking towards the same thing when it comes to all these, you know, you know, human factors or UX or you know, market research. Just at the end of the day is trying to answer questions with data from your actual your customers. And it’s trying to get at the need, I think it’s trying to find needs and try and discover needs, and then trying to help the people who make the products or make the services, make them better, make them more, not just making you know, Making people that’s not just, you know, listening to people about their problems of, you know, with, with a product or their problems with their day to day life, but really trying to understand the motivation and understand their feelings and their desires and kind of satisfying them with services and, and products.
Steve: So I want to ask a different question, following something you said before you talked about the sort of four areas and where culture was like a really important one. Yeah. And so you, I mean, from where I sit in California, I’m curious about culture in your organization, because not only are you know, in a specific industry, which, you know, based on my experience, like banking, has a certain culture, sort of historically around and you know, maybe that’s very much an American perspective. But, you know, a regulatory a regulated industry with certain history means there’s a certain approach to decisions and a certain approach to, quote, risk and so on. But you’re in a very different region of the world than I am. And I’m so and that makes me wonder, you know, what observations do you have about, you know, the regional business culture? And how that how that responds to innovation? And the changes that innovation is asking for, as well as financial banking kind of services culture as a factor that characterizes the culture that you’re working in.
Chris: Yeah, so I think culture is changing the culture is one of our core missions with one of fabrics core missions. So it’s a work in progress. You know, I think fabric and then fabric versus fab. I think within fabric, we kind of have in our own culture, it’s a little bit different than kind of the corporate, you know, buttoned up tie wearing jacket wearing kind of traditional stereotype of the bankers. I don’t see too much of that, that stereotype within the bank, you know, meeting with people in different departments. Um, but yeah, I think work I’ve worked in the corporate setting a couple times. And I think kind of, you know, there is there is a difference. There is a division between kind of the traditional, you know, normal kind of roles that you would think about when you think business versus kind of the UX or user research designer type people. And it says it’s, it’s something that yeah, it’s I feel like there should be more in integration, I would, I would like to see more integration, you know, in the future and to just from a business point of view, I think it’s good business to kind of have people like us at the table. And in many cases, it’s there. You know, we’re not we’re just not there yet. So, and it’s something I think that organizations that, you know, struggle with about how to kind of you hear a lot of people, you know, a lot of people in the industry, I don’t know if you get this as well, but talk about when they’re working in a big corporation like to evangelize, you know, that user research or just kind of UX in general, has to be evangelized, right. And kind of the, it’s kind of a little bit that you know, first off, it’s not great that it even needs to be evangelized. And then the second part is like How do you actually what’s the best way to, to evangelize? Yeah, we talked a little bit about, you know, it’s educating, I think that the way the approach that we have is kind of through education, you know, events, these kind of kind of cultural events and then and then just producing, you know, kick ass work.
Steve: Can you What do you think would be different if, you know, you’re at fabric within fab, but you were, this was a company in Manhattan, or this was a company in London. Do you think there’s any regional shifts that you would see?
Chris: Yeah, it’s an interesting question, um, regional shifts. So you know, it’s funny, I when I first moved out here, I was expecting it to be a lot more of a culture shock than I was like, then it actually was so. Yeah, I’m sure there would probably be some differences without a doubt, less sand. But I, I think I think for the most part though, it’s you know, it would there wouldn’t be much it’s funny Dubai actually has I think it’s 8080 or 85% expats right so there are very few locals and nationals UAE nationals. So the bank is comprised of a lot of people from a lot from all over the world, right so it’s a very diverse very multinational community. Not a not too many people from the States but a lot of people from England, a lot of people from Australia. India is there’s a high percentage so yeah, it’s kind of a melting pot, and it’s a really cool and interesting environment to work. But I think at the end of the day, yeah, it’s People are kind of people. So
Steve: do you have any sense of, you know, the business community whether it’s you know, outside your organization, but in the region you know, what the awareness or adoption of, you know, user experience user experience research, you know, elsewhere outside of fab.
Chris: Yeah. So it’s something that’s that is, I think taking off and growing within the within the region. I know for us personally, what we want to do is fab not only serves the UAE, but it also serves other countries in the MENA region like Egypt and like Saudi Arabia, and others, and I know that we as fabric have kind of goals to take certain elements of our of our toolkit in our service Kind of export them into, into these regions, right? So research being one of them, I think because of its kind of universality. And its way for, you know, you can kind of use it in many different contexts. So, so that’s definitely exciting. But, you know, it’s funny, I think user research and UX in general, is kind of a very niche, you know, industry. So even within the United States, you know, a lot of people don’t really know about it, whereas in this region, it’s almost like I think there’s a lot of people who don’t know about it about it as well. But I think that there is receptivity, they’re much more receptive to kind of embrace these kind of new and maybe kind of non traditional ways of approaching work. And I think that’s really exciting.
Steve: Yeah, sounds very positive. And we’ve talked in and around this a little bit, as you’ve talked about some of the ways that, you know, that fabric is working with teams, you know, throughout the bank and so on. When you start working with these teams, what kinds of disciplines or functions or roles do you most? Or is there a typical, you know, set of rules that you are collaborating with most directly?
Chris: Yeah, I would say I mean marketing for one because, you know, fabric and marketing, there’s kind of this dotted line between fabric, fabric reach research team and then the market research that we do. So that’s definitely a division that we work with very closely. We started working with product a lot. We’re doing some kind of foundational fundamental research project that we’re really, really excited about, on personal loans that hopefully, it can kind of spread and we can do this this kind of research, research stream with all different products that really start from the ground, ground floor and get those needs and kind of assess and evaluate the current state of the products and then kind of kind of move forward involved in evolving from there. We work with the digital Team A lot, a lot of times that the guys will come and request meetings for, you know, to do kind of benchmarking testing, like I said before, a lot of usability testing, touchpoint testing, sometimes card sorts, every now and again, some kind of upfront research about kind of ideas and content concept testing and this type of thing. But for the most part, it’s kind of its product, it’s the product testing. Let’s see this. Yeah, I mean, we’ve been working with a lot of different departments, which has been great. I think we had 25 or 28 different research projects completed last year. vast majority of them was from ppg, which is personal Banking Group. We have corporate banking, we haven’t been working with corporate all that much. But that might be changing in the next couple. next couple weeks, weeks and months ahead. But yeah, I think that’s weird. So there’s also a team that that that head that basically is single product team with it’s called pay it and it’s a it’s a digital wallet and we’ve been doing a lot of different types of ethnography work, you know, competitive landscape research, and you know, customer interviews and these types of things. So yeah, that’s been exciting too, because I think that the digital e wallet space is a new one. And I think it can be. It can it can there’s a lot of value in kind of thinking differently and really trying to figure out the needs of people and why kind of a digital wallet exists, or why they exist, and then kind of helping them shape and smooth the experience.
Steve: And are there situations where fabric is proactive, say, as opposed to responding to requests from a from one of these teams initiating some effort, some program from within fabric does that happen?
Chris: I think at the beginning, once we started kind of ramping up, we were so busy that we were just focusing on the products that I’m sorry, the projects that were that were kind of coming to us and There were times when we had to kind of deny or you know, tell people sorry, we’re kind of we’ve hit capacity, we have no more bandwidth. But I think with this kind of Corona situation, we’ve had a time to kind of level set and start thinking a lot more about, you know, spinning out projects internally. The prot, the personal loan project that I just that I mentioned in my last my last response to your last question, was something that we kind of were developing ourselves and when the product came to us, they were just like, What’s on your mind? What what’s kind of some, some new stuff that you got? What’s kind of what do you guys thinking about how, you know, we kind of want to kind of work with you like what can you kind of offer us you know, there wants to want to have this relationship and that was something that was internal, an internal idea or an internal model, turtle project that we kind of then took off the shelf and applied it to, to product when they kind of came knocking. So that was kind of kind of cool. I you know, it was to give a little bit of a background, it’s a new and innovative, new and innovative way to think about personas. And what we really wanted to do was we wanted to I mean, I think most people within the industry when they hear persona, it, they kind of, you know, they’re, I think it’s a very ugly, ugly words in some people’s minds. You know, but I think that there’s, there’s, there’s use and there’s value to them, but I, I see them as being kind of a little limiting or limited, right. So the idea for this project that we’re working on with product is kind of to do the personas the traditional way, which is where you go out and you get together. Either through focus groups or in depth interviews, a bunch of a bunch of users, a bunch of customers within that specific product. You talk to them about their lifes, you talk to them, you would get you understand, you know, their demographics, you understand their kind of, you know, their motivations, why they’re the other products, they’re using, you know, kind of a comprehensive 360 degree view. And from all these interviews and focus groups, you know, personas kind of emerge out of them. So the idea is, is to kind of take that traditional method and kind of mash it together with a method that’s kind of been now being developed over the past couple of years with using machine learning and AI by feeding all of you know, customer data into And an algorithm and have the algorithm then create the personas, you know, mathematically, and then if there’s some way for us to kind of merge those two, where you get kind of the, the best of both worlds of approach, because you know, in many cases, the traditional UX. Personas through research are kind of limiting. I mean, you can ask a lot more questions than you can with a datasheet, you can get a lot more kind of subjective internal motivate motivations, you can kind of capture those but the power of generalizing that is not the same as the power that the machine learning model has. And if there’s a way for us to kind of, you know, understand that those two are, there’s a way for us to kind of push those two methods together. I think that would be a really huge impact for the bank. So it’s kind of a combination of that persona piece with this idea of service design, which, in many, in my, in my opinion is very, very much a just another method of, of user of user research, where you go, and you talk to people, this is for personal loans, you talk to people about their experience, you know, taking out a personal loan, so usually people who, within the last, you know, three to six months have taken out a personal loan with fab. And you kind of map out the whole journey from kind of conceptualizing, to kind of get you know, that kind of motive motivation moment of like, I’m gonna need a personal loan, why they need one, and then going through the experience of, you know, coming to fab going, setting, you know, signing all the documents, and then, you know, lastly making the payments and kind of having that map and looking for gaps looking for pain, areas of pain, things that can be improved upon. And if we kind of have those two models, and we do those that piece of work for every product in the bank, we think that, you know, hopefully, we can really change the trajectory of the products and make them more about kind of user needs and user solving user problems then, then kind of how they’ve been set up so far.
Steve: That’s an amazing approach. I haven’t heard of anything like that before. You’re talking about? Yeah, just like you said about market researcher kind of blurring the boundaries or eliminating the boundaries between things that were sort of separate islands. And here you’re pulling in two very different kinds of things to create. One, one better approach. It’s very exciting to hear Yeah, it’s
Chris: it’s funny you say about marketing because I remember when I was meeting with so the merchant team came to us came to fabric. And we did just kind of a normal, you know, textbook, piece of user research on some credit card designs and then also asking them some questions about what would get them to sign up for a credit card. And, you know, they were really happy with the results. And then they were like, okay, we want to kind of grow the team, we want to kind of have you hire some dedicated researchers and kind of have this this dotted line between marketing and between UX and I was kind of like, wait, I, I don’t I don’t know like, I don’t know marketing. This is not my world. Like I don’t know if you have the right guy, but the more I kind of learned, and the more I talked to the marketing people and the kind of the decision makers, I learned that you know, there really is not bright lines between product and marketing, and, you know, and research and just kind of thinking holistically about how products Come about thinking about, you know holistically about the economics, you know, why? Why do products succeed? What makes what are they? For what, you know, what makes a good one? What makes a bad one? You know, at the end of the day, it all starts with understanding your customers. And that’s what we as researchers, do, we try to understand all the different facets of people and what would make them buy and what even what is it what is a product doing? What purpose does it serve? When you start there, then I think the marketing and the product and all of the those other parts of the business kind of just take care of themselves, it becomes much more easier to navigate and make decisions based on this fundamental understanding of the people that you’re serving.
Steve: And right now that thing that activity is But we call it research. It’s, and you know, people that do our job are the ones that do that. Yeah. And, I don’t know, I feel like in some ways, it’s a historical accidents like you when you describe, you know, this is what the process should be. And this is, you know, in order to get to these ideal results, and here’s the activities that kind of started off you know, the way that research is as a discipline has done evangelizing has done has, you know, invited other people to participate and sort of said, Hey, you know, everyone needs to do this. I mean, if you mentioned the D school right there D school is like classic for saying, Hey, we’re going to train everybody to do this not people with you know, the researcher title. This is everybody’s job.
Chris: I mean, I think it it’s a it’s a way of reopening Kind of reimagining kind of what businesses are doing, and why they exist and what their purposes are, right? I mean, it’s Yes, I think at the end of the day, it’s like, it has to do with monetary gains, and, you know, stakeholders, you know, you know, making sure your stock is going up and these types of these kind of economic questions, but, you know, I think, you know, when you say things like product, we talk about products, we talked about services, we talked about needs, and they can be really defined kind of in very broad strokes. And when you when you define them in broad strokes, then then the power and the use and the kind of impact and the relevance of understanding your customers, which is what as you’re saying is what we do, becomes, I think Paramount, right, because that’s the way that you can separate yourself from the competition. It’s not just about innovating for innovating sake, right, it’s innovating to me is to just find a need and solve it in a new way. And if you can do that, I think you’re, you’re poised to have to have success. The problem, of course, is some of the bureaucracy, some of the politics that kind of happens in you know, in the different layers within the business. And also the kind of tricky thing of, you know, researching humans, it’s, I love this quote, it was a physicist, guy named Lawrence Krauss said, I do physics because it’s easy. Right? Well, it’s clearly it’s not easy, right? But I think what he was getting at is when you’re when you’re researching humans, they’re tricky. They’re complicated. It’s a little bit messy, right. He kind of love He was his point was the elegance of physics. Right? You can describe parts of the world in very elegant mathematical ways. And it will always behave that way. Whereas people were unpredictable. And kind of getting at the root causes and those root needs and those root desires is tricky. But if you get them then then i think i think everything falls into place.
Steve: I would love to change directions. And, you know, hear a little about what your trajectory has been, you know, how you found research and you know, what kinds of roles you’ve had that kind of, you know, we all know about Manhattan a little bit, but yeah, describe more of that story.
Chris: Sure. So I kind of I kind of fell it fell into this accidentally, actually. And yeah, I love this question asking Others in the industry this one because it seems like you know that there isn’t a clear path into kind of the world of UX in the world of user research. Which maybe would be maybe that’s the case for all industries. I don’t know. I have a hunch that this one is a little bit different though. People get into it through kind of untraditional ways. So yeah, I was. I was going to grad school in New York. And I was getting interested in fascinated with science and I started working in a behavioral science lab. And I graduated, and I moved to Brooklyn and I didn’t have a job. And I was kind of running out of money and I was and I wanted to do research. I really liked the idea of kind of trying that out as a full time kind of career and the first job I interviewed ATS was a, like I was saying a small boutique firm on the Upper East Side of Manhattan that did Human Factors engineering research. And I saw the research paper the human factors for I had no idea. And it was like kind of a pretty low level, entry level position. And I went interviewed, I got the job. And I kind of just fell in love with it like, and I never really looked back since and kind of this idea of merging science and research and data with design was something to me that I never really thought about and, and kind of, you know, you know, for me when I when I thought when I think of design, I always would think of kind of just someone sitting in a room, you know, you know, twisting their moustache and coming up with ideas. I didn’t realize that you can actually kind of use science a US kind of methods, research methods to influence how products Shape products, right? Which is really what a lot of times what human factors engineering does. It, it’s, you know, if you if you looked at the D school model, it was the kind of lives in the testing area for, for a lot of for most of what it does. So I did that and I kind of after six months of working there, it was, like I said, a small a small firm, the my direct reports left kind of suddenly and the next in line. So, she reported to the CEO, a guy named Charles Morrow who has been in the industry for 45 years. And then so that left a big wide open role that he you know, said that now it’s mine and I was kind of again in moment of crisis like I don’t know if I can if I can handle this right like I’m only six months in, it was a very much like a sink or swim moment. And now here this, you know, person who really didn’t go to school for engineering and research is now kind of managing these huge projects for Amgen and Google I was I yeah, it was definitely harrowing, to say the least. But I kind of kind of made through it. And then a couple of years after I decided to move to the Midwest and I took a job working for more, more human factors, stuff working for a company called Medtronic that makes medical devices I worked in the neuromodulation. Department, working on a couple of different products kind of flew around the world, around the country, mostly talking to doctors, kind of getting their feedback and trying to help The shape certain medical devices. And then I kind of switched gears and got into the world of UX and UX research, actually that same mice the same directed my direct report of my first direct report, Emily, lady named Emily actually suggested we kept in touch after she left and she suggested that I start looking into this, this thing called UX research, it’s very similar uses the same methods of kind of the methods of that we employed for human factors employees. And I kind of thought that it was Yeah, it was I was getting a little bit bored of the physical products a little bit bored with the medical devices. You know, tech was kind of booming and it was kind of an exciting place to work. I thought that it’d be cool to kind of open up my tools. And start working on screens. I did a little bit of screens, kind of within the human factors, but definitely not as much. And I started then working in in a few different innovation labs, I worked for a company in Minnesota to call I’m sure you’ve heard of it Best Buy, how many people here have heard of it, and that was my last. My last role was Best Buy was before I moved out, moved out here to Dubai. To do to help them kind of creates UX research department within an embedded into the bank, which has been lots of fun, I think.
Steve: So outside of fabric and the work you’re doing for the Banco? Are there other things that you’re working on in your profession?
Chris: Yeah, so, um, since the past year, year and a half, I’ve been kind of kicking around an idea concept of trying to think about, you know, things like UX and jobs to be done theory and market research and human factors, all these different kind of disciplines that seem to be kind of putting a lot of a lot of weight on understanding the customer bringing the customer into the design process. Being customer obsessed. And it kind of kind of all kind of filled. I was thinking about writing a book about you UX research more technical, then I was working in Best Buy. And I kind of struck up a friendship with a guy named Jay reader who was a graduate from the Stanford D school and he kind of introduced me I broadwalk brought him the idea of the book and he kind of introduced me to different material, different kind of literature material that the D school was kind of putting out in their philosophy which I had heard of design thinking but I hadn’t really dove too deep in it. So we had a lot of interesting conversations about you know, design thinking and you know, books like need finding by dead panic, and the books by, by the Kelly brothers kind of I started kind of getting interested in that and, and design thinking I felt was very cool approach to kind of what user researchers were doing already, right when we do user research. We’re trying to make better products and we’re trying to come design help, you know, collect data to help designers and to help product people shape products and make and make really better experiences. And, you know, I found I found that, you know, it was really hard for that will that you know, a way that when you look at the design thinking framework, in many cases research is not so much talked about when you get kind of down into the details they do. Talk about how empathy which is kind of the first pillar the first step requires kind of a journalistic kind of anthropological mindset kind of going out and observing people and asking them questions and watching how they use the product and seeing the context and all of these kind of kind of more research stuff. And I thought that yeah, that this was a really cool model, but I thought that it kind of needed a little bit. It was kind of maybe outdated, a little bit. In many cases, it was it was used for physical products. And now I think, you know, because the world is becoming digitized. So there’s so many now, so many services and products that are now on a screen. And I think that kind of user research, especially for digital products, you know, I think that looking at them as a product I think in at the end of the day, I think some people when they look at, you know, a journey for example, have, you know a bunch of screens to get you to sign up for a loan that that is actually a product if you define if you can, you can actually define it that way. And when you start looking and seeing all the different products, you know that almost if anything, is customer facing and you know, you can use it or, you know, Facebook as a product or Candy Crush as a product, all these things, you can look at them through this kind of new lens as Okay, this is a product, this is a service, then the kind of the design, the D school design thinking kind of applies. And the problem I think one of the problems with the D school is that it just doesn’t go far enough into the methods and into the kind of deep dive that user researchers kind of do the kind of, you know, the ways that we kind of understand problems and the knowledge that we build through our methods is something that can Really, I think benefit the upfront beginning empathy part. So if people, and this was kind of the kind of the theory or the thesis of the book is that if you do your due diligence, and you do your research upfront, before, before, before you even have an idea about what the product is, that will actually help, you know, go waiting out into that kind of unknown area, and kind of relying and believing in research. That, that, that that’s the way to kind of kickstart these products, and then they these projects. And that’s kind of the kind of the idea.
Steve: So where were you at in that process? And do you have a title like, Is there a place we can find out more?
Chris: Yeah, so it’s still pretty new. I’m kind of it’s all outlined right now. That kind of the writing is began, but very slowly, so it’s the tentative title is the universal lens. And yeah, yeah, hopefully I, you know, I can kind of get it get it to completion I kind of have some more time on my hands now with everything that’s happened with kind of the work slowing down because of, you know, Corona and, and all this and social distancing. So hopefully by next year we can I can have something to share, you know, with the world so we’ll see.
Steve: What is the universal lens?
Chris: Yeah, so the, the universal lens is kind of the metaphor is you know, looking at the world and as I was rambling on, you know, before, if you look at the world as everything is a kind of a anything could be a product or service and kind of The way that that products are just basically solutions to needs, then that is kind of the universal lens that this kind of this kind of framework is applicable to whether you’re designing a medical device or your you know, you have a shoe running shoe or a bank app or a chair, it’s all kind of at the end of the day, understanding user needs. Then from based on what those needs are solving them with a service or solving them with a product that will that will at the end of the day, hopefully make that that product successful. You know, we talked a lot about experience. And I like to use the term smoothing the experience. But a and many, many times when you know we as researchers work on these projects, the Product themselves didn’t really seem like it was vetted. It seemed like it kind of like a you ask the question, what is the product for? And, you know, I think that there’s kind of I look at needs as, you know, proximate needs and almost, you know, kind of like. So proximate needs an ultimate needs and an ultimate need is kind of why the product exists in the first place. And then the proximate need is kind of the experience of using that product, right? So if you take for example, a hamburger, that’s a product and the ultimate need is the hunger that it satisfies, right? We as humans need to eat things and hamburgers are one of one of those things that we can eat. And that’s the kind of the universal need that we that it solves. But then there’s also the needs of you know, the needs of actually, you know, getting it, you know, getting to the to the place getting to the restaurant, then there’s the needs of having a good experience in line, being able to read the menu being able to take it to go if I wanted to. So there’s this kind of kind of nested needs within the kind of greater need of why the product exists. Both are important. But I think that product teams don’t always take both into account.
Steve: Well, Chris, thanks so much for being on the podcast. It’s great to hear what you’re working on what you’re thinking about, and just some of these bigger picture ideas, I think are really very interesting. So I’m thanks for taking the time.
Chris: Thanks, Steve. It was a pleasure.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 32. Chris Kovel of First Abu Dhabi Bank first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Noam Segal, the Director of Research at Wealthfront.
Everyone from PMs to designers, researchers, obviously, engineers, data scientists, marketing, we’re all trying to to understand our clients, we’re all taking part in that process in some way, shape or form. And so I view my role and user research’s role as an enabler, as a coach, as augmenting other efforts already happening in in the company, and really maximizing the returns we get on on the research we do. – Noam Segal
Show Links* Dolores Huerta and Alice Waters on City Arts & Lectures * Noam on LinkedIn * Noam on Twitter * Wealthfront * Cultural Probe * Moving with a Magic Thing * dscout * Airbnb * Intercom
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I just watched an interview with Alice Waters, who describes herself as a chef, author, food activist, and the owner of Chez Panisse Restaurant in Berkeley. She’s often referred to as the creator of California cuisine, back in the 70s. At one point in the conversation, she explained how when she meets people, she will sometimes begin by bringing out food, say fresh fruit, and taste the food together with the other person. She explained how this shared sensory experience was an alternate, and perhaps more effective way of creating a connection between people.
It really made me think, wow, is that something we could do in user research? Could you begin an interview by sharing a sensory experience with someone? It could be taste, via food as Alice Waters did, but it could be a touch experience, a moment of smelling, a shared point of listening. I would want to understand more about what Waters believes this accomplishes and if it’s conjecture or there’s any more evidence, and I don’t really have any idea how to introduce something like this into an already hesitant dynamic, that initial moment.
And with all (hopefully all?) research happening remotely at the moment, is there some sort of shared over distance aspect of this, a sensory experience that both parties could initiate, maybe it could be entirely pedestrian, such as feeling the glass of a mobile device, versus something celebratory like a piece of fruit selected by Alice Waters.
I don’t know what this would lead to but I’ll be curious to hear what happens for people that try it. As well, it serves to remind me that all too often I neglect including all the senses in how I process the world and how I engage with others.
This is another episode without either professional editor or transcriptionist. This podcast is my way to contribute at this particular moment, but I hope you can keep me and my practice in mind for collaborating on research, for coaching, for training, and other work to help advance the maturity of your organization’s research practice, wherever you’re at currently.
Now, let’s get to my interview with Noam Segal, who is the director of research at Wealthfront.
Noam, thanks for being on dollars to donuts. So it’s really great to have you here.
Noam: My pleasure. Thanks so much for having me.
Steve: So my typical way of beginning which we can do today is just to ask, ask you to introduce yourself.
Noam Segal: Sure. So my name is Noam Segal and I am currently on parental leave, actually, with my six month old, baby boy Dean. But when I’m not on leave, I’m director of user research at Wealthfront. So that’s what I’m currently up to. I’m originally from Israel, although I don’t sound like it because my parents are both English and somehow I inherited the British accent. And I moved to the US in 2012 with my wife for graduate school Originally, I got my PhD in psychology at the University of Illinois, Urbana Champaign. And when I graduated, we decided to move to the Bay Area and both pursue positions in tech. And I’ve been in UX research ever since and have worked for Airbnb, Intercom and again these days Wealthfront.
Steve: How did you find out about UX research?
Noam: So, as an Israeli, we have mandatory military service, and I served in a unit within the Israeli Air Force for most of my, my service. That particular unit is in charge of Israel’s missile defense systems. And when I left the military got discharged, I decided to join the military industries. Kind of the Israeli equivalent of maybe Lockheed Martin in general dynamics, those sorts of companies. And I focused on developing the UX, but we didn’t call it UX at the time of these missile defense systems, primarily the our weapon system, which is a system designed to protect as well, from medium to long range missiles. This could not be more different than what I do today. Very little relationship, but that’s when I really started appreciating and and desiring to be in this field of UX more, more broadly, that was my focus, I focused on the interface. I focused on understanding from the operators of these systems, what they what they need, and it was an incredible six year learning experience. So that’s kind of how I got into it in the first place. Later on I wanted to improve my my research abilities and knowledge and I was definitely considering going into academia for a while. So that led me to the Ph. D program. But at some point, in year four of the PhD I realized that I love two things. I love research and being a researcher and I love technology. And there is a profession that marries those two things quite perfectly. And that is UX research. So I decided to shift my focus back to industry and and going to work at first with Airbnb and carried on ever since.
Steve: Did you finish the PhD?
Noam: I did. I did. It was a hectic time in my life. I was in the middle of my fourth year in world Typically a six year program. That said, it’s typically a six year program because of how difficult the academic job market is rather than how long it actually takes to finish a dissertation. And I realized and there was some events in our lives, both personal professional that I want to take a different path from academia. And so I interviewed with several, especially Bay Area companies and was fortunate enough to get a couple of offers and decided to fast track the rest of my my dissertation. And I proposed the research I was going to do in January, and I defended my dissertation on June 1 of 2016. And less than 24 hours later, we were on a plane to San Francisco to stop life in the Bay Area. So it was a crazy period in our life. And I’m rather happy it was over because it was quite difficult. But yes, I did. I did graduate, it was very important for me to finish the program. And I benefited greatly from my studies, my relationship with my advisor, and all of my fellow students at Illinois, really appreciate that time, it taught me more than I could ever imagine.
Steve: So maybe I can just go back. So you kind of described this. And we have like, book ends, maybe have a narrative here, where at the beginning, you are, you know, in the, in the army and kind of looking at those things. And at the end of this chapter anyway, you’re getting on an airplane to San Francisco to enter the, you know, Bay Area, user research roles. But I’m wondering, going back to when you were in the army, you started as you said, You didn’t refer to it as UX. So I’m wondering, when did the research part like that this was a thing that needed to be done that could be done that you could pursue as an actual job. Is there a point at which that started to enter your consciousness?
Noam: Absolutely. So when I was in the military industries, I wasn’t in a research role, efficiently. And I also wasn’t particularly aware of the realm of UX research. But UX research is not really an existing profession in Israel, a separate function typically, in Israeli startup companies, it will be the founders or perhaps the designers that do the research. And so I was not in in a research role, but I was working in a very small team, trying to develop systems that were completely new to the world. incredibly complex, and the stakes couldn’t be higher. These are systems that are enabling Israel to defend itself from potential nuclear attacks. It really is life or death. I mean, in industry in tech, perhaps you’ve heard this before, we often use militaristic terms. Like we have war rooms if we launch a new product, or we conduct a post mortem, when something goes wrong and engineering have to figure out what happened. But I really was in in a company in a team where the stakes were incredibly high. And that requires research, whether you’re in that official role or not. I was in a unique position, where on the one hand, I was working in this team as a civilian, speaking to people who operate the system thinking about the user experience. This is the system where people have, in some cases just a few seconds to make very critical decisions. And failure is simply not an option. But on the other hand, I was also an operator myself. So I was still in the reserves, still serving in the military still doing one or two days a month in some cases in the unit as a reserve soldier. And so I got to see and experience these systems from those two, two sides of the coin, which was very, very interesting. And obviously, this happens to to us, as well here in the Bay Area, often we are using the systems we’re also developing, that’s quite common. But that was the first time that happened to me. And I think one thing that really struck me developing these systems and I think it’s part of the reason I decided to move on is that I really, truly believe that failure is a critical component of learning. And failure is not really an option when what you’re developing is missile missile defense system. Um, and that’s a very heavy weight to carry on your shoulders. For anyone who is part of that sort of team. It’s a huge responsibility. It’s not the type of intuitive work that we do in the Bay Area, you cannot fail often, and, and learn, you know, move quickly and break things. You can’t do that when you’re in that type of industry. And so on the one hand, it was an incredible experience and really my first venture into UX. And on the other hand, I really needed to move on in order to become a better researcher, and a better professional within the UX research community. So that’s a bit about my time there.
Steve: Yeah. And what did you do for undergraduate?
Noam: So I went to a small kind of liberal arts college in Israel. And I studied psychology as my major. And I took kind of Media Communications as my, my minor. I was in the first cohort of psychology studies at this college, they just opened the psychology school the way I started. And maybe maybe it’s a personality thing, but I love to be part of new programs, new things, new ventures. And then I had other options to go study at more established universities, which you may have heard of Tel Aviv University as an example. I really wanted to join this this new program, and I have to say it benefited me greatly. The learning experience was incredible. We were a small cohorts of people. We have very close relationships with our professors. And the fact that I went there was a big part of I think, why I ended up getting into graduate school in the US and being able to make this move and ended up where where I am today. So it was a wonderful undergraduate experience, very small campus. My first time moving out to my parents house and and experiencing a lot of lots of new things. I really appreciate that. That time and the psychology the foundations of psychology gave me which were very helpful later on. And until today, really.
Steve: So tell me what kind of company Wealthfront is.
Noam: So Wealthfront, I would say has been through a few different periods, different stages. It began as a platform where you can innovate vest your money passively, passively invest your money in in the markets later on Wealthfront added planning features to enable people to plan for their financial future. And most recently, Wealthfront has added a cash account, a high interest cash account. And later on in the year, we’ll also be launching a bank account for our clients. And so I would say in general, what we’re trying to do at Wealthfront is provide people with incredibly sophisticated tools to manage their finances, manage their investments, manage their cash, plan for their future, and hopefully, achieve their their life goals. Through through well planned and invested finances.
Steve: Yeah, what’s the research practice like at Wealthfront.
Noam: So it’s interesting, you should use the term research practice because research at Wealthfront is quite different than what I experienced elsewhere. First of all, when I joined Wealthfront, the research team was not the UX research team. It was the team in charge of the algorithms and the math powering our passive investment engine. And these days, that team is called the data science And there’s a research component to that. So in FinTech research actually means something different than in most other companies and that took getting used to for me. But research UX research at Wealthfront is an interesting topic because as a practice, it has existed since the company started. And everyone from the founders of the company to later on PMs and designers at Wealthfront have all been conducting variety of research, project methods, and staying connected and trying to understand and best help our clients and our potential clients. But user research as a function didn’t really exist that Wealthfront before I joined just over a year ago. So I’m the first person at Wealthfront with user research in his title. That’s maybe not completely accurate because my manager, a picture, who is VP of design at Wealthfront actually started out as a researcher, but she transitioned into a design role quite a while ago. And I was the first person since to join as a researcher. So as a practice, it’s existed for for over a decade now, as a function, what I’m trying to do Wealthfront is really build out research as a function, which means a bunch of a bunch of things. But really, the point is that research at Wealthfront is incredibly democratized. And I very much support that approach. And so by democratized I mean that we’re all running research trying to better understand our clients and potential clients. Everyone from pm to designers, researchers, obviously, engineers, data scientists, marketing, we’re all trying to to understand our clients, we’re all taking part in that process in some way, shape or form. And so I view my role in many cases and user research his role as an enabler as a coach, as augmenting other efforts already happening in in the company, and really maximizing the returns we get on on the research we do.
Steve: What are some things that you do in coach examples or it’s such a powerful word? What are ways that you are coaching?
Noam: Yeah, so we are currently two user researchers on the team. seem to be a third. Hopefully, we’re currently in a, in a weird situation all of us as we’re recording this with COVID-19 out there and a global pandemic. But we are growing the function of user research. Until then, and regardless of that, we hope to enable especially designers and PMs more than anyone else to do better research. And to do that twit were using a few different techniques. One of them is creating lots of materials in our internal company wiki around best practices for a variety of methods and research techniques. There’s no reason to reinvent the wheel every time and rethink how we should conduct a usability test or what best practices interviews are or what type of satisfaction metrics we can use, etc, etc. And so the first step there is to make sure those materials are out there for everyone to consume. And that’s a work in progress for us. The second thing is running workshops or interactive lessons, webinars, whatever whatever it takes to engage with various teams within the company, and enable them to learn in a group setting, about research best practices, and how they can level up their research. And then another technique or another way we do this is indeed, just one on one coaching, with with designers with with pm. We started doing this a bit. I hope when I get back from parental leave to continue doing that. That’s all of interest from many people in the company to have such sessions and get feedback on on their research sessions and what could be better, and what should what should remain the same because it’s already great. Our designers and PMs are well versed in research and they’ve conducted countless research sessions of all sorts. But there’s always something to learn for all of us. And so through those various methods, whether it’s on an internal wiki or whether it’s in a group setting, or whether it’s in a one on one coaching session, we believe that through those various ways of coaching and teaching, we can all do better research. Not to mention, teaching is maybe one of the best ways of learning, in my opinion. And so one benefit I feel of being a coach and an enabler and a teacher is that you have to constantly reevaluate and rethink what you’re doing. And then about the methods you’re using, about the best practices about the principles behind your research, everything you’re trying to, to accomplish. And I’m really happy about that because I’m not sure as a community within user research we’re doing that enough necessarily so i think it’s it’s a it’s a positive side effects of of taking that coaching approach.
Steve: So what is it that we’re not as a community overall doing enough?
Noam: Debatable, but I’m, I’m referring to revisiting the methods and techniques that we’re using as researchers. Use the research or UX research method Dollar discussions
Steve: Anything there anything specific that you’re thinking of that is unexamined that needs to be reconsidered or revisited?
Noam: Well, I’m curious to hear your opinion as well, I, I feel like many of the techniques or frameworks were using in user research. And one specific example I can give is Jobs To Be Done, which I really learned the most about in my time at intercom. Most of those techniques and methods are taken from other realms such as marketing. In the case of in the case of Jobs To Be Done, it really started out more than it was kind of imported into the world of software development and and product And I wonder if I do often wonder if there’s more room for user research specific methodological discussions and innovation around the methods we use? I think broadly speaking, I haven’t seen much of that. And I think it’s debatable whether that’s needed, or, or not. But I think that as we develop new tools, new ways of working remote research is a classic example right now, given the global pandemic, when there’s also a need to establish potentially new methods to go with that reality with those technologies and think about how we do how we do research. The fact that many of our methods and frameworks are imported from other realms, is maybe just partially a sign of how long UX UI Such has been going on as a distinct profession and function. But maybe it’s time for us to to branch out ourselves and dive more deeply into into those methods we use. So hopefully that answers some of the question them.
Steve: What’s the granularity of a method like Jobs To Be Done to me, and I know you’re talking about methods and frameworks, you know, I’m just thinking there’s, like remote research is an umbrella term for a certain set of things, Jobs To Be Done is. And I’m thinking about, you know, he talks about innovating in the methods. I’m just not sure. We all that’s such a great word and problematic because we all think it means something different. So. Right, I’m thinking about some of the things that have come up in remote research when people were saying and I’m not gonna have exactly right, but how can I see my screen and the other person’s screen if they’re on a mobile device, and then people have come up with various solutions and sort of technologies and ways to hook up different pieces together so that you can remotely have a certain kind of interaction is what we consider that and innovation in the methods.
Noam: So, I view that example as more of the medium than the method. Yeah. So if we take eye tracking, as a method, because I do consider eye tracking a method and to maybe one way to define the method given that is a way to, to I’m kind of losing my own train of thought here, but kind of, I was gonna say like a way to gather insights, but that’s also the medium right? So let’s kind of scratch that for And then. So, thinking about eye tracking, eye tracking can be done in person and it can also be done remotely. So, that component the in person or remote component that to me is not an inherent or core part of eye tracking as a method. I think what makes eye tracking a method or the point is that this enables us to glean certain insights by by measurement of where people are focusing their attention, what people are prioritizing what people are going back to, etc. and we can take that a few steps, a few steps further. We know already that what people say is only a citizen part of the picture and Other parts like body language, and other ways in which people convey sentiment are not conveyed in, in, in verbal form. And so I’m personally very interested in kind of sentiment analysis within interviews, because I think that would add a lot of richness to to what we do as researchers, and we’re starting to see the technologies that could enable such methods of sentiment analysis within qualitative research to happen. I haven’t seen much of that research go on, I’m certainly not doing it myself. I don’t think we have adopted the technologies that Wealthfront to conduct that type of research, but I find it intriguing. So that would be an example for me of other methods in this case, sentiment analysis. It’s a novel implementation of that method, because we’ve been doing sentiment analysis for text for a while now and and we can take a bunch of tweets or take a bunch of feedback from the App Store and analyze it for full service and sentiment, but within one on one interviews, whether remote or in person, I don’t think I’ve seen much if any of that happening. And so that would be an example of a methodological innovation in my mind, that we could be thinking about moving forward. That there are also you know, in the realm of psychology, for example, in academia, I mean, there are a bunch of researchers who are dedicated to to methodology, essentially, or things like developing new statistical methods or packaging. For platforms like AR, to enable others to analyze their data better, and do more rigorous science and get better results. We don’t really have that function. I would suggest in UX research, we have researchers, research ops has been on on the rise for a few years now as and is an incredibly important component of research. I don’t think I’ve seen anyone claim the title or show particular interest in UX research methodology as a focus as a professional focus, whether it’s within any particular company or research team or whether it’s, you know, a freelancer who doesn’t who isn’t affiliated with any particular company. So that to me is another another potential step our field could, could move towards to advance our methods. Not sure I planned to dive so deeply into that so tell me if you find it interesting.
Steve: Oh, I find it really really interesting. And, you know, it evokes for me, I mean, just your proposal that we’re not innovating within UX research around methods is just, I mean it’s a strong point of view and I don’t disagree I just hadn’t really ever thought of that. It just reminds me of, you know, in my trajectory, a very early era for me when it felt like there were lots of innovative methods. And so a couple of things come to mind that maybe I’ll just throw out and see, you know how this fits into your into your kind of hand waving. I’m waving my hands kind of an architecture of war. You know, what’s innovative? What’s sort of? What’s a method? What’s a medium? I’m thinking about Bill Gaver, who I think was at Xerox PARC or at EuroPARC at the time, I can’t really remember. I mean, those both might be wrong. But he came up with something called cultural probes. Which is, and I’m also not going to get this right. But it was, I mean, a form of a diary study, but you were kind of creating kits to send to people. So it was asynchronous. It was at a distance. It was analog, but not necessarily analog. I’m recalling it properly. And you know, it was just, and maybe some of those things. We don’t you know, something’s an innovation and then it just becomes every day. Like, I mean, I’m describing, putting a packet together of things and sending to people well You know, if you go back to when we had before digital cameras for a while there were disposable cameras and they were cheap and you could like just buy them and send them off to people and they would take pictures and send them back and you know, now we have discount there’s a platform for recording your own video. So the technology changes and you know, this idea of cultural probes which is like a pretty great term. I always thought it kind of it doesn’t feel innovative anymore, but it was I think it was you know, came out of came out of academia or kind of the academic labs in industry there was a group in Europe I wish I don’t I don’t off the top of my head know what the sources but I’ll look it up. They had a method called moving with a magic thing. I wish I hope I have that right. And that was basically if you want to understand that behavior, you handed somebody a block of wood. And I think you either described to them what functionality we have, or they don’t remember whether you determined or they determined, but they would carry it around for a while and then come back and report on. You know what it changed for them even though you were so just now we have, you know, we have a minimum viable product or we have, you know, a paper prototype, but we have lots of ways of sort of creating the appearance of capability. But this was very mean. Sort of a little hand wavy and very European and sort of rooted in a in a period in the, I don’t know, maybe the late 90s early 2000s those felts exciting to me at the time, and then they they are so I don’t know, I guess My question, you know, I don’t know if you’ve heard of those things or how they kind of fit into your, you know, thoughts about what methods are and what it means for methods to be innovative. And is that changing just you know, as decades fly by?
Noam: So it’s a question. I’m not even sure I have the answer to, I think we have to ask ourselves, where is the ball or what’s the tipping point where we can say that we have innovated on a method and diary studies are a perfect example you just mentioned them. Diary studies have been going on for decades now. And as you mentioned, they started out with people sending physical packets to to other people’s mailboxes and waiting for responses and just doing that, you know, a few times until they get the results they need. And indeed, these days, we Have modern fantastic platforms like dscout for diary studies, and they have completely digitized all of the process and have also added a bunch of functionality that was obviously not available to two research shows back in, in in, in previous decades such as video responses, for example, which offer an incredible richness to, to our, our clients, our users responses and to our understanding of what they need and what they want from our product. But I think that also begs the question, is that a fundamentally different method or is that an innovation on on the method that was used previously by sending people a letter or a packet in the mail or not? And I’m not sure what the answer to that is. I think it could be very, very subjective in my opinion. It’s it’s possible that let me rephrase that, in my opinion. I think it it’s fair to say that diary studies haven’t fundamentally changed in a while now. should they? I don’t know. But as a method, I’m not sure. It’s, it’s it’s evolved enough where we can say that we’ve completely innovated or we’ve reached a new era in diary studies.I wonder how you feel about that.
Steve: Yeah, I agree. And I think you’re I agree with you. Acknowledging that some of this is not it’s not answerable. At least, you know, I mean, I’m trying to I’m trying to get you to answer that’s my job here. But you also acknowledging like, yeah, we can’t really answer that, and I don’t disagree at all. And I think that’s a good, you know, I’m starting to server sort of see, like, yeah, diary studies or diary studies, and we do them differently and they’re richer and we have, you know, the pace is different and so on. But it’s still, it’s still the same kind of interaction with people. We’re trying to get a certain kind of information. You know, talk about sentiment analysis versus just, you know, an in person interview. That’s a very different type of data that’s being pulled into, into the research that, that I think we’re not getting now.
Noam: I’ll tell you, I’ll tell you what, though. I’m when I was in graduate school, I was in the social personality division within the psychology department, and obviously social and personality psychologists differ quite a bit in what they find most important social psychologists have focusing on the effects of the situation and the context people are in. Whereas personality psychologists are focusing on individual differences in characteristics and how they drive variation in in behavior and outcome in in real life. I think one thing that has potentially greatly affected and and maybe this is an innovation in, say diary studies, is the fact that people are still mostly self reporting, whether it’s in text or video, but they’re doing so on a mobile device. And they’re doing so in various places and contacts, which we know because, you know, potentially we can get that information as well. So these days, as we can all imagine, there’s a lot of information about each of us out there, online and our behaviors. It’s a very rich data set. It raises great ethical questions, security questions, other questions about how we all want to live our lives. But the fact is today, more than ever before, there’s a lot more data out there, not just about what we’re reporting and and what we’re answering in, whether it’s those written packets from the past or modern diary entries in the present through platforms like discounts. There’s also all of the context around that when it happened, wherever it happened, and countless other things. And to me that is potentially an innovation because it bridges between the two realms I was once part of and will always consider myself pause which is the social aspect and the individual aspects. And of course, we can add to that the cultural aspects and so forth and so forth. So, potentially, if, if, obviously, we get those data and people agreed to provide those data, we can get a much richer picture than ever before, about people’s behaviors about the context in which they happens in so many other things. And, and maybe that is a very meaningful step forward in how well we understand people.
Steve: It makes me realize and, you know, depending on your background, this is not going to be particularly novel observation, but a lot of research that, you know, we do in the field is focused on an individual. And, you know, we might occasionally look at a group of people, if we’re looking at something that takes place in a family or in a household, maybe some work processes, but we don’t seem to research across a network necessarily. I got at least I haven’t come across it.
Noam: That, yeah, this is something that absolutely resonates with me. Again, going back to my my academic days, I was working with an advisor who was doing a lot of what we call dyadic research or couples research, because we were working on romantic relationships, dating, online dating, attachment, stuff like that. And fast forward to to today at Wealthfront. Where we work where our work is about people’s financial lives. Then if you’re if you’re married or in a long term relationship, and you have joint finances, with your partner Now, then, obviously many of the questions on your mind, many of the needs you have are joint needs and joint questions. They are not individual needs or individual questions. But all too often, including at Wealthfront. And everywhere else I’ve worked, we do do the research at the individual level. And that is a very partial picture and a very biased picture of of people’s people’s realities. You can imagine that if you have to two people who are in a long term relationship and you ask each of them separately, about their finances and where they’re at and where they want to be. You may just get very different answers that don’t take the other person into account enough. You’ll also get very interesting answers when you ask each member Have that family to report on the other member of that family and what they think the other member believes about, in this case, finance. So, I definitely agree that one of the ways in which we can think, among other things about methodological innovation is the level at which we’re conducting the research. Is it the individual level? Is it the couple or dyadic? level, the group level? And when should we be focusing on that particular level? And also, should we be can we develop methods that are uniquely tailored to that level of of research? How can we conduct something like a joint interview in a way that will yield more robust insights? And does it need to be substantially different than how we’d interview Each of the individuals within that relationship
Steve: If I was just gonna wave my hands and imagine anything is possible, you know, having done a little bit about this topic and in my career as well. Seems like, you know, the family childhood family upbringing is a huge determining factor for a lot of beliefs and sort of why couples have different approaches because they have different backgrounds. I’m oversimplifying it, but boy if you wanted to, you know what, if you could maybe not literally go back in time, but it’d be interesting while we’re just you know, imagining Anything is possible to you know, interview the group that sharing the decision, but also learn about the, you know, the origins of those assumptions, rules, expectations, beliefs, and so on. From let’s just say it’s the family that they grew up in, you know, that I have no idea how you do that, from just a like a recruiting point of view or a logistical point of view or making sense of the data point of view. But just thinking about people as networks and where there are people as being in networks and where a lot of things that drive their behavior come from are not at the individual level. It does open it up in a way that I don’t know how to address.
Noam: Well, I too, struggle with that. And I, I struggled with even imagining what the methods would look like in order to get at the historical context of an individual or family, but I can tell you this. I have two children. And I have more photographs of my children than I do know than I ever imagined. I would have I have more data essentially about my children and I will always have that That data, then then I ever imagined would be would be possible that the richness of what I know about my children and could potentially share is, is crazy. Or another example is we can all go to Google or Facebook and look at years and years worth of history of where we’ve been, what we were up to, what you know, what we ate, and who we hung out with. And so, the next generation of researchers will certainly have access to an incredible trove of historical data. Hopefully, they get the proper permissions to access those data, and and respect the privacy of the individuals they’re researching. But assuming they do, I actually think it’ll be easier than ever before. To get such an incredible picture of what led the people wanted researching to the point where they’re at in their life. If you combine all of those data together with advanced analysis methods where we’d have the support of various fancy machine learning AI driven algorithms, then I actually do think that in just a few years, it will be feasible to understand people at a level where we’ve never been before, in particular, what you were just mentioning around where where everything began for those people and how they were raised and, and, and, and how they got to that personal and professional point in their lives. So I do see glimmers of or a glimpse into this potential future. I often wonder whether it will be useful or not, and how we’d analyze such data. And is this truly the future of our profession, but what I’m sure have is that too It will be a possibility, given everything that we we know about people and how we conduct ourselves these days with technology.
Steve: We’ve talked about a couple of things. So far and just very broadly, one is, you know, how research is working, you know, the roles that you’re playing, what you’re trying to help people accomplish at Wealthfront. And then, you know, your, your broader kind of observation that, that in terms of our methods, we’re not really focused on them as a practice, you know, as a profession, are there things that you can do? I like what you said, you know, this, that innovation and methods could come from anywhere. But is it something that there’s an opportunity for, for for Wealthfront, for you to be pushing methods ahead.
Noam: So I think that one of the beneficial results of democratizing research as extremely as possible and enabling everyone in the company whether it’s designers, PMs, marketers or others to conduct incredible research, and I’m fortunate enough to have colleagues who have amazing research chops and they do incredible research that frees up part of the time of the user research team. And and us as user researchers to focus on a some of the more strategic research projects that we have in the company. And it also will be it’s enabled us to focus on on research innovation, and either implementing tools that allow for better research to happen, for example, better knowledge management, of the research we already have, and are already doing and it also enables us potentially to focus More on developing tailored methods or frameworks that work for our context. And FinTech is in many ways, different or unique compared to to other areas. And there are certain things that we care deeply about. And so I think that the result of democratizing research as you assuming you accept that, that is a viable route to go down is that you are able to focus more on on other things, including that innovation I mentioned before. So we will definitely continue to try on the user research team to either implement new tools and platforms that are coming out that make us all better researchers, or innovate within the company. And and of course, if we can share those innovations with the broader community In order to develop methods that are better, a better fit better tailored to what we do, I can give. I can give an example here of, you know, something we care about deeply I would say in FinTech especially, is, of course, around the issue of trust. And the measurement of trust, which has, I would say, been a tricky thing, historically, for researchers, both in academia and industry. At Wealthfront, we are often handling people’s entire life savings investments. That’s a very big responsibility which we take incredibly seriously. And we want to understand how we can earn people’s trust, how we can build on that trust how we can augment it. We want to monitor the trust in us and and understand it deeply. And the measurement of trust. Well, measurement in general, is incredibly complex. And so developing great measurements for trust is definitely a goal, which I’m not sure we’ve, well I actually am sure we haven’t yet reached at Wellington, we aspire to do that. I’m not sure if anyone has. I think that as a research community, we can barely agree on how to measure satisfaction. And often we use deeply flawed methods just to do that. And in my opinion, measuring satisfaction is much easier than measuring something like trust. So more to come in the future, I hope including from us.
Steve: Maybe in the we have a few minutes left, it would be interesting, hopefully, to hear you. I’m going to start again in a few minutes that we have left I would love to hear a little bit about maybe a compare and contrast. The other places that you mentioned you’d worked Airbnb and intercom, you know, what are some of your perspectives on, you know, research being applied in very different organizations and what you experienced as a researcher in these different types of companies.
Noam: I’d love to do that. And by the way, I have as much time as you need because I feel like we covered a very specific thing for most of this conversation though I do find it fascinating. So for me, Airbnb was an incredible experience. I think what drew me to Airbnb more than anything was the mission at the time, which I believe is still Airbnb, his mission of belonging anywhere as an immigrant to this country, and as someone who’s traveled quite a bit and be To a few places around the world, I definitely know what it feels like not to belong somewhere. And I definitely know what it feels like to feel belonging. And I think that today more than ever before, it’s really important to break out of one’s bubble experience other cultures experience other places and hopefully experience that belonging. So, to me as a researcher, Airbnb, I had a very strong connection to to the mission. And there are some very unique aspects of research at Airbnb, for example. Airbnb is one of those products where the company’s aim is for people to use the app essentially as little as possible. And and venture out into the world the way Airbnb makes money. And grows its platform. And its benefits to the world is when people go out to travel not engage with with the app. And so a lot of the research at Airbnb isn’t just about the online app or web experience. It’s about the offline experience on the ground. It’s about the travel itself. It’s about things like the check in and the checkout experience, or what it’s like to go on an experience hosted by someone who lives in that city. And they happen to be experts at making pasta and they want to share their food passions with with the world. And so researching these offline experiences in a cross cultural settings in almost all countries across the world, is a very different experience than some people I would say have been in the UX research industry. Also, I credited with Be personally as a company founded by two designers and one non designer, but it’s a very design driven company. And in my mind, Airbnb is one of the best design schools in the world without officially being a design school. I learned most of what I know about design. And a lot of what I know about research for my time, at Airbnb. And the travel space is, is very challenging and very interesting. And I hope that in this time, when people aren’t traveling at all given a global pandemic, I hope we get back to that soon because it makes all of our lives a lot richer and a lot more interesting. And it’s some of the most fascinating research I’ve ever done. Another thing about about Airbnb, which is also a question I think in our industry, is how does the function of research operate within the larger organization? For example, at Airbnb researches is part of the design team. Whereas at Intercom for example, research was reporting into one part of the product function. We also have a bunch of different titles in our industry from experience research to design research to product research, user research and so many other things. And I know we discussed this a lot, but it does make a difference to one’s experience. Whether they work more closely with product and PMs or more closely with designers and, and design leaders. It really influences one’s experience, the things you can learn your your relationships with those functions and so forth.
Steve: But there’s a caveat to that I think it in looking at the two companies that you’re comparing and having not worked at either the missus, you know, definitely outside or hypothesizing but I could see be if I worked at Airbnb, I’d want to be working with him. He said, it’s a design. It’s a strong design company. And I think of intercom as a strong product company. And that, so it’s different where it’s different sort of who you report up to, or what organization or what roles you’re working closely with. But also that’s that is different in different organizations based on just the strength and maturity of those different practices.
Noam: Yes, I agree.
Steve: so just to clarify, You know, there’s a lot of different titles or the titles connected, is there a relationship between, you know what, what your job title is. And this this issue sort of, you know, who you report up to where the research organization sits, is that impacting the range of how titles are, are kind of applied to researchers.
Noam: So I think the titles we get as researchers are a function of a few things that are a function of the organization, their functional, its philosophy. They’re a function of how the organization itself is designed, and what the emphasis is on and what type of relationships the organization wants to cultivate more and how the company wants to be perceived, as well. I want to mention something about our work. At Wealthfront these days, some relationships are, in my opinion, so natural that it’s surprising that they don’t just occur within companies and don’t just happen and don’t just thrive without any particular intervention or organizational design. And yet more often than not, those relationships are missing at Wealthfront. We recently created something we call the research Guild. And the research guild is composed of user researchers like self consumer insights, which is typically the side of research that kind of belongs organizationally to marketing or perhaps in some cases, product and data science. Now, again, the relationship between user research, consumer insights and data science, in my opinion, is natural and obvious. And yet all too often, it’s not a thriving relationship, it might even be a very broken relationship. And so what we’re trying to create wealth and these days is is a much closer relationship between those functions Because ultimately, we are all trying to do the same thing, which is enable every single person in the company to understand our clients and potential gains and better to be closer to them. To hear them in a variety of ways. Whether qualitative or quantitative, it doesn’t matter. high level, we all have the same mission. And, and so maybe more, maybe something more important than the titles we have and constant discussions around what UX is or what UX isn’t or who is or who isn’t doing researcher, I think we might want to focus more on the collaboration between functions and the relationships between functions. And to really amplify these truly important crucial relationships between, for example, data science and, and user research. It makes all of our jobs better, it creates better outcome to create a better understanding of our users and our clients. And it’s, it’s one more thing I haven’t always experienced in my career and I wish I had so definitely trying to, to correct that now at Wealthfront and and create those, those beneficial cross functional relationships.
Steve: So how will this guild enable those relationships?
Noam: So what I’m again when I’m not on parental leave and actually working we have both leadership meetings for that guild with the managers or directors of each each group to keep each other in the know about projects and things going on and what we’re doing and how we can help each other and collaborate better. And we also have meetings between all of the guild members where we can provide each other with feedback on projects, whichever stage those projects are at, or ideas for new new directions or support, where support can be, can be provided. And anything else we can come up with. And just by having those couple of meetings, I think we’ve already seen so much benefit in terms of every project we conduct being that much stronger thanks to much broader feedback, much deeper feedback. And a variety of perspectives from people who have a very diverse set of skills, academic backgrounds industry experience, and to focus on on a variety of things in in their day to day.
Steve: And how did you come to use the term guild to describe this?
Noam: So I’m, I’m all about etymology, and very into into words, I think what drew me to the term guild and the reason I like that suggestion is that in most dictionary definitions of Guild, you have the notion of an association of people who are in the pursuit of a common goal. And I think that’s what’s important to remember that data scientists, consumer insights, or marketing research people, user research professionals, we are all in pursuit of a common goal at the, at the high level. And we should be collaborating as closely as possible in order to, to to reach that goal. And I think all too often we don’t understand each other’s work well enough, we don’t collaborate well enough and leverage each other’s work, we might sometimes repeat each other’s work or or you know, have like dual projects which are quite similar in their, in their outcomes. And so by focusing efforts on that common goal, and leveraging each other’s knowledge and skills. I think there’s something very powerful in that, which definitely exists in a bunch of companies in, in, in, in tech and otherwise, but could be could be stronger for sure.
Steve: So the thing about podcasts, I think, is that they, you know, we’re having a conversation on the day that we’re having it and someone may come to this at any point, you know, in the very near future or later on. So, you know, you’ve you’ve referenced our particular moments in time a couple of times or in the midst of a global pandemic. It’s really changing well as You know everything about everything for everybody right to work from home. People are sheltering in place right now in the Bay Area. And it’s forcing a lot of questions. Or, you know, for some and maybe not for others about research. You know, what we’re doing right now sort of how do we even think about how do we think about research our profession in a time where these other things are going on? And I’m wondering if that’s something that you’ve reflected on if you have a point of view about today about research today?
Noam: Yeah, I’ve definitely reflected on this, especially about a couple of points. The first is that I’ve, I’ve been passed off and seen lots of discussion around research ethics in this kind of period of time. People are potentially in also distress, a lot of anxiety. And so there’s the question of, it’s perhaps a philosophical question even of whether we should be approaching our users at a time like this. And what can we make of people’s responses when they’re in such a different and often distressed state of mine, given given the pandemic? My view on this is that it’s it’s crucial to continue to be such that such at such a time, certainly with more sensitivity and and a realization that there are probably limitations to to the research. But as someone who works in financial technology at a time when the financial markets are incredibly, incredibly volatile, I am personally happy that we are continuing at welfare to be attuned to what our clients need and what they hope for the future and to provide that for them. Because it’s exactly this type of situation where we want to to best serve our our clients and help them avoid potential mistakes that could be very costly to them. So I think there’s a need for much greater sensitivity at this kind of time. But I think researchers have an important role in pushing forward with research as much as possible. Maybe taking on additional projects that that benefit the health or medical services, community, and so forth. The other thing is, I think a big part of this past maybe decades and use research has been around research Ops, and how we can conduct research in a more lean fashion, fast paced, lightweight. For many companies, in person research travel, it’s a core part of the research practice. And yet today, we’re having to grapple with with this fundamental problem that we can’t do that anymore. And we’re forced to develop new tools and new ways of of getting to and connecting with our users, even when we can’t do that in person. And I think it’s important for every every company and every team and every researcher to think about those different contingencies and What can we do in a situation where we want to connect with the user, we want to understand them, but we have to do so in a in a remote interview or some other technique, which doesn’t include meeting in person. And this goes back to our discussion around methods and methodological innovation and what we can, what we can do around that. I personally hope that in the future, when maybe this time, augmented reality, virtual reality, other techniques become more of an established practice that we will be able to share spaces and share ideas and feel together and conduct kind of pseudo in person research even if we’re not. And I think there’s something very, very powerful in that. In fact, maybe that will drive part of the growth finally in in those types of methods, this type of event, I think it’s going to be very interesting to see the impact of this double pandemic on how we conduct our, our research, because ultimately, our profession is all about connecting with people and understanding them. And that is much, much harder to do when we’re all quarantined in our own houses, waiting for this to, to end.
Steve: Well, I thank you so much for your time. And, you know, it’s fun to go really deep into a few things. I appreciate you, you know, going all the way, all the way into the nooks and crannies of some of these ideas that you’re thinking about gives us a lot to chew on. So thank you so much for being on dollars to donuts. Thank you as well. It was a real pleasure.
Noam: I’ve been waiting for this. And I’m really glad you invited me. I appreciate it. Thank you.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 31. Noam Segal of Wealthfront first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Noam Segal, the Director of Research at Wealthfront.
Everyone from PMs to designers, researchers, obviously, engineers, data scientists, marketing, we’re all trying to to understand our clients, we’re all taking part in that process in some way, shape or form. And so I view my role and user research’s role as an enabler, as a coach, as augmenting other efforts already happening in in the company, and really maximizing the returns we get on on the research we do. – Noam Segal
Show Links* Dolores Huerta and Alice Waters on City Arts & Lectures * Noam on LinkedIn * Noam on Twitter * Wealthfront * Cultural Probe * Moving with a Magic Thing * dscout * Airbnb * Intercom
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I just watched an interview with Alice Waters, who describes herself as a chef, author, food activist, and the owner of Chez Panisse Restaurant in Berkeley. She’s often referred to as the creator of California cuisine, back in the 70s. At one point in the conversation, she explained how when she meets people, she will sometimes begin by bringing out food, say fresh fruit, and taste the food together with the other person. She explained how this shared sensory experience was an alternate, and perhaps more effective way of creating a connection between people.
It really made me think, wow, is that something we could do in user research? Could you begin an interview by sharing a sensory experience with someone? It could be taste, via food as Alice Waters did, but it could be a touch experience, a moment of smelling, a shared point of listening. I would want to understand more about what Waters believes this accomplishes and if it’s conjecture or there’s any more evidence, and I don’t really have any idea how to introduce something like this into an already hesitant dynamic, that initial moment.
And with all (hopefully all?) research happening remotely at the moment, is there some sort of shared over distance aspect of this, a sensory experience that both parties could initiate, maybe it could be entirely pedestrian, such as feeling the glass of a mobile device, versus something celebratory like a piece of fruit selected by Alice Waters.
I don’t know what this would lead to but I’ll be curious to hear what happens for people that try it. As well, it serves to remind me that all too often I neglect including all the senses in how I process the world and how I engage with others.
This is another episode without either professional editor or transcriptionist. This podcast is my way to contribute at this particular moment, but I hope you can keep me and my practice in mind for collaborating on research, for coaching, for training, and other work to help advance the maturity of your organization’s research practice, wherever you’re at currently.
Now, let’s get to my interview with Noam Segal, who is the director of research at Wealthfront.
Noam, thanks for being on dollars to donuts. So it’s really great to have you here.
Noam: My pleasure. Thanks so much for having me.
Steve: So my typical way of beginning which we can do today is just to ask, ask you to introduce yourself.
Noam Segal: Sure. So my name is Noam Segal and I am currently on parental leave, actually, with my six month old, baby boy Dean. But when I’m not on leave, I’m director of user research at Wealthfront. So that’s what I’m currently up to. I’m originally from Israel, although I don’t sound like it because my parents are both English and somehow I inherited the British accent. And I moved to the US in 2012 with my wife for graduate school Originally, I got my PhD in psychology at the University of Illinois, Urbana Champaign. And when I graduated, we decided to move to the Bay Area and both pursue positions in tech. And I’ve been in UX research ever since and have worked for Airbnb, Intercom and again these days Wealthfront.
Steve: How did you find out about UX research?
Noam: So, as an Israeli, we have mandatory military service, and I served in a unit within the Israeli Air Force for most of my, my service. That particular unit is in charge of Israel’s missile defense systems. And when I left the military got discharged, I decided to join the military industries. Kind of the Israeli equivalent of maybe Lockheed Martin in general dynamics, those sorts of companies. And I focused on developing the UX, but we didn’t call it UX at the time of these missile defense systems, primarily the our weapon system, which is a system designed to protect as well, from medium to long range missiles. This could not be more different than what I do today. Very little relationship, but that’s when I really started appreciating and and desiring to be in this field of UX more, more broadly, that was my focus, I focused on the interface. I focused on understanding from the operators of these systems, what they what they need, and it was an incredible six year learning experience. So that’s kind of how I got into it in the first place. Later on I wanted to improve my my research abilities and knowledge and I was definitely considering going into academia for a while. So that led me to the Ph. D program. But at some point, in year four of the PhD I realized that I love two things. I love research and being a researcher and I love technology. And there is a profession that marries those two things quite perfectly. And that is UX research. So I decided to shift my focus back to industry and and going to work at first with Airbnb and carried on ever since.
Steve: Did you finish the PhD?
Noam: I did. I did. It was a hectic time in my life. I was in the middle of my fourth year in world Typically a six year program. That said, it’s typically a six year program because of how difficult the academic job market is rather than how long it actually takes to finish a dissertation. And I realized and there was some events in our lives, both personal professional that I want to take a different path from academia. And so I interviewed with several, especially Bay Area companies and was fortunate enough to get a couple of offers and decided to fast track the rest of my my dissertation. And I proposed the research I was going to do in January, and I defended my dissertation on June 1 of 2016. And less than 24 hours later, we were on a plane to San Francisco to stop life in the Bay Area. So it was a crazy period in our life. And I’m rather happy it was over because it was quite difficult. But yes, I did. I did graduate, it was very important for me to finish the program. And I benefited greatly from my studies, my relationship with my advisor, and all of my fellow students at Illinois, really appreciate that time, it taught me more than I could ever imagine.
Steve: So maybe I can just go back. So you kind of described this. And we have like, book ends, maybe have a narrative here, where at the beginning, you are, you know, in the, in the army and kind of looking at those things. And at the end of this chapter anyway, you’re getting on an airplane to San Francisco to enter the, you know, Bay Area, user research roles. But I’m wondering, going back to when you were in the army, you started as you said, You didn’t refer to it as UX. So I’m wondering, when did the research part like that this was a thing that needed to be done that could be done that you could pursue as an actual job. Is there a point at which that started to enter your consciousness?
Noam: Absolutely. So when I was in the military industries, I wasn’t in a research role, efficiently. And I also wasn’t particularly aware of the realm of UX research. But UX research is not really an existing profession in Israel, a separate function typically, in Israeli startup companies, it will be the founders or perhaps the designers that do the research. And so I was not in in a research role, but I was working in a very small team, trying to develop systems that were completely new to the world. incredibly complex, and the stakes couldn’t be higher. These are systems that are enabling Israel to defend itself from potential nuclear attacks. It really is life or death. I mean, in industry in tech, perhaps you’ve heard this before, we often use militaristic terms. Like we have war rooms if we launch a new product, or we conduct a post mortem, when something goes wrong and engineering have to figure out what happened. But I really was in in a company in a team where the stakes were incredibly high. And that requires research, whether you’re in that official role or not. I was in a unique position, where on the one hand, I was working in this team as a civilian, speaking to people who operate the system thinking about the user experience. This is the system where people have, in some cases just a few seconds to make very critical decisions. And failure is simply not an option. But on the other hand, I was also an operator myself. So I was still in the reserves, still serving in the military still doing one or two days a month in some cases in the unit as a reserve soldier. And so I got to see and experience these systems from those two, two sides of the coin, which was very, very interesting. And obviously, this happens to to us, as well here in the Bay Area, often we are using the systems we’re also developing, that’s quite common. But that was the first time that happened to me. And I think one thing that really struck me developing these systems and I think it’s part of the reason I decided to move on is that I really, truly believe that failure is a critical component of learning. And failure is not really an option when what you’re developing is missile missile defense system. Um, and that’s a very heavy weight to carry on your shoulders. For anyone who is part of that sort of team. It’s a huge responsibility. It’s not the type of intuitive work that we do in the Bay Area, you cannot fail often, and, and learn, you know, move quickly and break things. You can’t do that when you’re in that type of industry. And so on the one hand, it was an incredible experience and really my first venture into UX. And on the other hand, I really needed to move on in order to become a better researcher, and a better professional within the UX research community. So that’s a bit about my time there.
Steve: Yeah. And what did you do for undergraduate?
Noam: So I went to a small kind of liberal arts college in Israel. And I studied psychology as my major. And I took kind of Media Communications as my, my minor. I was in the first cohort of psychology studies at this college, they just opened the psychology school the way I started. And maybe maybe it’s a personality thing, but I love to be part of new programs, new things, new ventures. And then I had other options to go study at more established universities, which you may have heard of Tel Aviv University as an example. I really wanted to join this this new program, and I have to say it benefited me greatly. The learning experience was incredible. We were a small cohorts of people. We have very close relationships with our professors. And the fact that I went there was a big part of I think, why I ended up getting into graduate school in the US and being able to make this move and ended up where where I am today. So it was a wonderful undergraduate experience, very small campus. My first time moving out to my parents house and and experiencing a lot of lots of new things. I really appreciate that. That time and the psychology the foundations of psychology gave me which were very helpful later on. And until today, really.
Steve: So tell me what kind of company Wealthfront is.
Noam: So Wealthfront, I would say has been through a few different periods, different stages. It began as a platform where you can innovate vest your money passively, passively invest your money in in the markets later on Wealthfront added planning features to enable people to plan for their financial future. And most recently, Wealthfront has added a cash account, a high interest cash account. And later on in the year, we’ll also be launching a bank account for our clients. And so I would say in general, what we’re trying to do at Wealthfront is provide people with incredibly sophisticated tools to manage their finances, manage their investments, manage their cash, plan for their future, and hopefully, achieve their their life goals. Through through well planned and invested finances.
Steve: Yeah, what’s the research practice like at Wealthfront.
Noam: So it’s interesting, you should use the term research practice because research at Wealthfront is quite different than what I experienced elsewhere. First of all, when I joined Wealthfront, the research team was not the UX research team. It was the team in charge of the algorithms and the math powering our passive investment engine. And these days, that team is called the data science And there’s a research component to that. So in FinTech research actually means something different than in most other companies and that took getting used to for me. But research UX research at Wealthfront is an interesting topic because as a practice, it has existed since the company started. And everyone from the founders of the company to later on PMs and designers at Wealthfront have all been conducting variety of research, project methods, and staying connected and trying to understand and best help our clients and our potential clients. But user research as a function didn’t really exist that Wealthfront before I joined just over a year ago. So I’m the first person at Wealthfront with user research in his title. That’s maybe not completely accurate because my manager, a picture, who is VP of design at Wealthfront actually started out as a researcher, but she transitioned into a design role quite a while ago. And I was the first person since to join as a researcher. So as a practice, it’s existed for for over a decade now, as a function, what I’m trying to do Wealthfront is really build out research as a function, which means a bunch of a bunch of things. But really, the point is that research at Wealthfront is incredibly democratized. And I very much support that approach. And so by democratized I mean that we’re all running research trying to better understand our clients and potential clients. Everyone from pm to designers, researchers, obviously, engineers, data scientists, marketing, we’re all trying to to understand our clients, we’re all taking part in that process in some way, shape or form. And so I view my role in many cases and user research his role as an enabler as a coach, as augmenting other efforts already happening in in the company, and really maximizing the returns we get on on the research we do.
Steve: What are some things that you do in coach examples or it’s such a powerful word? What are ways that you are coaching?
Noam: Yeah, so we are currently two user researchers on the team. seem to be a third. Hopefully, we’re currently in a, in a weird situation all of us as we’re recording this with COVID-19 out there and a global pandemic. But we are growing the function of user research. Until then, and regardless of that, we hope to enable especially designers and PMs more than anyone else to do better research. And to do that twit were using a few different techniques. One of them is creating lots of materials in our internal company wiki around best practices for a variety of methods and research techniques. There’s no reason to reinvent the wheel every time and rethink how we should conduct a usability test or what best practices interviews are or what type of satisfaction metrics we can use, etc, etc. And so the first step there is to make sure those materials are out there for everyone to consume. And that’s a work in progress for us. The second thing is running workshops or interactive lessons, webinars, whatever whatever it takes to engage with various teams within the company, and enable them to learn in a group setting, about research best practices, and how they can level up their research. And then another technique or another way we do this is indeed, just one on one coaching, with with designers with with pm. We started doing this a bit. I hope when I get back from parental leave to continue doing that. That’s all of interest from many people in the company to have such sessions and get feedback on on their research sessions and what could be better, and what should what should remain the same because it’s already great. Our designers and PMs are well versed in research and they’ve conducted countless research sessions of all sorts. But there’s always something to learn for all of us. And so through those various methods, whether it’s on an internal wiki or whether it’s in a group setting, or whether it’s in a one on one coaching session, we believe that through those various ways of coaching and teaching, we can all do better research. Not to mention, teaching is maybe one of the best ways of learning, in my opinion. And so one benefit I feel of being a coach and an enabler and a teacher is that you have to constantly reevaluate and rethink what you’re doing. And then about the methods you’re using, about the best practices about the principles behind your research, everything you’re trying to, to accomplish. And I’m really happy about that because I’m not sure as a community within user research we’re doing that enough necessarily so i think it’s it’s a it’s a positive side effects of of taking that coaching approach.
Steve: So what is it that we’re not as a community overall doing enough?
Noam: Debatable, but I’m, I’m referring to revisiting the methods and techniques that we’re using as researchers. Use the research or UX research method Dollar discussions
Steve: Anything there anything specific that you’re thinking of that is unexamined that needs to be reconsidered or revisited?
Noam: Well, I’m curious to hear your opinion as well, I, I feel like many of the techniques or frameworks were using in user research. And one specific example I can give is Jobs To Be Done, which I really learned the most about in my time at intercom. Most of those techniques and methods are taken from other realms such as marketing. In the case of in the case of Jobs To Be Done, it really started out more than it was kind of imported into the world of software development and and product And I wonder if I do often wonder if there’s more room for user research specific methodological discussions and innovation around the methods we use? I think broadly speaking, I haven’t seen much of that. And I think it’s debatable whether that’s needed, or, or not. But I think that as we develop new tools, new ways of working remote research is a classic example right now, given the global pandemic, when there’s also a need to establish potentially new methods to go with that reality with those technologies and think about how we do how we do research. The fact that many of our methods and frameworks are imported from other realms, is maybe just partially a sign of how long UX UI Such has been going on as a distinct profession and function. But maybe it’s time for us to to branch out ourselves and dive more deeply into into those methods we use. So hopefully that answers some of the question them.
Steve: What’s the granularity of a method like Jobs To Be Done to me, and I know you’re talking about methods and frameworks, you know, I’m just thinking there’s, like remote research is an umbrella term for a certain set of things, Jobs To Be Done is. And I’m thinking about, you know, he talks about innovating in the methods. I’m just not sure. We all that’s such a great word and problematic because we all think it means something different. So. Right, I’m thinking about some of the things that have come up in remote research when people were saying and I’m not gonna have exactly right, but how can I see my screen and the other person’s screen if they’re on a mobile device, and then people have come up with various solutions and sort of technologies and ways to hook up different pieces together so that you can remotely have a certain kind of interaction is what we consider that and innovation in the methods.
Noam: So, I view that example as more of the medium than the method. Yeah. So if we take eye tracking, as a method, because I do consider eye tracking a method and to maybe one way to define the method given that is a way to, to I’m kind of losing my own train of thought here, but kind of, I was gonna say like a way to gather insights, but that’s also the medium right? So let’s kind of scratch that for And then. So, thinking about eye tracking, eye tracking can be done in person and it can also be done remotely. So, that component the in person or remote component that to me is not an inherent or core part of eye tracking as a method. I think what makes eye tracking a method or the point is that this enables us to glean certain insights by by measurement of where people are focusing their attention, what people are prioritizing what people are going back to, etc. and we can take that a few steps, a few steps further. We know already that what people say is only a citizen part of the picture and Other parts like body language, and other ways in which people convey sentiment are not conveyed in, in, in verbal form. And so I’m personally very interested in kind of sentiment analysis within interviews, because I think that would add a lot of richness to to what we do as researchers, and we’re starting to see the technologies that could enable such methods of sentiment analysis within qualitative research to happen. I haven’t seen much of that research go on, I’m certainly not doing it myself. I don’t think we have adopted the technologies that Wealthfront to conduct that type of research, but I find it intriguing. So that would be an example for me of other methods in this case, sentiment analysis. It’s a novel implementation of that method, because we’ve been doing sentiment analysis for text for a while now and and we can take a bunch of tweets or take a bunch of feedback from the App Store and analyze it for full service and sentiment, but within one on one interviews, whether remote or in person, I don’t think I’ve seen much if any of that happening. And so that would be an example of a methodological innovation in my mind, that we could be thinking about moving forward. That there are also you know, in the realm of psychology, for example, in academia, I mean, there are a bunch of researchers who are dedicated to to methodology, essentially, or things like developing new statistical methods or packaging. For platforms like AR, to enable others to analyze their data better, and do more rigorous science and get better results. We don’t really have that function. I would suggest in UX research, we have researchers, research ops has been on on the rise for a few years now as and is an incredibly important component of research. I don’t think I’ve seen anyone claim the title or show particular interest in UX research methodology as a focus as a professional focus, whether it’s within any particular company or research team or whether it’s, you know, a freelancer who doesn’t who isn’t affiliated with any particular company. So that to me is another another potential step our field could, could move towards to advance our methods. Not sure I planned to dive so deeply into that so tell me if you find it interesting.
Steve: Oh, I find it really really interesting. And, you know, it evokes for me, I mean, just your proposal that we’re not innovating within UX research around methods is just, I mean it’s a strong point of view and I don’t disagree I just hadn’t really ever thought of that. It just reminds me of, you know, in my trajectory, a very early era for me when it felt like there were lots of innovative methods. And so a couple of things come to mind that maybe I’ll just throw out and see, you know how this fits into your into your kind of hand waving. I’m waving my hands kind of an architecture of war. You know, what’s innovative? What’s sort of? What’s a method? What’s a medium? I’m thinking about Bill Gaver, who I think was at Xerox PARC or at EuroPARC at the time, I can’t really remember. I mean, those both might be wrong. But he came up with something called cultural probes. Which is, and I’m also not going to get this right. But it was, I mean, a form of a diary study, but you were kind of creating kits to send to people. So it was asynchronous. It was at a distance. It was analog, but not necessarily analog. I’m recalling it properly. And you know, it was just, and maybe some of those things. We don’t you know, something’s an innovation and then it just becomes every day. Like, I mean, I’m describing, putting a packet together of things and sending to people well You know, if you go back to when we had before digital cameras for a while there were disposable cameras and they were cheap and you could like just buy them and send them off to people and they would take pictures and send them back and you know, now we have discount there’s a platform for recording your own video. So the technology changes and you know, this idea of cultural probes which is like a pretty great term. I always thought it kind of it doesn’t feel innovative anymore, but it was I think it was you know, came out of came out of academia or kind of the academic labs in industry there was a group in Europe I wish I don’t I don’t off the top of my head know what the sources but I’ll look it up. They had a method called moving with a magic thing. I wish I hope I have that right. And that was basically if you want to understand that behavior, you handed somebody a block of wood. And I think you either described to them what functionality we have, or they don’t remember whether you determined or they determined, but they would carry it around for a while and then come back and report on. You know what it changed for them even though you were so just now we have, you know, we have a minimum viable product or we have, you know, a paper prototype, but we have lots of ways of sort of creating the appearance of capability. But this was very mean. Sort of a little hand wavy and very European and sort of rooted in a in a period in the, I don’t know, maybe the late 90s early 2000s those felts exciting to me at the time, and then they they are so I don’t know, I guess My question, you know, I don’t know if you’ve heard of those things or how they kind of fit into your, you know, thoughts about what methods are and what it means for methods to be innovative. And is that changing just you know, as decades fly by?
Noam: So it’s a question. I’m not even sure I have the answer to, I think we have to ask ourselves, where is the ball or what’s the tipping point where we can say that we have innovated on a method and diary studies are a perfect example you just mentioned them. Diary studies have been going on for decades now. And as you mentioned, they started out with people sending physical packets to to other people’s mailboxes and waiting for responses and just doing that, you know, a few times until they get the results they need. And indeed, these days, we Have modern fantastic platforms like dscout for diary studies, and they have completely digitized all of the process and have also added a bunch of functionality that was obviously not available to two research shows back in, in in, in previous decades such as video responses, for example, which offer an incredible richness to, to our, our clients, our users responses and to our understanding of what they need and what they want from our product. But I think that also begs the question, is that a fundamentally different method or is that an innovation on on the method that was used previously by sending people a letter or a packet in the mail or not? And I’m not sure what the answer to that is. I think it could be very, very subjective in my opinion. It’s it’s possible that let me rephrase that, in my opinion. I think it it’s fair to say that diary studies haven’t fundamentally changed in a while now. should they? I don’t know. But as a method, I’m not sure. It’s, it’s it’s evolved enough where we can say that we’ve completely innovated or we’ve reached a new era in diary studies.I wonder how you feel about that.
Steve: Yeah, I agree. And I think you’re I agree with you. Acknowledging that some of this is not it’s not answerable. At least, you know, I mean, I’m trying to I’m trying to get you to answer that’s my job here. But you also acknowledging like, yeah, we can’t really answer that, and I don’t disagree at all. And I think that’s a good, you know, I’m starting to server sort of see, like, yeah, diary studies or diary studies, and we do them differently and they’re richer and we have, you know, the pace is different and so on. But it’s still, it’s still the same kind of interaction with people. We’re trying to get a certain kind of information. You know, talk about sentiment analysis versus just, you know, an in person interview. That’s a very different type of data that’s being pulled into, into the research that, that I think we’re not getting now.
Noam: I’ll tell you, I’ll tell you what, though. I’m when I was in graduate school, I was in the social personality division within the psychology department, and obviously social and personality psychologists differ quite a bit in what they find most important social psychologists have focusing on the effects of the situation and the context people are in. Whereas personality psychologists are focusing on individual differences in characteristics and how they drive variation in in behavior and outcome in in real life. I think one thing that has potentially greatly affected and and maybe this is an innovation in, say diary studies, is the fact that people are still mostly self reporting, whether it’s in text or video, but they’re doing so on a mobile device. And they’re doing so in various places and contacts, which we know because, you know, potentially we can get that information as well. So these days, as we can all imagine, there’s a lot of information about each of us out there, online and our behaviors. It’s a very rich data set. It raises great ethical questions, security questions, other questions about how we all want to live our lives. But the fact is today, more than ever before, there’s a lot more data out there, not just about what we’re reporting and and what we’re answering in, whether it’s those written packets from the past or modern diary entries in the present through platforms like discounts. There’s also all of the context around that when it happened, wherever it happened, and countless other things. And to me that is potentially an innovation because it bridges between the two realms I was once part of and will always consider myself pause which is the social aspect and the individual aspects. And of course, we can add to that the cultural aspects and so forth and so forth. So, potentially, if, if, obviously, we get those data and people agreed to provide those data, we can get a much richer picture than ever before, about people’s behaviors about the context in which they happens in so many other things. And, and maybe that is a very meaningful step forward in how well we understand people.
Steve: It makes me realize and, you know, depending on your background, this is not going to be particularly novel observation, but a lot of research that, you know, we do in the field is focused on an individual. And, you know, we might occasionally look at a group of people, if we’re looking at something that takes place in a family or in a household, maybe some work processes, but we don’t seem to research across a network necessarily. I got at least I haven’t come across it.
Noam: That, yeah, this is something that absolutely resonates with me. Again, going back to my my academic days, I was working with an advisor who was doing a lot of what we call dyadic research or couples research, because we were working on romantic relationships, dating, online dating, attachment, stuff like that. And fast forward to to today at Wealthfront. Where we work where our work is about people’s financial lives. Then if you’re if you’re married or in a long term relationship, and you have joint finances, with your partner Now, then, obviously many of the questions on your mind, many of the needs you have are joint needs and joint questions. They are not individual needs or individual questions. But all too often, including at Wealthfront. And everywhere else I’ve worked, we do do the research at the individual level. And that is a very partial picture and a very biased picture of of people’s people’s realities. You can imagine that if you have to two people who are in a long term relationship and you ask each of them separately, about their finances and where they’re at and where they want to be. You may just get very different answers that don’t take the other person into account enough. You’ll also get very interesting answers when you ask each member Have that family to report on the other member of that family and what they think the other member believes about, in this case, finance. So, I definitely agree that one of the ways in which we can think, among other things about methodological innovation is the level at which we’re conducting the research. Is it the individual level? Is it the couple or dyadic? level, the group level? And when should we be focusing on that particular level? And also, should we be can we develop methods that are uniquely tailored to that level of of research? How can we conduct something like a joint interview in a way that will yield more robust insights? And does it need to be substantially different than how we’d interview Each of the individuals within that relationship
Steve: If I was just gonna wave my hands and imagine anything is possible, you know, having done a little bit about this topic and in my career as well. Seems like, you know, the family childhood family upbringing is a huge determining factor for a lot of beliefs and sort of why couples have different approaches because they have different backgrounds. I’m oversimplifying it, but boy if you wanted to, you know what, if you could maybe not literally go back in time, but it’d be interesting while we’re just you know, imagining Anything is possible to you know, interview the group that sharing the decision, but also learn about the, you know, the origins of those assumptions, rules, expectations, beliefs, and so on. From let’s just say it’s the family that they grew up in, you know, that I have no idea how you do that, from just a like a recruiting point of view or a logistical point of view or making sense of the data point of view. But just thinking about people as networks and where there are people as being in networks and where a lot of things that drive their behavior come from are not at the individual level. It does open it up in a way that I don’t know how to address.
Noam: Well, I too, struggle with that. And I, I struggled with even imagining what the methods would look like in order to get at the historical context of an individual or family, but I can tell you this. I have two children. And I have more photographs of my children than I do know than I ever imagined. I would have I have more data essentially about my children and I will always have that That data, then then I ever imagined would be would be possible that the richness of what I know about my children and could potentially share is, is crazy. Or another example is we can all go to Google or Facebook and look at years and years worth of history of where we’ve been, what we were up to, what you know, what we ate, and who we hung out with. And so, the next generation of researchers will certainly have access to an incredible trove of historical data. Hopefully, they get the proper permissions to access those data, and and respect the privacy of the individuals they’re researching. But assuming they do, I actually think it’ll be easier than ever before. To get such an incredible picture of what led the people wanted researching to the point where they’re at in their life. If you combine all of those data together with advanced analysis methods where we’d have the support of various fancy machine learning AI driven algorithms, then I actually do think that in just a few years, it will be feasible to understand people at a level where we’ve never been before, in particular, what you were just mentioning around where where everything began for those people and how they were raised and, and, and, and how they got to that personal and professional point in their lives. So I do see glimmers of or a glimpse into this potential future. I often wonder whether it will be useful or not, and how we’d analyze such data. And is this truly the future of our profession, but what I’m sure have is that too It will be a possibility, given everything that we we know about people and how we conduct ourselves these days with technology.
Steve: We’ve talked about a couple of things. So far and just very broadly, one is, you know, how research is working, you know, the roles that you’re playing, what you’re trying to help people accomplish at Wealthfront. And then, you know, your, your broader kind of observation that, that in terms of our methods, we’re not really focused on them as a practice, you know, as a profession, are there things that you can do? I like what you said, you know, this, that innovation and methods could come from anywhere. But is it something that there’s an opportunity for, for for Wealthfront, for you to be pushing methods ahead.
Noam: So I think that one of the beneficial results of democratizing research as extremely as possible and enabling everyone in the company whether it’s designers, PMs, marketers or others to conduct incredible research, and I’m fortunate enough to have colleagues who have amazing research chops and they do incredible research that frees up part of the time of the user research team. And and us as user researchers to focus on a some of the more strategic research projects that we have in the company. And it also will be it’s enabled us to focus on on research innovation, and either implementing tools that allow for better research to happen, for example, better knowledge management, of the research we already have, and are already doing and it also enables us potentially to focus More on developing tailored methods or frameworks that work for our context. And FinTech is in many ways, different or unique compared to to other areas. And there are certain things that we care deeply about. And so I think that the result of democratizing research as you assuming you accept that, that is a viable route to go down is that you are able to focus more on on other things, including that innovation I mentioned before. So we will definitely continue to try on the user research team to either implement new tools and platforms that are coming out that make us all better researchers, or innovate within the company. And and of course, if we can share those innovations with the broader community In order to develop methods that are better, a better fit better tailored to what we do, I can give. I can give an example here of, you know, something we care about deeply I would say in FinTech especially, is, of course, around the issue of trust. And the measurement of trust, which has, I would say, been a tricky thing, historically, for researchers, both in academia and industry. At Wealthfront, we are often handling people’s entire life savings investments. That’s a very big responsibility which we take incredibly seriously. And we want to understand how we can earn people’s trust, how we can build on that trust how we can augment it. We want to monitor the trust in us and and understand it deeply. And the measurement of trust. Well, measurement in general, is incredibly complex. And so developing great measurements for trust is definitely a goal, which I’m not sure we’ve, well I actually am sure we haven’t yet reached at Wellington, we aspire to do that. I’m not sure if anyone has. I think that as a research community, we can barely agree on how to measure satisfaction. And often we use deeply flawed methods just to do that. And in my opinion, measuring satisfaction is much easier than measuring something like trust. So more to come in the future, I hope including from us.
Steve: Maybe in the we have a few minutes left, it would be interesting, hopefully, to hear you. I’m going to start again in a few minutes that we have left I would love to hear a little bit about maybe a compare and contrast. The other places that you mentioned you’d worked Airbnb and intercom, you know, what are some of your perspectives on, you know, research being applied in very different organizations and what you experienced as a researcher in these different types of companies.
Noam: I’d love to do that. And by the way, I have as much time as you need because I feel like we covered a very specific thing for most of this conversation though I do find it fascinating. So for me, Airbnb was an incredible experience. I think what drew me to Airbnb more than anything was the mission at the time, which I believe is still Airbnb, his mission of belonging anywhere as an immigrant to this country, and as someone who’s traveled quite a bit and be To a few places around the world, I definitely know what it feels like not to belong somewhere. And I definitely know what it feels like to feel belonging. And I think that today more than ever before, it’s really important to break out of one’s bubble experience other cultures experience other places and hopefully experience that belonging. So, to me as a researcher, Airbnb, I had a very strong connection to to the mission. And there are some very unique aspects of research at Airbnb, for example. Airbnb is one of those products where the company’s aim is for people to use the app essentially as little as possible. And and venture out into the world the way Airbnb makes money. And grows its platform. And its benefits to the world is when people go out to travel not engage with with the app. And so a lot of the research at Airbnb isn’t just about the online app or web experience. It’s about the offline experience on the ground. It’s about the travel itself. It’s about things like the check in and the checkout experience, or what it’s like to go on an experience hosted by someone who lives in that city. And they happen to be experts at making pasta and they want to share their food passions with with the world. And so researching these offline experiences in a cross cultural settings in almost all countries across the world, is a very different experience than some people I would say have been in the UX research industry. Also, I credited with Be personally as a company founded by two designers and one non designer, but it’s a very design driven company. And in my mind, Airbnb is one of the best design schools in the world without officially being a design school. I learned most of what I know about design. And a lot of what I know about research for my time, at Airbnb. And the travel space is, is very challenging and very interesting. And I hope that in this time, when people aren’t traveling at all given a global pandemic, I hope we get back to that soon because it makes all of our lives a lot richer and a lot more interesting. And it’s some of the most fascinating research I’ve ever done. Another thing about about Airbnb, which is also a question I think in our industry, is how does the function of research operate within the larger organization? For example, at Airbnb researches is part of the design team. Whereas at Intercom for example, research was reporting into one part of the product function. We also have a bunch of different titles in our industry from experience research to design research to product research, user research and so many other things. And I know we discussed this a lot, but it does make a difference to one’s experience. Whether they work more closely with product and PMs or more closely with designers and, and design leaders. It really influences one’s experience, the things you can learn your your relationships with those functions and so forth.
Steve: But there’s a caveat to that I think it in looking at the two companies that you’re comparing and having not worked at either the missus, you know, definitely outside or hypothesizing but I could see be if I worked at Airbnb, I’d want to be working with him. He said, it’s a design. It’s a strong design company. And I think of intercom as a strong product company. And that, so it’s different where it’s different sort of who you report up to, or what organization or what roles you’re working closely with. But also that’s that is different in different organizations based on just the strength and maturity of those different practices.
Noam: Yes, I agree.
Steve: so just to clarify, You know, there’s a lot of different titles or the titles connected, is there a relationship between, you know what, what your job title is. And this this issue sort of, you know, who you report up to where the research organization sits, is that impacting the range of how titles are, are kind of applied to researchers.
Noam: So I think the titles we get as researchers are a function of a few things that are a function of the organization, their functional, its philosophy. They’re a function of how the organization itself is designed, and what the emphasis is on and what type of relationships the organization wants to cultivate more and how the company wants to be perceived, as well. I want to mention something about our work. At Wealthfront these days, some relationships are, in my opinion, so natural that it’s surprising that they don’t just occur within companies and don’t just happen and don’t just thrive without any particular intervention or organizational design. And yet more often than not, those relationships are missing at Wealthfront. We recently created something we call the research Guild. And the research guild is composed of user researchers like self consumer insights, which is typically the side of research that kind of belongs organizationally to marketing or perhaps in some cases, product and data science. Now, again, the relationship between user research, consumer insights and data science, in my opinion, is natural and obvious. And yet all too often, it’s not a thriving relationship, it might even be a very broken relationship. And so what we’re trying to create wealth and these days is is a much closer relationship between those functions Because ultimately, we are all trying to do the same thing, which is enable every single person in the company to understand our clients and potential gains and better to be closer to them. To hear them in a variety of ways. Whether qualitative or quantitative, it doesn’t matter. high level, we all have the same mission. And, and so maybe more, maybe something more important than the titles we have and constant discussions around what UX is or what UX isn’t or who is or who isn’t doing researcher, I think we might want to focus more on the collaboration between functions and the relationships between functions. And to really amplify these truly important crucial relationships between, for example, data science and, and user research. It makes all of our jobs better, it creates better outcome to create a better understanding of our users and our clients. And it’s, it’s one more thing I haven’t always experienced in my career and I wish I had so definitely trying to, to correct that now at Wealthfront and and create those, those beneficial cross functional relationships.
Steve: So how will this guild enable those relationships?
Noam: So what I’m again when I’m not on parental leave and actually working we have both leadership meetings for that guild with the managers or directors of each each group to keep each other in the know about projects and things going on and what we’re doing and how we can help each other and collaborate better. And we also have meetings between all of the guild members where we can provide each other with feedback on projects, whichever stage those projects are at, or ideas for new new directions or support, where support can be, can be provided. And anything else we can come up with. And just by having those couple of meetings, I think we’ve already seen so much benefit in terms of every project we conduct being that much stronger thanks to much broader feedback, much deeper feedback. And a variety of perspectives from people who have a very diverse set of skills, academic backgrounds industry experience, and to focus on on a variety of things in in their day to day.
Steve: And how did you come to use the term guild to describe this?
Noam: So I’m, I’m all about etymology, and very into into words, I think what drew me to the term guild and the reason I like that suggestion is that in most dictionary definitions of Guild, you have the notion of an association of people who are in the pursuit of a common goal. And I think that’s what’s important to remember that data scientists, consumer insights, or marketing research people, user research professionals, we are all in pursuit of a common goal at the, at the high level. And we should be collaborating as closely as possible in order to, to to reach that goal. And I think all too often we don’t understand each other’s work well enough, we don’t collaborate well enough and leverage each other’s work, we might sometimes repeat each other’s work or or you know, have like dual projects which are quite similar in their, in their outcomes. And so by focusing efforts on that common goal, and leveraging each other’s knowledge and skills. I think there’s something very powerful in that, which definitely exists in a bunch of companies in, in, in, in tech and otherwise, but could be could be stronger for sure.
Steve: So the thing about podcasts, I think, is that they, you know, we’re having a conversation on the day that we’re having it and someone may come to this at any point, you know, in the very near future or later on. So, you know, you’ve you’ve referenced our particular moments in time a couple of times or in the midst of a global pandemic. It’s really changing well as You know everything about everything for everybody right to work from home. People are sheltering in place right now in the Bay Area. And it’s forcing a lot of questions. Or, you know, for some and maybe not for others about research. You know, what we’re doing right now sort of how do we even think about how do we think about research our profession in a time where these other things are going on? And I’m wondering if that’s something that you’ve reflected on if you have a point of view about today about research today?
Noam: Yeah, I’ve definitely reflected on this, especially about a couple of points. The first is that I’ve, I’ve been passed off and seen lots of discussion around research ethics in this kind of period of time. People are potentially in also distress, a lot of anxiety. And so there’s the question of, it’s perhaps a philosophical question even of whether we should be approaching our users at a time like this. And what can we make of people’s responses when they’re in such a different and often distressed state of mine, given given the pandemic? My view on this is that it’s it’s crucial to continue to be such that such at such a time, certainly with more sensitivity and and a realization that there are probably limitations to to the research. But as someone who works in financial technology at a time when the financial markets are incredibly, incredibly volatile, I am personally happy that we are continuing at welfare to be attuned to what our clients need and what they hope for the future and to provide that for them. Because it’s exactly this type of situation where we want to to best serve our our clients and help them avoid potential mistakes that could be very costly to them. So I think there’s a need for much greater sensitivity at this kind of time. But I think researchers have an important role in pushing forward with research as much as possible. Maybe taking on additional projects that that benefit the health or medical services, community, and so forth. The other thing is, I think a big part of this past maybe decades and use research has been around research Ops, and how we can conduct research in a more lean fashion, fast paced, lightweight. For many companies, in person research travel, it’s a core part of the research practice. And yet today, we’re having to grapple with with this fundamental problem that we can’t do that anymore. And we’re forced to develop new tools and new ways of of getting to and connecting with our users, even when we can’t do that in person. And I think it’s important for every every company and every team and every researcher to think about those different contingencies and What can we do in a situation where we want to connect with the user, we want to understand them, but we have to do so in a in a remote interview or some other technique, which doesn’t include meeting in person. And this goes back to our discussion around methods and methodological innovation and what we can, what we can do around that. I personally hope that in the future, when maybe this time, augmented reality, virtual reality, other techniques become more of an established practice that we will be able to share spaces and share ideas and feel together and conduct kind of pseudo in person research even if we’re not. And I think there’s something very, very powerful in that. In fact, maybe that will drive part of the growth finally in in those types of methods, this type of event, I think it’s going to be very interesting to see the impact of this double pandemic on how we conduct our, our research, because ultimately, our profession is all about connecting with people and understanding them. And that is much, much harder to do when we’re all quarantined in our own houses, waiting for this to, to end.
Steve: Well, I thank you so much for your time. And, you know, it’s fun to go really deep into a few things. I appreciate you, you know, going all the way, all the way into the nooks and crannies of some of these ideas that you’re thinking about gives us a lot to chew on. So thank you so much for being on dollars to donuts. Thank you as well. It was a real pleasure.
Noam: I’ve been waiting for this. And I’m really glad you invited me. I appreciate it. Thank you.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 31. Noam Segal of Wealthfront first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my interview with Noam Segal, the Director of Research at Wealthfront. Everyone from PMs to designers, researchers, obviously, engineers, data scientists, marketing, we’re all trying to to understand our clients, we’re all taking part in that process in some way, shape or form. And so I view my […]
The post 31. Noam Segal of Wealthfront first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Laith Ullaby, the Head of Research at Udemy. I’m really into the idea of questioning what we do. That can be the methods and that conversation about getting out of our comfort zone. It can be thinking about our relationships with stakeholders and trying to […]
The post 30. Laith Ulaby of Udemy first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Laith Ullaby, the Head of Research at Udemy.
I’m really into the idea of questioning what we do. That can be the methods and that conversation about getting out of our comfort zone. It can be thinking about our relationships with stakeholders and trying to reimagine and iterate on those. And it can be thinking about the historical trajectory of the field and the legacies that that has imbued us with. And I think that being ready to iterate and question the assumptions that a lot of those things are built on, is the thing that I’d really want folks to come away with. – Laith Ullaby
Show Links* Laith on LinkedIn * Udemy * UC Berkeley School of Information * See One, Do One, Teach One * Acquisition of Fjord * Partners At Teehan+Lax Join Facebook * Facebook Acq-Hires Part Of Design Firm Bolt | Peters * Adaptive Path Acquired By Capital One * Edward Tufte * Understanding Your Users by Kathy Baxter, Catherine Courage, and Kelly Caine * Insights Association Code of Standards and Ethics * Advancing Research conference * Advancing Research community * Research Skills Framework
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I had a profound moment this week. I was facilitating a meeting, not a user research setting at all. But we were having a fairly open-ended conversation. One of the people in the meeting shared some very specific advice for the rest of us, offering almost an impromptu speech or pep talk, filled with passion and encouragement. It was very inspiring. But as the facilitator, perhaps because of my work as a user researcher, I wondered if maybe that wasn’t enough. This person articulated their richly-realized state of being, and it may be easy for anyone else to dismiss it, oh, that’s just so-and-so, they’re just like that. But, we’re not born with insight about ourselves, or clarity about a new way of being, so I was curious – and I thought it would be helpful – to understand more about how this person accomplished this. How did they get here? And even though I wasn’t asking followups in this session, it seemed like an opportunity to shift my role slightly. And so I asked “How did you get to this stage, where you have this clarity in your approach?” This person paused, and said, “Well, if you really want to know” and then proceeded to share very personal details about their life, from hardship through to today and what they had overcome and how that had shaped them. The details aren’t important for my story, though, and they really belong to the individual who chose to share them, not me. Anyway, it was an impactful moment for all of us in the meeting. And it reminded me about a fundamental in user research, that you don’t know what’s behind a door, it could be something banal, I mean, it usually is, right, but you don’t know until you open it. And sometimes what you’ll find is new, surprising, and impactful. But if you assume that it’s always banal, you’ll never try to cross those thresholds, and never have the opportunity to hear those significant perspectives. I’m describing this as a doorway and that what we do as researchers do is step into that space, but I don’t think that’s sufficient to capture what it is that we do. Yes, we are stepping into something ourselves, but also, and probably most importantly, is that we’re making space for someone else, choosing to ask that question, to be interested, invites someone else to take a step forward themselves. You can call this a holding environment, or a safe space, but I think the additional point I want to make is that it’s not just what you do as a user researcher, it’s what you do for someone else.
I’m putting another episode here done without the benefit of a professional editor or transcriptionist. I’m focusing in the near term on how I can contribute but of course this is a difficult time for many people including those who operate a small business. It’s a challenge right now for Portigal Consulting. If you’re in a position to consider me as a partner, that’s appreciated. I lead research projects, I help companies build a more research practice including assessments, training, and coaching. Thank you!
Okay here we go with my conversation with Laith Ullaby, the Head of Research at Udemy.
Well, Laith, thanks so much for being on Dollars to Donuts. It’s great to have the chance to speak with you today.
Laith Ullaby: Pleasure to be here excited for the conversation.
Steve: All right, well, let’s, let’s get into it. Why don’t you introduce yourself?
Laith: Sure. So my name is Laith Ullaby. I’m currently the head of research at Udemy, which is an online education platform. I have a academic background in ethnography, and have bounced around the UX research world for a few years now. And then I’m also really passionate about teaching. So for the last few years, I’ve been teaching at the UC Berkeley School of Information and at the UC Berkeley Extension.
Steve: So what does it mean to have a degree in ethnography? Not? I feel like that’s not a phrase that everyone can say exactly.
Laith: Sure. So technically, the field that I studied was ethnomusicology. Which is using ethnographic methods to look at the intersection of music and popular culture, and so my focus was on how governments in the Middle East, were using technology to disseminate national identity through folk culture. So it was a little niche. But, you know, it was fun to sort of build that foundation and ethnographic research and think about how sort of the interplay between popular culture and technology.
Steve: Do you? Are there echoes of that in, you know, your daily practice? Like, can you draw the threads back to what your what that program was about for you what that research was about?
Laith: For sure, you know, I think that the, the, you know, when I think about user research, it’s really combining the strands from a lot of different places, different academic places, different you know, parts of sort of the applied world. And one of those is sort of the design anthropology world which I think is really steeped in ethnography and for me that really raises some of these really interesting questions about identity and community and how we sort of come to understand ourselves, as well as power relationships. So working on a marketplace for education, thinking cross culturally about, you know, who feels empowered to be a creator of knowledge, I think really hearkens back to some of the core areas of exploration that you might encounter in an academic program like that.
Steve: Maybe, can you say more about, you know, the organization where you’re at? What, what, what Udemy does, and a marketplace for education as a compelling phrase, maybe you can impact that?
Laith: Sure. Yeah, I’d love to. So, um, as opposed to some other, you know, awesome companies in the space. We’re not a publisher, we’re a marketplace. And so we have a platform for instructors to come and to find learners. So just to get a sense of the scale, we have 50 million students, 57,000 instructors and over 150,000 courses. So, you know, in other models, we might say, Hey, you know, here’s sort of the Python course that we think is cool. And in our model, you know, any instructor can come on and make any variation of a Python course. And then it’s those sort of marketplace dynamics like reviews and number of enrollments and social proof and those kind of things like you might do on an e commerce platform that folks would be familiar with, in which students then get to evaluate what’s the right course for their learning need. So it’s really exciting to sort of see that that proliferation and that there’s so many different approaches to you know, finding the right student, the right course with the right instructor at the right time. It can be overwhelming you know, you there’s obviously I’m sure a lot of folks minds are going to that, you know, moment of overload of too many options. So, it We do try and help out with that. But that’s sort of the the core value proposition of what Udemy does.
Steve: And so as a researcher in a marketplace company, I’m going to guess that there’s research focused on when you describe two players in the marketplace, right, the instructors and the students credit research is, would you describe research as kind of focusing on, you know, ultimately, one or the other?
Laith: Yeah, and that’s kind of the fun part is, you know, I think sometimes if we’re thinking about sort of the very, you know, naive version of user research, it’s, you know, we’re trying to help out the one user. And, you know, for us, it gets more complicated, because you have to think about that balance. And so, you know, you can make something a lot easier for the learners, but that could put a huge burden on the instructor. And so you have to be able to triangulate and sort of balance those. Similarly, we also have a b2b offering. So in that world, there’s also an administrator role or sort of a buyer role that can enter the fray as well. So we have to then sometimes think about the the balance and trying to find Win Win wins for any kind of change that we do. So it’s definitely a bigger challenge at times, but it’s a fun one.
Steve: So in that third example, the b2b so someone, this is a scenario where that Python course might be used by like a learning development person inside an organization to bring that in for training set the scenario.
Laith: Exactly. And so we have soft skills, we have technical skills, there’s there’s a whole range and that is a subscription model. So they would get access to our courses through that subscription model and make it available to their employees.
Steve: So what kinds of research you know, what kinds of research are you and your colleagues doing?
Laith: Um, one thing that I really love about Udemy is over two thirds of our learners are outside of the United States. So we really have this global presence and as a researcher, you know, just as a nerdy researcher that that presents a lot of fun stuff to look at. But it also means that we tend to rely a lot on remote research just to be able to get that that that global perspective, that global reach for the studies that we do, so remote as a lot, but it’s interesting, you bring this up, because I’ve been talking with my team a lot about how, you know, I think sometimes both individual researchers and organizations can kind of fall into a comfort zone with where, how and when they do research. And so we’ve been talking a lot about ways to kick ourselves out of our comfort zone and make sure that when we’re designing research, it’s, you know, truly the best way to answer that particular, you know, immediate research question that stares us in the face, rather than something that has a lower logistical overhead or just is something that we’re more confident and we’re comfortable doing.
Steve: So how do you create the conditions where, when you’re talking about risk, you didn’t use that word, but outside your comfort zone? How do you create the conditions where it’s okay to go outside the comfort zone for you, your team, the stakeholders that you work with?
Laith: Absolutely. And I think, you know, you hit the nail on the head with risk, you know, it’s not that folks oftentimes don’t want to do these things. It’s that, you know, they, they are used to doing things well, and they want to maintain that that same level that they uphold themselves to, I think there’s a few things that help there. One is, as I’ve helped to build upon my predecessor, who previously lead the research team and build the team further, really tried to have complimentary and overlapping skill sets within the team. So hopefully, it’s not something that no one on the team has done. It might be something that one person hasn’t done or a few people haven’t done. So this way we can share and teach each other. I’m a big fan of when they talk about in the medical practice, they talk about See one do one teach one. And that’s kind of how you know and and, you know, a new procedure in medicine. So I really tried to embrace that and say, Hey, folks, you know, how can we teach each other because you might have seen someone do something, and then you got to do it yourself. And now you get to teach someone, that’s when you’ve really kind of got that level of mastery. So when you’re constructing teams, I think it’s interesting to think about where those opportunities are, and that you’re creating the environment for that. I think we’re also lucky to work in a place where there’s just a lot of trust in the research team, folks have just done a good job of delivering high quality research, and that that gets you a tremendous amount of goodwill. And so I think that’s allowed us to have a little higher risk tolerance, because people know that, you know, we’re trying our best and we’re trying to deliver good research and if something, you know, by chance, didn’t go super great. You know, it was done with the sort of the best faith and intent and just generally the leadership, the product leadership and other folks are trying to you know, Encourage that general behavior in terms of the features that we’re launching or the the ways that we’re interacting with our marketing messages or things like that, too. So it helps to be at an organization that generally understands, you know, that risk is something there, and that’s where you get a lot of growth.
Steve: And buried within that as a, there’s a specific nugget that you just said, I want to kind of bring back up. You know, you implicitly answered How do you build trust? And you said, like, deliver great work and paraphrasing you but is that do I have that correct?
Laith: I think that’s a big part of it, for sure.
Steve: But then you kind of talked about goodwill, and maybe that’s maybe good work in genders goodwill, which then gives you some permission to not and no, no, no, I talked about failing but you’re talking about Yeah, getting outside the comfort zone and having trust to do that.
Laith: Yeah, I think that’s part of it. You know, hopefully there’s a little bit of a virtuous cycle, you know, that happens there. And I’d say the other part of it is, sometimes research can be a black box. And so I think that as people understand the component parts that go into research, it helps to build that empathy. And I know you know, empathy is a little bit of a loaded term and has been critiqued in some ways. And, and, but I think that just sort of at that base level of having a better understanding of what other folks are doing. And I think that you know, even just having folks observing research sessions or participating or the research team, facilitating non researchers, doing research helps to build up that kind of understanding of the complexity and the nuance that goes into, you know, really high quality research. So I think that’s part of it as well.
Steve: When you when you try to move research from a blackbox to something that has nuance. You’re asking more of your audience of your stakeholders, I want to I want you to understand that this is there’s more to this than what you knew. And maybe this is the benefit of working in a company that has learning as kind of part of its mission. But you’re asking more of your collaborators, if you want them to understand what you do is more than a black box. That is, that’s more work. That’s more brain cycles that you’re asking of them. I feel like this comes up in the, in the discussion of, you know, research is a thing that everyone’s doing. You just talked about, you know, facilitating teams guiding them and doing their own research. We we’ve, as a practice, we have invited a lot of people In. And yet, we still want to highlight the difference between what a non researcher can do and what we can do. But that means showing somebody that, hey, it’s not a black box. There’s a lot of detail here. I’m probably asking like eight questions at once.
Laith: Do you know where I’m going? I think I do. And I think this is something you know, every research or guy Ben has struggled with this in one form or another. So I think I get the gist of it. Um, I think for me, there’s a few things. The first is, it’s been a bit of an evolution for me, I think coming in as an someone with a bit of an academic background. And originally being more on the consultancy side. We sort of positioned herself as like separation of church and state of like design and research. And so we really kind of saw ourselves as on the user side. Maybe, you know, the metaphor I might use is something like we were the scientists or we were the sort of the testers in a way. And I think as I’ve moved In house and just fundamentally tried to had different relationships, I’ve had to reformulate how I think about my relationship with product teams. And I’m, you know, I’m someone who can’t code. I’m not a designer, I love all those kind of things, but I just don’t have that skill set. And I think I never, for a long time, I didn’t see myself as a builder. And one of the big evolutions in my career has has been saying, Hey, I’m part of a team that are building things. And so it’s not me sort of on the outside, you know, as a referee, but I’m part of this team that’s building something, and there has to be that kind of give and take. And I think that repositioning of myself has also helped to allow folks to reposition themselves in relation to me, and and has opened up some of those kinds of things. But you’re absolutely right. You know, researchers can’t just sort of demand that everyone drop all the things that they’re doing and have all the bandwidth in the role, world for, you know, extensive kind of things, because they are very good time intensive, especially the more elaborate research projects. So there’s a few things there, you know, again, echoing that idea of, you know, not just being empathetic with our users, but being empathetic with our colleagues and our co workers, I think is super important. I think that the principles of Human Centered Design in what we asked them to do is super important as well. So if you’re soliciting feedback on a discussion guide, um, have you made that a really easy to read? guide? Have you set up what kind of feedback you’re asking for them in a really clear and manageable way? Have you have you tried to maybe anticipate their workload? You know, I think all the things that we might think about doing in terms of building products we need to be able to do for our colleagues in different ways. But I think it also just shows that you know, for research to be really supercharged and have the impact that it can you know, it’s it’s a it’s a team sport, and you know, it’s So when we think about sort of those models of UX maturity or research maturity, you know, part of it is that the organization is willing to invest in it. When I, when I think about democratizing research, it’s it’s not just about letting a designer go, and sort of being the gatekeeper and letting them go do some usability sessions, but it’s about, you know, having them more involved in the process and feeling that we’re all designers. We’re all researchers, we’re all product managers. We’re all engineers. Now. We have specialized positions within that, but we’re building together and and i think that helps, but it’s, it’s still tough, you know, I don’t mean to act like I’ve nailed it, and I have all the answers, but that that mindset shift has at least helped a little bit.
Steve: Right, because you started off saying that, you know, you never thought of yourself as a builder. And but I hear you evolving that identity for yourself. And I heard two things. So let me actually want to check in with you. One is, is the collective view right? So, we are building something and I am part of I am part of the we, and if we are building then I, in this case, you take on some of that mantle of the identity of a builder because you’re part of that team after that, and I also heard that, you know, in, in living up to the principles of empathy for our colleagues, you are designing experiences for them through the experience through the artifacts that you’re creating, and the ways that you’re, you know, trying to make it effective for them to work with you. And that’s something that’s being built, you are a builder in the production in facilitation and production and artifacts, and all those sort of bits and pieces that make research happen.
Laith: So so like a lot of tech companies, you know, we do a lot of playing off the sort of name of the company so you know, people that work at Udemy or Udemy, mates and things like that. So, one thing that my team has worked on over the last few quarters is To make a an internal self serve research process that we call a u du u, so it’s, you know, a play on the U and the name of Udemy. But it’s sort of the idea that you can do it yourself. And we approached it in very much the same way that a product team might approach it, you know, in terms of, we basically had one of the researchers act as a PM, and we put together materials and we decided, you know, what was in scope and what was out of scope. We designed out, we sketched out the user flow, and what it might be, we tried to figure out how the different pieces of the experience would fit together, and try to, you know, create a portal through which folks could do this with as little friction as possible. And we very much approached it, how a design team or you know, a product team might make a product because we wanted to, you know, sort of practice what we preach and and embrace those principles, and we’re iterating on it, and we’re doing user testing on it and you know, we’re trying to do all those things, so that it can evolve and be the best way that folks can do research. themselves?
Steve: How did it feel for the team to do that to take on these, you know, put on some different hats and kind of act as if you you were these other types of roles?
Laith: Well, I will say, just because you brought up the issue of feeling, which I think is important, is I think that, you know, for different folks on the team, this was more and less out of their comfort zone. You know, I, I know that there can sometimes be that that gut reaction when we think about sort of opening the gates and letting non researchers, you know, run wild and so there was some good fun discussions about, you know, how we saw ourselves and how we talked about research and some folks really kind of had to grin and say, Hey, I’m going to trust the process and see how this goes. So that was a fun kind of thing to go through. Um, but definitely, it was great to see folks taking on these other roles, and you know, maybe putting some of their design chops to use that they maybe don’t think of themselves as a designer, but they’ve picked up some of those skills along different ways or put some of their chops together and and kind of see you know, put into practice what they’d seen great pm doing. So it was a really fun process. And I think it was a fun bonding process, we kind of did it as part of an off site. We have one awesome researcher, but she’s located in Dublin, Ireland, so we don’t always get to be as much spend as much time with her. So we did this, the bulk of this work sort of did a sprint, when she was visiting, you know, the main office in San Francisco. So it was also just a really fun way to get to know each other and our working styles because we’re in an embedded model. And so folks are working with teams. And so it was it was just really fun to be able to work on a project together and kind of learn from each other about how we approach product design and product development. So just as a team bonding experience, I thought we got tremendous value out of it on top of sort of the the output in addition.
Steve: You mentioned empathy for your colleagues, you know, trying to do the work of a world They don’t normally hold, I think would might illustrate that as well for everybody. Absolutely. What are the so just maybe shifting gears a little bit, you know, thinking about the, the populations, the user groups and sort of this, this space of learning of teaching and learning, you know, what’s are their unique ways that you have to approach you know, beyond the the global aspect, whether it’s methods or just, you know, the, the mindset you have to bring to these particular areas that you want to understand. Was there anything specific there?
Laith: Yeah, um, I think that there’s probably a similar thing that you might find with financial services, some forms of that or the fitness world where folks have a lot of aspiration to do a lot of learning But it takes a pretty big investment to, you know, learn a new skill, especially if it’s in a more self directed environment. So, a lot of times, it’s that classic thing of, you know, someone might if you talk to someone right after a purchase, they purchase a course, they’re imagining the sort of future where they’re, you know, diligently doing, you know, three hours a day in 10 hours on the weekend, and, you know, things like that. And, you know, unsurprisingly, you know, a lot of folks struggle with that, whether it’s their current work responsibilities or home responsibilities, or, you know, they hit a stumbling block, and they’re not sure how to how to get over it. So it can be a challenge to parse between sort of how people imagine themselves behaving and then reconciling that with maybe what we see in the data analytics or or sort of what they do. You know, there’s some interesting behavioral science things about the more you can empathize with your future. The more effective you can do at helping that future state. So, you know, there was a company in Mexico and they would digitize an aged version of your face when you were presented with, like how much you wanted to contribute to your retirement fund. And sort of being able to empathize with yourself in the future made people more likely to contribute to themselves basically their own retirement fund because they could sort of imagine themselves as not being distinct in the future, but being the same person. So there’s a lot of challenges there around motivation and intent versus behavior. That’s a little bit of a rambling answer, but it’s it’s a complicated research area to dive into.
Steve: What about the you know, how do you approach the the instructor, community or segments?
Laith: Yeah, um, You know, definitely, marketplace dynamics can be a challenge. You know, if you have an entrenched instructor who’s very popular and has, you know, hundreds of thousands of enrollees, enrollments, it can be tough to figure out well, you know, what if a new instructor comes on, so helping folks to kind of feel that they have a fair shot is super important. And then also, there’s a few elements, I think, that are really important about helping instructors get the right kind of feedback in both Udemy and other places I’ve worked, you know, if you ask folks, if they want more analytics, and you start listing options, they’ll just say like, yes, yes, yes, yes, yes. You know, people just feel that they want more data and they want more information. But at some point, it can be overwhelming, or if they don’t know what to do with it, sometimes folks can kind of feel guilty and you know, this numbers changing and I don’t know what I’m doing differently and things like that. So there’s a real challenge in helping to not just present in information on a dashboard. But the context, the understanding the nuance, and hopefully equip them with strategies or suggestions along the way. And they’re not just sort of twisting in the wind to sort of decipher the, the different data visualizations, you know, how they might do without that assistance.
Steve: Right. And there’s, there’s something maybe lurking underneath your stories about, you know, the gap between I mean, maybe it’s simplistic level, like what people say they want or how we can get people to agree that features are, are things that they want versus, you know, serving a serving a deeper need, which is harder to harder to extract and kind of have a deep understanding of, that’s absolutely the difference of what you’re trying to accomplish with research.
Laith: Absolutely. And, you know, that’s kind of the you know, in some ways, the Holy Grail A lot of the work that we do is is figuring out how to get at that, you know, underlying unmet need and things like that. And I’ve even seen, it’s been interesting recently, you know, a few teams working on similar product offerings or features, and then realizing, when we took a step back that, hey, these might actually be trying to solve some similar things, we might be able to combine these efforts. And I think that’s one thing that research was helpful in is sort of highlighting what the actual value people were getting out of some of these prototypes. And it helped to sort of triangulate you know, what sort of, you know, had the most juice, you know, that we thought could really deliver something unique.
Steve: You we talked about your team kind of indirectly Can you describe Kind of, yeah, what kind of what kind of skills and backgrounds you know, make up your team and, you know, in future what do you what do you look for when you When you meet prospective candidates for your for your team?
Laith: Yeah, it’s a great question. I think that we have a pretty diverse range of backgrounds, which is pretty cool. At Udemy the way we’re just organized, it’s market research and user research together. And, you know, this is something I think is really interesting in the industry is all the different permutations of org charts. And, you know, you know, where folks sit and things like that. I think sometimes it can get, you know, over indexed as something that’s important because I think sometimes it’s often just legacies of how companies grow. But it’s interesting and I think it is good to think about, so we have one market researcher, and then four user researchers and then myself and it’s actually been really fun because to to have that market research lens as part of the user research team. Because it really helps to highlight things about brand voice and brand positioning and Sort of that that more holistic experience of how someone might enter the product through different campaigns, and and sort of thinking about that broader journey. So it’s been fun to have that perspective, most of the places I’ve worked at before, you know, market research is in a different building. And it’s kind of a black box even internally. So it’s, it’s cool to have that that together. And that person tends to bring a little bit more of that the survey chops and and some of those more market research kind of things, not to say that user research doesn’t do surveys, but you know, sort of bread and butter and approaching some some different kinds of questions. And then on the user research team, you know, we’ve got a couple folks that have a, you know, master’s degree in HCI. And then other couple folks that, you know, came to the industry through through different routes, including someone who formerly used to be an engineer working on renewable energy in that kind of format. So, it’s cool to that we all bring in different things, and we’ve been at You know, some different organizations and sort of trying to figure out the DNA of those places and what we want to sort of copy and and what we hope to innovate on or what we see that you know, someone might be bringing in a different mousetrap. When I’m when I’m looking at candidates Udemy is very mission driven, even though it’s a company, a lot of folks really just believe in education and the power of education, and the global nature of what we do. So, you know, always want folks that are really bought in to, to the idea of what we’re doing. And then I think beyond that, it’s, you know, someone that can bring in something new and that can be new methods and new perspectives, new approaches, but um, you know, I don’t really see it as that there’s like a one single profile that is gonna is gonna mesh well. Collaboration, of course, is super important. And having that that flexibility To think about, you know, for each project what what sort of permutations or iterations you know is this something that’s going to be a little more quick and dirty is is something that’s going to be a little more rigorous. The last area that in hiring that I’ve sort of formulated in my mind is, I think about written communication, verbal communication and visual communication. And I really want to find folks who are really comfortable, at least one of those, like really good, and then hopefully, you know, decent at the other two, and, you know, willing to kind of work on them. But I think it’s important that you have one modality of communication that you know, you’re just really good at, because you know, so much of what we do is communication.
Steve: I want to loop back to you said all sorts of all sorts of interesting things. I want to loop back to one of them. Because you mentioned a few moments ago that that that people are embedded. But you also talked About the sources. Watch one. Sorry, see one. Do one teach one. Is that right? Yeah, yeah. Okay. So I’m just wondering about, you know, especially you got this market researcher with a, you know, a pretty unique skill set among your team. And you have all this diversity. So how the the researchers are are embedded they have product teams they’re specifically focused on. So how do you balance that focus and then harness the hardest that diversity and let people influence each other and teach each other and draw from draw from that team? You know, that you have the centralized part and you have the embedded part. I’m wondering what that looks like.
Laith: Yeah, and I you know, I think this is a challenge for for for folks in some organizations to sort of be a functional leader, but not a project. Leader, where the folks on my team are just so much more expert in the details of the features and pods that they’re working on. So, you know, sometimes I feel like, you know, oftentimes I feel like, you know, they’re teaching me or they’re, you know, showing me how things are working. And so I have to think about other ways that I can contribute or add value, obviously, you know, that can be with being a sounding board for thinking about methods and approaches. But But to your point, there’s also about sort of how I can create that sense of a research team, even though most of our day to day is kind of often embedded with, you know, awesome designers and engineers and marketers and you know, folks doing other things. So we have a few mechanisms. And, you know, I hate to say that the answer to everything is a meeting, but some of it is meetings. So we have a team meeting. We also have a bi weekly session. That’s a we call it researchers talk shop. And so we have a spreadsheet where people just throw in topics and we For a lunch and we try and dive into different research topics. So that’s a great way for, you know, someone’s been doing a lot of workshops recently, and they can kind of share what went well and what didn’t go well. And we do some of that in our research team as well. We have a office hours as well, so people can just stop in and talk with us, but we, you know, sort of hang out in the conference room. And that’s just an opportunity for us to kind of catch up a little as well, you know, because sometimes, you know, 10 people show up to office hours and sometimes it’s just one and just kind of hearing each other helping other folks and office hours you kind of get to hear a little about their approaches and things like that. Another mechanism that we have is we do try and pick some some bigger, juicy or questions that we are sort of more researcher initiated, and we do those collaboratively so that they you Edu initiative is a is an example of that. And we’ve tried to do a few other collaborative researcher initiated projects. Another thing is we have a little podcast club, so We’ll listen to podcast episodes and then just set up a quick little half hour to, to talk through it in different ways. And it’s just fun, you know, a chance for researchers to to talk about things through that lens and try and grow and get those other perspectives. For me, the nice thing about podcasts is it feels really low friction to just find like a 20 or 30 minute podcast. And, you know, folks can do that when they’re commuting to or from work and you can still have a really lively discussion, I found historically, it can be a lot harder to give folks PDFs or articles, you know, it can be harder to sort of find the time. So those are some of the mechanisms that we have to try and develop that that sense of community and exchange.
Steve: That’s there’s a lot of them. And so I know you’ve got one person in Dublin, for the people that are in San Francisco. Are you sitting in a co located or whatever the better way to say that would be
Laith: we’re all sitting on the same floor but you know, it’s a decent sized floor. So, you know, there’s folks that aren’t in visual visual range. So we’re kind of sitting near the product teams that we work with, we’re sort of the seating chart is by, you know, product teams rather than by functional teams. But you know, you know, you run into people constantly just in the halls and at lunch and running between meetings. So, definitely, I see everyone, you know, I’d say, at least a couple times every day, you know, even if it’s just passing.
Steve: And so for researchers that are embedded in these product teams, how, how is sort of the the, the plan for what they’re going to work on determined?
Laith: Yeah, it’s a great question. And shortly before I joined the company, or a little before I joined the company, the research team had switched from that more centralized model to an embedded model. And so some of this has been an iterative process. The other challenge is, you know, there’s just not infinite reading resources. So is the ratio of researchers to designers and PMS. You know, there’s usually one researcher juggling at least a couple PMS if not two, or three or even more. So it makes prioritization really a challenge. And I think before we’ve been a little reactive, and it’s hard to make sure that you’re working on the right things, if there’s just sort of like a queue coming in, and you’re trying to keep your head above water. So one of the things we’ve worked on and iterated on it quite a bit over every quarter is just to do a quarterly planning process, where folks go out as as teams are putting together their, you know, quarterly okrs. Researchers are also evaluating what the research asks will be over the next quarter. The researchers then take that all together, you know, from across the teams, you know, think about they do a little bit of a T shirt sizing, you know, what’s a small, medium and large project, think about their bandwidth, sort of think about timing And come up with sort of a proposal. You know, it’s very aspirational. But it’s still a really good, you know, exercise to go through. Now, what I think makes it has made it really effective is we then do a readout of these plans to leadership. So I’m really lucky to have a very actively involved CEO and CTO and head of product. And so it’s those folks and a few others in a room, and each researcher goes through what they’re intending to do over that quarter. Now, it’s really great for research just in terms of getting visibility, and, you know, getting some recognition because we also do a section where we sort of share what we thought were some of the winds, and do a little bit of a retro on the last quarter. So get some of the researchers a chance to highlight some of the great work that they’re doing to leadership. And then also just to get that feedback where they might talk about what they’re prioritizing, and then to get that color or get that really strategic perspective, from the CEO talking about, you know why he’s excited to hear us diving into this area or that And then the last part that I think is kind of powerful as well is then if there’s kind of push back a few weeks later, as a researcher is explaining, like, Oh, I’d really love to get to that, but I’m doing this, you know, they can kind of say, Well, you know, this is what our, you know, leadership team, you know, thought was really important. And it gives a little more gravitas to that sort of prioritization process. And, of course, we build in, you know, feedback cycles, you know, throughout as we’re developing this as well. So, it really hope hopefully gets folks on the same page. Um, you know, there can be a little overhead to this. So we are trying to make this as spelt and, and sort of low overhead. So that’s part of what we’re iterating on. And this might not scale if we were a team of 15 because there just wouldn’t be time to, you know, have the CEO sit there through 15 people’s presentations. So, at least for now, it’s been really valuable and, you know, we’re gonna have to think about as the team grows, how to modify it.
Steve: A lot there. There’s a lot. So the researchers themselves are, that are they’re the ones putting together their individual roadmaps.
Laith: Yeah. You know, going back to that question about, you know, building teams, you know, I want folks who who are, you know, I want to find awesome folks and empower them to do the work. I’m not, you know, overseeing or micromanaging or sort of telling them to do this or that, you know, they tend to bring me stuff and we chat about it. And I’m a sounding board, and we strategize and I give my perspective. But, you know, they’re the ones that are mediating the relationship with their, with their stakeholders. And so, you know, they really have to believe in whatever choices are being made about what they’re working on. And I also want them to feel that they have agency, you know, that they’re not just sort of, uh, you know, designers sometimes talk about sort of being like pixel pixel pushers. I don’t know what the what the right alliteration would be for researchers, you know, we could work on that, but I don’t want researchers to feel like they’re just sort of being dictated to about do this usability test or do this You know, the generative, you know, exploration or getting a survey, again, making them part of the process, and and they’re contributing to the OKR process, as well. So it’s really should be a symbiotic relationship. We’re not quite there, but I think we’re really moving in the right direction. And I’m excited about that, that that back and forth and the interplay
Steve: I think it’s a compelling vision that you’re, you’re illustrating, and I want to go slightly meta here. Sure. Maybe to meta I don’t know, you know, you. I mean, you said something that I think is quite common, right? The, the, the research resources are limited, or they’re, you know, to put another way, demand exceeds supply. And I think that’s common and I wonder, you know, outside of Udemy, like, if that’s sort of a truth in at least for tech, you know, in the kinds of organizations That means there’s lots of kinds of research, but in sort of tech overall, I think that’s a common, you know, you have different team sizes, you have different organization, you know, approaches to research, but in general, I hear this a lot. What do we what do we think of that like that? That’s a that’s sort of a universal truth that there’s a more there’s more needs than there are resources. You know, why? So what’s what is? Do you have any reflection on that?
Laith: I mean, there’s definitely I think we should be self critical enough to say we’re biased as researchers. So, you know, I see research need where perhaps other people don’t and I think that’s, that’s totally fair. And and I think it’s also the the tricky part is, are you optimizing for sort of peak demand load or average demand load, and that can be a hard thing to disentangle. And then Lastly, I think it’s what, where and how can research impact things? So, you know, I think, you know, are we already sort of cleaning up things and grinding out a lot of usability studies? Are we doing really conceptual, you know, blue sky kind of research? What kind of impact do those do different things have? are we are we, you know, what sets of stakeholders are we interfacing with? Um, you know, I think it’s a very different situation, if, you know, you’re getting into some really interesting things that overlap with back end engineering, and you’re doing interesting things with PMM. And, you know, local operations, you know, what I’m saying instead of just sort of the more core conventional stakeholders that you might have, I think that that exploring those kinds of things, or maybe things that I’m seeing that I wish I could do if I had more bandwidth, but the organism might not necessarily see that yet or see the value. And so it’s a matter of sort of picking the opportunities to try and demonstrate how big their menu of research is, and all the places that we can impact. So it is a tricky one. And you know, it’s probably better to have a few too many researchers, sorry, a few too few researchers then to go the other way.
Steve: And, you know, I think about I certainly when I started the field, there just there wasn’t a lot of demand. I mean, there wasn’t a lot, there wasn’t as much awareness there wasn’t as much, you know, process that includes these these tools and kind of incorporates what we bring. And it’s been interesting to see, you know, over time, how the demand is high. The, you know, the the The amount that that our work is integrated with, you know, other work that’s happening in the company is as grown and I think you’re just you’re articulating, you know, a future state perhaps where there’s even more, you know, more aspects of the work that a company is doing, are places where we can bring value. That’s like a, I don’t know what the shape of the curve is over time, but you can sort of see it evolving from, you know, who are you guys to, we have a lot of work for you to, you know, in the future, there’s way more than we can do then you then people are maybe ready for or that that, that we have demonstrated, you know, as a practice, you know, as, as broadly.
Laith: Absolutely. I mean, it as I’ve grown in this field, you know, I’ve really found that, you know, Human Centered Design has kind of permeated my life. You know, let’s say I’m trying to I set up a time to hang out with some friends. And you know, it’s sometimes hard to coordinate everything with like three or four friends. You know, when I’m typing up that email, you know, I think about, oh, a little, little visual hierarchy in, you know, as I typed this out to make people understand the options and what I’m asking for and to reduce the language and make it easier to scan, you know, that goes into me putting together that email. So I think we kind of have to embrace that, that philosophy of Human Centered Design is really powerful. And it’s been fun to your point of seeing some of the bigger companies where the internal, you know, human resources team, or the you know, real estate team might have a user researcher because they’re thinking about that that’s a product you know, how people access the benefits internally or what workspaces are like at a company or all these different kinds of applications. And that whether we call it user research or service design or you know, any of these kind of but that that that fundamental approach Can can benefit organizations in so much of what they’re doing. It is fun to think about that, that that future state and all the places that we can, you know, deliver value.
Steve: You know, and use that up top, you know, you came from academia, you had sort of just a variety of things that you’ve done along the way, in research and kind of before research, and I’m wondering if you kind of look at your history, but also, as you you know, have you as you grew into the field, you I think learned about the, the history of the field of user research. Do you see? I mean, are there seeds of what we’re talking about? If you look backwards in, you know, where research had been applied, how it kind of evolved or, you know, what you see when you look backwards?
Laith: Yeah, this this is a topic that I’m just continually fascinated by, and I would love your perspective on this, because I’m sure you know, a lot of this very well. And it’s been interesting. You know, I think sometimes When I’m doing my teaching, I almost get the sense my students kind of feel like this is how it is. And this is how it’s always been. And you know, sometimes I kind of have to remind them a little and be like, this is actually a very rapidly evolving and changing field. And you all are going to have a huge say, in sort of what this, you know, ultimately looks like or sort of what it what it might look like in a more mature state. And it really has undergone these different shifts in thinking about all the different strands that go into it, at a very, very simplified level. And not to say that these phases are not overlapping or exclusive, but it feels like, you know, previously there had sort of been the innovation labs, the Xerox parks, and that that had been sort of where a lot of this kind of work was being done. Obviously, there’s a older history when you’re thinking about World War One and two and human factors, and that’s another sort of strand in this. And then it feels like some of this work then became a little more focused around agencies. You know, you’ve got your radios and things like that. And we started to see that shift to in house research happening and calling it UX and things like that. And I think around then is also when we saw this shift to much more consumer oriented products. Before a lot of it, you know, had been very enterprise e or, you know, business oriented, where maybe someone, all their exposure to human computer interaction would be the one computer at work or something like that. And so we over this time, we’ve evolved to this place where technology is being much more omnipresent. And you’re not just dealing with a technology one manufacturer, but you’re starting to have to think about that the design changes on your phone are affecting the design changes on your computer, and those might be made by different software companies and hardware companies and all these kinds of things. Now, I think originally, a lot of those in house teams were embedded, or sorry, we’re centralized, like I sort of talked about a little before and then we’ve seen Seeing the shift to the more embedded teams and the real just tremendous growth that’s happened over I don’t know, the last five to eight years, and weaving together all these different strands from psychology and design and Information Science and anthropology and you know, more than I could sort of a name. And we’ve also seen the pipeline really emerged at these really great masters programs, you know, a lot of programs in library science, or have rebranded themselves as an Information Science program and are contributing to this. And of course, you know, human computer interaction programs and things like that. So it’s interesting to think about sort of what comes next. I started to see some more innovation labs kind of things popping up. So it’s funny how that might be coming back into vogue. And then the sort of rage of democratizing research and, and sort of figuring out how to scale the impact of research teams beyond just embedding more and more researchers interesting. As well as you know, sort of the the global nature of this, you know, feels like there’s a bit of a different UX culture in some parts of Europe, and you’re seeing emerging UX culture in in other parts of the world too. So that’s kind of my take that I’ve gathered through just a few articles and chatting with folks. But I’d actually be curious if that aligns a sort of a very rough arc with sort of your take of the field.
Steve: So this is the part of the interview. This is the danger of interviewing an interviewer. Yeah, I mean, I would love to sort of, maybe someone has done this, but I’d love there’s a visualization, it’s kind of it’s kind of a timeline you know, around with some of these major things happening and you know, as you talk about it, you know, the the emergence of design and consumer products as has elevated to design, right? The, the way the jobs for designers changed once there was an iPhone, I mean, going back the jobs are designers change once there was like a Macintosh computer, but and that, you know, we’re, I have felt for a lot of my career that and not necessarily a bad way but that that research is kind of following in the wake of design. And so when I joined the field it was it was following in the wake of industrial design, because that’s where all the innovation was happening. And, you know, they kind of they kind of weren’t, they were the innovation labs. But then the web comes along, and there’s just so much growth and you know, things that you talked about. And so, you know, user experience comes along as a phrase and suddenly we’re user experience researchers and researchers because we’re, we’re working on that in those industries. We’re working on digital stuff more exclusively. You know, and I think we, I mean, in your narrative you are, you’re any good history has to do it includes a lot of different as you call them strands. And sometimes I feel like it makes, it’s necessary, but it’s hard to see the research strands like you don’t want to decontextualize it. But I also I do want to decontextualize like I want to how do we talk about research without talking about anything else? Just as a you know, as a, as a as a thought exercise, like just to look at how research has changed, even though these other things have changed, and that’s why research has changed, but I’m certainly interested in the like, I think you’re going to your point about, you know, a lot of the work moving in house and as someone that’s still a consultant, I can tell you that’s that’s definitely true. It really has changed, my professional life changed dramatically. As that trend took hold, so I feel that one very specifically. And that’s a very research specific one, even though I think you see lots of creative professions that were historically done, those consultancies are done as agencies got moved in house. I feel like there’s that Apple caused some sea change in the ad industry when they Yeah, brought something in house. And I don’t know if that was sort of the the change point that led to what you’re describing with, you know, research overall.
Laith: I think that if I were to focus a little more on the research strand, because I think it’s a fair call out, I would say there’s two things and one is thinking about the position that the researcher has these of either product or what they’re working on. And it’s funny because I think Yeah, back in the day, and I’ll contradict myself a little about not about focusing on research, but you know, if you were a designer, you’d be like, Oh, I want to go work for fjord or tea handed lax or You know, these kind of big design agencies and they all got either acquired or Accu hired not all of them, but a lot of the really prestigious ones. You know, and and you saw that happening a little with research too, with with bolt Peters, you know, and things like that getting brought in house by the big companies Adaptive Path and things like that. So it has been interesting to see that, that that that transition, but I think we’ve inherited certain ways of approaching things, or at least I have, I know, coming from that consulting side, it was kind of a formal relationship that I had with the product and with the product teams. And you know, it’s sort of like having the big, bloated deck, you know, that Edward tufte A would, you know, you know, shake his finger at me for putting together and I’ve had to think about ways of thinking about like, Hey, you know, I don’t really need to go through a lot of that front matter that I might have had in this very formal deck for clients, with people that I have a much more casual relationship with. Similarly at Udemy when we first transferred from that embedded sort of Central To embedded forum, they’ve been sending out surveys, sort of satisfaction surveys to see like, how did the research project goes. And ultimately, you know, there are people that you’re in meetings with all day and you’re sitting next to them and all these kind of things. It feels kind of odd and distancing. To send that satisfaction survey. Now, we still need to have mechanisms for feedback and to figure out ways to do better, but we kind of needed to adapt the ways in which we work to reflect these different, you know, relationships. And I guess the other way I’d flip it around is to sort of to take a research, design thinking approach, which is, you know, if we were to redesign, design research today, and could somehow throw away the baggage or sort of the history of all these things, how would we design it differently? How would we redesign the way we work and our relationship with products, you know, allowing that there will probably be a spectrum of those things depending on the company and those kind of things. You know, it’d be, you know, as a further thought experiment, it would be kind of fun to think about, what are the things that we’ve carried forward? Because of these histories coming out of industrial design and out of human factors and out of these sort of evolutions? And what are the things that we might do differently if that if we’d been on a different path, or we’re just starting a new today?
Steve: And one of the, I think, one of the legacies and you know, whether it’s a strength or a weakness or both, is the the, the diversity of paths that have led people to research? You know, it’s still true today. And it was, yeah, like, I think, you know, if you go back in the olden times, there were few sort of credible paths, they had to sort of invent a path and now there are, I think, many paths that are somewhat credible, but it’s So lots of people have to find their way in from something else. Right. And I think there’s there’s, we’re the field welcomes more people from more places, it’s still a struggle to get in. And then I guess, just there, it was a different struggle in the past, but the that lack of definition of like, what background does a researcher have? Or what backgrounds plural does a research team need to have? I think that’s very exciting. But it also, it also seems like a huge amount of legacy code that we spent a lot of energy and emotion on that if we were looking at things to sort of toss away and reinvent from scratch. That would be that would be one that I would I would lobby for.
Laith: Yeah, it’s tough. And, you know, I think a consequence of that is the research culture at different organizations can be so different, you know, if you have one place A lot of the folks are master’s degrees in human computer interaction and not saying that that’s a bad thing, but it’s just a set of things. And you have another organization that, you know, more folks coming out of sort of design anthropology or applied anthropology, you know, they’re gonna have very different ways of framing problems and approaching things. And you might have another organization where it’s a lot of designers and engineers, who you know, learned and got into research, you know, through work experience, those would all have very different flavors. And yeah, to your point, it’s kind of like a great thing and it’s an exciting thing and it’s an enriching thing. But it can lead to folks talking past each other, or, you know, getting in over their heads in certain situations. You know, I do sometimes my main concern can be about the ethical conduct conducting of research, and that, you know, I really hope that everyone who gets to do this awesome line of work has the the proper you know, ethical, you know, You know, training to sort of keep their eye out for getting into situations that are not good for them or not good for the industry and are not good for the research subjects.
Steve: And where does that? I mean, I think this this speaks to the multi discipline. I guess that’s what you’re saying. So I’m just reiterating your point for you. Sorry. But like, yeah, where does that come from? How do research teams work, regardless of what their you know, what their composition is? How does? How do we mature the practice of the ethical aspect of the work?
Laith: It’s tough. I mean, I think that there there needs to be a degree of self policing, you know, whether it’s on slack communities or email listservs You know, I think it’s it’s okay to discuss things that have popped up and say, Hey, you know, I’m seeing the survey coming out from this or invitation to participate in research like this. And you know, I don’t know how I feel about About that, definitely, I’d say larger teams, which I think is where a lot of more junior folks are able to get their start. I would love to see them have a sort of a formal cadence of doing some some ethics training. I think it’s super important. And But yeah, I think there there is a bit of a vacuum there. You know, there’s some good articles and it does get covered in you know, the, the seminal, you know, works like the Baxter courage cane understanding user needs. You know, there’s great discussions in there about, you know, ethics and things like that. But um, I definitely think as an industry, we could be doing more.
Steve: And this is goes back to the strands thing, like the conversation about ethics in user experience, progresses a certain way and it’s easy for us as researchers to get sort of swept up in that but the issues are different. I think there are specific things as you’re kind of getting at.
Laith: Yeah, I think that the the things you’d watch out for if you’re doing academic psych experiments in grad school would be very different from if you’re doing anthropological field research, you know, let’s say with a marginalized community, but yet in industry, because of the wonderful interdisciplinary Enos, you know, you might have those folks sort of doing the other kind of research and not being fully abreast of the, you know, you know, if they have very advanced ethical training and research ethics training in their home discipline, you know, as they move and sort of get to experience the other strands, they might not be totally, you know, up to speed on on those other kinds of areas. So, even as folks that think they have really great you know, you know, Human Centered research ethics training, as we move into this UX research world, there needs to be something sort of specific for us. I will say that that you know, the UX pa has guidelines market research organizations have ethical guidelines. And I think those are a really fantastic starting point for the work that we do, because they’re, they’re focused on sort of this more applied or industry setting. But, um, you know, there’s a lot more that I think we can do collectively.
Steve: What do you see in your students in terms of their awareness or, you know, hunger for this kind of guidance?
Laith: Um, you know, I, I don’t know how representative Berkeley is, you know, I think it tends to approach sort of attract a little bit more of the mission driven or idealistic kind of folks. So, I know that whenever I’ve mentioned, you know, doing work with different kinds of communities or crossing socio economic status, you know, sort of working with folks in a broader spectrum, you know, not necessarily just, you know, Bay Area folks that have an iPhone, you know, 11 You know, I think that they really get excited to think about Those kind of things, and they really embrace the responsibility and recognize the privilege that they have. I have to say, I’m just so impressed and blown away by how sophisticated the students are, you know, I feel that they that they’re really attuned to some of these kind of things, and you just have to open the door a tiny bit, and they tend to really, you know, jump at the chance to sort of think through some of these really complex issues. And think about the ramifications of, you know, sound design of people that, you know, have different accents, or AR VR for accessibility and things like that, you know, they’re they’re connecting the dots really quickly. I just think it’s really important to give them sort of the frameworks through which to process these kind of things. So that’s kind of what I see to be my challenge.
Steve: Your challenge is to give them those frameworks.
Laith: Yeah, exactly. So they think it kind of helps. To see like the bigger picture a little easier.
Steve: Yeah. Well, this is super interesting. And we could obviously dig into all of this two times more than we already did. I guess, just to throw it to you like, what did I ask you about? Or do you want to talk about that? We didn’t get to?
Laith: I don’t know. Uh, you know, I’m a nerd. I can talk about anything for forever if you can’t tell from the conversation. But, you know, I appreciate you give me the chance to talk about some of the historical stuff. And just quite frankly, like outside of the interview would love it if you had any more perspective on that to share just because I think it’s interesting to think about that legacy. But I don’t really have any other you know, access to grind or anything. If you feel you you’ve got enough.
Steve: So I’ll throw one more question at you and then I’ll try for a perspective on you know, on the strands conversation I guess so so. So if this was, if our conversation on dollars to doughnuts was going to end up in podcast club, what would you want people to get out of it?
Laith: I think I’m really into the idea of sort of questioning
what we do. And, and, and again, that can be the methods and that conversation about getting out of our comfort zone. It can be thinking about our relationships with stakeholders and trying to reimagine and iterate on those. And it can be thinking about the historical trajectory of the field and the legacies that that has imbued us with. And I think that just sort of being ready to iterate and question the assumptions that a lot of those things are built on, is the thing that I’d really want folks to come away with.
Steve: Excellent. Can I said I would, I would try to offer something else. I’m the sort of looking forward looking back on the field piece. You know, and so, my, my perspective is both informed and I guess hampered by just time. Like I have been doing this for, you know, more than two decades, which is weird what to say when it comes out of my mouth, but it’s, it’s true. And I have just been feeling for the last little while that, you know, we’ve been on this trajectory, that sort of this gentle upward slide in terms of, I don’t know, skill building, like figuring stuff out building best practices, like a lot of things that we were, you know, making up as we went along, have been formalized, have been shared have been documented. There are, you know, for a talk I was doing, I just tried to gather up as many books about research that are out there and there is a lot like There’s so many of them. And that’s when it’s a good thing we’re, we’re talking about this expertise is being documented and shared. And I’ve just been feeling in the past year or so. And, you know, again, this is a stone, it’s only my perspective, it’s not a declaration of what’s real or true, but I’ve been feeling like that we’re like any sort of, I mean, maybe you have a better take on this as someone who helps people build skills or do learning, right, you, you advance at a certain rate, and then you kind of reach a plateau, and then you and you sort of you cruise at that plateau for a while, and then you hit the next, the next rung or the next, you know, movement upwards. And I feel like that in you know, in, in the profession that I’m part of, as I see it, I feel like that’s what’s about to happen or is happening. It’s happening in little pockets, and we’re just starting to to be bring it into the mainstream. And I mean, I am I mean as as as, as the day we have this conversation or a number of days away from a conference I’ve been involved in called advancing research. That’s part of the Rosenfeld media, advancing research community, which anyone is welcome to join. And I’ll put a link in. And I think the name is really great. And you know, that we we talked about, how do we position this conference and what does advancing research mean? And, you know, what are all there’s lots of great stories and great work that gets shared. But we’ve also heard it a lot over and over and over again. And you know, yeah, if you’re new, maybe you haven’t heard some of this, but there’s so much stuff that we haven’t heard that we need to be talking about, and you brought up ethics, you know, earlier in this conversation as one that like yeah, it’s here. It’s there, but it’s we haven’t really made it part of our it’s not part of the the baseline, it’s not part of the kind of the core fabric of what we all practice, it’s like a thing we need to do. And I’m, I feel like, you know, as we move from the plateau to like the next incline, that we’re making a collective concerted effort to bring these under recognized pieces in. And what’s exciting for me is you don’t have to look very far to see, oh, that thing that we’re not really doing well, somebody is working on that somebody has a point of view, somebody has built something or built a process or built a framework and that it brings me at least an opportunity to to, you know, to iterate or to to move to that next, you know, that sort of 2.0 3.0, whatever. I don’t know what the numbers are, but I feel like we’re on the cusp of that. And again, that is such a, that is my perspective. I can’t say that anyone else would feel that way. But it’s something that I am drawing a lot of excitement from, because I feel like oh yeah, I know how to do a lot of this stuff. And there’s things that I didn’t even know that I didn’t know how to do, that I should be doing. I don’t even know where considerations I’m not talking about, like, you know, our ever shifting library of tools and infrastructure, but, you know, really the, the, like, read like an iteration of best practices to be more inclusive to be more empathetic of our colleagues. And I feel so I feel very excited about the near term because I feel like that stuff is is coming out. Yeah, I don’t know if that if that how that matches with your own trajectory, which is obviously different than mine.
Laith: It’s so interesting. I’ve been having some really great conversations with do you know Steve Fadden.
Steve: Yeah, yeah,
Laith: yeah. So um, in you know, I think this is something that like organizations have struggled with where Like, what’s after a senior researcher level is a little bit like there be dragons, you know, it’s like kind of easy to articulate like an associate to a researcher to a senior researcher. But you know, if you don’t go the management route, you know, what makes you you know, what a principal or a staff researcher or whatever that organization calls it, you know, the language tends to get a little mushy. And what would distinguish that senior researcher from that next level, I think it’s something that we as an industry are like, to your point, we’re still we’re getting closer, we’re kind of grappling with it. And it’s a little tough to, to talk about, you know, what are those those things that really go into having that impact? That would sort of denote a level beyond senior. And you know, this is a little beyond the, I think some of the title inflation that might happen in the in the in tech as well. But there’s some real interesting challenges there. And I think there’s also a set of unmet need here where I’ve met a lot of folks that have kind of hit that plateau and it’s can be a little hard, especially if they don’t have a really involved manager to know about to figure out on their own, you know, what are those things are? How do I get the training? Or how do I get the experiences, to get out of my comfort zone or try the new things or be aware of the things that I don’t know that I don’t know, to figure out how to get to that next level, it can be a little bit of a glass ceiling. And to your point about inclusiveness, I think that that’s also an area where we’re bias can really creep in, in terms of, you know, allowing people to express what they bring to the table and make a case for themselves, that they’re having that impact or however they would be measured to sort of go on to that next level. So it I like that you brought that up and it’s a it’s share the perspective that’ll be exciting to sort of see how we collectively grapple with some of that.
Steve: I know that just today and again, our today is not the today of the person that’s hearing this but anyway The research ops community just published. I think it’s called like a research skills framework. So I saw that it was posted I haven’t seen what it Oh cool, what it consists of but I think at least it whatever it contains, it validates through another source the the issue that you’re identifying, and that this is a this is a thing that’s being considered and, you know, worked on by the research community at large.
Laith: Yeah, that’s awesome. And I love that, you know, a lot of times companies can’t share these kind of frameworks externally, which is which is a shame. But I love that that they’re able to come up with something that they can share publicly and it’s one thing I do love about this community like the the listserv is so helpful. People pay it forward, people pay it forward to me like yourself when I was first coming into this, this field, and I’ve been really lucky to help a few folks here and there as they have tried to enter the field and things Like that, and I just love that there’s such a propensity towards knowledge sharing, and I think it’s just going to benefit us all in the future. So that’s one thing I am very proud of, in terms of our community.
Steve: I think that’s an extremely positive up note too. So yeah, thanks late for taking the time and sharing so much interesting stuff and really for a great conversation. I really appreciate it.
Laith: Thank you for the invite. Really fun conversation and I appreciate the opportunity.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 30. Laith Ulaby of Udemy first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Laith Ullaby, the Head of Research at Udemy.
I’m really into the idea of questioning what we do. That can be the methods and that conversation about getting out of our comfort zone. It can be thinking about our relationships with stakeholders and trying to reimagine and iterate on those. And it can be thinking about the historical trajectory of the field and the legacies that that has imbued us with. And I think that being ready to iterate and question the assumptions that a lot of those things are built on, is the thing that I’d really want folks to come away with. – Laith Ullaby
Show Links* Laith on LinkedIn * Udemy * UC Berkeley School of Information * See One, Do One, Teach One * Acquisition of Fjord * Partners At Teehan+Lax Join Facebook * Facebook Acq-Hires Part Of Design Firm Bolt | Peters * Adaptive Path Acquired By Capital One * Edward Tufte * Understanding Your Users by Kathy Baxter, Catherine Courage, and Kelly Caine * Insights Association Code of Standards and Ethics * Advancing Research conference * Advancing Research community * Research Skills Framework
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I had a profound moment this week. I was facilitating a meeting, not a user research setting at all. But we were having a fairly open-ended conversation. One of the people in the meeting shared some very specific advice for the rest of us, offering almost an impromptu speech or pep talk, filled with passion and encouragement. It was very inspiring. But as the facilitator, perhaps because of my work as a user researcher, I wondered if maybe that wasn’t enough. This person articulated their richly-realized state of being, and it may be easy for anyone else to dismiss it, oh, that’s just so-and-so, they’re just like that. But, we’re not born with insight about ourselves, or clarity about a new way of being, so I was curious – and I thought it would be helpful – to understand more about how this person accomplished this. How did they get here? And even though I wasn’t asking followups in this session, it seemed like an opportunity to shift my role slightly. And so I asked “How did you get to this stage, where you have this clarity in your approach?” This person paused, and said, “Well, if you really want to know” and then proceeded to share very personal details about their life, from hardship through to today and what they had overcome and how that had shaped them. The details aren’t important for my story, though, and they really belong to the individual who chose to share them, not me. Anyway, it was an impactful moment for all of us in the meeting. And it reminded me about a fundamental in user research, that you don’t know what’s behind a door, it could be something banal, I mean, it usually is, right, but you don’t know until you open it. And sometimes what you’ll find is new, surprising, and impactful. But if you assume that it’s always banal, you’ll never try to cross those thresholds, and never have the opportunity to hear those significant perspectives. I’m describing this as a doorway and that what we do as researchers do is step into that space, but I don’t think that’s sufficient to capture what it is that we do. Yes, we are stepping into something ourselves, but also, and probably most importantly, is that we’re making space for someone else, choosing to ask that question, to be interested, invites someone else to take a step forward themselves. You can call this a holding environment, or a safe space, but I think the additional point I want to make is that it’s not just what you do as a user researcher, it’s what you do for someone else.
I’m putting another episode here done without the benefit of a professional editor or transcriptionist. I’m focusing in the near term on how I can contribute but of course this is a difficult time for many people including those who operate a small business. It’s a challenge right now for Portigal Consulting. If you’re in a position to consider me as a partner, that’s appreciated. I lead research projects, I help companies build a more research practice including assessments, training, and coaching. Thank you!
Okay here we go with my conversation with Laith Ullaby, the Head of Research at Udemy.
Well, Laith, thanks so much for being on Dollars to Donuts. It’s great to have the chance to speak with you today.
Laith Ullaby: Pleasure to be here excited for the conversation.
Steve: All right, well, let’s, let’s get into it. Why don’t you introduce yourself?
Laith: Sure. So my name is Laith Ullaby. I’m currently the head of research at Udemy, which is an online education platform. I have a academic background in ethnography, and have bounced around the UX research world for a few years now. And then I’m also really passionate about teaching. So for the last few years, I’ve been teaching at the UC Berkeley School of Information and at the UC Berkeley Extension.
Steve: So what does it mean to have a degree in ethnography? Not? I feel like that’s not a phrase that everyone can say exactly.
Laith: Sure. So technically, the field that I studied was ethnomusicology. Which is using ethnographic methods to look at the intersection of music and popular culture, and so my focus was on how governments in the Middle East, were using technology to disseminate national identity through folk culture. So it was a little niche. But, you know, it was fun to sort of build that foundation and ethnographic research and think about how sort of the interplay between popular culture and technology.
Steve: Do you? Are there echoes of that in, you know, your daily practice? Like, can you draw the threads back to what your what that program was about for you what that research was about?
Laith: For sure, you know, I think that the, the, you know, when I think about user research, it’s really combining the strands from a lot of different places, different academic places, different you know, parts of sort of the applied world. And one of those is sort of the design anthropology world which I think is really steeped in ethnography and for me that really raises some of these really interesting questions about identity and community and how we sort of come to understand ourselves, as well as power relationships. So working on a marketplace for education, thinking cross culturally about, you know, who feels empowered to be a creator of knowledge, I think really hearkens back to some of the core areas of exploration that you might encounter in an academic program like that.
Steve: Maybe, can you say more about, you know, the organization where you’re at? What, what, what Udemy does, and a marketplace for education as a compelling phrase, maybe you can impact that?
Laith: Sure. Yeah, I’d love to. So, um, as opposed to some other, you know, awesome companies in the space. We’re not a publisher, we’re a marketplace. And so we have a platform for instructors to come and to find learners. So just to get a sense of the scale, we have 50 million students, 57,000 instructors and over 150,000 courses. So, you know, in other models, we might say, Hey, you know, here’s sort of the Python course that we think is cool. And in our model, you know, any instructor can come on and make any variation of a Python course. And then it’s those sort of marketplace dynamics like reviews and number of enrollments and social proof and those kind of things like you might do on an e commerce platform that folks would be familiar with, in which students then get to evaluate what’s the right course for their learning need. So it’s really exciting to sort of see that that proliferation and that there’s so many different approaches to you know, finding the right student, the right course with the right instructor at the right time. It can be overwhelming you know, you there’s obviously I’m sure a lot of folks minds are going to that, you know, moment of overload of too many options. So, it We do try and help out with that. But that’s sort of the the core value proposition of what Udemy does.
Steve: And so as a researcher in a marketplace company, I’m going to guess that there’s research focused on when you describe two players in the marketplace, right, the instructors and the students credit research is, would you describe research as kind of focusing on, you know, ultimately, one or the other?
Laith: Yeah, and that’s kind of the fun part is, you know, I think sometimes if we’re thinking about sort of the very, you know, naive version of user research, it’s, you know, we’re trying to help out the one user. And, you know, for us, it gets more complicated, because you have to think about that balance. And so, you know, you can make something a lot easier for the learners, but that could put a huge burden on the instructor. And so you have to be able to triangulate and sort of balance those. Similarly, we also have a b2b offering. So in that world, there’s also an administrator role or sort of a buyer role that can enter the fray as well. So we have to then sometimes think about the the balance and trying to find Win Win wins for any kind of change that we do. So it’s definitely a bigger challenge at times, but it’s a fun one.
Steve: So in that third example, the b2b so someone, this is a scenario where that Python course might be used by like a learning development person inside an organization to bring that in for training set the scenario.
Laith: Exactly. And so we have soft skills, we have technical skills, there’s there’s a whole range and that is a subscription model. So they would get access to our courses through that subscription model and make it available to their employees.
Steve: So what kinds of research you know, what kinds of research are you and your colleagues doing?
Laith: Um, one thing that I really love about Udemy is over two thirds of our learners are outside of the United States. So we really have this global presence and as a researcher, you know, just as a nerdy researcher that that presents a lot of fun stuff to look at. But it also means that we tend to rely a lot on remote research just to be able to get that that that global perspective, that global reach for the studies that we do, so remote as a lot, but it’s interesting, you bring this up, because I’ve been talking with my team a lot about how, you know, I think sometimes both individual researchers and organizations can kind of fall into a comfort zone with where, how and when they do research. And so we’ve been talking a lot about ways to kick ourselves out of our comfort zone and make sure that when we’re designing research, it’s, you know, truly the best way to answer that particular, you know, immediate research question that stares us in the face, rather than something that has a lower logistical overhead or just is something that we’re more confident and we’re comfortable doing.
Steve: So how do you create the conditions where, when you’re talking about risk, you didn’t use that word, but outside your comfort zone? How do you create the conditions where it’s okay to go outside the comfort zone for you, your team, the stakeholders that you work with?
Laith: Absolutely. And I think, you know, you hit the nail on the head with risk, you know, it’s not that folks oftentimes don’t want to do these things. It’s that, you know, they, they are used to doing things well, and they want to maintain that that same level that they uphold themselves to, I think there’s a few things that help there. One is, as I’ve helped to build upon my predecessor, who previously lead the research team and build the team further, really tried to have complimentary and overlapping skill sets within the team. So hopefully, it’s not something that no one on the team has done. It might be something that one person hasn’t done or a few people haven’t done. So this way we can share and teach each other. I’m a big fan of when they talk about in the medical practice, they talk about See one do one teach one. And that’s kind of how you know and and, you know, a new procedure in medicine. So I really tried to embrace that and say, Hey, folks, you know, how can we teach each other because you might have seen someone do something, and then you got to do it yourself. And now you get to teach someone, that’s when you’ve really kind of got that level of mastery. So when you’re constructing teams, I think it’s interesting to think about where those opportunities are, and that you’re creating the environment for that. I think we’re also lucky to work in a place where there’s just a lot of trust in the research team, folks have just done a good job of delivering high quality research, and that that gets you a tremendous amount of goodwill. And so I think that’s allowed us to have a little higher risk tolerance, because people know that, you know, we’re trying our best and we’re trying to deliver good research and if something, you know, by chance, didn’t go super great. You know, it was done with the sort of the best faith and intent and just generally the leadership, the product leadership and other folks are trying to you know, Encourage that general behavior in terms of the features that we’re launching or the the ways that we’re interacting with our marketing messages or things like that, too. So it helps to be at an organization that generally understands, you know, that risk is something there, and that’s where you get a lot of growth.
Steve: And buried within that as a, there’s a specific nugget that you just said, I want to kind of bring back up. You know, you implicitly answered How do you build trust? And you said, like, deliver great work and paraphrasing you but is that do I have that correct?
Laith: I think that’s a big part of it, for sure.
Steve: But then you kind of talked about goodwill, and maybe that’s maybe good work in genders goodwill, which then gives you some permission to not and no, no, no, I talked about failing but you’re talking about Yeah, getting outside the comfort zone and having trust to do that.
Laith: Yeah, I think that’s part of it. You know, hopefully there’s a little bit of a virtuous cycle, you know, that happens there. And I’d say the other part of it is, sometimes research can be a black box. And so I think that as people understand the component parts that go into research, it helps to build that empathy. And I know you know, empathy is a little bit of a loaded term and has been critiqued in some ways. And, and, but I think that just sort of at that base level of having a better understanding of what other folks are doing. And I think that you know, even just having folks observing research sessions or participating or the research team, facilitating non researchers, doing research helps to build up that kind of understanding of the complexity and the nuance that goes into, you know, really high quality research. So I think that’s part of it as well.
Steve: When you when you try to move research from a blackbox to something that has nuance. You’re asking more of your audience of your stakeholders, I want to I want you to understand that this is there’s more to this than what you knew. And maybe this is the benefit of working in a company that has learning as kind of part of its mission. But you’re asking more of your collaborators, if you want them to understand what you do is more than a black box. That is, that’s more work. That’s more brain cycles that you’re asking of them. I feel like this comes up in the, in the discussion of, you know, research is a thing that everyone’s doing. You just talked about, you know, facilitating teams guiding them and doing their own research. We we’ve, as a practice, we have invited a lot of people In. And yet, we still want to highlight the difference between what a non researcher can do and what we can do. But that means showing somebody that, hey, it’s not a black box. There’s a lot of detail here. I’m probably asking like eight questions at once.
Laith: Do you know where I’m going? I think I do. And I think this is something you know, every research or guy Ben has struggled with this in one form or another. So I think I get the gist of it. Um, I think for me, there’s a few things. The first is, it’s been a bit of an evolution for me, I think coming in as an someone with a bit of an academic background. And originally being more on the consultancy side. We sort of positioned herself as like separation of church and state of like design and research. And so we really kind of saw ourselves as on the user side. Maybe, you know, the metaphor I might use is something like we were the scientists or we were the sort of the testers in a way. And I think as I’ve moved In house and just fundamentally tried to had different relationships, I’ve had to reformulate how I think about my relationship with product teams. And I’m, you know, I’m someone who can’t code. I’m not a designer, I love all those kind of things, but I just don’t have that skill set. And I think I never, for a long time, I didn’t see myself as a builder. And one of the big evolutions in my career has has been saying, Hey, I’m part of a team that are building things. And so it’s not me sort of on the outside, you know, as a referee, but I’m part of this team that’s building something, and there has to be that kind of give and take. And I think that repositioning of myself has also helped to allow folks to reposition themselves in relation to me, and and has opened up some of those kinds of things. But you’re absolutely right. You know, researchers can’t just sort of demand that everyone drop all the things that they’re doing and have all the bandwidth in the role, world for, you know, extensive kind of things, because they are very good time intensive, especially the more elaborate research projects. So there’s a few things there, you know, again, echoing that idea of, you know, not just being empathetic with our users, but being empathetic with our colleagues and our co workers, I think is super important. I think that the principles of Human Centered Design in what we asked them to do is super important as well. So if you’re soliciting feedback on a discussion guide, um, have you made that a really easy to read? guide? Have you set up what kind of feedback you’re asking for them in a really clear and manageable way? Have you have you tried to maybe anticipate their workload? You know, I think all the things that we might think about doing in terms of building products we need to be able to do for our colleagues in different ways. But I think it also just shows that you know, for research to be really supercharged and have the impact that it can you know, it’s it’s a it’s a team sport, and you know, it’s So when we think about sort of those models of UX maturity or research maturity, you know, part of it is that the organization is willing to invest in it. When I, when I think about democratizing research, it’s it’s not just about letting a designer go, and sort of being the gatekeeper and letting them go do some usability sessions, but it’s about, you know, having them more involved in the process and feeling that we’re all designers. We’re all researchers, we’re all product managers. We’re all engineers. Now. We have specialized positions within that, but we’re building together and and i think that helps, but it’s, it’s still tough, you know, I don’t mean to act like I’ve nailed it, and I have all the answers, but that that mindset shift has at least helped a little bit.
Steve: Right, because you started off saying that, you know, you never thought of yourself as a builder. And but I hear you evolving that identity for yourself. And I heard two things. So let me actually want to check in with you. One is, is the collective view right? So, we are building something and I am part of I am part of the we, and if we are building then I, in this case, you take on some of that mantle of the identity of a builder because you’re part of that team after that, and I also heard that, you know, in, in living up to the principles of empathy for our colleagues, you are designing experiences for them through the experience through the artifacts that you’re creating, and the ways that you’re, you know, trying to make it effective for them to work with you. And that’s something that’s being built, you are a builder in the production in facilitation and production and artifacts, and all those sort of bits and pieces that make research happen.
Laith: So so like a lot of tech companies, you know, we do a lot of playing off the sort of name of the company so you know, people that work at Udemy or Udemy, mates and things like that. So, one thing that my team has worked on over the last few quarters is To make a an internal self serve research process that we call a u du u, so it’s, you know, a play on the U and the name of Udemy. But it’s sort of the idea that you can do it yourself. And we approached it in very much the same way that a product team might approach it, you know, in terms of, we basically had one of the researchers act as a PM, and we put together materials and we decided, you know, what was in scope and what was out of scope. We designed out, we sketched out the user flow, and what it might be, we tried to figure out how the different pieces of the experience would fit together, and try to, you know, create a portal through which folks could do this with as little friction as possible. And we very much approached it, how a design team or you know, a product team might make a product because we wanted to, you know, sort of practice what we preach and and embrace those principles, and we’re iterating on it, and we’re doing user testing on it and you know, we’re trying to do all those things, so that it can evolve and be the best way that folks can do research. themselves?
Steve: How did it feel for the team to do that to take on these, you know, put on some different hats and kind of act as if you you were these other types of roles?
Laith: Well, I will say, just because you brought up the issue of feeling, which I think is important, is I think that, you know, for different folks on the team, this was more and less out of their comfort zone. You know, I, I know that there can sometimes be that that gut reaction when we think about sort of opening the gates and letting non researchers, you know, run wild and so there was some good fun discussions about, you know, how we saw ourselves and how we talked about research and some folks really kind of had to grin and say, Hey, I’m going to trust the process and see how this goes. So that was a fun kind of thing to go through. Um, but definitely, it was great to see folks taking on these other roles, and you know, maybe putting some of their design chops to use that they maybe don’t think of themselves as a designer, but they’ve picked up some of those skills along different ways or put some of their chops together and and kind of see you know, put into practice what they’d seen great pm doing. So it was a really fun process. And I think it was a fun bonding process, we kind of did it as part of an off site. We have one awesome researcher, but she’s located in Dublin, Ireland, so we don’t always get to be as much spend as much time with her. So we did this, the bulk of this work sort of did a sprint, when she was visiting, you know, the main office in San Francisco. So it was also just a really fun way to get to know each other and our working styles because we’re in an embedded model. And so folks are working with teams. And so it was it was just really fun to be able to work on a project together and kind of learn from each other about how we approach product design and product development. So just as a team bonding experience, I thought we got tremendous value out of it on top of sort of the the output in addition.
Steve: You mentioned empathy for your colleagues, you know, trying to do the work of a world They don’t normally hold, I think would might illustrate that as well for everybody. Absolutely. What are the so just maybe shifting gears a little bit, you know, thinking about the, the populations, the user groups and sort of this, this space of learning of teaching and learning, you know, what’s are their unique ways that you have to approach you know, beyond the the global aspect, whether it’s methods or just, you know, the, the mindset you have to bring to these particular areas that you want to understand. Was there anything specific there?
Laith: Yeah, um, I think that there’s probably a similar thing that you might find with financial services, some forms of that or the fitness world where folks have a lot of aspiration to do a lot of learning But it takes a pretty big investment to, you know, learn a new skill, especially if it’s in a more self directed environment. So, a lot of times, it’s that classic thing of, you know, someone might if you talk to someone right after a purchase, they purchase a course, they’re imagining the sort of future where they’re, you know, diligently doing, you know, three hours a day in 10 hours on the weekend, and, you know, things like that. And, you know, unsurprisingly, you know, a lot of folks struggle with that, whether it’s their current work responsibilities or home responsibilities, or, you know, they hit a stumbling block, and they’re not sure how to how to get over it. So it can be a challenge to parse between sort of how people imagine themselves behaving and then reconciling that with maybe what we see in the data analytics or or sort of what they do. You know, there’s some interesting behavioral science things about the more you can empathize with your future. The more effective you can do at helping that future state. So, you know, there was a company in Mexico and they would digitize an aged version of your face when you were presented with, like how much you wanted to contribute to your retirement fund. And sort of being able to empathize with yourself in the future made people more likely to contribute to themselves basically their own retirement fund because they could sort of imagine themselves as not being distinct in the future, but being the same person. So there’s a lot of challenges there around motivation and intent versus behavior. That’s a little bit of a rambling answer, but it’s it’s a complicated research area to dive into.
Steve: What about the you know, how do you approach the the instructor, community or segments?
Laith: Yeah, um, You know, definitely, marketplace dynamics can be a challenge. You know, if you have an entrenched instructor who’s very popular and has, you know, hundreds of thousands of enrollees, enrollments, it can be tough to figure out well, you know, what if a new instructor comes on, so helping folks to kind of feel that they have a fair shot is super important. And then also, there’s a few elements, I think, that are really important about helping instructors get the right kind of feedback in both Udemy and other places I’ve worked, you know, if you ask folks, if they want more analytics, and you start listing options, they’ll just say like, yes, yes, yes, yes, yes. You know, people just feel that they want more data and they want more information. But at some point, it can be overwhelming, or if they don’t know what to do with it, sometimes folks can kind of feel guilty and you know, this numbers changing and I don’t know what I’m doing differently and things like that. So there’s a real challenge in helping to not just present in information on a dashboard. But the context, the understanding the nuance, and hopefully equip them with strategies or suggestions along the way. And they’re not just sort of twisting in the wind to sort of decipher the, the different data visualizations, you know, how they might do without that assistance.
Steve: Right. And there’s, there’s something maybe lurking underneath your stories about, you know, the gap between I mean, maybe it’s simplistic level, like what people say they want or how we can get people to agree that features are, are things that they want versus, you know, serving a serving a deeper need, which is harder to harder to extract and kind of have a deep understanding of, that’s absolutely the difference of what you’re trying to accomplish with research.
Laith: Absolutely. And, you know, that’s kind of the you know, in some ways, the Holy Grail A lot of the work that we do is is figuring out how to get at that, you know, underlying unmet need and things like that. And I’ve even seen, it’s been interesting recently, you know, a few teams working on similar product offerings or features, and then realizing, when we took a step back that, hey, these might actually be trying to solve some similar things, we might be able to combine these efforts. And I think that’s one thing that research was helpful in is sort of highlighting what the actual value people were getting out of some of these prototypes. And it helped to sort of triangulate you know, what sort of, you know, had the most juice, you know, that we thought could really deliver something unique.
Steve: You we talked about your team kind of indirectly Can you describe Kind of, yeah, what kind of what kind of skills and backgrounds you know, make up your team and, you know, in future what do you what do you look for when you When you meet prospective candidates for your for your team?
Laith: Yeah, it’s a great question. I think that we have a pretty diverse range of backgrounds, which is pretty cool. At Udemy the way we’re just organized, it’s market research and user research together. And, you know, this is something I think is really interesting in the industry is all the different permutations of org charts. And, you know, you know, where folks sit and things like that. I think sometimes it can get, you know, over indexed as something that’s important because I think sometimes it’s often just legacies of how companies grow. But it’s interesting and I think it is good to think about, so we have one market researcher, and then four user researchers and then myself and it’s actually been really fun because to to have that market research lens as part of the user research team. Because it really helps to highlight things about brand voice and brand positioning and Sort of that that more holistic experience of how someone might enter the product through different campaigns, and and sort of thinking about that broader journey. So it’s been fun to have that perspective, most of the places I’ve worked at before, you know, market research is in a different building. And it’s kind of a black box even internally. So it’s, it’s cool to have that that together. And that person tends to bring a little bit more of that the survey chops and and some of those more market research kind of things, not to say that user research doesn’t do surveys, but you know, sort of bread and butter and approaching some some different kinds of questions. And then on the user research team, you know, we’ve got a couple folks that have a, you know, master’s degree in HCI. And then other couple folks that, you know, came to the industry through through different routes, including someone who formerly used to be an engineer working on renewable energy in that kind of format. So, it’s cool to that we all bring in different things, and we’ve been at You know, some different organizations and sort of trying to figure out the DNA of those places and what we want to sort of copy and and what we hope to innovate on or what we see that you know, someone might be bringing in a different mousetrap. When I’m when I’m looking at candidates Udemy is very mission driven, even though it’s a company, a lot of folks really just believe in education and the power of education, and the global nature of what we do. So, you know, always want folks that are really bought in to, to the idea of what we’re doing. And then I think beyond that, it’s, you know, someone that can bring in something new and that can be new methods and new perspectives, new approaches, but um, you know, I don’t really see it as that there’s like a one single profile that is gonna is gonna mesh well. Collaboration, of course, is super important. And having that that flexibility To think about, you know, for each project what what sort of permutations or iterations you know is this something that’s going to be a little more quick and dirty is is something that’s going to be a little more rigorous. The last area that in hiring that I’ve sort of formulated in my mind is, I think about written communication, verbal communication and visual communication. And I really want to find folks who are really comfortable, at least one of those, like really good, and then hopefully, you know, decent at the other two, and, you know, willing to kind of work on them. But I think it’s important that you have one modality of communication that you know, you’re just really good at, because you know, so much of what we do is communication.
Steve: I want to loop back to you said all sorts of all sorts of interesting things. I want to loop back to one of them. Because you mentioned a few moments ago that that that people are embedded. But you also talked About the sources. Watch one. Sorry, see one. Do one teach one. Is that right? Yeah, yeah. Okay. So I’m just wondering about, you know, especially you got this market researcher with a, you know, a pretty unique skill set among your team. And you have all this diversity. So how the the researchers are are embedded they have product teams they’re specifically focused on. So how do you balance that focus and then harness the hardest that diversity and let people influence each other and teach each other and draw from draw from that team? You know, that you have the centralized part and you have the embedded part. I’m wondering what that looks like.
Laith: Yeah, and I you know, I think this is a challenge for for for folks in some organizations to sort of be a functional leader, but not a project. Leader, where the folks on my team are just so much more expert in the details of the features and pods that they’re working on. So, you know, sometimes I feel like, you know, oftentimes I feel like, you know, they’re teaching me or they’re, you know, showing me how things are working. And so I have to think about other ways that I can contribute or add value, obviously, you know, that can be with being a sounding board for thinking about methods and approaches. But But to your point, there’s also about sort of how I can create that sense of a research team, even though most of our day to day is kind of often embedded with, you know, awesome designers and engineers and marketers and you know, folks doing other things. So we have a few mechanisms. And, you know, I hate to say that the answer to everything is a meeting, but some of it is meetings. So we have a team meeting. We also have a bi weekly session. That’s a we call it researchers talk shop. And so we have a spreadsheet where people just throw in topics and we For a lunch and we try and dive into different research topics. So that’s a great way for, you know, someone’s been doing a lot of workshops recently, and they can kind of share what went well and what didn’t go well. And we do some of that in our research team as well. We have a office hours as well, so people can just stop in and talk with us, but we, you know, sort of hang out in the conference room. And that’s just an opportunity for us to kind of catch up a little as well, you know, because sometimes, you know, 10 people show up to office hours and sometimes it’s just one and just kind of hearing each other helping other folks and office hours you kind of get to hear a little about their approaches and things like that. Another mechanism that we have is we do try and pick some some bigger, juicy or questions that we are sort of more researcher initiated, and we do those collaboratively so that they you Edu initiative is a is an example of that. And we’ve tried to do a few other collaborative researcher initiated projects. Another thing is we have a little podcast club, so We’ll listen to podcast episodes and then just set up a quick little half hour to, to talk through it in different ways. And it’s just fun, you know, a chance for researchers to to talk about things through that lens and try and grow and get those other perspectives. For me, the nice thing about podcasts is it feels really low friction to just find like a 20 or 30 minute podcast. And, you know, folks can do that when they’re commuting to or from work and you can still have a really lively discussion, I found historically, it can be a lot harder to give folks PDFs or articles, you know, it can be harder to sort of find the time. So those are some of the mechanisms that we have to try and develop that that sense of community and exchange.
Steve: That’s there’s a lot of them. And so I know you’ve got one person in Dublin, for the people that are in San Francisco. Are you sitting in a co located or whatever the better way to say that would be
Laith: we’re all sitting on the same floor but you know, it’s a decent sized floor. So, you know, there’s folks that aren’t in visual visual range. So we’re kind of sitting near the product teams that we work with, we’re sort of the seating chart is by, you know, product teams rather than by functional teams. But you know, you know, you run into people constantly just in the halls and at lunch and running between meetings. So, definitely, I see everyone, you know, I’d say, at least a couple times every day, you know, even if it’s just passing.
Steve: And so for researchers that are embedded in these product teams, how, how is sort of the the, the plan for what they’re going to work on determined?
Laith: Yeah, it’s a great question. And shortly before I joined the company, or a little before I joined the company, the research team had switched from that more centralized model to an embedded model. And so some of this has been an iterative process. The other challenge is, you know, there’s just not infinite reading resources. So is the ratio of researchers to designers and PMS. You know, there’s usually one researcher juggling at least a couple PMS if not two, or three or even more. So it makes prioritization really a challenge. And I think before we’ve been a little reactive, and it’s hard to make sure that you’re working on the right things, if there’s just sort of like a queue coming in, and you’re trying to keep your head above water. So one of the things we’ve worked on and iterated on it quite a bit over every quarter is just to do a quarterly planning process, where folks go out as as teams are putting together their, you know, quarterly okrs. Researchers are also evaluating what the research asks will be over the next quarter. The researchers then take that all together, you know, from across the teams, you know, think about they do a little bit of a T shirt sizing, you know, what’s a small, medium and large project, think about their bandwidth, sort of think about timing And come up with sort of a proposal. You know, it’s very aspirational. But it’s still a really good, you know, exercise to go through. Now, what I think makes it has made it really effective is we then do a readout of these plans to leadership. So I’m really lucky to have a very actively involved CEO and CTO and head of product. And so it’s those folks and a few others in a room, and each researcher goes through what they’re intending to do over that quarter. Now, it’s really great for research just in terms of getting visibility, and, you know, getting some recognition because we also do a section where we sort of share what we thought were some of the winds, and do a little bit of a retro on the last quarter. So get some of the researchers a chance to highlight some of the great work that they’re doing to leadership. And then also just to get that feedback where they might talk about what they’re prioritizing, and then to get that color or get that really strategic perspective, from the CEO talking about, you know why he’s excited to hear us diving into this area or that And then the last part that I think is kind of powerful as well is then if there’s kind of push back a few weeks later, as a researcher is explaining, like, Oh, I’d really love to get to that, but I’m doing this, you know, they can kind of say, Well, you know, this is what our, you know, leadership team, you know, thought was really important. And it gives a little more gravitas to that sort of prioritization process. And, of course, we build in, you know, feedback cycles, you know, throughout as we’re developing this as well. So, it really hope hopefully gets folks on the same page. Um, you know, there can be a little overhead to this. So we are trying to make this as spelt and, and sort of low overhead. So that’s part of what we’re iterating on. And this might not scale if we were a team of 15 because there just wouldn’t be time to, you know, have the CEO sit there through 15 people’s presentations. So, at least for now, it’s been really valuable and, you know, we’re gonna have to think about as the team grows, how to modify it.
Steve: A lot there. There’s a lot. So the researchers themselves are, that are they’re the ones putting together their individual roadmaps.
Laith: Yeah. You know, going back to that question about, you know, building teams, you know, I want folks who who are, you know, I want to find awesome folks and empower them to do the work. I’m not, you know, overseeing or micromanaging or sort of telling them to do this or that, you know, they tend to bring me stuff and we chat about it. And I’m a sounding board, and we strategize and I give my perspective. But, you know, they’re the ones that are mediating the relationship with their, with their stakeholders. And so, you know, they really have to believe in whatever choices are being made about what they’re working on. And I also want them to feel that they have agency, you know, that they’re not just sort of, uh, you know, designers sometimes talk about sort of being like pixel pixel pushers. I don’t know what the what the right alliteration would be for researchers, you know, we could work on that, but I don’t want researchers to feel like they’re just sort of being dictated to about do this usability test or do this You know, the generative, you know, exploration or getting a survey, again, making them part of the process, and and they’re contributing to the OKR process, as well. So it’s really should be a symbiotic relationship. We’re not quite there, but I think we’re really moving in the right direction. And I’m excited about that, that that back and forth and the interplay
Steve: I think it’s a compelling vision that you’re, you’re illustrating, and I want to go slightly meta here. Sure. Maybe to meta I don’t know, you know, you. I mean, you said something that I think is quite common, right? The, the, the research resources are limited, or they’re, you know, to put another way, demand exceeds supply. And I think that’s common and I wonder, you know, outside of Udemy, like, if that’s sort of a truth in at least for tech, you know, in the kinds of organizations That means there’s lots of kinds of research, but in sort of tech overall, I think that’s a common, you know, you have different team sizes, you have different organization, you know, approaches to research, but in general, I hear this a lot. What do we what do we think of that like that? That’s a that’s sort of a universal truth that there’s a more there’s more needs than there are resources. You know, why? So what’s what is? Do you have any reflection on that?
Laith: I mean, there’s definitely I think we should be self critical enough to say we’re biased as researchers. So, you know, I see research need where perhaps other people don’t and I think that’s, that’s totally fair. And and I think it’s also the the tricky part is, are you optimizing for sort of peak demand load or average demand load, and that can be a hard thing to disentangle. And then Lastly, I think it’s what, where and how can research impact things? So, you know, I think, you know, are we already sort of cleaning up things and grinding out a lot of usability studies? Are we doing really conceptual, you know, blue sky kind of research? What kind of impact do those do different things have? are we are we, you know, what sets of stakeholders are we interfacing with? Um, you know, I think it’s a very different situation, if, you know, you’re getting into some really interesting things that overlap with back end engineering, and you’re doing interesting things with PMM. And, you know, local operations, you know, what I’m saying instead of just sort of the more core conventional stakeholders that you might have, I think that that exploring those kinds of things, or maybe things that I’m seeing that I wish I could do if I had more bandwidth, but the organism might not necessarily see that yet or see the value. And so it’s a matter of sort of picking the opportunities to try and demonstrate how big their menu of research is, and all the places that we can impact. So it is a tricky one. And you know, it’s probably better to have a few too many researchers, sorry, a few too few researchers then to go the other way.
Steve: And, you know, I think about I certainly when I started the field, there just there wasn’t a lot of demand. I mean, there wasn’t a lot, there wasn’t as much awareness there wasn’t as much, you know, process that includes these these tools and kind of incorporates what we bring. And it’s been interesting to see, you know, over time, how the demand is high. The, you know, the the The amount that that our work is integrated with, you know, other work that’s happening in the company is as grown and I think you’re just you’re articulating, you know, a future state perhaps where there’s even more, you know, more aspects of the work that a company is doing, are places where we can bring value. That’s like a, I don’t know what the shape of the curve is over time, but you can sort of see it evolving from, you know, who are you guys to, we have a lot of work for you to, you know, in the future, there’s way more than we can do then you then people are maybe ready for or that that, that we have demonstrated, you know, as a practice, you know, as, as broadly.
Laith: Absolutely. I mean, it as I’ve grown in this field, you know, I’ve really found that, you know, Human Centered Design has kind of permeated my life. You know, let’s say I’m trying to I set up a time to hang out with some friends. And you know, it’s sometimes hard to coordinate everything with like three or four friends. You know, when I’m typing up that email, you know, I think about, oh, a little, little visual hierarchy in, you know, as I typed this out to make people understand the options and what I’m asking for and to reduce the language and make it easier to scan, you know, that goes into me putting together that email. So I think we kind of have to embrace that, that philosophy of Human Centered Design is really powerful. And it’s been fun to your point of seeing some of the bigger companies where the internal, you know, human resources team, or the you know, real estate team might have a user researcher because they’re thinking about that that’s a product you know, how people access the benefits internally or what workspaces are like at a company or all these different kinds of applications. And that whether we call it user research or service design or you know, any of these kind of but that that that fundamental approach Can can benefit organizations in so much of what they’re doing. It is fun to think about that, that that future state and all the places that we can, you know, deliver value.
Steve: You know, and use that up top, you know, you came from academia, you had sort of just a variety of things that you’ve done along the way, in research and kind of before research, and I’m wondering if you kind of look at your history, but also, as you you know, have you as you grew into the field, you I think learned about the, the history of the field of user research. Do you see? I mean, are there seeds of what we’re talking about? If you look backwards in, you know, where research had been applied, how it kind of evolved or, you know, what you see when you look backwards?
Laith: Yeah, this this is a topic that I’m just continually fascinated by, and I would love your perspective on this, because I’m sure you know, a lot of this very well. And it’s been interesting. You know, I think sometimes When I’m doing my teaching, I almost get the sense my students kind of feel like this is how it is. And this is how it’s always been. And you know, sometimes I kind of have to remind them a little and be like, this is actually a very rapidly evolving and changing field. And you all are going to have a huge say, in sort of what this, you know, ultimately looks like or sort of what it what it might look like in a more mature state. And it really has undergone these different shifts in thinking about all the different strands that go into it, at a very, very simplified level. And not to say that these phases are not overlapping or exclusive, but it feels like, you know, previously there had sort of been the innovation labs, the Xerox parks, and that that had been sort of where a lot of this kind of work was being done. Obviously, there’s a older history when you’re thinking about World War One and two and human factors, and that’s another sort of strand in this. And then it feels like some of this work then became a little more focused around agencies. You know, you’ve got your radios and things like that. And we started to see that shift to in house research happening and calling it UX and things like that. And I think around then is also when we saw this shift to much more consumer oriented products. Before a lot of it, you know, had been very enterprise e or, you know, business oriented, where maybe someone, all their exposure to human computer interaction would be the one computer at work or something like that. And so we over this time, we’ve evolved to this place where technology is being much more omnipresent. And you’re not just dealing with a technology one manufacturer, but you’re starting to have to think about that the design changes on your phone are affecting the design changes on your computer, and those might be made by different software companies and hardware companies and all these kinds of things. Now, I think originally, a lot of those in house teams were embedded, or sorry, we’re centralized, like I sort of talked about a little before and then we’ve seen Seeing the shift to the more embedded teams and the real just tremendous growth that’s happened over I don’t know, the last five to eight years, and weaving together all these different strands from psychology and design and Information Science and anthropology and you know, more than I could sort of a name. And we’ve also seen the pipeline really emerged at these really great masters programs, you know, a lot of programs in library science, or have rebranded themselves as an Information Science program and are contributing to this. And of course, you know, human computer interaction programs and things like that. So it’s interesting to think about sort of what comes next. I started to see some more innovation labs kind of things popping up. So it’s funny how that might be coming back into vogue. And then the sort of rage of democratizing research and, and sort of figuring out how to scale the impact of research teams beyond just embedding more and more researchers interesting. As well as you know, sort of the the global nature of this, you know, feels like there’s a bit of a different UX culture in some parts of Europe, and you’re seeing emerging UX culture in in other parts of the world too. So that’s kind of my take that I’ve gathered through just a few articles and chatting with folks. But I’d actually be curious if that aligns a sort of a very rough arc with sort of your take of the field.
Steve: So this is the part of the interview. This is the danger of interviewing an interviewer. Yeah, I mean, I would love to sort of, maybe someone has done this, but I’d love there’s a visualization, it’s kind of it’s kind of a timeline you know, around with some of these major things happening and you know, as you talk about it, you know, the the emergence of design and consumer products as has elevated to design, right? The, the way the jobs for designers changed once there was an iPhone, I mean, going back the jobs are designers change once there was like a Macintosh computer, but and that, you know, we’re, I have felt for a lot of my career that and not necessarily a bad way but that that research is kind of following in the wake of design. And so when I joined the field it was it was following in the wake of industrial design, because that’s where all the innovation was happening. And, you know, they kind of they kind of weren’t, they were the innovation labs. But then the web comes along, and there’s just so much growth and you know, things that you talked about. And so, you know, user experience comes along as a phrase and suddenly we’re user experience researchers and researchers because we’re, we’re working on that in those industries. We’re working on digital stuff more exclusively. You know, and I think we, I mean, in your narrative you are, you’re any good history has to do it includes a lot of different as you call them strands. And sometimes I feel like it makes, it’s necessary, but it’s hard to see the research strands like you don’t want to decontextualize it. But I also I do want to decontextualize like I want to how do we talk about research without talking about anything else? Just as a you know, as a, as a as a thought exercise, like just to look at how research has changed, even though these other things have changed, and that’s why research has changed, but I’m certainly interested in the like, I think you’re going to your point about, you know, a lot of the work moving in house and as someone that’s still a consultant, I can tell you that’s that’s definitely true. It really has changed, my professional life changed dramatically. As that trend took hold, so I feel that one very specifically. And that’s a very research specific one, even though I think you see lots of creative professions that were historically done, those consultancies are done as agencies got moved in house. I feel like there’s that Apple caused some sea change in the ad industry when they Yeah, brought something in house. And I don’t know if that was sort of the the change point that led to what you’re describing with, you know, research overall.
Laith: I think that if I were to focus a little more on the research strand, because I think it’s a fair call out, I would say there’s two things and one is thinking about the position that the researcher has these of either product or what they’re working on. And it’s funny because I think Yeah, back in the day, and I’ll contradict myself a little about not about focusing on research, but you know, if you were a designer, you’d be like, Oh, I want to go work for fjord or tea handed lax or You know, these kind of big design agencies and they all got either acquired or Accu hired not all of them, but a lot of the really prestigious ones. You know, and and you saw that happening a little with research too, with with bolt Peters, you know, and things like that getting brought in house by the big companies Adaptive Path and things like that. So it has been interesting to see that, that that that transition, but I think we’ve inherited certain ways of approaching things, or at least I have, I know, coming from that consulting side, it was kind of a formal relationship that I had with the product and with the product teams. And you know, it’s sort of like having the big, bloated deck, you know, that Edward tufte A would, you know, you know, shake his finger at me for putting together and I’ve had to think about ways of thinking about like, Hey, you know, I don’t really need to go through a lot of that front matter that I might have had in this very formal deck for clients, with people that I have a much more casual relationship with. Similarly at Udemy when we first transferred from that embedded sort of Central To embedded forum, they’ve been sending out surveys, sort of satisfaction surveys to see like, how did the research project goes. And ultimately, you know, there are people that you’re in meetings with all day and you’re sitting next to them and all these kind of things. It feels kind of odd and distancing. To send that satisfaction survey. Now, we still need to have mechanisms for feedback and to figure out ways to do better, but we kind of needed to adapt the ways in which we work to reflect these different, you know, relationships. And I guess the other way I’d flip it around is to sort of to take a research, design thinking approach, which is, you know, if we were to redesign, design research today, and could somehow throw away the baggage or sort of the history of all these things, how would we design it differently? How would we redesign the way we work and our relationship with products, you know, allowing that there will probably be a spectrum of those things depending on the company and those kind of things. You know, it’d be, you know, as a further thought experiment, it would be kind of fun to think about, what are the things that we’ve carried forward? Because of these histories coming out of industrial design and out of human factors and out of these sort of evolutions? And what are the things that we might do differently if that if we’d been on a different path, or we’re just starting a new today?
Steve: And one of the, I think, one of the legacies and you know, whether it’s a strength or a weakness or both, is the the, the diversity of paths that have led people to research? You know, it’s still true today. And it was, yeah, like, I think, you know, if you go back in the olden times, there were few sort of credible paths, they had to sort of invent a path and now there are, I think, many paths that are somewhat credible, but it’s So lots of people have to find their way in from something else. Right. And I think there’s there’s, we’re the field welcomes more people from more places, it’s still a struggle to get in. And then I guess, just there, it was a different struggle in the past, but the that lack of definition of like, what background does a researcher have? Or what backgrounds plural does a research team need to have? I think that’s very exciting. But it also, it also seems like a huge amount of legacy code that we spent a lot of energy and emotion on that if we were looking at things to sort of toss away and reinvent from scratch. That would be that would be one that I would I would lobby for.
Laith: Yeah, it’s tough. And, you know, I think a consequence of that is the research culture at different organizations can be so different, you know, if you have one place A lot of the folks are master’s degrees in human computer interaction and not saying that that’s a bad thing, but it’s just a set of things. And you have another organization that, you know, more folks coming out of sort of design anthropology or applied anthropology, you know, they’re gonna have very different ways of framing problems and approaching things. And you might have another organization where it’s a lot of designers and engineers, who you know, learned and got into research, you know, through work experience, those would all have very different flavors. And yeah, to your point, it’s kind of like a great thing and it’s an exciting thing and it’s an enriching thing. But it can lead to folks talking past each other, or, you know, getting in over their heads in certain situations. You know, I do sometimes my main concern can be about the ethical conduct conducting of research, and that, you know, I really hope that everyone who gets to do this awesome line of work has the the proper you know, ethical, you know, You know, training to sort of keep their eye out for getting into situations that are not good for them or not good for the industry and are not good for the research subjects.
Steve: And where does that? I mean, I think this this speaks to the multi discipline. I guess that’s what you’re saying. So I’m just reiterating your point for you. Sorry. But like, yeah, where does that come from? How do research teams work, regardless of what their you know, what their composition is? How does? How do we mature the practice of the ethical aspect of the work?
Laith: It’s tough. I mean, I think that there there needs to be a degree of self policing, you know, whether it’s on slack communities or email listservs You know, I think it’s it’s okay to discuss things that have popped up and say, Hey, you know, I’m seeing the survey coming out from this or invitation to participate in research like this. And you know, I don’t know how I feel about About that, definitely, I’d say larger teams, which I think is where a lot of more junior folks are able to get their start. I would love to see them have a sort of a formal cadence of doing some some ethics training. I think it’s super important. And But yeah, I think there there is a bit of a vacuum there. You know, there’s some good articles and it does get covered in you know, the, the seminal, you know, works like the Baxter courage cane understanding user needs. You know, there’s great discussions in there about, you know, ethics and things like that. But um, I definitely think as an industry, we could be doing more.
Steve: And this is goes back to the strands thing, like the conversation about ethics in user experience, progresses a certain way and it’s easy for us as researchers to get sort of swept up in that but the issues are different. I think there are specific things as you’re kind of getting at.
Laith: Yeah, I think that the the things you’d watch out for if you’re doing academic psych experiments in grad school would be very different from if you’re doing anthropological field research, you know, let’s say with a marginalized community, but yet in industry, because of the wonderful interdisciplinary Enos, you know, you might have those folks sort of doing the other kind of research and not being fully abreast of the, you know, you know, if they have very advanced ethical training and research ethics training in their home discipline, you know, as they move and sort of get to experience the other strands, they might not be totally, you know, up to speed on on those other kinds of areas. So, even as folks that think they have really great you know, you know, Human Centered research ethics training, as we move into this UX research world, there needs to be something sort of specific for us. I will say that that you know, the UX pa has guidelines market research organizations have ethical guidelines. And I think those are a really fantastic starting point for the work that we do, because they’re, they’re focused on sort of this more applied or industry setting. But, um, you know, there’s a lot more that I think we can do collectively.
Steve: What do you see in your students in terms of their awareness or, you know, hunger for this kind of guidance?
Laith: Um, you know, I, I don’t know how representative Berkeley is, you know, I think it tends to approach sort of attract a little bit more of the mission driven or idealistic kind of folks. So, I know that whenever I’ve mentioned, you know, doing work with different kinds of communities or crossing socio economic status, you know, sort of working with folks in a broader spectrum, you know, not necessarily just, you know, Bay Area folks that have an iPhone, you know, 11 You know, I think that they really get excited to think about Those kind of things, and they really embrace the responsibility and recognize the privilege that they have. I have to say, I’m just so impressed and blown away by how sophisticated the students are, you know, I feel that they that they’re really attuned to some of these kind of things, and you just have to open the door a tiny bit, and they tend to really, you know, jump at the chance to sort of think through some of these really complex issues. And think about the ramifications of, you know, sound design of people that, you know, have different accents, or AR VR for accessibility and things like that, you know, they’re they’re connecting the dots really quickly. I just think it’s really important to give them sort of the frameworks through which to process these kind of things. So that’s kind of what I see to be my challenge.
Steve: Your challenge is to give them those frameworks.
Laith: Yeah, exactly. So they think it kind of helps. To see like the bigger picture a little easier.
Steve: Yeah. Well, this is super interesting. And we could obviously dig into all of this two times more than we already did. I guess, just to throw it to you like, what did I ask you about? Or do you want to talk about that? We didn’t get to?
Laith: I don’t know. Uh, you know, I’m a nerd. I can talk about anything for forever if you can’t tell from the conversation. But, you know, I appreciate you give me the chance to talk about some of the historical stuff. And just quite frankly, like outside of the interview would love it if you had any more perspective on that to share just because I think it’s interesting to think about that legacy. But I don’t really have any other you know, access to grind or anything. If you feel you you’ve got enough.
Steve: So I’ll throw one more question at you and then I’ll try for a perspective on you know, on the strands conversation I guess so so. So if this was, if our conversation on dollars to doughnuts was going to end up in podcast club, what would you want people to get out of it?
Laith: I think I’m really into the idea of sort of questioning
what we do. And, and, and again, that can be the methods and that conversation about getting out of our comfort zone. It can be thinking about our relationships with stakeholders and trying to reimagine and iterate on those. And it can be thinking about the historical trajectory of the field and the legacies that that has imbued us with. And I think that just sort of being ready to iterate and question the assumptions that a lot of those things are built on, is the thing that I’d really want folks to come away with.
Steve: Excellent. Can I said I would, I would try to offer something else. I’m the sort of looking forward looking back on the field piece. You know, and so, my, my perspective is both informed and I guess hampered by just time. Like I have been doing this for, you know, more than two decades, which is weird what to say when it comes out of my mouth, but it’s, it’s true. And I have just been feeling for the last little while that, you know, we’ve been on this trajectory, that sort of this gentle upward slide in terms of, I don’t know, skill building, like figuring stuff out building best practices, like a lot of things that we were, you know, making up as we went along, have been formalized, have been shared have been documented. There are, you know, for a talk I was doing, I just tried to gather up as many books about research that are out there and there is a lot like There’s so many of them. And that’s when it’s a good thing we’re, we’re talking about this expertise is being documented and shared. And I’ve just been feeling in the past year or so. And, you know, again, this is a stone, it’s only my perspective, it’s not a declaration of what’s real or true, but I’ve been feeling like that we’re like any sort of, I mean, maybe you have a better take on this as someone who helps people build skills or do learning, right, you, you advance at a certain rate, and then you kind of reach a plateau, and then you and you sort of you cruise at that plateau for a while, and then you hit the next, the next rung or the next, you know, movement upwards. And I feel like that in you know, in, in the profession that I’m part of, as I see it, I feel like that’s what’s about to happen or is happening. It’s happening in little pockets, and we’re just starting to to be bring it into the mainstream. And I mean, I am I mean as as as, as the day we have this conversation or a number of days away from a conference I’ve been involved in called advancing research. That’s part of the Rosenfeld media, advancing research community, which anyone is welcome to join. And I’ll put a link in. And I think the name is really great. And you know, that we we talked about, how do we position this conference and what does advancing research mean? And, you know, what are all there’s lots of great stories and great work that gets shared. But we’ve also heard it a lot over and over and over again. And you know, yeah, if you’re new, maybe you haven’t heard some of this, but there’s so much stuff that we haven’t heard that we need to be talking about, and you brought up ethics, you know, earlier in this conversation as one that like yeah, it’s here. It’s there, but it’s we haven’t really made it part of our it’s not part of the the baseline, it’s not part of the kind of the core fabric of what we all practice, it’s like a thing we need to do. And I’m, I feel like, you know, as we move from the plateau to like the next incline, that we’re making a collective concerted effort to bring these under recognized pieces in. And what’s exciting for me is you don’t have to look very far to see, oh, that thing that we’re not really doing well, somebody is working on that somebody has a point of view, somebody has built something or built a process or built a framework and that it brings me at least an opportunity to to, you know, to iterate or to to move to that next, you know, that sort of 2.0 3.0, whatever. I don’t know what the numbers are, but I feel like we’re on the cusp of that. And again, that is such a, that is my perspective. I can’t say that anyone else would feel that way. But it’s something that I am drawing a lot of excitement from, because I feel like oh yeah, I know how to do a lot of this stuff. And there’s things that I didn’t even know that I didn’t know how to do, that I should be doing. I don’t even know where considerations I’m not talking about, like, you know, our ever shifting library of tools and infrastructure, but, you know, really the, the, like, read like an iteration of best practices to be more inclusive to be more empathetic of our colleagues. And I feel so I feel very excited about the near term because I feel like that stuff is is coming out. Yeah, I don’t know if that if that how that matches with your own trajectory, which is obviously different than mine.
Laith: It’s so interesting. I’ve been having some really great conversations with do you know Steve Fadden.
Steve: Yeah, yeah,
Laith: yeah. So um, in you know, I think this is something that like organizations have struggled with where Like, what’s after a senior researcher level is a little bit like there be dragons, you know, it’s like kind of easy to articulate like an associate to a researcher to a senior researcher. But you know, if you don’t go the management route, you know, what makes you you know, what a principal or a staff researcher or whatever that organization calls it, you know, the language tends to get a little mushy. And what would distinguish that senior researcher from that next level, I think it’s something that we as an industry are like, to your point, we’re still we’re getting closer, we’re kind of grappling with it. And it’s a little tough to, to talk about, you know, what are those those things that really go into having that impact? That would sort of denote a level beyond senior. And you know, this is a little beyond the, I think some of the title inflation that might happen in the in the in tech as well. But there’s some real interesting challenges there. And I think there’s also a set of unmet need here where I’ve met a lot of folks that have kind of hit that plateau and it’s can be a little hard, especially if they don’t have a really involved manager to know about to figure out on their own, you know, what are those things are? How do I get the training? Or how do I get the experiences, to get out of my comfort zone or try the new things or be aware of the things that I don’t know that I don’t know, to figure out how to get to that next level, it can be a little bit of a glass ceiling. And to your point about inclusiveness, I think that that’s also an area where we’re bias can really creep in, in terms of, you know, allowing people to express what they bring to the table and make a case for themselves, that they’re having that impact or however they would be measured to sort of go on to that next level. So it I like that you brought that up and it’s a it’s share the perspective that’ll be exciting to sort of see how we collectively grapple with some of that.
Steve: I know that just today and again, our today is not the today of the person that’s hearing this but anyway The research ops community just published. I think it’s called like a research skills framework. So I saw that it was posted I haven’t seen what it Oh cool, what it consists of but I think at least it whatever it contains, it validates through another source the the issue that you’re identifying, and that this is a this is a thing that’s being considered and, you know, worked on by the research community at large.
Laith: Yeah, that’s awesome. And I love that, you know, a lot of times companies can’t share these kind of frameworks externally, which is which is a shame. But I love that that they’re able to come up with something that they can share publicly and it’s one thing I do love about this community like the the listserv is so helpful. People pay it forward, people pay it forward to me like yourself when I was first coming into this, this field, and I’ve been really lucky to help a few folks here and there as they have tried to enter the field and things Like that, and I just love that there’s such a propensity towards knowledge sharing, and I think it’s just going to benefit us all in the future. So that’s one thing I am very proud of, in terms of our community.
Steve: I think that’s an extremely positive up note too. So yeah, thanks late for taking the time and sharing so much interesting stuff and really for a great conversation. I really appreciate it.
Laith: Thank you for the invite. Really fun conversation and I appreciate the opportunity.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 30. Laith Ulaby of Udemy first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Kathryn Campbell, the Director of Research & Insights at Ticketmaster.
Whenever there is availability of somebody that might normally work on the marketplace side, they might tag team on an account manager project and that helps to inform them about that product. It gives them a little bit more purview. It facilitates internal sharing of learnings because we are a very large, complex organization. So that flexibility is both more satisfying to the researchers, but also benefits the product teams in the long run. – Kathryn Campbell
Show Links* MoveOn * Kathryn on LinkedIn * Kathryn on Twitter * Ticketmaster * Live Nation * Amy Howe, President and COO at Ticketmaster * Kathryn Frederick, CMO at Ticketmaster * Customer Insights Center of Excellence * Ogilvy (formerly Ogilvy & Mather) * Jakob Nielsen: Why You Only Need to Test with 5 Users * Dustin Guinee, Lead User Experience Researcher at Ticketmaster * Hilary Bienstock, Cal State University, Fullerton * Brent Jefferson Lowe, Senior Manager, UX Research at Ticketmaster
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Over the past few years I’ve been volunteering with MoveOn a progressive organization. My contribution is to be part of their texting team, where they identify an issue or a cause and a group of us volunteers send text message about that topic to their membership. Last week I was part of a campaign raise awareness of best practices for protection against coronavirus.
We don’t send text messages from our phones, it’s through a browser interface. A texter sees a pre-addressed, and pre-written message which they send, one at a time, usually thousands in total. When people respond, the volunteer can classify that response, which will produce a relevant response that we might customize to ensure it makes the most sense.
MoveOn collects responses and future campaigns (say, for someone to call their representative about an issue, or to get out of the vote) a built based on what they learn from looking at their data.
In this recent experience, we were asking people if they were practicing social distancing and sharing al link with resources and information. The responses that they were expecting were essentially, “yes I am”, “no I can’t (or don’t think I need to)”, “I need medical help” and “thank you.” The first day I participated, I heard from a number of people who were medical professionals. Now even though we are working on a dashboard with pulldowns menus, people are entering text on their phones, and have no idea what our interface is. They wouldn’t necessarily share their responses according to our predetermined categories. MoveOn runs a busy Slack channel where during any campaign you can get help in categorizing a response, or dealing with a situation that may arise.
So the next day when I was texting, I noticed two changes. The “thank you” response had been changed to “you’re welcome” which was a big improvement. Even though the “yes” and “no” responses were labels for what the person told us, if they said “thank you” as their response, your brain isn’t looking for a “thank you” label in order to send the text “you’re welcome” – you start looking for “you’re welcome” because that’s what you’re trying to say. The other change was a new response category for “I’m a medical professional.”
Whether they were looking at the responses in a table, or whether they had one of the volunteers raise this as an issue, I don’t know. I was interested in their ability to pay attention to the data and update their tools. It reminded me of survey research, not something I would say I’m particular expert in or anything. And I know in surveys it’s a best practice to do a pilot study, and maybe that’s where you catch something like this. It just reaffirmed for me that creating categories for people’s behavior and opinions is always a hypothesis, and deciding ahead of time how to categorize responses, before you’ve got a big pile of responses is always going to be flawed. MoveOn is able to make changes on the fly, even during a campaign, and I’m sure it messes up their data collection somewhat, but their priority is on the experience of the campaign, not their data. Survey researchers probably don’t have that opportunity. But it’s powerful to consider, well, what if they did?
Well, here’s another episode! As before, this was assembled quickly, with the budget I have available for production, which is zero. There’s no professional editor, there’s no professional transcript. The audio quality is the best we could get, when people are working at home, without access to all of their gear, using an Internet under strain, and so on.
And just a reminder that you can support this podcast by supporting my business. This is an uncertain time, and many small businesses are feeling the pinch and don’t know what’s coming. I will appreciate you keeping me in mind now and in the future as a collaborator in conducting user research, and as a resource for helping grow and develop your company’s capabilities. Please reach out!
All right, let’s get to my interview with Kathryn Campbell, the Director of Research & Insights at Ticketmaster.
Steve: Kathryn, thanks so much for being on dollars to donuts. I’m looking forward to the chance to speak with you.
Kathryn Campbell: Hi, Steve. Yeah, thank you. I’m looking forward to the conversation as well. I’m a big fan of your podcast.
Steve: And now you’re on the podcast. So
Kathryn: living the dream.
Steve: Yeah, living the dream. Well, we’ll see. We’re starting to live the dream. I would love for you to introduce yourself to begin.
Kathryn: Okay, great. Kathryn Campbell. I am the director of research and insights for Ticketmaster, which is a subsidiary of Live Nation, the world’s largest live entertainment company.
Steve: So in a minute, maybe we can talk about the research that you’re doing. But can you set the context a little bit about Ticketmaster itself?
Kathryn: Absolutely. So most of your listeners are probably familiar with Ticketmaster and if Had some interactions with us over the years. And we’re a 40 year old company, and a long time innovator and leader in the live event ticketing space. So most of us think about Ticketmaster, from the perspective of the fans, and the fans are a critical user group for us very near and dear to our hearts. And the people at Ticketmaster. Mostly come there because we are fans ourselves. But we also have a number of other customer groups that we serve. That includes the artists and their managers and promoters. It includes the venues and the surfaces that work with the venues like parking and refreshments. And we also have direct relationships with most of the major sports leagues in the US and another country. So football, basketball, hockey, all those kinds of things. So as you dig into it, you realize there’s actually a very Complex ecosystem of customers to serve.
Steve: Oh, so you do research with fans, artists, venues, leagues? I think those were the kind of groups you mentioned. What? Yeah, what does it look like? I mean, you know, within what you can describe, I’m wonder if there’s any aspects of some of those types of research that maybe are things we wouldn’t have heard about.
Kathryn: Sure. One of the things that I really enjoy about my job is working. We have a subsidiary called tm music that works directly with the artists and their teams. And it’s trying to get more money into their pockets rather than into brokers or scalpers pockets. So working with them on innovative solutions that allow them to ensure that people that say members Their fan clubs can pre order tickets, or receive priority in the waiting room or queue versus other people that we have not identified and might have reason to believe could be resellers. So that’s one area where we get to work. We also get to work with venues. And of course, we tend to think of the big arenas. But there’s also the relatively small theaters that do a lot of touring productions and helping them be more efficient and have a better relationship with their subscribers as well as one time buyers. And then with the lakes, it’s helping them get to know their customers better. One of the things that takes you a while to realize about ticketing is that in the old world, you only actually knew who about a fifth to a fourth of the people in your stadium or arena were most of them you know, one person might buy four tickets. So football teams might be selling out, but they don’t really know who a lot of the people are that are in that stadium, how far they’re coming from, what the composition of their group is. So there’s lots of interesting dimensions to the research that we do with those different segments.
Steve: Great. So a ticket buyer is not the same as a event goer.
Kathryn: Exactly. Yeah, it’s a different
Steve: two different stages there. Oh, well. What? You know, how do you given the different types of I mean, customers, users, etc, that you’re trying to learn about? Does that impact, you know, how you organize your own group and in order to work with different different parts of your organization and then different customers?
Kathryn: Absolutely. And, you know, this is an ongoing debate, I think, in the research world is whether you have a centralized group, you know, Internal agency, which is what our model is, or if you embed researchers in the lines of business so that they can work more closely with the stakeholders. What I feel is best for both the company and for the team members is kind of a hybrid model. I really like the mutual support and training benefits and the opportunity to stretch and grow of having everyone on a centralized team. But some of these areas are very complex. And so our internal stakeholders would like to have some continuity over time and work with the same person and not have to re educate them about the complexities of their product each time. So what I’m always trying to do is to maintain continuity to the greatest degree possible on the products and stakeholders, but also allow the individual researcher to feel like they are growing and stretching and developing new skills. Being exposed to new kinds of projects. So that is a constant balancing act.
Steve: So this may be a really annoying semantic question, or not even a question, but okay.
I liked the phrase that you use when you talked about embedded, you talked about embedding in lines of business. And sometimes when people talk, right, embedded, they talk about embedding on product teams. And those may be the same thing. But in terms of what they what picture they paint in my head. It I don’t know, is a product team and a line of business when we think for the purposes of embedding, is that a distinction without a difference, right?
Kathryn: So Well, not necessarily. I mean, you’re right within a line of business, there’s going to be multiple product teams and some of our products cross multiple lines of business. So for example, the person product that we use that support season ticket holders is called account manager. And it is basically a white label for the leaks or the theater theaters to manage their subscription customers, right. So that’s within one line of business, which is actually our enterprise, our b2b line of business. However, those people, the individual seasons, subscribers, season ticket holders are still fans, and they also want to buy maybe individual tickets, right? And they want to be able to manage their tickets in a centralized way. And they can easily get confused if there’s a different experience and account manager than there is in their account on ticketmaster.com. So, you’re right that it gets muddy. It’s not the same thing. But in terms of the underlying question about How do you balance that? One thing that we try to do is let’s say, we have one person who is an expert, which we do we actually have two people that are experts on account manager. However, whenever there is availability of somebody that might normally work on our what we call the marketplace side, the b2c side, they might tag team on an account manager project and that helps to inform them about that product. make them more aware of the complexities for a fam that has membership, you know, an accountant both and they can also inform that account manager team about some of the things that we’re doing on the marketplace side, right. So it gives them a little bit more purview. It facilitates internal sharing of learnings because we are as you’re beginning to get a sense we are evolving. Very large, complex organization. And we’re very fast moving. And so that flexibility i think is both more satisfying to the researchers, but also benefits the product teams in the long run.
Steve: It’s an interesting, it’s a, it’s a, I think, a brilliant idea to do sort of knowledge transfer or knowledge distribution, there’s probably a smarter way to say that by moving people, the people contain information, and you move the people around and the information, the knowledge, the expertise gets shared that way. As opposed to, you know, I feel like sometimes we will. There’s a there’s an idea, we should extract the information that’s in people and then centralize that, but you’re kind of, you’re distributing the people. Because the collaboration, you know, lets that expertise come to the surface and have impact in ways that you wouldn’t necessarily be able to predict.
Kathryn: Yeah, I think that’s absolutely true. I remember listening to someone talking about how that was a strength that Google the way that people tended to move between different groups and the informal knowledge sharing and transfer there. I think with large, complex organizations that whether we like it or not that informal knowledge sharing is at least as important, if not more important than the more formal documentation, which is always going to be less rich and behind the times, right. And I also think, you know, at the risk of digressing that this whole issue of really caring for your team and their career is one that I think is too often overlooked. You know, even on the agency side, it’s always easier to satisfy your client, whether it’s an internal or an external client by having them keep seeing the same face. But no, who wants to be that person that only knows about One thing and keeps doing the same thing year after year. And you know whether or not you want to build out your portfolio because you want to eventually leave or maybe you just want to stay interested in engaged and challenged. I think it’s really incumbent on us as leaders to facilitate that growth. And that is a lot easier to do. If you deliberately encourage this kind of movement across different project types and skill sets.
Steve: What else do you do to take care of your team?
Kathryn: Oh, gosh, we do a lot of things. You know, in our weekly meeting, one technique that I developed a few years ago when I was working with a boutique agency primitive spark was to start the weekly team meeting by really sharing on a more personal level. What’s present for you, you know, are things going great and you want to share something? Do you have a sick kid at home And you’re worried about them. We really try to bring in a bit of that humanity to our professional understanding of each other, because it helps you understand where people are coming from. We also really try to encourage thinking as a team in all respects, we actively try to load balance. So you know, with research, you have the best laid plans, right? have everything laid out and then a product gets delayed, or it gets accelerated. And so you might find yourself either with three major projects that you need to do interviews on this week or nothing for three weeks. And so we’re constantly encouraging the team members to chip in and help each other and again that facilitates that ability to knowledge, share and maintain both personal growth and professional efficiency. Time,
Steve: whenever there. I mean, it seems like in some organizations there’s, you know, a lot of measurement of where you spend your time or where you don’t spend your time that may have the it may work against what you’re talking about, right? What How are people incentivized or just incentivizes spend their time? So, I’m wondering, I don’t mean I don’t know sort of how it works at Ticketmaster, you can obviously, say this is the goal. But is there are there measures in place to reward is the wrong word, but to support that kind of operationally?
Kathryn: Yeah, it’s a really good point because we get back what we incentivize and what we measure, right. So I do not keep a tight track of hours per week, or, you know, we don’t have billable hours because we’re an internal agency that that Not what the focus is on so much as a couple things. One is what kind of research we’re doing. I’ve heard you in prior podcasts, ask about focus on generative versus evaluative research. And so that is a focus for us. We want to continuously move upstream where we can have more value in terms of, you know, the quote to build the right it before you worry about building it, right. So we do have quarterly, okay, ours, objectives and key results around what percentage of your work is generative. To that end, we have embarked on a democratizing research program over the last year where we’re training the product designers. In how to do their own UX research, you prototype research, which has a number of benefits for everybody. One is it’s faster for the designer, they can do quicker, smaller studies, if they don’t have to wait for one of us to be available. They get closer to their end customer and get to understand them better. And it also means that we don’t get into the situation that happens with many UX research departments, where we’re just constantly every week or two churning out new studies, but never moving upstream to the bigger product concepts, right? So this way, we can kind of push off some of that more routine, prototype testing work. And we’re now focused on doing concepting focus groups or surveys to size market interest in something or facilitating a design workshop or or those kinds of
Steve: So I wanted to ask them about where projects come from. You’ve made space kind of made bandwidth for your team by empowering others to take on a certain set of projects. And now you listed a bunch of things that, you know, you’re free to go and do. Where do those projects originate?
Kathryn: Right? It’s a great question. And I think that feeling empowered to take some ownership of that process is probably the inflection point in terms of really adding value to an organization with research. So I am extremely fortunate because ticket masters president Amy Howe, and our cmo cat Fredrik out when I joined two years ago, both of them explicitly said to me, I do not want you to be an order taker. I do not want research to Be order takers. And so that said to me, You are empowered to go out there. And so we do a couple things. One is as projects come in, and we have a process by which things come into the queue, and then they’re reviewed and groomed weekly, and that sort of thing we talked about, okay, what here best aligns with a key corporate priority? And Ticketmaster is pretty good about sharing what the key business priorities are that people should be focused on. But then occasionally, we also say, Gosh, I wonder I keep hearing about this new thing, and it seems like a really important thing, and it seems like something we should be doing research on and nobody’s asked us to do a search. So we feel very comfortable pursuing those stakeholders and saying, gosh, you know, wouldn’t, wouldn’t it be a good idea to get some end user validation that this is the right set of requirements before we built this, or to help ensure you know that that we’ve captured all of the requirements here. We can do that we also have built in a model of periodically checking in with the most senior leadership at the company to tell them what we have been working on. And also what’s coming up, and to say, are the projects that we’re planning on doing for the next quarter aligned with where you think the real value for the organization is? And so that can also be a point of alignment. But yeah, increasingly, we find ourselves suggesting things that really should have research done on them. And sometimes, in fact, we just go ahead and do stuff. And I think that as long as you’re not just pursuing, kind of cool vanity projects, but you’re using as your North Star, what the origin This nation says are truly the key business priorities. I don’t see that that whatever turn out badly for you.
Steve: So in the situation where you are, you know, you’ve identified something that a group is working on. And you approach those stakeholders and say, Hey, we we could do this research that would help you at this point. I guess my, the question I’m really after is, you know, that this thing that a lot of researchers Express stress about the, you know, no one listens to me. They don’t take our recommendations, we can’t kind of impact the team, they go ahead and do what they’re going to do anyway. So how did the How does the How does the taking action on the research part play out when you are, you know, not being an order taker when you’re being proactive and kind of coming to them? What do you see happening as a result of the research?
Kathryn: I think that we as researchers need to be willing to be flexible. I don’t like to encourage stakeholders who constantly come in at the last minute when they could have allowed for more planning time. But sometimes things really do emerge at the last minute or become very urgent. And so we need to be willing to say, hey, what can we do to try to make research work within the time constraints and the other needs that this project has. And I think that if you are willing to be an empathetic partner in that respect that that earns you some trust in the relationship and they can see hey, you’re knocking yourself out to make this work. So I should go ahead and use it. And then the other thing is some stakeholders just get it more than others do and one of the best pieces of advice I got from analyst at Forrester Research when we were talking about how to build a culture that respects research, what she said, Hey, you know, work with the people that really value you create fabulous case studies, and then use them as advocates because that will have a lot more impact on the recalcitrant stakeholders then you constantly saying, Hey, why don’t you use research? research is good, right? So, you know, when I have to prioritize who to make room for I definitely prioritize people that historically have acted on our recommendations. Yeah.
Steve: That’s, that’s really good. I just want to highlight the first part of that, and make sure I understood that. You’re saying that if you adapt and you know, are flexible and are sort of demonstrably digging in to support someone, that that sets them up to be more receptive to the results of the work that you’re doing for them.
Kathryn: That’s that’s been my experience, you know, let’s face it 90% of the time. The reason someone does not embrace research is, well, sometimes they say to themselves, well, I won’t learn anything new. But the biggest fear is normally this is going to take too long. I’m already behind. I have to hit this deadline. That’s what I’m going to be measured by. It’s back to what we talked about earlier. people behave based on what they’re measured on. And product owners and certainly project managers tend to be measured on Did you deliver on time, right? So if I go to that stakeholder and I said, Look, I am really worried about the unknowns in this new product and rolling it out too quickly to a large audience without doing some more user testing with the beta product. We’re willing to take our whole team and work in the evening or over the weekend and pull this together because we feel so strongly that that would be a good thing to do. Well, that kind of puts the stakeholder in the position where if they say no, now they’re going to be held accountable if big things show up, right? They’re making a very conscious choice to say no. And the good news is, you know, I won’t say how long I’ve been in research, but it’s been quite quite some time. And in all of that time, there has literally never, ever even one time than a research project where I didn’t learn anything. There’s always a surprise and in our business, sometimes that little surprise can turn out to be something quite significant. So for us, you know, now in the last year a big product innovation for us has been around secure mobile ticketing, which means that your phone is your ticket, right? I don’t know if you’ve had experience now with going to events and everything’s on your phone. Well, if you happen to have, let’s say, an Android device with a heavy cover on it, it turns out that that impedes the scanner working correctly in certain circumstances. So there may only be if we do a quick beta test with 100 people, there might only be two or three people there. But when you scale that up to all of the people that are going to any event in the next year, all of a sudden you’re talking about having a big impact on our call center, for example. So you know, highlighting Oh god, I’m glad we combat. And and you know promulgating that story of how we went to bat and how we found those insights. You know, I think it builds a sense that it is in everyone’s best interest to work with us. Now, the flip side is, I also constantly am tracking down and hounding the Senior Product people to say, what’s coming up this quarter. What’s coming up this half of the year? Have your priorities shifted, let us know as soon as possible because I don’t like living in a state of chaos. I don’t like constantly having to shift all the priorities because of something being coming up at the last minute if in fact, we could have known about it in advance. So now when they see me in the hallway, they immediately go through their mental checklist of, oh, is there anything that I should tell Kathryn and her team about?
Steve: Right, because you want people saying Oh, when they So you’re coming down?
Kathryn: Well, in a good way,
Steve: in a good way, the good kind of Oh, here comes Kathryn you’re painting a picture for us. I mean, the you mentioned that, you know, when you first started talking with the leadership and protect the master about taking on this role, they had a they framed the position as something where you had agency where you were kind of going to do things as you said, not being an order taker. I mean, this, I would imagine, you have to guess here, but what what do you think created the conditions for them, that they understood the power of research and someone like you to, to work in order to work that way? How did they see that light?
Kathryn: You know, my hypothesis about that is both of those women are very smart, very data driven executives. So they We’re not particularly knowledgeable about research, we do have Ticketmaster has an extremely robust analytics group. So we invest a lot in site analytics and marketing analytics and have very in depth reporting and good turnaround times for learnings. So I think that they just wanted to extend that from the behavioral, you know, what are people doing data that we were very rich in into the Why are people doing that side that we were not rich in? So I think that having a data driven organization that only had half of the insights that they needed is probably what created that willingness to or that embrace of the research group.
Steve: And how does the work that your team does? How does that set With the the people or the work of of analytics?
Kathryn: Oh, that’s a great question because it’s kind of an unusual setup and yet I do think it’s the direction that the industry should move over time. So both micro, which is research and insights primary research, we sit next to an equal to the site and marketing analytics group. So that together we form a combined insight center of excellence that reports up through marketing. And Forrester has been promoting this idea of an insight Center of Excellence for some time of bringing together all of those different sources of information about your customers. The site data, marketing data, UX research, survey research, qualitative research into one place And when I hear the stories, the war stories from my colleagues at other companies, it really helps me appreciate how wonderful this construct is. Because just the other day, I was talking with a friend who’s in charge of user research for a global company, and he said, you know, any time our work starts to move in the direction of maybe doing a survey, all of a sudden the marketing research team wants to get involved, but they’re a completely separate group reporting to different people. They’re not as knowledgeable about the background on the study, you know, it becomes a big deal to engage them and yet, they don’t want us to do anything that looks like a survey if they’re not engaged. So you create these territorial issues and obstacles to free, flowing information and you’ve been unmuted. For a long time, you know how it is it’s kind of like unwinding the sweater where you start out in one place, and then you learn something interesting. And so you start to follow that down. And so things keep evolving, you know, the appropriate methodology and keeps evolving. So having everything centralized, the only thing that’s not centralized that I think is a key part of the insight into our customers is fan support and client support. And we work very closely with those groups. We exchange reporting, and we also check in with each other when an issue seems to be arising to see you know, what each other is hearing. But pulling all of that together, I think makes the company just much more insightful and more nimble to act on insights.
Steve: Maybe we can shift a little bit be interesting to hear about, you know, the makeup of your team, whatever, what kinds of backgrounds that people have coming in? Yeah.
Kathryn: So, you can imagine given the wide scope of research that we do, it’s, it’s a mosaic of people with different backgrounds and different skills. There’s a certain core skill set that everyone has to have in terms of You know, basic one on one interviews, usability testing, and straightforward survey research design. But beyond that, we definitely have areas of expertise, we have one person who really loves and embraces research Ops, and that is a godsend to all the rest of us, because it just makes everything much more organized and all the rest of us more efficient. We have a couple people, including my number two person who come from design backgrounds, and so they are especially good at thinking through how to make the information design as impactful as possible. So we’ve really been up leveling the quality of our reporting to make things more aesthetically pleasing and also to help focus the reader on the couple of key numbers or quotes or Whatever that they should really come away remembering. We have we just brought on someone who is expert in advanced analytics. So like I said, What I’m looking for is everybody has a basic, a couple of the same tools in their tool belt. And then across the team as a whole, were able to offer richness of skills in a wide variety of areas.
Steve: And when you talk to prospective hires are the kinds of things maybe beyond you know, the skills necessarily, but I don’t know. How do you think about trying to assess if somebody would be a good addition to your team?
Kathryn: Yeah.
I think that really the personality characteristics are more important in most cases than the hard skills although sometimes you need a particular Hard skill. It’s really important that people be smart and think quickly on their feet, you know,how how intensely you have to focus to shift when an interview shifts or to look at data and think about what it’s really telling you and whether or not there’s alternative explanations for that. So that that kind of skill set a mental nimbleness is very important. It’s very important to me, in particular, that they show initiative. I don’t want to be a micromanager especially, you know, when you’re working remotely and right now a lot of us are working remotely, you need to trust that people are going to pick up things and run with it. I think that the basic people skills likability and very strong verbal and written communication skills are pretty key. Once again, where we’re servicing Senior stakeholders across a wide variety of businesses and sometimes our insights might mean multimillion dollar shifts and investment. So a person has to be credible and take feedback seriously. You know, one of the things that we practice a lot is it helping our clients recognize the actions that are implied in an insight, but not overstating what the research itself told us, right? So it would indicate that this is a good new hypothesis for us to test and explore, maybe through research, maybe through a B testing, you know, but without saying, Oh, yeah, this is what fans want us to do if that was not exactly what the research says. So that kind of discernment is pretty important to maintain credibility as a researcher.
Steve: Among the researchers, today, what kind of relationship do you all have to? I guess live entertainment, right? The business that the ticket master is in, what’s the culture among your team relative to that?
Kathryn: It’s pretty fun. I have to tell you, it really is, um, you know, the hand employee handbook says, you know, in terms of dress code, it says, you know, come to work is your authentic self. And so, you know, people come to work and they’re in band t shirts, or they’re wearing a jersey for their favorite sports team. We have dogs, you know, dog friendly policy. So a lot of people bring dogs to work, when there’s a major sporting event, you know, the World Cup or something like that. We throw it up on the TVs in the office. So we are all fans and we really do empathize with and more To maximize the joy in the live event experience. And the flip side is we work. It’s a very, very, very high velocity environment. Ticketmaster is actually one of the largest e commerce companies in the world. And as we sometimes say, you know, every Tuesday is like, Black Friday, because, you know, when you have one big global music musician after another coming out with their tours, and you have 20 million people trying to get 400,000 seats all at the same time. You know, there’s a lot of stress on systems. So, yeah, it’s an intense environment, but it’s also a fun environment.
Steve: Do people, do people still line up to buy tickets that’s up maybe showing my age or a nice
Kathryn: They do but they line up virtually. they line up in a virtual waiting room. And yeah, and it’s it’s intense out there, you know, they’re stressed out and we have a lot of compassion and empathy for that, you know, somebody is so frustrated because they’re, you know, biggest star is announced a tour and they really want to see them. And the reality is that even if the systems work perfectly and very frequently, the systems don’t work perfectly. There’s just you know, demand outstrips supply. So, in certain circumstances, you know, when BTS goes on tour when Taylor Swift goes on tour, not everybody is going to get tickets, at least not immediately, you know, maybe eventually but it’s not easy. So again, working through fairly sophisticated tools to try Try to root out bots we actually get we turn away more than 1 billion with a B bots a month. You know, resellers and bad actors are constantly hitting our systems trying to scoop up all the tickets so that they can resell them at a much higher margin and we are spending all of our time trying to outwit them and trying to deliver real fans tickets to the events that they’d like to see within their pocketbook if at all possible.
Steve: So demand you know, for successful artists demand is always gonna outstrip supply, but you’re trying to make sure that as much supplies as possible is available for the for the fans.
Kathryn: That’s the goal.
Steve: I’d like to I’m going to shift gears.
Kathryn: you’re the host, you get to do that.
Steve: All right. I would love to hear a bit more about, you know, we’ve talked about, you know, your work most recently, but can you talk a little about your background, how you found research and how you found, you know, the kind of research that you’re doing now? Sure.
Kathryn: It’s an unusual background for UX research, but I think it is the perfect background for what I’m doing now and the direction that I think research is moving. I actually started out working for advertising agencies doing marketing research, doing segmentation studies, message testing, focus groups, you know, various brand measurements and became a brand strategist. I was is actually a partner for Ogilvy and Mather ad agency over brand strategy. And what I realized as the internet changed from being viewed as just a media channel to being really the whole business for some companies, I realized that the biggest problem with some of the brands I was working on was their poor user experience. And so I wanted to learn more about the elements of good user experience and what made a site intuitive as part of understanding how to align that experience with their brand promise. And over time, I just got more and more interested in user experience. And so that became both through self education and reading and taking classes. I never got a degree in HCI But I migrated into UX research. And because I had the prior background, I could also recognize when some questions came up, and I would say, Hey, this is not, this is not the kind of thing that Jakob Nielsen meant when he said you only needed five users. This is actually a survey research question that you’re thinking about here. And this is how we should go about that, or this is a conjoint pricing question. So developed an unusually broad research skill set, which is, as I said, perfect for what I’m doing today. I’ve given a lot of thought to why, what the obstacles are to having more integrated research departments like the one that we’re building at Ticketmaster. And I think the problem is, people come up through different silos. I mean, the research mindset, broadly speaking, is appropriate across all these different techniques, it shouldn’t stop with, you know, at the edge of one tool and now you need a completely different kind of training, researches research at a fundamental level. But I think the fact that some people start like I did in marketing research in the marketing department and some people start in UX research either in it or design or the product group, and they just don’t have much exposure to the other side. I think that that legacy of corporate organization is really what is holding us back more than anything intrinsic that says that we couldn’t work more closely together.
Steve: Hence the insight Center of Excellence.
Kathryn: Exactly.
Steve: Yeah. But then I think about you know, your, your earlier point about, you know, people, people moving themselves between different Different groups is learning for them, but also, they are pollinating or cross pollinating. And so you’re someone that’s moved from the advertising world in the marketing research world to the, as you said, the user experience world right now they are separate worlds. Yeah, I guess my question is just one that reiterates what you said, which was that it was, it was a, it was good for you to be able to do that and set you up. So
Kathryn: I think it’s good for all of us. Really, you know, I think the fact that we tend to transition careers is not just a fact of life, I think that it actually strengthens us. You know, so Dustin connais. At Ticketmaster, he’s Senior Research manager and he actually had started a pod of user researchers in the enterprise Design Group at Ticketmaster. Before I arrived, Dustin was a designer. And he took Hillary beanstalks class over at I think it’s Cal State Fullerton, and got so interested in user research that that became his passion. And he transitioned into becoming UX research and talked the design leader into letting him hire more people in that area. So now he’s part of this larger group. But he’s one of the people I was referring to that really helps elevate all of us. A lot of researchers are good at identifying insights, but we’re not fabulous at the way that we communicate those insights certainly visually. I’m one of those people. And so Dustin has been developing templates and guidelines so that our work isn’t ugly and tedious. To make it through, people are proud to share our decks and the key information jumps off the page at them. And, you know, we wouldn’t have that capability if Dustin hadn’t migrated his career in that way.
Steve: Well, he sounds like you to a certain extent, sort of discovering something that’s adjacent, getting very excited about it, investing in developing it themselves and then bringing, bringing not just the skills that they’ve learned, but that larger perspective, that larger set of skills and mindsets to help because research is a multifaceted thing. I think we keep talking about that. And so the diversity of skills is only enhanced by people kind of coming in from elsewhere and bringing their their previous kind of wardrobe or kits with them.
Kathryn: I agree. 100%. You know, I think that people that go into Research are almost by definition intellectually curious. And, you know, so we all benefit from the other things that are interesting to them, right?
Steve: I mean, one of the things that I’ve reflected on, you know, just given that I’ve been in this profession for a long time, and you know, I was in this profession when no one really knew what it was and including me that now you can’t now there are undergraduate programs or graduate programs, there’s a thing with a name that people can find, and they kind of come into it with a definition of what they what it is and what they’re going to bring to it. Of course, that doesn’t mean so that sort of the shape of the field has evolved because people are choosing it earlier, as opposed to finding it later. And I
Kathryn: yes, I mean, I agree with that. And I am right there with you. I share that that surprised at how Oh, wow. This is now Real Thing and none of us that started out, you know, came to it with that kind of background or at least very few. And yet, you know, if you go to research meetups, like I actually am the organizer for LA UX research meetup. Here in Los Angeles, there are still quite a few people that want to enter the field from adjacent fields, right. People that were designers, people that were content strategists, people that did maybe academic research. And so even with that clearer path and training opportunity, I don’t think it will ever necessarily end that there will be people trying to figure out how to leverage the life experience that they’ve had that does still bring a lot of value of a transition into a new skill set.
Steve: I mean, I feel like life experience is, and I don’t mean the amount of life experience you have, I just mean, the, you know, back to your, you know, your meetings where people talk about what’s going on with with them, you know, a culture you work in where people can bring their authentic selves, you know, that, to me seems so supportive of reach research, because it’s a mean depending on your method, but there’s a large amount of human to human that’s in it. And it is that makes it messy, but also very real. And if you don’t have space to be yourself at work, if you’re not creating an opportunity to kind of connect, you know, backstage in front stage or whatever your metaphor is, then that makes it hard to I think, to do research as well as it possibly can as you possibly can. So, as you said, your life experience whether that’s, you know, anything, I mean anything, any life experience that you have had, as well as sort of discipline and education professional experience to me, feel Kind of feeds the whole beast
Kathryn: 100% 100% you know, we also have another Senior Research manager on my team Brent Lowe. He came from a theater background and you know, so he brings an energy and exuberance to facilitating brainstorming sessions and meetings and helps to coach some of the people that are perhaps a little bit more introverted by nature and how to maintain that high level of energy that, you know, facilitates people contributing lots of ideas, right? So it leveraging whatever background and passion you have and being able to find a way to nurture and express that in your profession to me, that’s that’s a win win for everybody. Right.
Steve: And within your team, people, you know, and you said this before, we were talking more about about maybe methods or projects, I guess, but that also is an opportunity for each other to learn. I mean, work with a theatre person on the same project, you’re going to just you can’t help but be exposed to and inspired by different approaches to anything.
Kathryn: I agree.
Steve: And so you mentioned that you you had when you were working in advertising, you know, you you spent time in the agency world. How do you contrast that a life inside a corporation versus life inside an agency?
Kathryn: Yeah, that’s a good one. It was a big transition for me. I spent a lot of my career in agencies either large or small. And you know, the fun thing about agency life is usually everybody’s very smart, very creative. You’re constantly learning something new because you have Learn about the industry for each of your different clients. And that’s all fun. And I was I was nervous to be completely honest with you when I went to work for a huge corporation, like Ticketmaster Live Nation. And the good news is that ticket master is as I was explaining earlier, so complex, I don’t think I’m gonna get bored there anytime soon. You know, there’s, there’s still lots and lots and lots to learn. And it is kind of fun to rather than just tell somebody what they should do and give them the idea and hand it off to them. It can be it can be frustrating and it can be time consuming, and it can take a lot of perseverance to see that through. But there’s also a real joy in that Being part of something that comes out the other end, then you say, wow, you know, I played apartness, building that product in the first place, or I made a major contribution to how user friendly it is. And so that creative aspect of it is a little bit different. It’s not as much time on the blue sky brainstorming and a little bit more on the Okay, so how do we still pursue this given these technical integration considerations or these other challenges that in the real world, you have to work through, right. But it’s been very fulfilling. It’s been a great experience so far.
Steve: Is there anything that we should try to cover in this conversation that we haven’t got to?
Kathryn: Those were a lot that you hit on a lot of great questions. So it’s been a real joy chatting with you. I’m just kind of curious, you talk with so many people. Are there any trends that you find emerging or find very interesting in user research right now?
Steve: I mean, part of the fun of doing this podcast is talking with people who are I mean, it’s the design intent of the podcast, people who are who are in leadership roles. And you know, at so where the organization has invested to a certain point, because I think if you sort of hang around with researchers as I do, you hear a lot of the same concerns. So it’s interesting to talk with you and hear how either you’ve built around them or the conditions were created differently. So you know, you’re describing a situation where the research team is proactive. But that’s by design. It’s. So you are sort of you were given that opportunity. And I don’t mean to diminish your accomplishment that way, but the culture, the culture, the leadership kind of started off there. And I think a lot of people want to get to that stage. And it just makes me think about you know, like, there’s a lot of there’s a lot of frequently asked questions, I think that and we talked about them, maybe using some different language, but you know, having influence getting time, you know, having buy in isn’t there’s a lot of anxiety about that. I, I hear that from people in so many different environments that I go into. And I feel like more and more, I’m realizing, and maybe everybody else already knows this, but I’m realizing it’s like it’s hard to manage up on those things. It’s hard to, you know, create a culture that isn’t there. It’s hard to sort of change leadership. shifts mind. It’s that the there’s a lot of these are big picture things, how do we get everyone working on the same thing? Or how do we not duplicate research? Well, we have to devote resources to managing that. And I feel like because there’s a lot of scrappy research practices out there. Sort of under resourced, without leadership without guidance, you know, not completely bought into there’s a lot of hope for you know, magic fixes to systemic problems. So, you know, when I hear how people have worked hard to either find or build, you know, cultures, investment supports, that doesn’t have those problems. I don’t mean to say that everything is perfect for you or anybody that that I talked to, but it sort of drives home that you kind of can then there should be some folksy saying about how you you Can’t get the thing that you need without really going without really having what it takes to pull that off. So, if you’re a solo researcher and you work kind of taking, you know, taking orders, you know, for some designers and you’re by yourself, you can’t really change the corporate culture and buy in around research in the belief because that, you know, often needs to be there beforehand. So it’s a it’s a heavy lift for those folks. And I know they want it. So yeah, yeah, I mean, I’m telling you what I learned from you, I guess it how it fits,
Kathryn: right. Well, no, I, I really agree with what you’re saying, you know, there’s that old saw. Pick your parents carefully or pick your parents? Well, you know, the secret to so many things in life is good genes, right. So I think that what you’re highlighting for folks that are looking for work is You know, especially earlier in your career, it’s easy to get caught up in the title and the salary and that kind of thing. But there’s, there’s some of these bigger issues that you’re touching on Steve that are really so much more important. You know, what, what is the attitude toward research? Are people looking forward to you and what you’re providing? Or are they crossing their arms and being defensive about it? Do they view research as being an essential ingredient in success? You know, I sometimes say to people, I know that you’re just focused on getting out the door as quickly as possible, you know, moving fast and breaking things, but I genuinely believe that I am the fastest path to you launching a successfully adopted product in the marketplace and by I mean the research team, right. So you know, looking at Those things asking some questions kind of testing that out. If you are earlier in your career, and you can find a company that will really value and encourage you in that career, then that’s going to be a lot better for you in the long run and a lot more satisfying, personally, than making an extra $5,000 or having a senior in front of your title or or that kind of thing. I mean, that’s my perspective. What What do you think?
Steve: I mean, I would say yes, and some people may like the challenge, some in but you know, I guess go into these things with your eyes open. Yeah, so if this is not a place, go ahead.
Kathryn: Oh, no, no, I was just going to say I do think this is one of the drivers behind the trend in the last couple of years of teaching UX designers and UX researchers to better understand the language of business. I imagine you are familiar with, you know, these workshops and things that are starting to crop up. And there is some truth to that, that if a person in product leadership is focused on, you know, we were talking about before, what are you measured on? Okay, well, you’re measured on, you know, did you hit stay within the budget for this product at launch on time? What was the initial adoption? And the initial adoption is going to be the last thing to be measured? You find that out after the fact, right? So if we only ever talk about user experience as well, we want our fans to have a positive user experience that makes it seem like a nice to have, right? Yes, if time allows, of course, we’d like to do that. So I do think that if you’re in one of those less hospitable, less enlightened environments, that learning how to frame your work in a way that makes it clear that it is directly part of that team’s success and that you are all about I use terms like mitigating risk, enhancing adoption, reducing wasted dev effort, right? Those are the ways that I talk about research because it makes it really clear. And, you know, one thing I left out of my background is along the way, I got an MBA in business, you know. So it makes it a little bit easier for me to think about the way to talk about what I do in the benefit oriented terms that my stakeholders will understand.
Steve: I feel like what pulled research along you know, for many years, when we started having phrases like UX, which we didn’t always have, was was designed was user experience, and that that was kind of the foot in the door for a lot of technology companies. And that it, those people in that work highlighted the need for research and so we’ve kind of been pulled along. And, you know, I think you’re highlighting another what feels like a transition to me that research is part of the organization, whatever part of the organization, all of the parts of the organization, or can impact all of those. And that’s the reframe, you’re talking about, from, hey, research, can we can help create a good user experience too. We can help, right, you know, change the how the business model we can help. Right, reduce inefficiencies in engineering. Those are all that’s not that’s research sort of stepping out of the shadow of its big brother and saying, like, Hey, we’re, we’re a function that does all this stuff. And that’s I feel like that is part of the future that you’re that you’re looking towards and kind of helping to move to feel towards
Kathryn: Yeah, a great, good way to put it.
Steve: All right, well, if we’re almost at wrap up, I want to tell you about my experience working at Ticketmaster.
Kathryn: Okay.
Steve: Because I worked as an undergraduate computer science student I, I worked at a subset was a subsidiary It was a company in in Toronto called bass, which was best available seating service. Bought bikes got bought by Ticketmaster long ago. It was bass ticket master for a while then just became like, part of ticketmaster.ca when I was a night run operator, so there was no there was no internet. So it was it was lining up at the record stores that had terminals, and it
Kathryn: right right.
Steve: And it was the phones. So I was in a little office that had like the phone banks. You know, if you think of like a 70s movie with like a computer room with like the raised floor and the glass walls, you had to like to like badge in into it. I worked well, I worked in like an office outside there. And so at the end of the night, like I would go in and was like a, I don’t know, like a 6pm to like 1am shift. And you basically had to wait for all of the nodes of the network to close like so whenever the club sold its last ticket or the planets ran his last show, all those things would go offline. And then there were all these like men and women who had to run scripts that would then print out I think every record for the day. So you just print out these huge so that was their audit trail, I guess. I mean, I didn’t have any mental model at the time, but you just go print out all these things like 132 character tractor, whatever the metric was the super wide. Super fast printers that were in this room and you have to, you know, tight tape all these reports and take the tractor feeds off them and staple them and put them everybody’s mailbox and then do a disk backup, which was these dishwasher sized appliances you’d open the top and there was a harddrive in there which looked like like a giant kind of cake container. unscrew them physically move, you know, disc a to there was two there was a some kind of backup thing, you’d move one to the other and take one off the shelf and like run this, like 25 minute job to kind of back everything up. And it was I mean, I don’t know if it was archaic at the time, but it’s obviously just like a weird history thing there. But of course even even the tech people were like so into music, so into bands just exactly like what you’re describing there. They had that same culture. And it was fun to be part of that. It wasn’t fun to be moving disc packs around it, like 1am. But
Kathryn: anyway, pretty cold, right?
Steve: Yes, temperature control.
Kathryn: Oh, that is great. That is a great story. Well, you’ll have to, I’d be happy to get your ticket master t shirt. It sounds like you’re an OG
Steve; like old old OG. Yeah.
Steve: Well, this was really fun. Thank you for taking the time and, you know, sharing your perspective on all the things that you’ve accomplished and kind of your your vision for the field. I think
Kathryn: I really enjoyed this. Thanks. Thanks so much. Yes.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 29. Kathryn Campbell of Ticketmaster first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Kathryn Campbell, the Director of Research & Insights at Ticketmaster.
Whenever there is availability of somebody that might normally work on the marketplace side, they might tag team on an account manager project and that helps to inform them about that product. It gives them a little bit more purview. It facilitates internal sharing of learnings because we are a very large, complex organization. So that flexibility is both more satisfying to the researchers, but also benefits the product teams in the long run. – Kathryn Campbell
Show Links* MoveOn * Kathryn on LinkedIn * Kathryn on Twitter * Ticketmaster * Live Nation * Amy Howe, President and COO at Ticketmaster * Kathryn Frederick, CMO at Ticketmaster * Customer Insights Center of Excellence * Ogilvy (formerly Ogilvy & Mather) * Jakob Nielsen: Why You Only Need to Test with 5 Users * Dustin Guinee, Lead User Experience Researcher at Ticketmaster * Hilary Bienstock, Cal State University, Fullerton * Brent Jefferson Lowe, Senior Manager, UX Research at Ticketmaster
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Over the past few years I’ve been volunteering with MoveOn a progressive organization. My contribution is to be part of their texting team, where they identify an issue or a cause and a group of us volunteers send text message about that topic to their membership. Last week I was part of a campaign raise awareness of best practices for protection against coronavirus.
We don’t send text messages from our phones, it’s through a browser interface. A texter sees a pre-addressed, and pre-written message which they send, one at a time, usually thousands in total. When people respond, the volunteer can classify that response, which will produce a relevant response that we might customize to ensure it makes the most sense.
MoveOn collects responses and future campaigns (say, for someone to call their representative about an issue, or to get out of the vote) a built based on what they learn from looking at their data.
In this recent experience, we were asking people if they were practicing social distancing and sharing al link with resources and information. The responses that they were expecting were essentially, “yes I am”, “no I can’t (or don’t think I need to)”, “I need medical help” and “thank you.” The first day I participated, I heard from a number of people who were medical professionals. Now even though we are working on a dashboard with pulldowns menus, people are entering text on their phones, and have no idea what our interface is. They wouldn’t necessarily share their responses according to our predetermined categories. MoveOn runs a busy Slack channel where during any campaign you can get help in categorizing a response, or dealing with a situation that may arise.
So the next day when I was texting, I noticed two changes. The “thank you” response had been changed to “you’re welcome” which was a big improvement. Even though the “yes” and “no” responses were labels for what the person told us, if they said “thank you” as their response, your brain isn’t looking for a “thank you” label in order to send the text “you’re welcome” – you start looking for “you’re welcome” because that’s what you’re trying to say. The other change was a new response category for “I’m a medical professional.”
Whether they were looking at the responses in a table, or whether they had one of the volunteers raise this as an issue, I don’t know. I was interested in their ability to pay attention to the data and update their tools. It reminded me of survey research, not something I would say I’m particular expert in or anything. And I know in surveys it’s a best practice to do a pilot study, and maybe that’s where you catch something like this. It just reaffirmed for me that creating categories for people’s behavior and opinions is always a hypothesis, and deciding ahead of time how to categorize responses, before you’ve got a big pile of responses is always going to be flawed. MoveOn is able to make changes on the fly, even during a campaign, and I’m sure it messes up their data collection somewhat, but their priority is on the experience of the campaign, not their data. Survey researchers probably don’t have that opportunity. But it’s powerful to consider, well, what if they did?
Well, here’s another episode! As before, this was assembled quickly, with the budget I have available for production, which is zero. There’s no professional editor, there’s no professional transcript. The audio quality is the best we could get, when people are working at home, without access to all of their gear, using an Internet under strain, and so on.
And just a reminder that you can support this podcast by supporting my business. This is an uncertain time, and many small businesses are feeling the pinch and don’t know what’s coming. I will appreciate you keeping me in mind now and in the future as a collaborator in conducting user research, and as a resource for helping grow and develop your company’s capabilities. Please reach out!
All right, let’s get to my interview with Kathryn Campbell, the Director of Research & Insights at Ticketmaster.
Steve: Kathryn, thanks so much for being on dollars to donuts. I’m looking forward to the chance to speak with you.
Kathryn Campbell: Hi, Steve. Yeah, thank you. I’m looking forward to the conversation as well. I’m a big fan of your podcast.
Steve: And now you’re on the podcast. So
Kathryn: living the dream.
Steve: Yeah, living the dream. Well, we’ll see. We’re starting to live the dream. I would love for you to introduce yourself to begin.
Kathryn: Okay, great. Kathryn Campbell. I am the director of research and insights for Ticketmaster, which is a subsidiary of Live Nation, the world’s largest live entertainment company.
Steve: So in a minute, maybe we can talk about the research that you’re doing. But can you set the context a little bit about Ticketmaster itself?
Kathryn: Absolutely. So most of your listeners are probably familiar with Ticketmaster and if Had some interactions with us over the years. And we’re a 40 year old company, and a long time innovator and leader in the live event ticketing space. So most of us think about Ticketmaster, from the perspective of the fans, and the fans are a critical user group for us very near and dear to our hearts. And the people at Ticketmaster. Mostly come there because we are fans ourselves. But we also have a number of other customer groups that we serve. That includes the artists and their managers and promoters. It includes the venues and the surfaces that work with the venues like parking and refreshments. And we also have direct relationships with most of the major sports leagues in the US and another country. So football, basketball, hockey, all those kinds of things. So as you dig into it, you realize there’s actually a very Complex ecosystem of customers to serve.
Steve: Oh, so you do research with fans, artists, venues, leagues? I think those were the kind of groups you mentioned. What? Yeah, what does it look like? I mean, you know, within what you can describe, I’m wonder if there’s any aspects of some of those types of research that maybe are things we wouldn’t have heard about.
Kathryn: Sure. One of the things that I really enjoy about my job is working. We have a subsidiary called tm music that works directly with the artists and their teams. And it’s trying to get more money into their pockets rather than into brokers or scalpers pockets. So working with them on innovative solutions that allow them to ensure that people that say members Their fan clubs can pre order tickets, or receive priority in the waiting room or queue versus other people that we have not identified and might have reason to believe could be resellers. So that’s one area where we get to work. We also get to work with venues. And of course, we tend to think of the big arenas. But there’s also the relatively small theaters that do a lot of touring productions and helping them be more efficient and have a better relationship with their subscribers as well as one time buyers. And then with the lakes, it’s helping them get to know their customers better. One of the things that takes you a while to realize about ticketing is that in the old world, you only actually knew who about a fifth to a fourth of the people in your stadium or arena were most of them you know, one person might buy four tickets. So football teams might be selling out, but they don’t really know who a lot of the people are that are in that stadium, how far they’re coming from, what the composition of their group is. So there’s lots of interesting dimensions to the research that we do with those different segments.
Steve: Great. So a ticket buyer is not the same as a event goer.
Kathryn: Exactly. Yeah, it’s a different
Steve: two different stages there. Oh, well. What? You know, how do you given the different types of I mean, customers, users, etc, that you’re trying to learn about? Does that impact, you know, how you organize your own group and in order to work with different different parts of your organization and then different customers?
Kathryn: Absolutely. And, you know, this is an ongoing debate, I think, in the research world is whether you have a centralized group, you know, Internal agency, which is what our model is, or if you embed researchers in the lines of business so that they can work more closely with the stakeholders. What I feel is best for both the company and for the team members is kind of a hybrid model. I really like the mutual support and training benefits and the opportunity to stretch and grow of having everyone on a centralized team. But some of these areas are very complex. And so our internal stakeholders would like to have some continuity over time and work with the same person and not have to re educate them about the complexities of their product each time. So what I’m always trying to do is to maintain continuity to the greatest degree possible on the products and stakeholders, but also allow the individual researcher to feel like they are growing and stretching and developing new skills. Being exposed to new kinds of projects. So that is a constant balancing act.
Steve: So this may be a really annoying semantic question, or not even a question, but okay.
I liked the phrase that you use when you talked about embedded, you talked about embedding in lines of business. And sometimes when people talk, right, embedded, they talk about embedding on product teams. And those may be the same thing. But in terms of what they what picture they paint in my head. It I don’t know, is a product team and a line of business when we think for the purposes of embedding, is that a distinction without a difference, right?
Kathryn: So Well, not necessarily. I mean, you’re right within a line of business, there’s going to be multiple product teams and some of our products cross multiple lines of business. So for example, the person product that we use that support season ticket holders is called account manager. And it is basically a white label for the leaks or the theater theaters to manage their subscription customers, right. So that’s within one line of business, which is actually our enterprise, our b2b line of business. However, those people, the individual seasons, subscribers, season ticket holders are still fans, and they also want to buy maybe individual tickets, right? And they want to be able to manage their tickets in a centralized way. And they can easily get confused if there’s a different experience and account manager than there is in their account on ticketmaster.com. So, you’re right that it gets muddy. It’s not the same thing. But in terms of the underlying question about How do you balance that? One thing that we try to do is let’s say, we have one person who is an expert, which we do we actually have two people that are experts on account manager. However, whenever there is availability of somebody that might normally work on our what we call the marketplace side, the b2c side, they might tag team on an account manager project and that helps to inform them about that product. make them more aware of the complexities for a fam that has membership, you know, an accountant both and they can also inform that account manager team about some of the things that we’re doing on the marketplace side, right. So it gives them a little bit more purview. It facilitates internal sharing of learnings because we are as you’re beginning to get a sense we are evolving. Very large, complex organization. And we’re very fast moving. And so that flexibility i think is both more satisfying to the researchers, but also benefits the product teams in the long run.
Steve: It’s an interesting, it’s a, it’s a, I think, a brilliant idea to do sort of knowledge transfer or knowledge distribution, there’s probably a smarter way to say that by moving people, the people contain information, and you move the people around and the information, the knowledge, the expertise gets shared that way. As opposed to, you know, I feel like sometimes we will. There’s a there’s an idea, we should extract the information that’s in people and then centralize that, but you’re kind of, you’re distributing the people. Because the collaboration, you know, lets that expertise come to the surface and have impact in ways that you wouldn’t necessarily be able to predict.
Kathryn: Yeah, I think that’s absolutely true. I remember listening to someone talking about how that was a strength that Google the way that people tended to move between different groups and the informal knowledge sharing and transfer there. I think with large, complex organizations that whether we like it or not that informal knowledge sharing is at least as important, if not more important than the more formal documentation, which is always going to be less rich and behind the times, right. And I also think, you know, at the risk of digressing that this whole issue of really caring for your team and their career is one that I think is too often overlooked. You know, even on the agency side, it’s always easier to satisfy your client, whether it’s an internal or an external client by having them keep seeing the same face. But no, who wants to be that person that only knows about One thing and keeps doing the same thing year after year. And you know whether or not you want to build out your portfolio because you want to eventually leave or maybe you just want to stay interested in engaged and challenged. I think it’s really incumbent on us as leaders to facilitate that growth. And that is a lot easier to do. If you deliberately encourage this kind of movement across different project types and skill sets.
Steve: What else do you do to take care of your team?
Kathryn: Oh, gosh, we do a lot of things. You know, in our weekly meeting, one technique that I developed a few years ago when I was working with a boutique agency primitive spark was to start the weekly team meeting by really sharing on a more personal level. What’s present for you, you know, are things going great and you want to share something? Do you have a sick kid at home And you’re worried about them. We really try to bring in a bit of that humanity to our professional understanding of each other, because it helps you understand where people are coming from. We also really try to encourage thinking as a team in all respects, we actively try to load balance. So you know, with research, you have the best laid plans, right? have everything laid out and then a product gets delayed, or it gets accelerated. And so you might find yourself either with three major projects that you need to do interviews on this week or nothing for three weeks. And so we’re constantly encouraging the team members to chip in and help each other and again that facilitates that ability to knowledge, share and maintain both personal growth and professional efficiency. Time,
Steve: whenever there. I mean, it seems like in some organizations there’s, you know, a lot of measurement of where you spend your time or where you don’t spend your time that may have the it may work against what you’re talking about, right? What How are people incentivized or just incentivizes spend their time? So, I’m wondering, I don’t mean I don’t know sort of how it works at Ticketmaster, you can obviously, say this is the goal. But is there are there measures in place to reward is the wrong word, but to support that kind of operationally?
Kathryn: Yeah, it’s a really good point because we get back what we incentivize and what we measure, right. So I do not keep a tight track of hours per week, or, you know, we don’t have billable hours because we’re an internal agency that that Not what the focus is on so much as a couple things. One is what kind of research we’re doing. I’ve heard you in prior podcasts, ask about focus on generative versus evaluative research. And so that is a focus for us. We want to continuously move upstream where we can have more value in terms of, you know, the quote to build the right it before you worry about building it, right. So we do have quarterly, okay, ours, objectives and key results around what percentage of your work is generative. To that end, we have embarked on a democratizing research program over the last year where we’re training the product designers. In how to do their own UX research, you prototype research, which has a number of benefits for everybody. One is it’s faster for the designer, they can do quicker, smaller studies, if they don’t have to wait for one of us to be available. They get closer to their end customer and get to understand them better. And it also means that we don’t get into the situation that happens with many UX research departments, where we’re just constantly every week or two churning out new studies, but never moving upstream to the bigger product concepts, right? So this way, we can kind of push off some of that more routine, prototype testing work. And we’re now focused on doing concepting focus groups or surveys to size market interest in something or facilitating a design workshop or or those kinds of
Steve: So I wanted to ask them about where projects come from. You’ve made space kind of made bandwidth for your team by empowering others to take on a certain set of projects. And now you listed a bunch of things that, you know, you’re free to go and do. Where do those projects originate?
Kathryn: Right? It’s a great question. And I think that feeling empowered to take some ownership of that process is probably the inflection point in terms of really adding value to an organization with research. So I am extremely fortunate because ticket masters president Amy Howe, and our cmo cat Fredrik out when I joined two years ago, both of them explicitly said to me, I do not want you to be an order taker. I do not want research to Be order takers. And so that said to me, You are empowered to go out there. And so we do a couple things. One is as projects come in, and we have a process by which things come into the queue, and then they’re reviewed and groomed weekly, and that sort of thing we talked about, okay, what here best aligns with a key corporate priority? And Ticketmaster is pretty good about sharing what the key business priorities are that people should be focused on. But then occasionally, we also say, Gosh, I wonder I keep hearing about this new thing, and it seems like a really important thing, and it seems like something we should be doing research on and nobody’s asked us to do a search. So we feel very comfortable pursuing those stakeholders and saying, gosh, you know, wouldn’t, wouldn’t it be a good idea to get some end user validation that this is the right set of requirements before we built this, or to help ensure you know that that we’ve captured all of the requirements here. We can do that we also have built in a model of periodically checking in with the most senior leadership at the company to tell them what we have been working on. And also what’s coming up, and to say, are the projects that we’re planning on doing for the next quarter aligned with where you think the real value for the organization is? And so that can also be a point of alignment. But yeah, increasingly, we find ourselves suggesting things that really should have research done on them. And sometimes, in fact, we just go ahead and do stuff. And I think that as long as you’re not just pursuing, kind of cool vanity projects, but you’re using as your North Star, what the origin This nation says are truly the key business priorities. I don’t see that that whatever turn out badly for you.
Steve: So in the situation where you are, you know, you’ve identified something that a group is working on. And you approach those stakeholders and say, Hey, we we could do this research that would help you at this point. I guess my, the question I’m really after is, you know, that this thing that a lot of researchers Express stress about the, you know, no one listens to me. They don’t take our recommendations, we can’t kind of impact the team, they go ahead and do what they’re going to do anyway. So how did the How does the How does the taking action on the research part play out when you are, you know, not being an order taker when you’re being proactive and kind of coming to them? What do you see happening as a result of the research?
Kathryn: I think that we as researchers need to be willing to be flexible. I don’t like to encourage stakeholders who constantly come in at the last minute when they could have allowed for more planning time. But sometimes things really do emerge at the last minute or become very urgent. And so we need to be willing to say, hey, what can we do to try to make research work within the time constraints and the other needs that this project has. And I think that if you are willing to be an empathetic partner in that respect that that earns you some trust in the relationship and they can see hey, you’re knocking yourself out to make this work. So I should go ahead and use it. And then the other thing is some stakeholders just get it more than others do and one of the best pieces of advice I got from analyst at Forrester Research when we were talking about how to build a culture that respects research, what she said, Hey, you know, work with the people that really value you create fabulous case studies, and then use them as advocates because that will have a lot more impact on the recalcitrant stakeholders then you constantly saying, Hey, why don’t you use research? research is good, right? So, you know, when I have to prioritize who to make room for I definitely prioritize people that historically have acted on our recommendations. Yeah.
Steve: That’s, that’s really good. I just want to highlight the first part of that, and make sure I understood that. You’re saying that if you adapt and you know, are flexible and are sort of demonstrably digging in to support someone, that that sets them up to be more receptive to the results of the work that you’re doing for them.
Kathryn: That’s that’s been my experience, you know, let’s face it 90% of the time. The reason someone does not embrace research is, well, sometimes they say to themselves, well, I won’t learn anything new. But the biggest fear is normally this is going to take too long. I’m already behind. I have to hit this deadline. That’s what I’m going to be measured by. It’s back to what we talked about earlier. people behave based on what they’re measured on. And product owners and certainly project managers tend to be measured on Did you deliver on time, right? So if I go to that stakeholder and I said, Look, I am really worried about the unknowns in this new product and rolling it out too quickly to a large audience without doing some more user testing with the beta product. We’re willing to take our whole team and work in the evening or over the weekend and pull this together because we feel so strongly that that would be a good thing to do. Well, that kind of puts the stakeholder in the position where if they say no, now they’re going to be held accountable if big things show up, right? They’re making a very conscious choice to say no. And the good news is, you know, I won’t say how long I’ve been in research, but it’s been quite quite some time. And in all of that time, there has literally never, ever even one time than a research project where I didn’t learn anything. There’s always a surprise and in our business, sometimes that little surprise can turn out to be something quite significant. So for us, you know, now in the last year a big product innovation for us has been around secure mobile ticketing, which means that your phone is your ticket, right? I don’t know if you’ve had experience now with going to events and everything’s on your phone. Well, if you happen to have, let’s say, an Android device with a heavy cover on it, it turns out that that impedes the scanner working correctly in certain circumstances. So there may only be if we do a quick beta test with 100 people, there might only be two or three people there. But when you scale that up to all of the people that are going to any event in the next year, all of a sudden you’re talking about having a big impact on our call center, for example. So you know, highlighting Oh god, I’m glad we combat. And and you know promulgating that story of how we went to bat and how we found those insights. You know, I think it builds a sense that it is in everyone’s best interest to work with us. Now, the flip side is, I also constantly am tracking down and hounding the Senior Product people to say, what’s coming up this quarter. What’s coming up this half of the year? Have your priorities shifted, let us know as soon as possible because I don’t like living in a state of chaos. I don’t like constantly having to shift all the priorities because of something being coming up at the last minute if in fact, we could have known about it in advance. So now when they see me in the hallway, they immediately go through their mental checklist of, oh, is there anything that I should tell Kathryn and her team about?
Steve: Right, because you want people saying Oh, when they So you’re coming down?
Kathryn: Well, in a good way,
Steve: in a good way, the good kind of Oh, here comes Kathryn you’re painting a picture for us. I mean, the you mentioned that, you know, when you first started talking with the leadership and protect the master about taking on this role, they had a they framed the position as something where you had agency where you were kind of going to do things as you said, not being an order taker. I mean, this, I would imagine, you have to guess here, but what what do you think created the conditions for them, that they understood the power of research and someone like you to, to work in order to work that way? How did they see that light?
Kathryn: You know, my hypothesis about that is both of those women are very smart, very data driven executives. So they We’re not particularly knowledgeable about research, we do have Ticketmaster has an extremely robust analytics group. So we invest a lot in site analytics and marketing analytics and have very in depth reporting and good turnaround times for learnings. So I think that they just wanted to extend that from the behavioral, you know, what are people doing data that we were very rich in into the Why are people doing that side that we were not rich in? So I think that having a data driven organization that only had half of the insights that they needed is probably what created that willingness to or that embrace of the research group.
Steve: And how does the work that your team does? How does that set With the the people or the work of of analytics?
Kathryn: Oh, that’s a great question because it’s kind of an unusual setup and yet I do think it’s the direction that the industry should move over time. So both micro, which is research and insights primary research, we sit next to an equal to the site and marketing analytics group. So that together we form a combined insight center of excellence that reports up through marketing. And Forrester has been promoting this idea of an insight Center of Excellence for some time of bringing together all of those different sources of information about your customers. The site data, marketing data, UX research, survey research, qualitative research into one place And when I hear the stories, the war stories from my colleagues at other companies, it really helps me appreciate how wonderful this construct is. Because just the other day, I was talking with a friend who’s in charge of user research for a global company, and he said, you know, any time our work starts to move in the direction of maybe doing a survey, all of a sudden the marketing research team wants to get involved, but they’re a completely separate group reporting to different people. They’re not as knowledgeable about the background on the study, you know, it becomes a big deal to engage them and yet, they don’t want us to do anything that looks like a survey if they’re not engaged. So you create these territorial issues and obstacles to free, flowing information and you’ve been unmuted. For a long time, you know how it is it’s kind of like unwinding the sweater where you start out in one place, and then you learn something interesting. And so you start to follow that down. And so things keep evolving, you know, the appropriate methodology and keeps evolving. So having everything centralized, the only thing that’s not centralized that I think is a key part of the insight into our customers is fan support and client support. And we work very closely with those groups. We exchange reporting, and we also check in with each other when an issue seems to be arising to see you know, what each other is hearing. But pulling all of that together, I think makes the company just much more insightful and more nimble to act on insights.
Steve: Maybe we can shift a little bit be interesting to hear about, you know, the makeup of your team, whatever, what kinds of backgrounds that people have coming in? Yeah.
Kathryn: So, you can imagine given the wide scope of research that we do, it’s, it’s a mosaic of people with different backgrounds and different skills. There’s a certain core skill set that everyone has to have in terms of You know, basic one on one interviews, usability testing, and straightforward survey research design. But beyond that, we definitely have areas of expertise, we have one person who really loves and embraces research Ops, and that is a godsend to all the rest of us, because it just makes everything much more organized and all the rest of us more efficient. We have a couple people, including my number two person who come from design backgrounds, and so they are especially good at thinking through how to make the information design as impactful as possible. So we’ve really been up leveling the quality of our reporting to make things more aesthetically pleasing and also to help focus the reader on the couple of key numbers or quotes or Whatever that they should really come away remembering. We have we just brought on someone who is expert in advanced analytics. So like I said, What I’m looking for is everybody has a basic, a couple of the same tools in their tool belt. And then across the team as a whole, were able to offer richness of skills in a wide variety of areas.
Steve: And when you talk to prospective hires are the kinds of things maybe beyond you know, the skills necessarily, but I don’t know. How do you think about trying to assess if somebody would be a good addition to your team?
Kathryn: Yeah.
I think that really the personality characteristics are more important in most cases than the hard skills although sometimes you need a particular Hard skill. It’s really important that people be smart and think quickly on their feet, you know,how how intensely you have to focus to shift when an interview shifts or to look at data and think about what it’s really telling you and whether or not there’s alternative explanations for that. So that that kind of skill set a mental nimbleness is very important. It’s very important to me, in particular, that they show initiative. I don’t want to be a micromanager especially, you know, when you’re working remotely and right now a lot of us are working remotely, you need to trust that people are going to pick up things and run with it. I think that the basic people skills likability and very strong verbal and written communication skills are pretty key. Once again, where we’re servicing Senior stakeholders across a wide variety of businesses and sometimes our insights might mean multimillion dollar shifts and investment. So a person has to be credible and take feedback seriously. You know, one of the things that we practice a lot is it helping our clients recognize the actions that are implied in an insight, but not overstating what the research itself told us, right? So it would indicate that this is a good new hypothesis for us to test and explore, maybe through research, maybe through a B testing, you know, but without saying, Oh, yeah, this is what fans want us to do if that was not exactly what the research says. So that kind of discernment is pretty important to maintain credibility as a researcher.
Steve: Among the researchers, today, what kind of relationship do you all have to? I guess live entertainment, right? The business that the ticket master is in, what’s the culture among your team relative to that?
Kathryn: It’s pretty fun. I have to tell you, it really is, um, you know, the hand employee handbook says, you know, in terms of dress code, it says, you know, come to work is your authentic self. And so, you know, people come to work and they’re in band t shirts, or they’re wearing a jersey for their favorite sports team. We have dogs, you know, dog friendly policy. So a lot of people bring dogs to work, when there’s a major sporting event, you know, the World Cup or something like that. We throw it up on the TVs in the office. So we are all fans and we really do empathize with and more To maximize the joy in the live event experience. And the flip side is we work. It’s a very, very, very high velocity environment. Ticketmaster is actually one of the largest e commerce companies in the world. And as we sometimes say, you know, every Tuesday is like, Black Friday, because, you know, when you have one big global music musician after another coming out with their tours, and you have 20 million people trying to get 400,000 seats all at the same time. You know, there’s a lot of stress on systems. So, yeah, it’s an intense environment, but it’s also a fun environment.
Steve: Do people, do people still line up to buy tickets that’s up maybe showing my age or a nice
Kathryn: They do but they line up virtually. they line up in a virtual waiting room. And yeah, and it’s it’s intense out there, you know, they’re stressed out and we have a lot of compassion and empathy for that, you know, somebody is so frustrated because they’re, you know, biggest star is announced a tour and they really want to see them. And the reality is that even if the systems work perfectly and very frequently, the systems don’t work perfectly. There’s just you know, demand outstrips supply. So, in certain circumstances, you know, when BTS goes on tour when Taylor Swift goes on tour, not everybody is going to get tickets, at least not immediately, you know, maybe eventually but it’s not easy. So again, working through fairly sophisticated tools to try Try to root out bots we actually get we turn away more than 1 billion with a B bots a month. You know, resellers and bad actors are constantly hitting our systems trying to scoop up all the tickets so that they can resell them at a much higher margin and we are spending all of our time trying to outwit them and trying to deliver real fans tickets to the events that they’d like to see within their pocketbook if at all possible.
Steve: So demand you know, for successful artists demand is always gonna outstrip supply, but you’re trying to make sure that as much supplies as possible is available for the for the fans.
Kathryn: That’s the goal.
Steve: I’d like to I’m going to shift gears.
Kathryn: you’re the host, you get to do that.
Steve: All right. I would love to hear a bit more about, you know, we’ve talked about, you know, your work most recently, but can you talk a little about your background, how you found research and how you found, you know, the kind of research that you’re doing now? Sure.
Kathryn: It’s an unusual background for UX research, but I think it is the perfect background for what I’m doing now and the direction that I think research is moving. I actually started out working for advertising agencies doing marketing research, doing segmentation studies, message testing, focus groups, you know, various brand measurements and became a brand strategist. I was is actually a partner for Ogilvy and Mather ad agency over brand strategy. And what I realized as the internet changed from being viewed as just a media channel to being really the whole business for some companies, I realized that the biggest problem with some of the brands I was working on was their poor user experience. And so I wanted to learn more about the elements of good user experience and what made a site intuitive as part of understanding how to align that experience with their brand promise. And over time, I just got more and more interested in user experience. And so that became both through self education and reading and taking classes. I never got a degree in HCI But I migrated into UX research. And because I had the prior background, I could also recognize when some questions came up, and I would say, Hey, this is not, this is not the kind of thing that Jakob Nielsen meant when he said you only needed five users. This is actually a survey research question that you’re thinking about here. And this is how we should go about that, or this is a conjoint pricing question. So developed an unusually broad research skill set, which is, as I said, perfect for what I’m doing today. I’ve given a lot of thought to why, what the obstacles are to having more integrated research departments like the one that we’re building at Ticketmaster. And I think the problem is, people come up through different silos. I mean, the research mindset, broadly speaking, is appropriate across all these different techniques, it shouldn’t stop with, you know, at the edge of one tool and now you need a completely different kind of training, researches research at a fundamental level. But I think the fact that some people start like I did in marketing research in the marketing department and some people start in UX research either in it or design or the product group, and they just don’t have much exposure to the other side. I think that that legacy of corporate organization is really what is holding us back more than anything intrinsic that says that we couldn’t work more closely together.
Steve: Hence the insight Center of Excellence.
Kathryn: Exactly.
Steve: Yeah. But then I think about you know, your, your earlier point about, you know, people, people moving themselves between different Different groups is learning for them, but also, they are pollinating or cross pollinating. And so you’re someone that’s moved from the advertising world in the marketing research world to the, as you said, the user experience world right now they are separate worlds. Yeah, I guess my question is just one that reiterates what you said, which was that it was, it was a, it was good for you to be able to do that and set you up. So
Kathryn: I think it’s good for all of us. Really, you know, I think the fact that we tend to transition careers is not just a fact of life, I think that it actually strengthens us. You know, so Dustin connais. At Ticketmaster, he’s Senior Research manager and he actually had started a pod of user researchers in the enterprise Design Group at Ticketmaster. Before I arrived, Dustin was a designer. And he took Hillary beanstalks class over at I think it’s Cal State Fullerton, and got so interested in user research that that became his passion. And he transitioned into becoming UX research and talked the design leader into letting him hire more people in that area. So now he’s part of this larger group. But he’s one of the people I was referring to that really helps elevate all of us. A lot of researchers are good at identifying insights, but we’re not fabulous at the way that we communicate those insights certainly visually. I’m one of those people. And so Dustin has been developing templates and guidelines so that our work isn’t ugly and tedious. To make it through, people are proud to share our decks and the key information jumps off the page at them. And, you know, we wouldn’t have that capability if Dustin hadn’t migrated his career in that way.
Steve: Well, he sounds like you to a certain extent, sort of discovering something that’s adjacent, getting very excited about it, investing in developing it themselves and then bringing, bringing not just the skills that they’ve learned, but that larger perspective, that larger set of skills and mindsets to help because research is a multifaceted thing. I think we keep talking about that. And so the diversity of skills is only enhanced by people kind of coming in from elsewhere and bringing their their previous kind of wardrobe or kits with them.
Kathryn: I agree. 100%. You know, I think that people that go into Research are almost by definition intellectually curious. And, you know, so we all benefit from the other things that are interesting to them, right?
Steve: I mean, one of the things that I’ve reflected on, you know, just given that I’ve been in this profession for a long time, and you know, I was in this profession when no one really knew what it was and including me that now you can’t now there are undergraduate programs or graduate programs, there’s a thing with a name that people can find, and they kind of come into it with a definition of what they what it is and what they’re going to bring to it. Of course, that doesn’t mean so that sort of the shape of the field has evolved because people are choosing it earlier, as opposed to finding it later. And I
Kathryn: yes, I mean, I agree with that. And I am right there with you. I share that that surprised at how Oh, wow. This is now Real Thing and none of us that started out, you know, came to it with that kind of background or at least very few. And yet, you know, if you go to research meetups, like I actually am the organizer for LA UX research meetup. Here in Los Angeles, there are still quite a few people that want to enter the field from adjacent fields, right. People that were designers, people that were content strategists, people that did maybe academic research. And so even with that clearer path and training opportunity, I don’t think it will ever necessarily end that there will be people trying to figure out how to leverage the life experience that they’ve had that does still bring a lot of value of a transition into a new skill set.
Steve: I mean, I feel like life experience is, and I don’t mean the amount of life experience you have, I just mean, the, you know, back to your, you know, your meetings where people talk about what’s going on with with them, you know, a culture you work in where people can bring their authentic selves, you know, that, to me seems so supportive of reach research, because it’s a mean depending on your method, but there’s a large amount of human to human that’s in it. And it is that makes it messy, but also very real. And if you don’t have space to be yourself at work, if you’re not creating an opportunity to kind of connect, you know, backstage in front stage or whatever your metaphor is, then that makes it hard to I think, to do research as well as it possibly can as you possibly can. So, as you said, your life experience whether that’s, you know, anything, I mean anything, any life experience that you have had, as well as sort of discipline and education professional experience to me, feel Kind of feeds the whole beast
Kathryn: 100% 100% you know, we also have another Senior Research manager on my team Brent Lowe. He came from a theater background and you know, so he brings an energy and exuberance to facilitating brainstorming sessions and meetings and helps to coach some of the people that are perhaps a little bit more introverted by nature and how to maintain that high level of energy that, you know, facilitates people contributing lots of ideas, right? So it leveraging whatever background and passion you have and being able to find a way to nurture and express that in your profession to me, that’s that’s a win win for everybody. Right.
Steve: And within your team, people, you know, and you said this before, we were talking more about about maybe methods or projects, I guess, but that also is an opportunity for each other to learn. I mean, work with a theatre person on the same project, you’re going to just you can’t help but be exposed to and inspired by different approaches to anything.
Kathryn: I agree.
Steve: And so you mentioned that you you had when you were working in advertising, you know, you you spent time in the agency world. How do you contrast that a life inside a corporation versus life inside an agency?
Kathryn: Yeah, that’s a good one. It was a big transition for me. I spent a lot of my career in agencies either large or small. And you know, the fun thing about agency life is usually everybody’s very smart, very creative. You’re constantly learning something new because you have Learn about the industry for each of your different clients. And that’s all fun. And I was I was nervous to be completely honest with you when I went to work for a huge corporation, like Ticketmaster Live Nation. And the good news is that ticket master is as I was explaining earlier, so complex, I don’t think I’m gonna get bored there anytime soon. You know, there’s, there’s still lots and lots and lots to learn. And it is kind of fun to rather than just tell somebody what they should do and give them the idea and hand it off to them. It can be it can be frustrating and it can be time consuming, and it can take a lot of perseverance to see that through. But there’s also a real joy in that Being part of something that comes out the other end, then you say, wow, you know, I played apartness, building that product in the first place, or I made a major contribution to how user friendly it is. And so that creative aspect of it is a little bit different. It’s not as much time on the blue sky brainstorming and a little bit more on the Okay, so how do we still pursue this given these technical integration considerations or these other challenges that in the real world, you have to work through, right. But it’s been very fulfilling. It’s been a great experience so far.
Steve: Is there anything that we should try to cover in this conversation that we haven’t got to?
Kathryn: Those were a lot that you hit on a lot of great questions. So it’s been a real joy chatting with you. I’m just kind of curious, you talk with so many people. Are there any trends that you find emerging or find very interesting in user research right now?
Steve: I mean, part of the fun of doing this podcast is talking with people who are I mean, it’s the design intent of the podcast, people who are who are in leadership roles. And you know, at so where the organization has invested to a certain point, because I think if you sort of hang around with researchers as I do, you hear a lot of the same concerns. So it’s interesting to talk with you and hear how either you’ve built around them or the conditions were created differently. So you know, you’re describing a situation where the research team is proactive. But that’s by design. It’s. So you are sort of you were given that opportunity. And I don’t mean to diminish your accomplishment that way, but the culture, the culture, the leadership kind of started off there. And I think a lot of people want to get to that stage. And it just makes me think about you know, like, there’s a lot of there’s a lot of frequently asked questions, I think that and we talked about them, maybe using some different language, but you know, having influence getting time, you know, having buy in isn’t there’s a lot of anxiety about that. I, I hear that from people in so many different environments that I go into. And I feel like more and more, I’m realizing, and maybe everybody else already knows this, but I’m realizing it’s like it’s hard to manage up on those things. It’s hard to, you know, create a culture that isn’t there. It’s hard to sort of change leadership. shifts mind. It’s that the there’s a lot of these are big picture things, how do we get everyone working on the same thing? Or how do we not duplicate research? Well, we have to devote resources to managing that. And I feel like because there’s a lot of scrappy research practices out there. Sort of under resourced, without leadership without guidance, you know, not completely bought into there’s a lot of hope for you know, magic fixes to systemic problems. So, you know, when I hear how people have worked hard to either find or build, you know, cultures, investment supports, that doesn’t have those problems. I don’t mean to say that everything is perfect for you or anybody that that I talked to, but it sort of drives home that you kind of can then there should be some folksy saying about how you you Can’t get the thing that you need without really going without really having what it takes to pull that off. So, if you’re a solo researcher and you work kind of taking, you know, taking orders, you know, for some designers and you’re by yourself, you can’t really change the corporate culture and buy in around research in the belief because that, you know, often needs to be there beforehand. So it’s a it’s a heavy lift for those folks. And I know they want it. So yeah, yeah, I mean, I’m telling you what I learned from you, I guess it how it fits,
Kathryn: right. Well, no, I, I really agree with what you’re saying, you know, there’s that old saw. Pick your parents carefully or pick your parents? Well, you know, the secret to so many things in life is good genes, right. So I think that what you’re highlighting for folks that are looking for work is You know, especially earlier in your career, it’s easy to get caught up in the title and the salary and that kind of thing. But there’s, there’s some of these bigger issues that you’re touching on Steve that are really so much more important. You know, what, what is the attitude toward research? Are people looking forward to you and what you’re providing? Or are they crossing their arms and being defensive about it? Do they view research as being an essential ingredient in success? You know, I sometimes say to people, I know that you’re just focused on getting out the door as quickly as possible, you know, moving fast and breaking things, but I genuinely believe that I am the fastest path to you launching a successfully adopted product in the marketplace and by I mean the research team, right. So you know, looking at Those things asking some questions kind of testing that out. If you are earlier in your career, and you can find a company that will really value and encourage you in that career, then that’s going to be a lot better for you in the long run and a lot more satisfying, personally, than making an extra $5,000 or having a senior in front of your title or or that kind of thing. I mean, that’s my perspective. What What do you think?
Steve: I mean, I would say yes, and some people may like the challenge, some in but you know, I guess go into these things with your eyes open. Yeah, so if this is not a place, go ahead.
Kathryn: Oh, no, no, I was just going to say I do think this is one of the drivers behind the trend in the last couple of years of teaching UX designers and UX researchers to better understand the language of business. I imagine you are familiar with, you know, these workshops and things that are starting to crop up. And there is some truth to that, that if a person in product leadership is focused on, you know, we were talking about before, what are you measured on? Okay, well, you’re measured on, you know, did you hit stay within the budget for this product at launch on time? What was the initial adoption? And the initial adoption is going to be the last thing to be measured? You find that out after the fact, right? So if we only ever talk about user experience as well, we want our fans to have a positive user experience that makes it seem like a nice to have, right? Yes, if time allows, of course, we’d like to do that. So I do think that if you’re in one of those less hospitable, less enlightened environments, that learning how to frame your work in a way that makes it clear that it is directly part of that team’s success and that you are all about I use terms like mitigating risk, enhancing adoption, reducing wasted dev effort, right? Those are the ways that I talk about research because it makes it really clear. And, you know, one thing I left out of my background is along the way, I got an MBA in business, you know. So it makes it a little bit easier for me to think about the way to talk about what I do in the benefit oriented terms that my stakeholders will understand.
Steve: I feel like what pulled research along you know, for many years, when we started having phrases like UX, which we didn’t always have, was was designed was user experience, and that that was kind of the foot in the door for a lot of technology companies. And that it, those people in that work highlighted the need for research and so we’ve kind of been pulled along. And, you know, I think you’re highlighting another what feels like a transition to me that research is part of the organization, whatever part of the organization, all of the parts of the organization, or can impact all of those. And that’s the reframe, you’re talking about, from, hey, research, can we can help create a good user experience too. We can help, right, you know, change the how the business model we can help. Right, reduce inefficiencies in engineering. Those are all that’s not that’s research sort of stepping out of the shadow of its big brother and saying, like, Hey, we’re, we’re a function that does all this stuff. And that’s I feel like that is part of the future that you’re that you’re looking towards and kind of helping to move to feel towards
Kathryn: Yeah, a great, good way to put it.
Steve: All right, well, if we’re almost at wrap up, I want to tell you about my experience working at Ticketmaster.
Kathryn: Okay.
Steve: Because I worked as an undergraduate computer science student I, I worked at a subset was a subsidiary It was a company in in Toronto called bass, which was best available seating service. Bought bikes got bought by Ticketmaster long ago. It was bass ticket master for a while then just became like, part of ticketmaster.ca when I was a night run operator, so there was no there was no internet. So it was it was lining up at the record stores that had terminals, and it
Kathryn: right right.
Steve: And it was the phones. So I was in a little office that had like the phone banks. You know, if you think of like a 70s movie with like a computer room with like the raised floor and the glass walls, you had to like to like badge in into it. I worked well, I worked in like an office outside there. And so at the end of the night, like I would go in and was like a, I don’t know, like a 6pm to like 1am shift. And you basically had to wait for all of the nodes of the network to close like so whenever the club sold its last ticket or the planets ran his last show, all those things would go offline. And then there were all these like men and women who had to run scripts that would then print out I think every record for the day. So you just print out these huge so that was their audit trail, I guess. I mean, I didn’t have any mental model at the time, but you just go print out all these things like 132 character tractor, whatever the metric was the super wide. Super fast printers that were in this room and you have to, you know, tight tape all these reports and take the tractor feeds off them and staple them and put them everybody’s mailbox and then do a disk backup, which was these dishwasher sized appliances you’d open the top and there was a harddrive in there which looked like like a giant kind of cake container. unscrew them physically move, you know, disc a to there was two there was a some kind of backup thing, you’d move one to the other and take one off the shelf and like run this, like 25 minute job to kind of back everything up. And it was I mean, I don’t know if it was archaic at the time, but it’s obviously just like a weird history thing there. But of course even even the tech people were like so into music, so into bands just exactly like what you’re describing there. They had that same culture. And it was fun to be part of that. It wasn’t fun to be moving disc packs around it, like 1am. But
Kathryn: anyway, pretty cold, right?
Steve: Yes, temperature control.
Kathryn: Oh, that is great. That is a great story. Well, you’ll have to, I’d be happy to get your ticket master t shirt. It sounds like you’re an OG
Steve; like old old OG. Yeah.
Steve: Well, this was really fun. Thank you for taking the time and, you know, sharing your perspective on all the things that you’ve accomplished and kind of your your vision for the field. I think
Kathryn: I really enjoyed this. Thanks. Thanks so much. Yes.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 29. Kathryn Campbell of Ticketmaster first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Kathryn Campbell, the Director of Research & Insights at Ticketmaster. Whenever there is availability of somebody that might normally work on the marketplace side, they might tag team on an account manager project and that helps to inform them about that product. It gives them […]
The post 29. Kathryn Campbell of Ticketmaster first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Laura Faulkner, the Head of Research at Rackspace.
I’ve never just sat and done just what I was asked to do. I’m always looking for something new, something else. It’s probably just part of how I’m built but it’s also a conscious choice of, of just doing my my current job is simply not enough for me and even just that extra 10 minutes of curiosity or desire to see something else or learn something new while still doing my job and delivering on that, it’s just opened so many doors helped me step into a lot of opportunities and learn new things. – Laura Faulkner
Show Links* Public Speaking * Laura on LinkedIn * Laura on Twitter * Rackspace * Blink by Malcolm Gladwell * Applied Research Laboratories
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I just watched Martin Scorcese’s 2010 documentary “Public Speaking” a profile of author Fran Leibowitz. I honestly didn’t know very much about her, but she’s a smart and acerbic social critic, whose persona, and perhaps her life is built around her strong opinions and her skill in expressing those opinions. She repatedly emphasized in the film how there are too many books, and thus too many poorly written books; she’s pretty critical of the ability of most people to have an interesting story to share and to have the ability to communicate it in a valuable and valid way. While her career is indeed public speaking in one form or another, for the rest of society, she advises public listening. But really this is because she wants to be listened to, she doesn’t necessarily want to listen, she wants to talk. This is in no way meant to be an indictment of her – she very clearly exists in a world of her own making. While the mode of the times, and indeed given the work that many of us do, emphasizes the importance of everyone’s stores, and we put real effort into inviting people in and giving them space. I mean, this approach, this mindset is at the core of user research! But I found it remarkably refreshing to hear a different point of view.
It reminds me of an experience I had in the field last year interviewing a head neurologist, someone who was very successful, who had written multiple books and taught surgery around the world. Before this project, I was kind of warned to expect a lot of ego from surgeons especially, and I guess in some ways this particular interview subject was the platonic ideal of that. He would not sit through my carefully scripted concept presentation. He interrupted. He pointed out his bookshelf of published books, he told us about his children attending an Ivy League school. He gave examples of where he’d been influential, how many times he’d taught, how much money he had saved the hospital with his process innovations. I had to adapt my approach for him so that we could get through the session. After we left, my colleague, who worked my frequently in this domain half apologized and half-checked-in with me to see if I was okay or not, given how quote awful and difficult the interview had been. But I didn’t have that feeling at all. The interview was amazing! We learned so much from him! He was incredibly accomplished, and very very opinionated. And like Fran Liebowitz, a total New Yorker about it.
But I wasn’t having lunch with him, I wasn’t trying to be friends with him. I was trying to learn from him. And I approached him – as I do everyone I interview – as an expert. He was just more able to act in that role that most people are.
I am very sensitive to people who don’t listen to me, or who let their ego run all over me, or otherwise disrespect me. But this was an interview situation, and his so-called arrogance and ego was amazingly helpful and honestly didn’t feel disrespectful in any way, so it was just fascinating to see how someone else in the same session felt about it.
I’m glad there are people like Fran Liebowitz and this surgeon. They are in the world to do amazing work, to inspire people, to make a difference. That doesn’t mean I want to hang out with them and try to tell them about myself. But that’s okay, isn’t it?
So, we’re back with some more episodes. As of this recording, we’re in a particular bizarre period. This episode was assembled quickly, with the budget I have available for this, which is zero dollars. I’ve foregone my professional editor and my professional transcriptionist. You will hear some digital noise and distortion on my part of the upcoming interview, but it comes and goes, and I’m not talking too much, so I hope you don’t get too bothered by it.
It is a challenging time for businesses of all types, and certainly for small consultancies. It’s a challenging time for my own practice. I hope if you’re listening to this that you’re in a position to consider me as a potential collaborator for user research, training, coaching, and facilitation. Thanks!
All right, let’s get to my interview with Laura Faulkner, the Head of Research at Rackspace.
Steve: Laura, thanks for being on Dollars to Donuts. It’s great to speak with you.
Laura Faulkner: Thanks, my pleasure.
Steve: So I think it’s a great starting point would be to ask you to introduce yourself. Tell me and everybody what about your work and we can kind of dig in from there.
Laura: So I’m Laura Faulkner. I have a doctorate in research psychology focusing specifically on human computer interaction. I got that from the University of Texas about 10 years ago as I was already working in the field. As of just a few weeks ago, I celebrated 24 years in this field. So I was in it before it was a job. And when we were just doing it and practicing it alongside whatever else we were doing, especially in technology now I’m the head of research at Rackspace, which is rather fun in in a couple of ways. One is that it’s an enterprise organization. And we do really deep technologies essentially provide the internet and manage that for folks. So we’re doing really complex interfaces. And I love that I love that problem space with all those bizarre technical details and add lots of complexity to manage. And our to make that easy for customers who like are sitting in their chairs, working hours and hours every day, and often in high stress situations. It’s just really fulfilling. So I’ve been here about three years, three and a half years and our team works as as an internal agency really, for the entire organization of Rackspace. So we’ll do pretty much any kind of research for anybody who asks
Steve: That’s a nice position to be in or not? I don’t know. We’ll do research for anybody who asks you, how did that come about?
Laura: So within the organization, we were we sit alongside design and where we work with them and and we have some close partners with a content team. So we’re like always at the ready for them. But that ends up being just a fraction of our work. We will, will work for product to the any of the product managers or product teams that need to want to market test their product ideas, and the directions that they’re trying to go or to prioritize their products to discover demand in the market, what’s going to make things easy and best for them. And then we’ll do larger scale market research. We’re currently running a large scale market survey, because it it lets us flex to whatever questions that that any of right anybody in Rackspace has that is that where there’s unknowns and they can’t move forward to make a decision that we’re all about that we’re our mission statement is to spark fast, confident decisions. And so that’s where we come in. And sometimes it’s just to confirm unknowns that certain teams are having, or to break ties. We break ties a lot with the data and and also to support business cases based on what’s best for our customers and our users.
Steve: Stay over the 24 years. I was gonna ask about this later, but now you’re making me think like the arc of, you know, your, your experience in the profession over 24 years, but also your observation of what the professional looks like. you’re describing an interesting state. I think, you know, this this mission statement and the The positioning and the interest and the provision, I guess, to help the whole company kind of wherever this, this, you know, critical decision, information is needed. You’re there to do it. You know, how, if you look at that positioning where you are now and kind of what you’re able to provide? How, how would you describe the arc of just this one part of it? Like, who and how do we work with and what’s kind of how are we? I keep saying positioning, but but I guess that’s really what it’s about. How do you think that has evolved over 24 years?
Laura: That’s a great question. And I really I get really excited about this question because I see this as one of the possible futures of our profession. So we we began and even the DNA of the team I currently lead began holy just like as design testing kind of organization at the beginning of my career. It was really a unicorn position. It’s like there was the design and the research. And I, I like I have a special like passion for the research part because I have degrees in sociology, anthropology and psychology. And so I just love love love studying human beings and I love research methods is like finding that that question and applying the best method to it. So as the the need has evolved, and William say as the field and the practice have evolved as a as the need has evolved, for answers from people who begin to be seen, as reliable, as valuable as as somebody who can really help and help get the right answers, the freedom to apply the best method possible. Is is what’s so exciting about how we’re working today and I see this as that potential future one potential future for our field is the opportunity to drop it having to everything having to be specific methods and having this one toolkit that’s just user experience and open up to the broader possibilities of any kind of method that’s going to help deliver the best experience in the end. And and I’ve just having watched that happened and I also just, I landed in rather a dream position here at Rackspace in that I, I inherited a team that was very deeply educated, there’s masters and PhDs is is what we just happen to have. It’s not what scare anybody about that. That’s what you need to practice in the field because that’s not accurate, but because we had a lot of deep experience in a lot of different kinds of methods and the ability to understand applying a finding a best fit method for Whatever question is out there and whatever way that you need to prove that are convinced it’s true. It’s just really opened up our possibilities and it’s also accelerated our cadence. Because we don’t we get very little pushback on the back end once we’ve delivered our results because we’ve chosen a great method at the beginning that actually gets to the heart of the questions.
Steve: So in this this futures, that we’re talking about, where do these methods come from?
Laura: a broad range of of research capabilities for me, they came from my education in sociology, anthropology and psychology. So, all of those different approaches such as being able, the very fact that you can know how to ask a question well, means that you can put it in multiple different formats. Perhaps it’s a large scale surveys, because you need, you need a you need a shallow data set that just has a whole lot of yes and no answers upfront. Or maybe the best way to answer the question that you come up with is to have a deep qualitative interview where you generate a large number of, of what I call them vertical data points, which is that you create these deep, deep pits of thousands of unstructured data points. So, gosh, don’t remember did that? Did I answer your question? They’re asking me again.
Steve: Well, I’ll say I mean, you started to I mean, it’s where did the where did the methods come from? And I think your answer starts to beg the other question, as was what sort of the granularity of the phrase method? You know, are there millions of methods are there eights like? What is that what’s underneath the term method?
Laura: Wow, that is such a great question. And now I have to like take my take an expert mind and try to quantify it because it’s Something that after all these years I do rather naturally. And so I don’t know if you’ve read Malcolm Gladwell blink, but it talks about, experts can’t always put a word on how they do something. So let me see if I can parse that out. What do I mean by method? So we’ll take the problem, what’s the problem to be solved? Like from a business perspective, what’s the business value that needs to be delivered? We look at that from a project perspective. And then we then we drill down on that, well, what is holding you back? What are the unknowns holding you back? And maybe they just they need to know what are like to get some sentiment and they just need some words on some sentiment. Maybe they need to know about does this workflow work or does it break down and that would be qualitative tests where you’re watching people or a remote test, where you get a lot of data where you’re gonna, like recorded online, and you look at multiple people having done task testing and that sort of thing. And then maybe it’s, look, our CEO just doesn’t believe this because it it only has a few people and and he’s just really wanting to see the numbers. And so then we would convert whatever questions and methods we have up to. Okay, let’s, you know, let’s we’ll, we’ll do a survey, which by the way, surveys are hard. I think it’s one of the hardest kinds of research to get right. And it takes the longest it takes the longest to get it right before you release it, even though it’s fast on the data collection end and the analysis end. So does that begin to get at what you’re asking about methods? does it begin to codify What is it my head?
Steve: Yeah, it was. So maybe I can reflect back a couple of things. And then maybe I’ll try I’ll try asking you a question with hypothesis, which we might call leading, or you might call good interviewing. I don’t know, we’ll see how we go.
Laura: Sounds good to me. I love that you’re asking this by the way, love it.
Steve: Cuz I feel like the when you started off talking about the team and the, you know, the deep level of education and your own backgrounds, and then even you saying earlier on, like, Look, we do market research, we do all these things that sort of sometimes exist under other umbrellas. It seems like, you know, approach a in, you know, building up a super set of methods is look at a range of disciplines, whether they’re professional disciplines or academic disciplines. And you have, you know, your set to choose from just gets larger. I guess the, the leading question is, is was to ask what you think about inventing methods, generating new methods.
Laura: Oh, my gosh. Absolutely love that. And and it’s really necessary to this approach and and can get you like the best possible thing. So a great example of that recently, we we did a study where that we were studying a concept of domain trusts, which is where you have ad administrators have to hand over the permission to do administrative tasks to other people. And so what they wanted to know what would people trust this with? Would they would they be willing to do it? And so we, the, the original method, and the the, the, the stakeholder who came and asked for in that case was very adamant about how they wanted to do it. And in that case, we went ahead and rolled with it and then they will also wanted to like do some of it themselves and so we part of how we scale it. In the organizations that sometimes we will also mentor DIY research. And this person has done some sessions. And so we were willing to do it. So we went with it, but we weren’t sure that the task testing method was going to get to that underlying question of trust. They believed that it would come up naturally as they were going through and that if they really questioned it, and it was kind of a way that they thought it was a way to fake them out, if they really questioned that, that these system administrators would trust somebody else to have the permissions and then not just trust somebody else, but somebody else is somebody else that they might not know about. So like, I pass it off to to Joe and Joe passes it off to Mary, but I don’t know that Joe ever passed it off to Mary. And so so that that was the original method and it was one of those cases where it failed miserably. It looked it on the surface. It looked like they all trusted it. But it was very clear that they didn’t quite get it. And by showing them the interface in the interfacing eat seeming easy, because it was beautifully designed, the workflow was great. There, it was misleading. And we were quite sure that something was missing. And they came back and said, Well, this isn’t they didn’t answer the questions we needed. So we went into a room and we just brainstormed and thought through, okay, how can we get at the psychology of this trust thing, without showing them something tangible like or having them work through a tangible workflow that that’s beautifully designed to this easy so that we’re not testing the design. And so suddenly, we started, like mapping out on the whiteboard. And we came up with this method of creating diagrams of people and these apps. diagrams of people and messages and connections and created these stories around it. And we had them put themselves in the story. So we would show like this one box and there’s a little icon and it’s them and we put their name on it as they told us their name. And then we said, okay, who is somebody that you might turn over permissions to, that you might allow to? And so they would say that and we built out the diagram as they talked like, Okay, well, here’s these other icons, and these are, these are these other three people, you’re going to turn it over, so like your level B people. And so we went from there and continued the story and we had them tell us a story of their own organization, we would live map out the organization as they talked about it. And so then we got down to the really down to the heart of it. And we we had, like person number five in their in their first first degree connections. Say Jay Okay, you’ve Turn it over to them. Now they’ve turned it over, they have these, these now, third degree connections from you. And they’re turning it over to them. And they said, Well, that’s probably okay. Because they wouldn’t turn it over to anybody that wasn’t trustworthy and this and that. And they then they said the key word, and I, and I assume I would know who that was. And I’d be able to see who that was, because then I could go back and check and see who that was. And that was the key. That was the key that broke the whole thing open, which is that the truth was, they were not willing to have the mechanism work in that manner. Were without without their permission, though, that was another piece that they would often just volunteer and say, it’s like, well, I’m assuming that my first degree connections will come back to me and say, Okay, I want to grant permissions to this person, or you’re going to notify me that they’ve done that and tell me who and and we blew up in the whole model that way and It turned out to be exactly what they needed. So that was like live creating a model. And that we actually did live in an abstract way through the combination of these diagrams and the storytelling.
Steve: And were you showing the this beautifully designed interface in this round of sessions?
Laura: We, we did it at the very end as a like a bonus piece, like, okay, now given that, and then what we did was we said, okay, now assume that those problems were solved. Now, let’s experience this. And so we that we just got extra data for them about the about the design in that process.
Steve: Can you say a little more about, you know, going into the room with a whiteboard and thinking through how like, you know, the process of creation of this new method? You know, and maybe that’s another unkind question for me, but I don’t know. Can you reflect on that process?
Laura: Not at all, not at all that that really begins to get to it. So this is where you have some understanding of how a human is built and how a human works and thinks just really works and comes in handy. It’s like and then also understanding one of the key aspects of, of any research design in any research method is being able to, to know that you’re measuring the right thing. And so how do you get to that you’re measuring the right thing and you’re getting at what the actual human behavior is going to be. And so that particular day, it’s like what we were looking at is like, okay, we knew that showing the interface was getting in the way of getting at where where’s that fear button in there? It’s like we were trying to see Is there a fear button in there and have we pushed it and so it this was, this was a combination of like the the minds in the room that understood how human beings work. We have a couple people with Human Factors degrees, and and then others are just, you know, have experienced so much experience in the user experience field. And then just some common sense is like About what? what would what would get at that that fear that concern that which is what we’re actually trying to measure. So that’s really the big question there is like, what are we trying to measure? Looking at your various methods and saying, Okay, if we use this method, whoops, this thing might get in the way. Or if we use that method, this thing might get in the way. Just like when you do surveys, the order of your questions can change the results that you get. So there’s one bit of guidance that I like to apply, which is that in a survey, you don’t ask the demo, or if you can avoid it. You avoid asking the demographic questions up front that have you telling them about themselves, because that can actually bias how they think about what they’re getting ready to do because you and they have Suddenly put them into a certain box and they’ve stepped into that point of view, you can get us may be able to get a slightly more pure response. If you if you just dive in and do the, the the other the basic questions that you’re trying to get at first and then add the demographic on at the end. So yeah, that’s that’s some of how we do it is make sure that that you’re getting at what you need to get at and that whatever you’re doing is not getting in the way of getting at what you need to get at. Mm hmm.
Steve: And then part of the story, I think is, you know, is your collaboration with the the other your stakeholders, your clients, I mean, however you kind of label them who, who have something in mind who have a goal in mind for this research. And so how do you how do you have that conversation? You know, maybe if we, you just told the story, but like, maybe we could talk about what happened before the part of the story that you’ve told But how to how to projects like this get formed.
Laura: That that is absolutely one of my favorite favorite things. So after many years of pain of, of studies like that, that again is a really good example of one where this happened and that that we didn’t do it. We didn’t set the foundation well up front. For, for both what was really needed we believed we’d gotten it that but turns out as we had the conversation after the first study failed, we we we went back to our disciplines conversation. So after some years of having those kinds of experiences in my career of studying, not getting what they needed after we did all of this work, is that I now ask I have a series of questions that I ask up front, it’s a weakness as we call it, our research intake process. We do it in about 20 minutes. So anytime somebody comes to us for research Was it? Oh, that’s great. We’re so excited about what you want to learn. Let’s have a 20 minute talk. And that’s we keep it really simple and lightweight for them. And there are six areas that we ask about and drill down on the first really being the most important one. The first and most important question, and I’ll drill down into each of these in a little more detail is what problem are we trying to solve? So a lot of times they’ll, they’ll think that they have one thing that they need, and turns out there’s another so what problem they’re trying to solve? Who is asking them to do it? Who are they accountable to? Because that can change the questions too. Are there any constraints to the to like the solutions or anything? Is there? Is there something that that’s in the way is like a technical constraint? Or is there a business constraint? And then are those negotiable? Are there any risks? What do we know or already have because sometimes they don’t need a test at all because they already have it. And then what is the timeline on this? And are there dependencies in that timeline? So, so those are those are our six, what What problem? are we solving? Who’s asking you to do it your stake your stakeholders? constraints, risks? What do we know? Or have? And what’s the timeline?
Steve: Yeah, that sounds like that’s 20 minutes to, to cover that.
Laura: Yeah, we’re really good at it now. I think our first ones maybe took an hour. But we’ve really got it down. Because the first one we are able to get down to, really to the heart of it very, very fast. And it’s the single most important if we get to nothing else, the one that we have to have in terms of a conversation that you can’t just ask over email, you have to really hold their feet to the fire is that what problem are we trying to solve? So it will sometimes have people come in and say, well, we need to deliver this new ticketing application or we need to make improvements to that ticketing application, or we need to test our design. So that’s not really the business problem that you’re trying to solve. That’s, that’s what you’re wanting us to do is is test those or, or that’s what you’re going to deliver. But what is the business problem that that application is trying to solve? And so in this domain trust case, it’s like we we needed to get down to what their real concern was not was, would they do this? Would they be willing to do it? And you know, is does this workflow work? It was more, does this frighten them? Is there something about this that they’re worried about? That could go terribly wrong? And then so we really have to hold them to to that on the way out, and what business value does this deliver? What are you trying to do? Who are you solving it for? So there’s kind of three questions and what problem we’re trying to solve. what problem are you trying To solve, that’s the that’s question number one, who are you solving it for? So are is it for a certain kind of customer or an internal person or a certain kind of user? Who’s the what’s that persona? And then what is the desired outcome? What do you hope will happen not as a result of the test, but as a result of actually, like releasing this thing that you envision that you want to release out into the wild? So if we do nothing else, we’re really going to get them down on that. And they often think they can do that in a in a one line email, and then they find out in the conversation that it’s not quite that simple. I think one of the best compliments that we’ve had is that we have one, what one repeat customer, he’s a product manager who comes to us all the time. And he said, Okay, so I need you to ask me those questions that make product managers cry.
Steve: Is that just take those All these questions are is where to where’s the tiers for them here? That problem one seems really interesting.
Laura: Yes, that’s definitely that’s the first and biggest. And if we get to nothing else that one can deliver all sorts of value, then we’ll also you know, then we also get them to this, the stakeholders who are who’s asking you to deliver this because what they really need can often change with, they may be driven by something external. One of the frustrations that we sometimes have, when we don’t ask this question is that we’ll go and do this and deliver it. And they’ll go a certain direction, but really, they or they don’t believe you or they won’t take our they won’t, they won’t. They won’t take our findings and they won’t do anything with it. And we find out after the fact, oh, their boss asked them to do this specific thing. And so if we know both who what, who they’re accountable for like with this product manager, it’s like After a while, it’s like I got to know who his boss was, or whether the his boss or whether the engineers were asking him to do this thing. And based on those two things, we could structure the study where it would actually get the answers that would convince that person. So we actually do that skip over of our stakeholder by asking them who’s your stakeholder, and then we arm them with the data that will help them be convincing to their other to their person. So that’s, that’s really another layer.
Steve: And you mentioned this, but I feel like it bears reiterating that. I mean, none of these questions or anything about the how we will approach this and, and, and I think, and I think you said this as well, that’s often the starting point for people who approach you, right, we need to do step x, or task y. And you’re, you’re setting that aside and asking a lot of very specific whys and how’s and whens and who’s That’s correct.
Laura: So yeah, this this is all developed not just out of pain in user experience testing over the years, but also out of having gotten to be a program manager and being able to think from that space and in various times in places where I had engineers as well as as user experience and other folks working with me and for me, and and so it was really this this concept of the, who are we accountable for? What are the constraints? What are the risks that that sort of thing does actually come straight out of business management experience? But now we’ve armed researchers with Yeah,
Steve: I can’t help but think that there are, you know, either small teams or you know, research teams of one or just lots of researchers who their role is very much To take assignments, and that may be a methodological assignment or problem assignments. And you’re describing, you know, a context for research at Rackspace, where that dynamic has, has changed entirely and you’re there to, you know, I guess work as a partner and, you know, former foreman approach together, you’re guiding them, you’re facilitating, you’re not being directed. Go do this. So we need it tomorrow. If I’m characterizing where you’re at, right now, I’m wondering if you have thoughts about, you know, how does, is there a way to get from A to B, like, you know, if A is Yeah, delegated assignments and B is facilitative, empowered partner who’s you know, who has a mission statement? What is what does it look like to go from one to the other?
Laura: So, so we have found that even our most beginner, Junior researchers are able to have this conversation because one, it’s kept very short. And we don’t reveal these things. It’s like we asked them more from that empathetic perspective, like Rao, what do you what do you need to do in this? And what do you know about this? Just as a discovery, it may not change a whole lot of what you do, but it can provide really valuable context for what you do. So somebody may have asked you here, can you do a usability test on this really quick for me? And, and yeah, we just need you to run this and run these workflows to go ahead and ask these questions up front. It really builds out the goals and objectives sections of your test plan, so that you get to a real goal of what is the ultimate goal from of this thing that you’re delivering, it could actually end up changing you Your script and how you ask those questions. So even if you’re a very frontline researcher who’s just doing what people asked this, this can really inform your your test planning and give it great context.
Steve: It’s interesting to think about that case, you sort of not, like you said, You’re not really revealing how this works. It’s not Well, before we agree to do the usability tests, we’re going to we need to ask these questions. It’s it’s kind of a yes. And that’s great. Tell me Tell me more. Tell me more. Tell me more.
Laura: Yes, it’s taking what we do from a we asked great questions, perspective and from but from this this at the beginning planning point of view and place In the fight in the study,
Steve: yes. And would there be a scenario where, you know, this frontline researcher is being asked to do usability tests, ask these questions. You know, might they identify a different approach? And if so, what would they do?
Laura: Absolutely. And so this is where you really get to shine is where and and, you know, because a lot of what we’re about is keeping people from making really serious mistakes and delivering things that don’t work. And so this is where you really have that opportunity to, to guide encourage mentor your stakeholder, even if they’re bigger than you by by saying, okay, so if we, if we do that, that user experience test, here’s what might happen is that you would get this and these kinds of answers. But if we were to test this this other way, and it’s good to kind of build it out. First thing and get an idea in your head, we don’t typically reveal the method at that time, we typically have them know the method within about an hour after this after we’ve said and thought about it. And sometimes we know instantly and we’ll reveal it, but rarely do we actually reveal the method in that situation. We’re taking in we’re being a sponge, and we’re encouraging them to think in a different sort of way. And, and, and it’s only it’s then when we’re looking at how that might work and and how it might have a method that they were asking for isn’t going to quite work. And that happens a lot like when we have people come to us say, we need a survey. And we we did there were times when we used to just say well, actually, you probably don’t need a survey, that that didn’t work, telling them that they’re wrong. And redirecting them without context didn’t work. hearing them first. So by asking these curious questions and diving down, honestly, we found those people really enjoy the conversation. Because when they get to talk about their work and their challenges, who they’re accountable to, and what success looks like for them in this project or this effort, and sometimes they’ll actually walk out of that those just those few questions, thinking about their own work a little differently. So that’s the first part is that we’ve listened and we applied empathy to that, and we’re very kind when we do this, this experience experiment and this experience and so that’s, that’s really the first foundational piece is that we’re, we’re giving them an opportunity to speak about things that they nobody thought to ask them. And so that’s, that’s step one in the success measures of this and then second is that then if we develop another method, that that that we then go have another conversation like okay, so looking at this, it looks like you’re This is going to get you better what you’re going to want and they may or may not buy that No, I just need this, this run like this. It’s like, okay, run an experimental session, run a pilot session with both and see how they work. Sometimes the stakeholders are right. And I think we’re, we’re more on point most of the time simply because that’s, you know, that’s our superpower. We were the were the methods people. But there are times when, when we thought it needed to be a certain way, and we were wrong. But when we are more effective when we’re delivering something more effective, we can show it, then we have this opportunity to do it. In that particular case with our domain trust case study. It was simply that we went back and told them we didn’t say, okay, you all screwed up by saying, making us do this method and all of that we actually took it on ourselves. We said, Look, we we heard you, we believe that that maybe there should be a different approach. We and we didn’t follow that up strongly. Would you be willing and we’re going to try this other experiment, we’re going to go do these other studies, and we’re going to see if we can get you exactly what you need. We’ve developed this other method. And so in that case, they felt the pain and they were willing and by the time we got there, they were like sponges with with with our data. And yeah, really answered the question. So that’s a that’s a Those were the steps, which is that empathetic listening upfront and really using it the conversation as an opportunity to discover and to hear. And then, and then, if they’re insistent, go ahead and do it, perhaps run a pilot of the alternate method and see if those look different and if that gets them closer to what they want. And then if there is a fail, then you have the new method in your back pocket and kind of apply it at that time.
Steve: You know, I have this not particularly brilliant theory, but I guess this observation that user research, given who’s in the profession, how it works. It’s a helping profession. You know, the way that reference librarians are or you know, some medical people or something, and I mean, I hear that in your story that that, that this is not about being adversarial, towards whatever client stakeholders people in other departments or disciplines, you’re, you’re really approaching them to help them and it becomes you just it’s framed very collaboratively. It seems to me that, that starting point of that kind of philosophy, like you said, of being very kind and using the empathy and we use those words in research all the time, it sort of gets thrown around a lot, but when you describe how it’s implemented and how it’s sort of a philosophy to live by and work by, it’s really exciting to see how that creates a working relationship that doesn’t have a lot of the a lot of the challenges a lot of researchers raise in their own Work, you’ve kind of created a culture or context of cooperation that just is flavored very, very differently because of that.
Laura: That’s That’s right. And thank you for seeing and recognizing that, that that is how that works. And we actually have we have one more like, one of our superpowers in our toolkit for that, is that after we had a big push back situation, where product managers got really upset by our results, and then we found out that that he was under other pressures. And that was that we didn’t even know about and the fact our results like they were correct, and they were right, and they needed to be applied and used and they knew that but it they they just had a real reactive situation to it. And in that case, this this is on another project than that one. In that case. We really see sat down. And we did a post mortem ourselves about that interaction and how that there was some unfortunate Bad Blood really created between us and that, that person and and he was just really wounded for a while. And so we, we set but we were bright, but we were right about it, they and they needed to do it and they and they have to do this that they and they did they applied the results. But still it was like we had that relationship fail. And so we we wrestled with this for some hours and an offside over my kitchen table one day. And we finally came up with so we were what we’re doing is like coming up with with core value approaches that we could apply to to solve a situation like this and to respond to it. And and even before that, how could we seek from not creating it in the first place. And the big thing was, is that we wanted to do that without giving up our power, our authority. as researchers and our knowledge, so it wasn’t just about caving and saying, okay, you’re right now our data is wrong or whatever, or you can take it or leave it or whatever it is that that we might approach. It’s like, Okay, what can we do in a situation like this that’s proactive without giving away our authority. And the two words that we finally came up with, which are our watchword, our bill, whether we’re be curious. Be curious. That is one of our research superpowers is that when a stakeholder is pushing back on our results, or on our method, or anything about that, we might start getting defensive on the inside. And we will stop and recognize that it’s like, okay, we don’t want to give up our authority as the methods experts as the data experts as the findings, experts. But in that moment, what can we do? It’s like, oh, there may be Something more going on for this person or in this situation that we simply don’t understand. And there’s a reason why they might be pushing back in that way. And so we turn it around into a question. It’s like, wow, that’s it, you know that that seemed hard for you What? What is that about that? What are you afraid will happen? Is there somebody that might come back at you or push back on you? Is there a deadline you might not make? Is this going to make things more complex? So we we pause and and seek to be curious, and that has made a huge difference in our stakeholder relationships. So even the most basic researcher can be curious about the people who are asking them to do something. It’s a simple tool.
Steve: That’s really wonderful. It reminds me of so many of my own experiences where I have an interaction that is weird or uncomfortable or unpleasant, you know, even a minor thing and I might Hold on to it for a while and then try to have that realization that it’s even though I’m experiencing it, it’s probably not about me. Right? There’s something going on. And so, I mean, I think your team is, has got a really lovely framing for it. And then we can take the opportunity to answer that question like find out be curious, what is that thing that’s going on to, I would imagine there’s some self care in that because it sort of proves that they don’t hate you, or they don’t think you’re stupid or whatever the thing we do to ourselves when we get those signals, but also back to your kindness and your framing around helping if you understand what the issue is for that person, or even giving them the chance to articulate it, maybe they haven’t articulated it before. you’re providing value in every direction and a number of different levels with that with that mission.
Laura: Thank you. And that that that is correct. It’s like having that, that opportunity to speak about those. And we learn so much more about the business and about how it works and or about the technology and how everything works or the product direction by doing this. The other element of that that was starting to talk about a little bit was that retaining your own power in this. So we also in that same offsite session really looked at, okay, how do we retain and keep, you know, so yes, we’ve listened, we’re curious now. And the data is still the data. It’s like the answer is still the answer. And so there’s a couple of things that we do is that one, we’ve now taken the heat off of that by by using this phrase, the data so the data is telling us, the data is telling us and so instead of saying Yeah, but the users are saying this or we found this in the study, and this was this was the the output and whatever it was that and really starting to get our backup about that. We we use that more generic and in really, like research based answer of, yeah, I hear you with all of those challenges and things. And this will be complex. And we just want you to know that the data is telling us this, the data is telling us this. And that way, it’s it’s agnostic, and it doesn’t say you have to go do this. It’s like we also the third thing we do is we treat them as grownups who are accountable for their own jobs. And so the ultimate decision on what to do we leave to them. And so then the final thing that we do other than So, you know, saying this is what the data is telling us, and then leaving Leo letting them giving them the courtesy of making their decision. And and living with that, whatever it is, the last thing that we do is we retain our data. And we will, sometimes we will even take special care after a study is done on one that somebody is pushed back on. And we’ll make it even more attractive and pleasant and more complete of a report. So even if we’ve done like just a quick turn report will actually turn it into something more formal archival. And then we save that. And I’ve actually had as a situation with this actually happened more than once. But there was a large one that for a large corporation that I was consulting with. And this was a large scale refactor of the main navigation of their whole site. And it was intended to increase click throughs to their product sets. And we tested this and we actually traveled nationwide tested, you know, 18 year olds to 80 year olds and, you know, all walks of life and everything. And so we gotten all this stuff. And the data told us that there weren’t these critical fails in this and that it was it was going to fail. And they were they were on their timeline. There were business reasons they wanted to release it. And so they they rushed and did it. So, you know, we told them here’s the findings. We let gave them the courtesy and and here’s the bad things that might happen. We gave them the courtesy of making their decision they did. And then they went back to look at the at the analytics and discovered that not only did click throughs not go up by the 30% that they estimated. The click throughs went down by 30%. So something had gone drastically wrong. And they spent like the first 18 hours or so scrambling with technology and all sorts of things and all sorts of finger pointing and fight since then somebody said maybe we should ask research and so they came back to research and they said Do you have any ideas about what may have gone wrong? We said yes, actually. So we we documented in this report. And here are the things that that were risky that that you can look at. And they looked at those as like, well, there was some dissent in the organization about how did these get through, we didn’t go there. We just gave them the data, the information, we helped them interpret it helped them come up with designs to fix it. And they did fix it. And within 36 hours, the click throughs had not only corrected, but they had gone up the original 30% that was intended for the project. And we never went back to that we never said I told you so we lead with the data says tells us this, gave them the courtesy of making the decision and then coming back and being ready in case the risk the bad thing does happen. That that we we gave it to them without judgment and and say yeah, here’s how to fix it. said this is this is not only very humane it’s great business
Steve: there’s something here about you know, when are the moments when people are willing to hear like we are we are sometimes change agents we are sometimes telling people things that yeah the break their worldview.
Laura: Yes, that’s a great way to say it.
Steve: So, you know, you can sometimes you can choose your moments and I mean, in that story, you couldn’t really choose your moment it was kind of the moment came to you and you were you were prepared to respond. But your earlier your earlier attempts to to influence and sort of change minds. I mean, really didn’t didn’t land but you were ready when the when it came back around.
Laura: That’s correct. And most of the time where the the research results are supported, And the bad thing does happen. And then sometimes we you know, because we’re intentionally observing this, and we’re really going for high quality for our customers and end users. Almost every researcher has discovered that there are times when it maybe that bad thing happens, but it doesn’t have as big of impact as we thought it was. Would if they didn’t, that was with somebody not fixing something. And so sometimes, yeah, sometimes that happens, too. And, but most of the time, that’s, that’s what we’ll do. And actually, we we do find because we’ve had that in tech conversation up front. We, we have the tech conversation upfront, we review test plans with our stakeholders before we run them, and we invite them to the sessions so that, you know they’re in there too. We’re very transparent. We even put all of our working documents in shared directories, even our working reports and things we don’t hide anything from them. Most of the time, they don’t go look at the details, but the fact that we’ve made it transparent is created a lot of trust. And then ultimately, you know, sharing the data with them most of the time, they’re convinced because we’ve set up such a good foundation up to there. And then when they’re not we have this failsafe mechanism for for how to handle when they don’t take the advice.
Steve: In US, you said a couple times, no, you’re treating them like adults, the decision about what to do or not to do is theirs.
Laura: Well, yes. And it’s not just that it’s like recognizing that, that there’s other drivers that that maybe they’ve not articulated in all those conversations, or maybe they haven’t even identified themselves. We just don’t always know what those are.
Steve: And I think you’re surfacing. You know, what is what does success look like for researchers? And you’ve, I think you’ve articulated in a specific way that I haven’t necessarily heard that there’s a lot of there’s a lot pressure we put on ourselves as a practice to have influence in you know, in that word means so many things, but one of those is people doing what we think they should do. But saying, oh, that that they’re adults and that they’re going to make their own decision and there’s factors that we don’t know. right i mean even your mission statement about where you use the word spark is kind of what you’re doing. I mean spark spark plays a certain role in a fire being led or not led but it’s it’s right there kind of outside to make it happen but it’s not the fire right the fire is the is the next thing that happens.
Laura: Yes, I’m so that’s the spark fast, confident decisions, because that’s what we’re about. And so that’s actually another the fast part is another element of the method is that sometimes we need a faster method to to get at at the same sort of thing. So that would be another consideration. But ultimately, yeah, it’s like we’re trying to find where is the unknown in the audience. is keeping them from moving forward and moving forward to the right thing in the right direction. And yeah, once we’ve discovered that we, by setting it up that way up front, and having that in mind that, that we have more success influence on the back end.
Steve: When you spoke earlier in this conversation about, I think bringing tools and mindsets from business management to research, because we’ve, that’s what we’ve talked a lot about, right is sort of the management of making research, effective and successful inside your organization with all of your colleagues.
Laura: Yes. So the two pieces I brought specifically from business were the constraints and the risks, because those are two things that you really have to well and then what is the business outcome when we ask about the desired outcome in the problem solving conversation, and question we’re going for what is the business outcome, like no, not that just that we need to release these ticketing changes. It’s like, Why? What do you hope will happen as a result of those? So that’s one aspect is what’s the business outcome? Or the organizational or whatever it is that, you know, that’s driving the outcomes, the bigger ones? And then the constraints. It’s like, what are the technical business constraints? And are they negotiable? It’s like this is that that’s comes basic, from basic business management that really anybody can begin to can learn to understand and begin to understand is that there are sometimes there’s just you know, there’s there is a hard schedule or budget constraint or legacy technology constraint. We all hit that with a lot, don’t we? But then then the question to ask is, you know, not not just to say, well, that’s just going to work and you can’t give them what they want. Before you go there, you say, or, you know, give them what they need. The users are going to have to have this bigger investment you say, okay, is that negotiable And to what extent is that negotiable, that technology or that, and then that you, you begin to get them thinking in those other directions. So then then the third one, then is is the risk. So I use a basic risk management model, which risk risk assessment management model, we asked what are the potential risks of the of this solution? What is the likely you know what bad things will happen? So, that’s a simple way to ask it, what what bad things might happen if this gets released? Because a lot of times when they’re asking you to test something, they’re not necessarily thinking about the the potential implications of it. Or maybe that’s what they’re actually trying to get at in the test. They just didn’t really say it. It’s like, we’re afraid that this workflow will, will break down at this point. It’s like, well, what bad things will happen? What’s the risk? What, what’s about bad things would happen because they don’t always articulate that part. They’ll just say that it won’t work. And then Okay, what’s the likelihood of that is that really Something that would happen, is that a high medium or low likelihood that it’s going to happen? And then what? What would be the impact? It’s like, okay, yeah, that bad thing might happen, like, there’s this usability problem, and it would be there, and they’d have to work around it. And they would still have another way to do it. So that would be a low impact. There’s not a big, yeah, the bad thing might happen. But the bad thing that will happen as a result is is really not so huge. Or they might have a data loss or the system will crash or somebody will die like that those those would be really high impact. So once we have that high, medium, low likelihood, high, medium low impact, and we know what that risk is, then we can have that conversation. How do we mitigate that? So how can we mitigate that and sometimes I’ll actually change their solution live while they’re talking to you simply by asking these questions. And sometimes we even ask ourselves these questions about our own studies. It’s like is there a risk of this study so so we asked the the risk The likelihood high, medium low, the impact high, medium low. If both of those are high Oh, then we may need to add more participants cuz, because if this is a really risky thing, we need to find all possible capability reasons that are possible failures. It’s like maybe the first 15 people won’t reveal it. So we need number 16. So if it’s a really high high impact that that’s one of the decision points it tells us is we need to add more participants potentially, if it’s a really low impact, sure, we might be able to get feedback from six, we might and we only get feedback from three or hallway testing if it’s a low likelihood, low impact. And then how can we mitigate that? How can we fix that? So? So yeah, it’s it’s a great little conversation to have, and it’s not really complicated. Risk Management sounds like such a big, scary business II thing, but the risk, the likelihood impact and mitigation. It’s really rather rather simple. It’s so valuable
Steve: not just to step back a level a little bit, and we started off, you know, you talked about your, your, your degree in research psychology and you mentioned sociology, you mentioned anthropology and by the end of the conversation, we’re talking about business. Business. And so you know, for you in your career, like where did that come from? How did you pull that in, in addition to all the other things that you’ve kind of incorporated in from the social sciences? Where did the business stuff come from?
Laura: By taking opportunities that came along and by being curious, so that specifically for me how that happened was that I was in a defense consulting organization with the University of Texas at Austin, the Applied Research Laboratories, a very old and respected laboratory that started with the Navy and then expanded to other services. And there was a there was a pitch that was We got to do just people coming in. And I thought, well, I want to pitch user experience to them. And so I managed to weasel five minutes in with these people who were looking for high value, things that they could invest in toward this specific effort that they wanted. And have that whole day and the little five minutes, I was able to wrangle out of somebody else’s 30 minutes session, because I came in late to the game. They were lit up by this whole user experience concept, and it exactly matched on what they needed. And so because they had a need, and I presented it, I brought the contract and so I started asking my boss, how do I do this? How do I get a contract? What do we do? And so I started learning about the business aspects just because I went put myself out there. There was an opportunity, I’ll tell you, I was terrified. But it was it was like but I was also like, thrilled exhilarated like being at the top of the hill on a roller coaster, and the best way and So I took that on and and by doing that I began to, to have to learn that sort of thing. So that was the start of it. But the really big pivot was then when, in addition to that I, they they had a project that had a manager who left and they needed somebody just to pick it up really fast and keep it going. They thought it was going to die in six months, because it was not going well. The stakeholder wasn’t happy. So again, I was curious, it’s like, Okay, why is the stakeholder not happy? Well, it’s because things were not delivering on time or and it was like what all these requests for like a year overdue there were $100,000 over budget, and they just didn’t see the value anymore, except for this one guy doing little ad hoc fixes for them. So I started looking at their contract and seeing what they signed up for and discovered that, that we were doing a whole lot of things that were out of scope, but we weren’t doing the things that they signed up to And it was really simple way to begin to learn business and budgets and, and to that sort of thing was I was just curious. And I just started asking the question like we do as researchers, why, why? Why is why are they unhappy and I had empathetic conversations with them. And so when I when I started having those empathetic conversations with the people who were in charge of these projects, and even as a researcher start asking them well, so why are you concerned about that? What that what’s that about? What kind of pressures are there? That’s where I began to learn about this and in in, like, I go out, try to find, well, how do you manage this and this sort of thing, and I paid attention when they somebody did a risk study at some point and I was like, Oh, what’s that? So I went in and said, Can I listen in on your on your post mortem, you know, on your on that risk that happened, that bad thing that happened? And I that’s where I learned those those basic steps. Even though they weren’t basic in that situation, but because we’re user experience people, to boil them down to those basic questions, that was how I did it. That’s how it’s in this one tiny little part of this conversation and that really anybody can do it. So it’s like taking the best of who we are as user experience people, and especially as researchers, which is paying attention, asking questions and boiling down things to their simplest, simplest forms, and then applying those to make a difference. I’m really proud of myself. I’m glad you asked all these questions.
Steve: You know, we keep coming back to how be curious. It’s just as a mantra for good collaboration and for developing skills and you know, we we bump up against all kinds of stuff that may be intimidating. But you’ve, you’ve showed how to learn by being curious. And as you said, you know, paying attention and boiling it down. So all the skills of a researcher being applied to all these different contexts, it’s a really it’s a it’s inspiring, and I think but also very actionable. I really like it for that.
Laura: Yes. And then I, I’ve never just sat and done just what I was asked to do. It’s like I’m always looking for something new. Something else is probably just part of how I’m built but it’s also a conscious choice of, of just doing my my current job is it’s simply not enough for me and that even just that extra 10 minutes of curiosity or desire to see something else or learn something new while still doing my job and delivering on that, that that that’s just opened so many doors And, and and, you know, helped me step into a lot of opportunities and learn new things.
Steve: Is there anything we didn’t talk about today that we should
Laura: we could talk about all sorts of things. So that this this is a lot. And this is really a lot of the heart of of how my own practices developed, how I would love to see the field develop just in terms of, of one, you know, that that applying the best possible methods in being free to use any or invent them based on the real questions and to know those real questions to ask up front. And then how to treat people along the way. And grab the opportunities along the way, I think. Yeah, that’s that we’ve summed up a lot of my practice really, really well. Well, it’s, it’s really beautiful. As I said, I find it really inspirational. So thank you, Laura, for being so open and sharing so much good information. I really enjoyed the chance to speak with you today.
Laura: Oh, thank you so much for asking. It’s my pleasure indeed.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit portigal.com/podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 28. Laura Faulkner of Rackspace first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Laura Faulkner, the Head of Research at Rackspace. I’ve never just sat and done just what I was asked to do. I’m always looking for something new, something else. It’s probably just part of how I’m built but it’s also a conscious choice of, of […]
The post 28. Laura Faulkner of Rackspace first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Laura Faulkner, the Head of Research at Rackspace.
I’ve never just sat and done just what I was asked to do. I’m always looking for something new, something else. It’s probably just part of how I’m built but it’s also a conscious choice of, of just doing my my current job is simply not enough for me and even just that extra 10 minutes of curiosity or desire to see something else or learn something new while still doing my job and delivering on that, it’s just opened so many doors helped me step into a lot of opportunities and learn new things. – Laura Faulkner
Show Links* Public Speaking * Laura on LinkedIn * Laura on Twitter * Rackspace * Blink by Malcolm Gladwell * Applied Research Laboratories
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I just watched Martin Scorcese’s 2010 documentary “Public Speaking” a profile of author Fran Leibowitz. I honestly didn’t know very much about her, but she’s a smart and acerbic social critic, whose persona, and perhaps her life is built around her strong opinions and her skill in expressing those opinions. She repatedly emphasized in the film how there are too many books, and thus too many poorly written books; she’s pretty critical of the ability of most people to have an interesting story to share and to have the ability to communicate it in a valuable and valid way. While her career is indeed public speaking in one form or another, for the rest of society, she advises public listening. But really this is because she wants to be listened to, she doesn’t necessarily want to listen, she wants to talk. This is in no way meant to be an indictment of her – she very clearly exists in a world of her own making. While the mode of the times, and indeed given the work that many of us do, emphasizes the importance of everyone’s stores, and we put real effort into inviting people in and giving them space. I mean, this approach, this mindset is at the core of user research! But I found it remarkably refreshing to hear a different point of view.
It reminds me of an experience I had in the field last year interviewing a head neurologist, someone who was very successful, who had written multiple books and taught surgery around the world. Before this project, I was kind of warned to expect a lot of ego from surgeons especially, and I guess in some ways this particular interview subject was the platonic ideal of that. He would not sit through my carefully scripted concept presentation. He interrupted. He pointed out his bookshelf of published books, he told us about his children attending an Ivy League school. He gave examples of where he’d been influential, how many times he’d taught, how much money he had saved the hospital with his process innovations. I had to adapt my approach for him so that we could get through the session. After we left, my colleague, who worked my frequently in this domain half apologized and half-checked-in with me to see if I was okay or not, given how quote awful and difficult the interview had been. But I didn’t have that feeling at all. The interview was amazing! We learned so much from him! He was incredibly accomplished, and very very opinionated. And like Fran Liebowitz, a total New Yorker about it.
But I wasn’t having lunch with him, I wasn’t trying to be friends with him. I was trying to learn from him. And I approached him – as I do everyone I interview – as an expert. He was just more able to act in that role that most people are.
I am very sensitive to people who don’t listen to me, or who let their ego run all over me, or otherwise disrespect me. But this was an interview situation, and his so-called arrogance and ego was amazingly helpful and honestly didn’t feel disrespectful in any way, so it was just fascinating to see how someone else in the same session felt about it.
I’m glad there are people like Fran Liebowitz and this surgeon. They are in the world to do amazing work, to inspire people, to make a difference. That doesn’t mean I want to hang out with them and try to tell them about myself. But that’s okay, isn’t it?
So, we’re back with some more episodes. As of this recording, we’re in a particular bizarre period. This episode was assembled quickly, with the budget I have available for this, which is zero dollars. I’ve foregone my professional editor and my professional transcriptionist. You will hear some digital noise and distortion on my part of the upcoming interview, but it comes and goes, and I’m not talking too much, so I hope you don’t get too bothered by it.
It is a challenging time for businesses of all types, and certainly for small consultancies. It’s a challenging time for my own practice. I hope if you’re listening to this that you’re in a position to consider me as a potential collaborator for user research, training, coaching, and facilitation. Thanks!
All right, let’s get to my interview with Laura Faulkner, the Head of Research at Rackspace.
Steve: Laura, thanks for being on Dollars to Donuts. It’s great to speak with you.
Laura Faulkner: Thanks, my pleasure.
Steve: So I think it’s a great starting point would be to ask you to introduce yourself. Tell me and everybody what about your work and we can kind of dig in from there.
Laura: So I’m Laura Faulkner. I have a doctorate in research psychology focusing specifically on human computer interaction. I got that from the University of Texas about 10 years ago as I was already working in the field. As of just a few weeks ago, I celebrated 24 years in this field. So I was in it before it was a job. And when we were just doing it and practicing it alongside whatever else we were doing, especially in technology now I’m the head of research at Rackspace, which is rather fun in in a couple of ways. One is that it’s an enterprise organization. And we do really deep technologies essentially provide the internet and manage that for folks. So we’re doing really complex interfaces. And I love that I love that problem space with all those bizarre technical details and add lots of complexity to manage. And our to make that easy for customers who like are sitting in their chairs, working hours and hours every day, and often in high stress situations. It’s just really fulfilling. So I’ve been here about three years, three and a half years and our team works as as an internal agency really, for the entire organization of Rackspace. So we’ll do pretty much any kind of research for anybody who asks
Steve: That’s a nice position to be in or not? I don’t know. We’ll do research for anybody who asks you, how did that come about?
Laura: So within the organization, we were we sit alongside design and where we work with them and and we have some close partners with a content team. So we’re like always at the ready for them. But that ends up being just a fraction of our work. We will, will work for product to the any of the product managers or product teams that need to want to market test their product ideas, and the directions that they’re trying to go or to prioritize their products to discover demand in the market, what’s going to make things easy and best for them. And then we’ll do larger scale market research. We’re currently running a large scale market survey, because it it lets us flex to whatever questions that that any of right anybody in Rackspace has that is that where there’s unknowns and they can’t move forward to make a decision that we’re all about that we’re our mission statement is to spark fast, confident decisions. And so that’s where we come in. And sometimes it’s just to confirm unknowns that certain teams are having, or to break ties. We break ties a lot with the data and and also to support business cases based on what’s best for our customers and our users.
Steve: Stay over the 24 years. I was gonna ask about this later, but now you’re making me think like the arc of, you know, your, your experience in the profession over 24 years, but also your observation of what the professional looks like. you’re describing an interesting state. I think, you know, this this mission statement and the The positioning and the interest and the provision, I guess, to help the whole company kind of wherever this, this, you know, critical decision, information is needed. You’re there to do it. You know, how, if you look at that positioning where you are now and kind of what you’re able to provide? How, how would you describe the arc of just this one part of it? Like, who and how do we work with and what’s kind of how are we? I keep saying positioning, but but I guess that’s really what it’s about. How do you think that has evolved over 24 years?
Laura: That’s a great question. And I really I get really excited about this question because I see this as one of the possible futures of our profession. So we we began and even the DNA of the team I currently lead began holy just like as design testing kind of organization at the beginning of my career. It was really a unicorn position. It’s like there was the design and the research. And I, I like I have a special like passion for the research part because I have degrees in sociology, anthropology and psychology. And so I just love love love studying human beings and I love research methods is like finding that that question and applying the best method to it. So as the the need has evolved, and William say as the field and the practice have evolved as a as the need has evolved, for answers from people who begin to be seen, as reliable, as valuable as as somebody who can really help and help get the right answers, the freedom to apply the best method possible. Is is what’s so exciting about how we’re working today and I see this as that potential future one potential future for our field is the opportunity to drop it having to everything having to be specific methods and having this one toolkit that’s just user experience and open up to the broader possibilities of any kind of method that’s going to help deliver the best experience in the end. And and I’ve just having watched that happened and I also just, I landed in rather a dream position here at Rackspace in that I, I inherited a team that was very deeply educated, there’s masters and PhDs is is what we just happen to have. It’s not what scare anybody about that. That’s what you need to practice in the field because that’s not accurate, but because we had a lot of deep experience in a lot of different kinds of methods and the ability to understand applying a finding a best fit method for Whatever question is out there and whatever way that you need to prove that are convinced it’s true. It’s just really opened up our possibilities and it’s also accelerated our cadence. Because we don’t we get very little pushback on the back end once we’ve delivered our results because we’ve chosen a great method at the beginning that actually gets to the heart of the questions.
Steve: So in this this futures, that we’re talking about, where do these methods come from?
Laura: a broad range of of research capabilities for me, they came from my education in sociology, anthropology and psychology. So, all of those different approaches such as being able, the very fact that you can know how to ask a question well, means that you can put it in multiple different formats. Perhaps it’s a large scale surveys, because you need, you need a you need a shallow data set that just has a whole lot of yes and no answers upfront. Or maybe the best way to answer the question that you come up with is to have a deep qualitative interview where you generate a large number of, of what I call them vertical data points, which is that you create these deep, deep pits of thousands of unstructured data points. So, gosh, don’t remember did that? Did I answer your question? They’re asking me again.
Steve: Well, I’ll say I mean, you started to I mean, it’s where did the where did the methods come from? And I think your answer starts to beg the other question, as was what sort of the granularity of the phrase method? You know, are there millions of methods are there eights like? What is that what’s underneath the term method?
Laura: Wow, that is such a great question. And now I have to like take my take an expert mind and try to quantify it because it’s Something that after all these years I do rather naturally. And so I don’t know if you’ve read Malcolm Gladwell blink, but it talks about, experts can’t always put a word on how they do something. So let me see if I can parse that out. What do I mean by method? So we’ll take the problem, what’s the problem to be solved? Like from a business perspective, what’s the business value that needs to be delivered? We look at that from a project perspective. And then we then we drill down on that, well, what is holding you back? What are the unknowns holding you back? And maybe they just they need to know what are like to get some sentiment and they just need some words on some sentiment. Maybe they need to know about does this workflow work or does it break down and that would be qualitative tests where you’re watching people or a remote test, where you get a lot of data where you’re gonna, like recorded online, and you look at multiple people having done task testing and that sort of thing. And then maybe it’s, look, our CEO just doesn’t believe this because it it only has a few people and and he’s just really wanting to see the numbers. And so then we would convert whatever questions and methods we have up to. Okay, let’s, you know, let’s we’ll, we’ll do a survey, which by the way, surveys are hard. I think it’s one of the hardest kinds of research to get right. And it takes the longest it takes the longest to get it right before you release it, even though it’s fast on the data collection end and the analysis end. So does that begin to get at what you’re asking about methods? does it begin to codify What is it my head?
Steve: Yeah, it was. So maybe I can reflect back a couple of things. And then maybe I’ll try I’ll try asking you a question with hypothesis, which we might call leading, or you might call good interviewing. I don’t know, we’ll see how we go.
Laura: Sounds good to me. I love that you’re asking this by the way, love it.
Steve: Cuz I feel like the when you started off talking about the team and the, you know, the deep level of education and your own backgrounds, and then even you saying earlier on, like, Look, we do market research, we do all these things that sort of sometimes exist under other umbrellas. It seems like, you know, approach a in, you know, building up a super set of methods is look at a range of disciplines, whether they’re professional disciplines or academic disciplines. And you have, you know, your set to choose from just gets larger. I guess the, the leading question is, is was to ask what you think about inventing methods, generating new methods.
Laura: Oh, my gosh. Absolutely love that. And and it’s really necessary to this approach and and can get you like the best possible thing. So a great example of that recently, we we did a study where that we were studying a concept of domain trusts, which is where you have ad administrators have to hand over the permission to do administrative tasks to other people. And so what they wanted to know what would people trust this with? Would they would they be willing to do it? And so we, the, the original method, and the the, the, the stakeholder who came and asked for in that case was very adamant about how they wanted to do it. And in that case, we went ahead and rolled with it and then they will also wanted to like do some of it themselves and so we part of how we scale it. In the organizations that sometimes we will also mentor DIY research. And this person has done some sessions. And so we were willing to do it. So we went with it, but we weren’t sure that the task testing method was going to get to that underlying question of trust. They believed that it would come up naturally as they were going through and that if they really questioned it, and it was kind of a way that they thought it was a way to fake them out, if they really questioned that, that these system administrators would trust somebody else to have the permissions and then not just trust somebody else, but somebody else is somebody else that they might not know about. So like, I pass it off to to Joe and Joe passes it off to Mary, but I don’t know that Joe ever passed it off to Mary. And so so that that was the original method and it was one of those cases where it failed miserably. It looked it on the surface. It looked like they all trusted it. But it was very clear that they didn’t quite get it. And by showing them the interface in the interfacing eat seeming easy, because it was beautifully designed, the workflow was great. There, it was misleading. And we were quite sure that something was missing. And they came back and said, Well, this isn’t they didn’t answer the questions we needed. So we went into a room and we just brainstormed and thought through, okay, how can we get at the psychology of this trust thing, without showing them something tangible like or having them work through a tangible workflow that that’s beautifully designed to this easy so that we’re not testing the design. And so suddenly, we started, like mapping out on the whiteboard. And we came up with this method of creating diagrams of people and these apps. diagrams of people and messages and connections and created these stories around it. And we had them put themselves in the story. So we would show like this one box and there’s a little icon and it’s them and we put their name on it as they told us their name. And then we said, okay, who is somebody that you might turn over permissions to, that you might allow to? And so they would say that and we built out the diagram as they talked like, Okay, well, here’s these other icons, and these are, these are these other three people, you’re going to turn it over, so like your level B people. And so we went from there and continued the story and we had them tell us a story of their own organization, we would live map out the organization as they talked about it. And so then we got down to the really down to the heart of it. And we we had, like person number five in their in their first first degree connections. Say Jay Okay, you’ve Turn it over to them. Now they’ve turned it over, they have these, these now, third degree connections from you. And they’re turning it over to them. And they said, Well, that’s probably okay. Because they wouldn’t turn it over to anybody that wasn’t trustworthy and this and that. And they then they said the key word, and I, and I assume I would know who that was. And I’d be able to see who that was, because then I could go back and check and see who that was. And that was the key. That was the key that broke the whole thing open, which is that the truth was, they were not willing to have the mechanism work in that manner. Were without without their permission, though, that was another piece that they would often just volunteer and say, it’s like, well, I’m assuming that my first degree connections will come back to me and say, Okay, I want to grant permissions to this person, or you’re going to notify me that they’ve done that and tell me who and and we blew up in the whole model that way and It turned out to be exactly what they needed. So that was like live creating a model. And that we actually did live in an abstract way through the combination of these diagrams and the storytelling.
Steve: And were you showing the this beautifully designed interface in this round of sessions?
Laura: We, we did it at the very end as a like a bonus piece, like, okay, now given that, and then what we did was we said, okay, now assume that those problems were solved. Now, let’s experience this. And so we that we just got extra data for them about the about the design in that process.
Steve: Can you say a little more about, you know, going into the room with a whiteboard and thinking through how like, you know, the process of creation of this new method? You know, and maybe that’s another unkind question for me, but I don’t know. Can you reflect on that process?
Laura: Not at all, not at all that that really begins to get to it. So this is where you have some understanding of how a human is built and how a human works and thinks just really works and comes in handy. It’s like and then also understanding one of the key aspects of, of any research design in any research method is being able to, to know that you’re measuring the right thing. And so how do you get to that you’re measuring the right thing and you’re getting at what the actual human behavior is going to be. And so that particular day, it’s like what we were looking at is like, okay, we knew that showing the interface was getting in the way of getting at where where’s that fear button in there? It’s like we were trying to see Is there a fear button in there and have we pushed it and so it this was, this was a combination of like the the minds in the room that understood how human beings work. We have a couple people with Human Factors degrees, and and then others are just, you know, have experienced so much experience in the user experience field. And then just some common sense is like About what? what would what would get at that that fear that concern that which is what we’re actually trying to measure. So that’s really the big question there is like, what are we trying to measure? Looking at your various methods and saying, Okay, if we use this method, whoops, this thing might get in the way. Or if we use that method, this thing might get in the way. Just like when you do surveys, the order of your questions can change the results that you get. So there’s one bit of guidance that I like to apply, which is that in a survey, you don’t ask the demo, or if you can avoid it. You avoid asking the demographic questions up front that have you telling them about themselves, because that can actually bias how they think about what they’re getting ready to do because you and they have Suddenly put them into a certain box and they’ve stepped into that point of view, you can get us may be able to get a slightly more pure response. If you if you just dive in and do the, the the other the basic questions that you’re trying to get at first and then add the demographic on at the end. So yeah, that’s that’s some of how we do it is make sure that that you’re getting at what you need to get at and that whatever you’re doing is not getting in the way of getting at what you need to get at. Mm hmm.
Steve: And then part of the story, I think is, you know, is your collaboration with the the other your stakeholders, your clients, I mean, however you kind of label them who, who have something in mind who have a goal in mind for this research. And so how do you how do you have that conversation? You know, maybe if we, you just told the story, but like, maybe we could talk about what happened before the part of the story that you’ve told But how to how to projects like this get formed.
Laura: That that is absolutely one of my favorite favorite things. So after many years of pain of, of studies like that, that again is a really good example of one where this happened and that that we didn’t do it. We didn’t set the foundation well up front. For, for both what was really needed we believed we’d gotten it that but turns out as we had the conversation after the first study failed, we we we went back to our disciplines conversation. So after some years of having those kinds of experiences in my career of studying, not getting what they needed after we did all of this work, is that I now ask I have a series of questions that I ask up front, it’s a weakness as we call it, our research intake process. We do it in about 20 minutes. So anytime somebody comes to us for research Was it? Oh, that’s great. We’re so excited about what you want to learn. Let’s have a 20 minute talk. And that’s we keep it really simple and lightweight for them. And there are six areas that we ask about and drill down on the first really being the most important one. The first and most important question, and I’ll drill down into each of these in a little more detail is what problem are we trying to solve? So a lot of times they’ll, they’ll think that they have one thing that they need, and turns out there’s another so what problem they’re trying to solve? Who is asking them to do it? Who are they accountable to? Because that can change the questions too. Are there any constraints to the to like the solutions or anything? Is there? Is there something that that’s in the way is like a technical constraint? Or is there a business constraint? And then are those negotiable? Are there any risks? What do we know or already have because sometimes they don’t need a test at all because they already have it. And then what is the timeline on this? And are there dependencies in that timeline? So, so those are those are our six, what What problem? are we solving? Who’s asking you to do it your stake your stakeholders? constraints, risks? What do we know? Or have? And what’s the timeline?
Steve: Yeah, that sounds like that’s 20 minutes to, to cover that.
Laura: Yeah, we’re really good at it now. I think our first ones maybe took an hour. But we’ve really got it down. Because the first one we are able to get down to, really to the heart of it very, very fast. And it’s the single most important if we get to nothing else, the one that we have to have in terms of a conversation that you can’t just ask over email, you have to really hold their feet to the fire is that what problem are we trying to solve? So it will sometimes have people come in and say, well, we need to deliver this new ticketing application or we need to make improvements to that ticketing application, or we need to test our design. So that’s not really the business problem that you’re trying to solve. That’s, that’s what you’re wanting us to do is is test those or, or that’s what you’re going to deliver. But what is the business problem that that application is trying to solve? And so in this domain trust case, it’s like we we needed to get down to what their real concern was not was, would they do this? Would they be willing to do it? And you know, is does this workflow work? It was more, does this frighten them? Is there something about this that they’re worried about? That could go terribly wrong? And then so we really have to hold them to to that on the way out, and what business value does this deliver? What are you trying to do? Who are you solving it for? So there’s kind of three questions and what problem we’re trying to solve. what problem are you trying To solve, that’s the that’s question number one, who are you solving it for? So are is it for a certain kind of customer or an internal person or a certain kind of user? Who’s the what’s that persona? And then what is the desired outcome? What do you hope will happen not as a result of the test, but as a result of actually, like releasing this thing that you envision that you want to release out into the wild? So if we do nothing else, we’re really going to get them down on that. And they often think they can do that in a in a one line email, and then they find out in the conversation that it’s not quite that simple. I think one of the best compliments that we’ve had is that we have one, what one repeat customer, he’s a product manager who comes to us all the time. And he said, Okay, so I need you to ask me those questions that make product managers cry.
Steve: Is that just take those All these questions are is where to where’s the tiers for them here? That problem one seems really interesting.
Laura: Yes, that’s definitely that’s the first and biggest. And if we get to nothing else that one can deliver all sorts of value, then we’ll also you know, then we also get them to this, the stakeholders who are who’s asking you to deliver this because what they really need can often change with, they may be driven by something external. One of the frustrations that we sometimes have, when we don’t ask this question is that we’ll go and do this and deliver it. And they’ll go a certain direction, but really, they or they don’t believe you or they won’t take our they won’t, they won’t. They won’t take our findings and they won’t do anything with it. And we find out after the fact, oh, their boss asked them to do this specific thing. And so if we know both who what, who they’re accountable for like with this product manager, it’s like After a while, it’s like I got to know who his boss was, or whether the his boss or whether the engineers were asking him to do this thing. And based on those two things, we could structure the study where it would actually get the answers that would convince that person. So we actually do that skip over of our stakeholder by asking them who’s your stakeholder, and then we arm them with the data that will help them be convincing to their other to their person. So that’s, that’s really another layer.
Steve: And you mentioned this, but I feel like it bears reiterating that. I mean, none of these questions or anything about the how we will approach this and, and, and I think, and I think you said this as well, that’s often the starting point for people who approach you, right, we need to do step x, or task y. And you’re, you’re setting that aside and asking a lot of very specific whys and how’s and whens and who’s That’s correct.
Laura: So yeah, this this is all developed not just out of pain in user experience testing over the years, but also out of having gotten to be a program manager and being able to think from that space and in various times in places where I had engineers as well as as user experience and other folks working with me and for me, and and so it was really this this concept of the, who are we accountable for? What are the constraints? What are the risks that that sort of thing does actually come straight out of business management experience? But now we’ve armed researchers with Yeah,
Steve: I can’t help but think that there are, you know, either small teams or you know, research teams of one or just lots of researchers who their role is very much To take assignments, and that may be a methodological assignment or problem assignments. And you’re describing, you know, a context for research at Rackspace, where that dynamic has, has changed entirely and you’re there to, you know, I guess work as a partner and, you know, former foreman approach together, you’re guiding them, you’re facilitating, you’re not being directed. Go do this. So we need it tomorrow. If I’m characterizing where you’re at, right now, I’m wondering if you have thoughts about, you know, how does, is there a way to get from A to B, like, you know, if A is Yeah, delegated assignments and B is facilitative, empowered partner who’s you know, who has a mission statement? What is what does it look like to go from one to the other?
Laura: So, so we have found that even our most beginner, Junior researchers are able to have this conversation because one, it’s kept very short. And we don’t reveal these things. It’s like we asked them more from that empathetic perspective, like Rao, what do you what do you need to do in this? And what do you know about this? Just as a discovery, it may not change a whole lot of what you do, but it can provide really valuable context for what you do. So somebody may have asked you here, can you do a usability test on this really quick for me? And, and yeah, we just need you to run this and run these workflows to go ahead and ask these questions up front. It really builds out the goals and objectives sections of your test plan, so that you get to a real goal of what is the ultimate goal from of this thing that you’re delivering, it could actually end up changing you Your script and how you ask those questions. So even if you’re a very frontline researcher who’s just doing what people asked this, this can really inform your your test planning and give it great context.
Steve: It’s interesting to think about that case, you sort of not, like you said, You’re not really revealing how this works. It’s not Well, before we agree to do the usability tests, we’re going to we need to ask these questions. It’s it’s kind of a yes. And that’s great. Tell me Tell me more. Tell me more. Tell me more.
Laura: Yes, it’s taking what we do from a we asked great questions, perspective and from but from this this at the beginning planning point of view and place In the fight in the study,
Steve: yes. And would there be a scenario where, you know, this frontline researcher is being asked to do usability tests, ask these questions. You know, might they identify a different approach? And if so, what would they do?
Laura: Absolutely. And so this is where you really get to shine is where and and, you know, because a lot of what we’re about is keeping people from making really serious mistakes and delivering things that don’t work. And so this is where you really have that opportunity to, to guide encourage mentor your stakeholder, even if they’re bigger than you by by saying, okay, so if we, if we do that, that user experience test, here’s what might happen is that you would get this and these kinds of answers. But if we were to test this this other way, and it’s good to kind of build it out. First thing and get an idea in your head, we don’t typically reveal the method at that time, we typically have them know the method within about an hour after this after we’ve said and thought about it. And sometimes we know instantly and we’ll reveal it, but rarely do we actually reveal the method in that situation. We’re taking in we’re being a sponge, and we’re encouraging them to think in a different sort of way. And, and, and it’s only it’s then when we’re looking at how that might work and and how it might have a method that they were asking for isn’t going to quite work. And that happens a lot like when we have people come to us say, we need a survey. And we we did there were times when we used to just say well, actually, you probably don’t need a survey, that that didn’t work, telling them that they’re wrong. And redirecting them without context didn’t work. hearing them first. So by asking these curious questions and diving down, honestly, we found those people really enjoy the conversation. Because when they get to talk about their work and their challenges, who they’re accountable to, and what success looks like for them in this project or this effort, and sometimes they’ll actually walk out of that those just those few questions, thinking about their own work a little differently. So that’s the first part is that we’ve listened and we applied empathy to that, and we’re very kind when we do this, this experience experiment and this experience and so that’s, that’s really the first foundational piece is that we’re, we’re giving them an opportunity to speak about things that they nobody thought to ask them. And so that’s, that’s step one in the success measures of this and then second is that then if we develop another method, that that that we then go have another conversation like okay, so looking at this, it looks like you’re This is going to get you better what you’re going to want and they may or may not buy that No, I just need this, this run like this. It’s like, okay, run an experimental session, run a pilot session with both and see how they work. Sometimes the stakeholders are right. And I think we’re, we’re more on point most of the time simply because that’s, you know, that’s our superpower. We were the were the methods people. But there are times when, when we thought it needed to be a certain way, and we were wrong. But when we are more effective when we’re delivering something more effective, we can show it, then we have this opportunity to do it. In that particular case with our domain trust case study. It was simply that we went back and told them we didn’t say, okay, you all screwed up by saying, making us do this method and all of that we actually took it on ourselves. We said, Look, we we heard you, we believe that that maybe there should be a different approach. We and we didn’t follow that up strongly. Would you be willing and we’re going to try this other experiment, we’re going to go do these other studies, and we’re going to see if we can get you exactly what you need. We’ve developed this other method. And so in that case, they felt the pain and they were willing and by the time we got there, they were like sponges with with with our data. And yeah, really answered the question. So that’s a that’s a Those were the steps, which is that empathetic listening upfront and really using it the conversation as an opportunity to discover and to hear. And then, and then, if they’re insistent, go ahead and do it, perhaps run a pilot of the alternate method and see if those look different and if that gets them closer to what they want. And then if there is a fail, then you have the new method in your back pocket and kind of apply it at that time.
Steve: You know, I have this not particularly brilliant theory, but I guess this observation that user research, given who’s in the profession, how it works. It’s a helping profession. You know, the way that reference librarians are or you know, some medical people or something, and I mean, I hear that in your story that that, that this is not about being adversarial, towards whatever client stakeholders people in other departments or disciplines, you’re, you’re really approaching them to help them and it becomes you just it’s framed very collaboratively. It seems to me that, that starting point of that kind of philosophy, like you said, of being very kind and using the empathy and we use those words in research all the time, it sort of gets thrown around a lot, but when you describe how it’s implemented and how it’s sort of a philosophy to live by and work by, it’s really exciting to see how that creates a working relationship that doesn’t have a lot of the a lot of the challenges a lot of researchers raise in their own Work, you’ve kind of created a culture or context of cooperation that just is flavored very, very differently because of that.
Laura: That’s That’s right. And thank you for seeing and recognizing that, that that is how that works. And we actually have we have one more like, one of our superpowers in our toolkit for that, is that after we had a big push back situation, where product managers got really upset by our results, and then we found out that that he was under other pressures. And that was that we didn’t even know about and the fact our results like they were correct, and they were right, and they needed to be applied and used and they knew that but it they they just had a real reactive situation to it. And in that case, this this is on another project than that one. In that case. We really see sat down. And we did a post mortem ourselves about that interaction and how that there was some unfortunate Bad Blood really created between us and that, that person and and he was just really wounded for a while. And so we, we set but we were bright, but we were right about it, they and they needed to do it and they and they have to do this that they and they did they applied the results. But still it was like we had that relationship fail. And so we we wrestled with this for some hours and an offside over my kitchen table one day. And we finally came up with so we were what we’re doing is like coming up with with core value approaches that we could apply to to solve a situation like this and to respond to it. And and even before that, how could we seek from not creating it in the first place. And the big thing was, is that we wanted to do that without giving up our power, our authority. as researchers and our knowledge, so it wasn’t just about caving and saying, okay, you’re right now our data is wrong or whatever, or you can take it or leave it or whatever it is that that we might approach. It’s like, Okay, what can we do in a situation like this that’s proactive without giving away our authority. And the two words that we finally came up with, which are our watchword, our bill, whether we’re be curious. Be curious. That is one of our research superpowers is that when a stakeholder is pushing back on our results, or on our method, or anything about that, we might start getting defensive on the inside. And we will stop and recognize that it’s like, okay, we don’t want to give up our authority as the methods experts as the data experts as the findings, experts. But in that moment, what can we do? It’s like, oh, there may be Something more going on for this person or in this situation that we simply don’t understand. And there’s a reason why they might be pushing back in that way. And so we turn it around into a question. It’s like, wow, that’s it, you know that that seemed hard for you What? What is that about that? What are you afraid will happen? Is there somebody that might come back at you or push back on you? Is there a deadline you might not make? Is this going to make things more complex? So we we pause and and seek to be curious, and that has made a huge difference in our stakeholder relationships. So even the most basic researcher can be curious about the people who are asking them to do something. It’s a simple tool.
Steve: That’s really wonderful. It reminds me of so many of my own experiences where I have an interaction that is weird or uncomfortable or unpleasant, you know, even a minor thing and I might Hold on to it for a while and then try to have that realization that it’s even though I’m experiencing it, it’s probably not about me. Right? There’s something going on. And so, I mean, I think your team is, has got a really lovely framing for it. And then we can take the opportunity to answer that question like find out be curious, what is that thing that’s going on to, I would imagine there’s some self care in that because it sort of proves that they don’t hate you, or they don’t think you’re stupid or whatever the thing we do to ourselves when we get those signals, but also back to your kindness and your framing around helping if you understand what the issue is for that person, or even giving them the chance to articulate it, maybe they haven’t articulated it before. you’re providing value in every direction and a number of different levels with that with that mission.
Laura: Thank you. And that that that is correct. It’s like having that, that opportunity to speak about those. And we learn so much more about the business and about how it works and or about the technology and how everything works or the product direction by doing this. The other element of that that was starting to talk about a little bit was that retaining your own power in this. So we also in that same offsite session really looked at, okay, how do we retain and keep, you know, so yes, we’ve listened, we’re curious now. And the data is still the data. It’s like the answer is still the answer. And so there’s a couple of things that we do is that one, we’ve now taken the heat off of that by by using this phrase, the data so the data is telling us, the data is telling us and so instead of saying Yeah, but the users are saying this or we found this in the study, and this was this was the the output and whatever it was that and really starting to get our backup about that. We we use that more generic and in really, like research based answer of, yeah, I hear you with all of those challenges and things. And this will be complex. And we just want you to know that the data is telling us this, the data is telling us this. And that way, it’s it’s agnostic, and it doesn’t say you have to go do this. It’s like we also the third thing we do is we treat them as grownups who are accountable for their own jobs. And so the ultimate decision on what to do we leave to them. And so then the final thing that we do other than So, you know, saying this is what the data is telling us, and then leaving Leo letting them giving them the courtesy of making their decision. And and living with that, whatever it is, the last thing that we do is we retain our data. And we will, sometimes we will even take special care after a study is done on one that somebody is pushed back on. And we’ll make it even more attractive and pleasant and more complete of a report. So even if we’ve done like just a quick turn report will actually turn it into something more formal archival. And then we save that. And I’ve actually had as a situation with this actually happened more than once. But there was a large one that for a large corporation that I was consulting with. And this was a large scale refactor of the main navigation of their whole site. And it was intended to increase click throughs to their product sets. And we tested this and we actually traveled nationwide tested, you know, 18 year olds to 80 year olds and, you know, all walks of life and everything. And so we gotten all this stuff. And the data told us that there weren’t these critical fails in this and that it was it was going to fail. And they were they were on their timeline. There were business reasons they wanted to release it. And so they they rushed and did it. So, you know, we told them here’s the findings. We let gave them the courtesy and and here’s the bad things that might happen. We gave them the courtesy of making their decision they did. And then they went back to look at the at the analytics and discovered that not only did click throughs not go up by the 30% that they estimated. The click throughs went down by 30%. So something had gone drastically wrong. And they spent like the first 18 hours or so scrambling with technology and all sorts of things and all sorts of finger pointing and fight since then somebody said maybe we should ask research and so they came back to research and they said Do you have any ideas about what may have gone wrong? We said yes, actually. So we we documented in this report. And here are the things that that were risky that that you can look at. And they looked at those as like, well, there was some dissent in the organization about how did these get through, we didn’t go there. We just gave them the data, the information, we helped them interpret it helped them come up with designs to fix it. And they did fix it. And within 36 hours, the click throughs had not only corrected, but they had gone up the original 30% that was intended for the project. And we never went back to that we never said I told you so we lead with the data says tells us this, gave them the courtesy of making the decision and then coming back and being ready in case the risk the bad thing does happen. That that we we gave it to them without judgment and and say yeah, here’s how to fix it. said this is this is not only very humane it’s great business
Steve: there’s something here about you know, when are the moments when people are willing to hear like we are we are sometimes change agents we are sometimes telling people things that yeah the break their worldview.
Laura: Yes, that’s a great way to say it.
Steve: So, you know, you can sometimes you can choose your moments and I mean, in that story, you couldn’t really choose your moment it was kind of the moment came to you and you were you were prepared to respond. But your earlier your earlier attempts to to influence and sort of change minds. I mean, really didn’t didn’t land but you were ready when the when it came back around.
Laura: That’s correct. And most of the time where the the research results are supported, And the bad thing does happen. And then sometimes we you know, because we’re intentionally observing this, and we’re really going for high quality for our customers and end users. Almost every researcher has discovered that there are times when it maybe that bad thing happens, but it doesn’t have as big of impact as we thought it was. Would if they didn’t, that was with somebody not fixing something. And so sometimes, yeah, sometimes that happens, too. And, but most of the time, that’s, that’s what we’ll do. And actually, we we do find because we’ve had that in tech conversation up front. We, we have the tech conversation upfront, we review test plans with our stakeholders before we run them, and we invite them to the sessions so that, you know they’re in there too. We’re very transparent. We even put all of our working documents in shared directories, even our working reports and things we don’t hide anything from them. Most of the time, they don’t go look at the details, but the fact that we’ve made it transparent is created a lot of trust. And then ultimately, you know, sharing the data with them most of the time, they’re convinced because we’ve set up such a good foundation up to there. And then when they’re not we have this failsafe mechanism for for how to handle when they don’t take the advice.
Steve: In US, you said a couple times, no, you’re treating them like adults, the decision about what to do or not to do is theirs.
Laura: Well, yes. And it’s not just that it’s like recognizing that, that there’s other drivers that that maybe they’ve not articulated in all those conversations, or maybe they haven’t even identified themselves. We just don’t always know what those are.
Steve: And I think you’re surfacing. You know, what is what does success look like for researchers? And you’ve, I think you’ve articulated in a specific way that I haven’t necessarily heard that there’s a lot of there’s a lot pressure we put on ourselves as a practice to have influence in you know, in that word means so many things, but one of those is people doing what we think they should do. But saying, oh, that that they’re adults and that they’re going to make their own decision and there’s factors that we don’t know. right i mean even your mission statement about where you use the word spark is kind of what you’re doing. I mean spark spark plays a certain role in a fire being led or not led but it’s it’s right there kind of outside to make it happen but it’s not the fire right the fire is the is the next thing that happens.
Laura: Yes, I’m so that’s the spark fast, confident decisions, because that’s what we’re about. And so that’s actually another the fast part is another element of the method is that sometimes we need a faster method to to get at at the same sort of thing. So that would be another consideration. But ultimately, yeah, it’s like we’re trying to find where is the unknown in the audience. is keeping them from moving forward and moving forward to the right thing in the right direction. And yeah, once we’ve discovered that we, by setting it up that way up front, and having that in mind that, that we have more success influence on the back end.
Steve: When you spoke earlier in this conversation about, I think bringing tools and mindsets from business management to research, because we’ve, that’s what we’ve talked a lot about, right is sort of the management of making research, effective and successful inside your organization with all of your colleagues.
Laura: Yes. So the two pieces I brought specifically from business were the constraints and the risks, because those are two things that you really have to well and then what is the business outcome when we ask about the desired outcome in the problem solving conversation, and question we’re going for what is the business outcome, like no, not that just that we need to release these ticketing changes. It’s like, Why? What do you hope will happen as a result of those? So that’s one aspect is what’s the business outcome? Or the organizational or whatever it is that, you know, that’s driving the outcomes, the bigger ones? And then the constraints. It’s like, what are the technical business constraints? And are they negotiable? It’s like this is that that’s comes basic, from basic business management that really anybody can begin to can learn to understand and begin to understand is that there are sometimes there’s just you know, there’s there is a hard schedule or budget constraint or legacy technology constraint. We all hit that with a lot, don’t we? But then then the question to ask is, you know, not not just to say, well, that’s just going to work and you can’t give them what they want. Before you go there, you say, or, you know, give them what they need. The users are going to have to have this bigger investment you say, okay, is that negotiable And to what extent is that negotiable, that technology or that, and then that you, you begin to get them thinking in those other directions. So then then the third one, then is is the risk. So I use a basic risk management model, which risk risk assessment management model, we asked what are the potential risks of the of this solution? What is the likely you know what bad things will happen? So, that’s a simple way to ask it, what what bad things might happen if this gets released? Because a lot of times when they’re asking you to test something, they’re not necessarily thinking about the the potential implications of it. Or maybe that’s what they’re actually trying to get at in the test. They just didn’t really say it. It’s like, we’re afraid that this workflow will, will break down at this point. It’s like, well, what bad things will happen? What’s the risk? What, what’s about bad things would happen because they don’t always articulate that part. They’ll just say that it won’t work. And then Okay, what’s the likelihood of that is that really Something that would happen, is that a high medium or low likelihood that it’s going to happen? And then what? What would be the impact? It’s like, okay, yeah, that bad thing might happen, like, there’s this usability problem, and it would be there, and they’d have to work around it. And they would still have another way to do it. So that would be a low impact. There’s not a big, yeah, the bad thing might happen. But the bad thing that will happen as a result is is really not so huge. Or they might have a data loss or the system will crash or somebody will die like that those those would be really high impact. So once we have that high, medium, low likelihood, high, medium low impact, and we know what that risk is, then we can have that conversation. How do we mitigate that? So how can we mitigate that and sometimes I’ll actually change their solution live while they’re talking to you simply by asking these questions. And sometimes we even ask ourselves these questions about our own studies. It’s like is there a risk of this study so so we asked the the risk The likelihood high, medium low, the impact high, medium low. If both of those are high Oh, then we may need to add more participants cuz, because if this is a really risky thing, we need to find all possible capability reasons that are possible failures. It’s like maybe the first 15 people won’t reveal it. So we need number 16. So if it’s a really high high impact that that’s one of the decision points it tells us is we need to add more participants potentially, if it’s a really low impact, sure, we might be able to get feedback from six, we might and we only get feedback from three or hallway testing if it’s a low likelihood, low impact. And then how can we mitigate that? How can we fix that? So? So yeah, it’s it’s a great little conversation to have, and it’s not really complicated. Risk Management sounds like such a big, scary business II thing, but the risk, the likelihood impact and mitigation. It’s really rather rather simple. It’s so valuable
Steve: not just to step back a level a little bit, and we started off, you know, you talked about your, your, your degree in research psychology and you mentioned sociology, you mentioned anthropology and by the end of the conversation, we’re talking about business. Business. And so you know, for you in your career, like where did that come from? How did you pull that in, in addition to all the other things that you’ve kind of incorporated in from the social sciences? Where did the business stuff come from?
Laura: By taking opportunities that came along and by being curious, so that specifically for me how that happened was that I was in a defense consulting organization with the University of Texas at Austin, the Applied Research Laboratories, a very old and respected laboratory that started with the Navy and then expanded to other services. And there was a there was a pitch that was We got to do just people coming in. And I thought, well, I want to pitch user experience to them. And so I managed to weasel five minutes in with these people who were looking for high value, things that they could invest in toward this specific effort that they wanted. And have that whole day and the little five minutes, I was able to wrangle out of somebody else’s 30 minutes session, because I came in late to the game. They were lit up by this whole user experience concept, and it exactly matched on what they needed. And so because they had a need, and I presented it, I brought the contract and so I started asking my boss, how do I do this? How do I get a contract? What do we do? And so I started learning about the business aspects just because I went put myself out there. There was an opportunity, I’ll tell you, I was terrified. But it was it was like but I was also like, thrilled exhilarated like being at the top of the hill on a roller coaster, and the best way and So I took that on and and by doing that I began to, to have to learn that sort of thing. So that was the start of it. But the really big pivot was then when, in addition to that I, they they had a project that had a manager who left and they needed somebody just to pick it up really fast and keep it going. They thought it was going to die in six months, because it was not going well. The stakeholder wasn’t happy. So again, I was curious, it’s like, Okay, why is the stakeholder not happy? Well, it’s because things were not delivering on time or and it was like what all these requests for like a year overdue there were $100,000 over budget, and they just didn’t see the value anymore, except for this one guy doing little ad hoc fixes for them. So I started looking at their contract and seeing what they signed up for and discovered that, that we were doing a whole lot of things that were out of scope, but we weren’t doing the things that they signed up to And it was really simple way to begin to learn business and budgets and, and to that sort of thing was I was just curious. And I just started asking the question like we do as researchers, why, why? Why is why are they unhappy and I had empathetic conversations with them. And so when I when I started having those empathetic conversations with the people who were in charge of these projects, and even as a researcher start asking them well, so why are you concerned about that? What that what’s that about? What kind of pressures are there? That’s where I began to learn about this and in in, like, I go out, try to find, well, how do you manage this and this sort of thing, and I paid attention when they somebody did a risk study at some point and I was like, Oh, what’s that? So I went in and said, Can I listen in on your on your post mortem, you know, on your on that risk that happened, that bad thing that happened? And I that’s where I learned those those basic steps. Even though they weren’t basic in that situation, but because we’re user experience people, to boil them down to those basic questions, that was how I did it. That’s how it’s in this one tiny little part of this conversation and that really anybody can do it. So it’s like taking the best of who we are as user experience people, and especially as researchers, which is paying attention, asking questions and boiling down things to their simplest, simplest forms, and then applying those to make a difference. I’m really proud of myself. I’m glad you asked all these questions.
Steve: You know, we keep coming back to how be curious. It’s just as a mantra for good collaboration and for developing skills and you know, we we bump up against all kinds of stuff that may be intimidating. But you’ve, you’ve showed how to learn by being curious. And as you said, you know, paying attention and boiling it down. So all the skills of a researcher being applied to all these different contexts, it’s a really it’s a it’s inspiring, and I think but also very actionable. I really like it for that.
Laura: Yes. And then I, I’ve never just sat and done just what I was asked to do. It’s like I’m always looking for something new. Something else is probably just part of how I’m built but it’s also a conscious choice of, of just doing my my current job is it’s simply not enough for me and that even just that extra 10 minutes of curiosity or desire to see something else or learn something new while still doing my job and delivering on that, that that that’s just opened so many doors And, and and, you know, helped me step into a lot of opportunities and learn new things.
Steve: Is there anything we didn’t talk about today that we should
Laura: we could talk about all sorts of things. So that this this is a lot. And this is really a lot of the heart of of how my own practices developed, how I would love to see the field develop just in terms of, of one, you know, that that applying the best possible methods in being free to use any or invent them based on the real questions and to know those real questions to ask up front. And then how to treat people along the way. And grab the opportunities along the way, I think. Yeah, that’s that we’ve summed up a lot of my practice really, really well. Well, it’s, it’s really beautiful. As I said, I find it really inspirational. So thank you, Laura, for being so open and sharing so much good information. I really enjoyed the chance to speak with you today.
Laura: Oh, thank you so much for asking. It’s my pleasure indeed.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit portigal.com/podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 28. Laura Faulkner of Rackspace first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I chat with Colin MacArthur, the Head of Design Research at the Canadian Digital Service. We talk about bureaucracy hacking, spreading the gospel of research throughout government, and embedding researchers in complex domains. Often the idiosyncrasies in people’s research and the sort of surprises that don’t fit within […]
The post 27. Colin MacArthur of the Canadian Digital Service first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I chat with Colin MacArthur, the Head of Design Research at the Canadian Digital Service. We talk about bureaucracy hacking, spreading the gospel of research throughout government, and embedding researchers in complex domains.
Often the idiosyncrasies in people’s research and the sort of surprises that don’t fit within the template are the most important things that our researchers find. – Colin MacArthur
Show Links* It Choose You * Miranda July * The Future * PennySaver * Advancing Research conference * Colin on LinkedIn * Colin on Twitter * Canadian Digital Service * Treasury Board of Canada * Scott Brison, Canada’s First Minister of Digital Government * Public opinion research * Snowball sampling * Randomized controlled trial * Stand-up meeting * Homepage Usability by Jakob Nielsen and Marie Tahir * U.S. National Park Service * Wizard of Oz * ResearchOps
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I just read the 2011 book “It Chooses You” by filmmaker and artist Miranda July. It’s one of the best books about ethnographic research that isn’t really actually about ethnographic research. In the book she describes a period of her life where she was creatively stalled in finishing the screenplay for her film “The Future.” As a way to either get unblocked or just avoid what she should be working on, she develops another project, to call people who have placed ads in the free classified newspaper the PennySaver, and arrange to come to their homes and interview them.
She reports on each of the interviews, including excerpts of the transcripts, and some amazing photographs. The interviews are sort of about the thing being sold, but because she’s getting outside of her cultural bubble, she takes a wider view, asking people about a period in their life when they were happy and whether or not they used a computer (since even in 2011 a newspaper with classified ads was a relic of a previous era).
These interviews are confounding, hilarious, disturbing, touching – everything you’d hope. And July is honest about what makes her uncomfortable, about her own failures to properly exhibit empathy when it’s needed, or her challenge in exercising caution in some dodgy situations while still being open to connecting with strangers. She incorporates her feelings about her own life as she hears from people about their hopes and their reflections back on their lives, lived well or not so well. She articulates her own judgements about the people she met and how that informs her current thinking about her own life and her aspirations for her future.
In one chapter she meets Beverly, a woman with Bengal leopard babies and birds and sheep and dogs. Beverly was clearly excited for Miranda’s visit, and prepared an enormous amount of fruit-and-marshmallow salad which neither July nor her crew want, but accept out of politeness, eager to get away from Beverly and her home, but then head straight to a gas station and throw the marshmallow salad in the trash, covering it up with newspaper in case Beverly stops by. Reading it, I felt my own judgement of Miranda for her judgement of Beverly, but I can imagine doing the exact same thing in a similar circumstance, and I appreciate July’s ability to observe her own judgment and infuse it with compassion at the same time. Ultimately, she views her struggles to connect as her own personal failure, saying “the fullness of Beverly’s life was menacing to me – there was no room for invention, no place for the kind of fictional conjuring that makes me feel useful, or feel anything at all. She wanted me to just actually be there and eat fruit with her.” In articulating something so nuanced and personal, we learn an awful lot about Miranda July as well as all the people, like Beverly, that she meets.
I can’t believe it took me this long to finally read this book, and I can’t recommend it highly enough.
Through Dollars to Donuts, I’m gathering stories about maturing research teams, looking for best practices, emergent approaches, and insights about organizational culture. This is of course highly related to the services I offer as a consultant. In addition to leading research studies in collaboration with my clients, I also help organizations plan and strategize about how to improve the impact research is having. Whether that’s working as a coach for individuals or teams, or running workshops, or advising on best practices, or leading training sessions, there’s a number of different ways you can engage with me. Please get in touch and let’s explore what the ideal collaboration would look like.
You can email me about this, or with feedback about the podcast, at donuts@portigal.com.
Coming up soon is the first Advancing Research conference. Rosenfeld Media is putting on this conference March 30th through April 1st 2020, at the Museum of the Moving Image in Queens New York. I’ve been working on the curation team helping to assemble the program and it’s looking like a really fantastic event. I’ll put the URL for the conference in the show notes. You can use my discount code Portigal dash ar10 to save 10 percent on your registration. I hope to see you there!
All right, on to my interview with Colin MacArthur, the Head of Design Research at the Canadian Digital Service.
Well, thanks so much for being on the podcast. I’m really happy to get to speak with you.
Colin MacArthur: My pleasure. Thanks so much for having me.
Steve: All right, let’s just start with who you are? What role do you have? What organization do you work for? I’ll just throw all the launching stuff onto you and we’ll go from there.
Colin: Absolutely. My name is Colin MacArthur. I’m the Head of Design Research at the Canadian Digital Service. We are an office of the Treasury Board of Canada that works with departments across the Canadian federal government to improve the way they serve their constituents. We do that on my team using design research and by helping people inside the government get closer to the folks who they’re trying to service. We do that in partnership with designers and engineers and product managers and policy experts on interdisciplinary teams, but the perspective that the researchers bring is a hard and fast focus on the people who we’re trying to serve and what their experience with our services is like.
Steve: Why is this under Treasury?
Colin: (laughs) Good question. The Canadian federal government, or government of Canada is organized with departments and then with a number of central agencies. And Treasury Board is one of those central agencies. Its name is a little odd in that it’s not actually the department of finance. It’s not the Ministry of Finance. It plays a central role in sort of managing and consulting with other departments about how they run themselves. It’s a management board of government. And so we are in kind of an interesting place because from Treasury Board we have a view of lots of interesting things happening across government. We’re positioned kind of naturally to give advice and also learn from departments that we work with and then to work across lots of different departments in a way that would be a little more unusual if we were nestled inside a dedicated department itself. So, Treasury Board is sort of one of the central agencies of the government and that’s why we’ve ended up where we are.
Steve: Is that where the Canadian Digital Service originated?
Colin: That’s exactly right. So, we’re relatively new. Founded just a couple of years ago and we started right in the same place we still are, inside Treasury Board, reporting to Canada’s first Minister for Digital Government who was also the President of the Treasury Board.
Steve: What’s the relationship between digital service and, I don’t know, what would you call like regular service. Like the things that government does. Because you included policy in kind of the mix of people that you collaborate with. So, everyone else you listed seemed – design, engineering, research, PM – seems very – yeah, this is how sort of software is made. How digital services are made. But policy – and this is from someone outside government, so maybe it’s a naïve question, but policy just sort of begs the question for me, like oh what’s digital versus just services?
Colin: Yeah. What a good question. I think the way we choose to look at it is we’re interested in improving services, period. So that means the elements of those services that are online, but also the elements that are offline and drift into paper processes and drift into things that are more typical policy problems. But in this day and age it’s pretty hard to have a meaningful discussion about service in general without talking about the digital pieces of those services. So, when we put together teams to work with departments we absolutely come with a digital emphasis. That’s one of the strengths that we can bring. That’s one of the pools of expertise that we have. But we’re just as interested in looking at the non-digital sides of that service. And in reality, they all fit together and attempts to kind of separate them out into the website and the paper part are often pretty hard and don’t end very well for the people we’re trying to serve. So, we kind of view ourselves as tied up in both and we try to staff our teams with expertise that allows us to do that. That said, our name is the Canadian Digital Service and I think often our entrée is our digital skills. But we try to be more than that. We don’t think digital problems can really be solved by just looking at the technology piece.
Steve: That seems to me analogous with so many efforts to do transformation or introduce new services or new products – you know private and public sector around the world where – I mean you’re kind of hinting at the power of the like the construct of digital. That it invites maybe a different mindset in approaching a problem that can exceed beyond the boundaries of what is digital. Or, as you said, they’re intertwined.
Colin: Yeah, absolutely. I think we view the computer part as only part of the digital mindset and approach that we try to bring. I think departments appreciate that. I think that we are often able to look at a problem from different angles because we’re not just looking at the IT systems involved. We’re looking at all the pieces related to the problem they have.
Steve: So, what is your role specifically?
Colin: My role as head of design research is to first of all lead and coach the researchers on our team. So, I help them as they’re embedded in product teams chart the direction they want to take with their research, improve the quality and the quantity of their research. Make sure their research is kind of fitting into the product lifecycle in the ways that we hope. And also make sure they’re developing as professionals. We are happy that folks don’t just come here to build services. They come here to build skills and we do that with our researchers, like with all of our other disciplines. So, helping and supporting the researchers is one big piece of my job.
Another is to convene the government wide community practice around design research. So, we know that design research happens in places other than CDS. We know that research is a tool that lots of other departments are using. But what I noticed when I first arrived was that there were very few conversations between people doing research in different departments, or even within a particularly large department. So, I use my role and my limited additional time to convene folks across government of Canada who do this kind of work and talk about the challenges of doing it and talk about ways that we’ve worked through some of those challenges. The joy of that is getting to see all of the sometimes hard to spot places where people are doing interesting things. You know taking interesting methodological approaches, having new discussions that aren’t visible if you don’t get them all in a room and get them to start talking. So, convening that government wide community of practice is another key part of the job. And related to that is trying to create some tools and some policy change that helps that whole community move forward. We try not just to talk about what the problems are and swap stories on the solutions. We try to learn from that community and then use our position and knowledge to nudge forward policy changes or government sort of practice changes that can help researchers across government be more effective.
Steve: Do you have an example of a policy change or process change?
Colin: Sure. So, one of the central pieces of research regulation in the government of Canada is something called the public opinion research rules and guidance, known as POR. I’ll try not to use that acronym, but public opinion research is this kind of class of research that the government has a process for managing – very thoughtfully and deliberately over a relatively – I don’t want to say slow, but certainly a longer timescale than most user research or design research would happen on. And so one of the key areas of confusion we saw when we started doing design research with partner departments was wait, can you really do this without going through the whole process for public opinion research? Isn’t what you’re doing just public opinion research? Why are these things different? Why should we believe you that they’re different?
And so we went to our colleagues in another part of Treasury Board who actually own the policy on public opinion research, and we said to them, “look, these are the kinds of things we do. These are the kinds of questions we ask. If these are the kinds of questions involved is this really public opinion research?” And their response was, “if that’s really all it is then probably not.” And what was great was that we could then work with them to write a clarification to the guidance around the policy which really just meant like updating a webpage to have a section that talks about POR relates to user or design research and kind of further explain to departments that they could be doing user design research that didn’t need to fall within this typical public opinion research cycle. And that kind of guidance is so important to loosening structural challenges to doing research across government, right? That kind of guidance from the center is what helps people trying to do research in the edge of their departments make the argument to their manager that this isn’t quite as risky as they thought it might be.
Steve: The perceived risk is in doing the research? Is that what you mean?
Colin: Exactly. And I would say risk is probably a strong word. People say “well we have these rules for doing public opinion research. I don’t really know about any other kinds of research, so you better follow these rules.” And I think that what we’re trying to do is say “well there are actually some other kinds of research that are useful and not quite the same thing”. And think about those and realize that maybe what you’re trying to do falls within those umbrellas and thus doesn’t need to go through the same process and make management and the executive layer of departments a little more comfortable with that fact.
Steve: I think implicit, and maybe you said this explicitly, the kind of processes and procedures and things that would be required to do “design research” are less onerous than to do public opinion research. Is that correct?
Colin: That’s right. I think that public opinion research is – is I think often interested in policy and government level questions about public opinions, right? And design research is often focused on questions like what’s it like to interact with this department or this service? And those are different things that require different methods and different rules of evidence and so what we try to do is keep people from getting them mixed up in one another. Now I think if my colleagues from other parts of Treasure Board were here, they would remind us that there are some forms of design research that can kind of verge into typical public opinion research and then it’s important to engage the public opinion research process. But there are also lots of things like usability testing of a new service interface that are clearly not in the realm of public opinion research and therefore we’re really happy to encourage departments to do.
Steve: So, in this convening that you describe, sort of finding these, you also mentioned that there are maybe hard to find – I may be putting a word in here – surprising…
Colin: Yeah.
Steve: …sort of areas of research. How have you come to find those people and that work?
Colin: Well you know it’s funny Steve. I think as we were setting up the community of practice, I realized that recruiting participants for research was kind of the best practice I could have for recruiting members of a government design research community of practice. So, like when you’re recruiting people for research, you put out a call for the kinds of folks you’re interested in, but you also – you snowball, right? So, we started with people that we knew and we said hey, do you know anyone else that does this kind of work, or is interested in these kinds of questions and do they know anyone else and do they know anyone else. Often the third or fourth degree from there, you get to folks who are like, “oh, I didn’t even realize what I was doing was design research, but it is and I’m excited to find this group of people who’s all trying to do something similar.”
Steve: So, by bringing these people together, some of whom wouldn’t even have identified with the labels that we would put on what you’re doing and creating a chance to share and improve practices, this seems like it spreads far beyond what design research within Canadian Digital Service would be involved, right?
Colin: Right.
Steve: The reach is much broader.
Colin: That’s absolutely right and that’s the reason why we did it, right. So, our mandate isn’t just to build services with departments. It’s to continue to improve how the government delivers service more generally. And one of the ways we do that is through finding the folks involved in doing what we do and trying to enable them, right? Give them more tools, whether they’re policy tools or methodological tools. Whether it’s – give them a space to vent or give them a space to celebrate. One of the hard things, I think, about pushing something like design research within government is it’s – it can be hard to find a place where there’s a group of people like you who are really excited about the kind of work you’re doing. And so I think there’s some practical benefits of the community and there’s also some sort of emotional support that happens in the community of practice that’s really heartwarming.
Steve: Are there ways to find – this is maybe the counter example, or the other part of the set – so that teams or departments or groups that should be doing this, that maybe don’t know that it exists, or don’t know that it’s accessible to them, that it’s reasonable or feasible, but would help them in their efforts to improve the way that they serve the people that they’re serving? How do they fit into – I mean it’s kind of a boil the ocean question since you’re already finding everyone that is doing it, but sort of opportunities to build the practice, whether you all are providing that, or you’re enabling them the way you’re enabling the people that you’ve convened? How do you think about that?
Colin: Yeah. It seems like your question is really like what about the people that aren’t doing research and should? And they are also an important group. So, I’ll say a couple of things about them. I think that most major departments do have some groups of people trying to do research by some name, right? So, they’re trying to do client experience improvement, or your digital transformation group has a group of user centric specialists, right? So, there are people trying to do something sort of related to design research, at the least. I think what we try to do is find those folks and then we try to introduce them to the body of knowledge that’s specific to conducting research. Like we believe research is a craft in and of itself and it’s hard. And it takes some work, but it’s also accessible and something many folks can learn. So, with that in mind we try to locate the folks who are working in the generally related areas and inspire and equip them to do this kind of work, or continue to improve how they do this kind of work. You know we talk about the challenges to not only design research, but sort of digital best practice, at multiple levels. One of them is the policy or sort of structural level and we – things like our POR guidance clarification, the public opinion research clarifications, those certainly help folks, but they’re not enough, right? They – folks also need the skill to do the work and they need opportunities to learn. We try to create lots of informal opportunities to see what other people are doing. And then, you know, the bottom layer, beyond just policy and skill, is, for lack of better word, inspiration. Showing that it’s possible, right? Like showing that this work can happen within the government, within all of the unique factors of the government, and that it’s useful. And so I think we try to not just give folks kind of on the edge some policy tools, but we also try to expose them to the skillset and we frankly just try to show it’s possible and continue to cheer them on as they push it forward.
Steve: That’s good. I think we’ll probably come back to the ways that you’re working with teams and departments, but I want to go back to one of the other things you said early on. You were kind of describing the two main things that you’re involved in. We’ve talked about sort of convening the community of practice aspect here, but you also talked about leading and coaching researchers. You were really emphasizing building skills was almost a core value. Like this is a thing that you’re really thinking about being an outcome for people that work in research. Where does that come from? That is not a universally held belief I think in groups of researchers.
Colin: Yeah, well I think part of it is born out of the broader mission that we’ve been discussing, right? So, I think we are not just here to do research that helps create better services. We’re here to help the gospel of research spread around the government. And so in order to do that I think I need to care both about continuously building the skills of our staff and then taking those same tools and making them available across the government. So, for me it’s hard to figure out how I could say, “ah, broader government you should be building your skills in this way” if I wasn’t also practicing that as a leader in our own community. And I’ll say we’re not just interested in sort of helping folks build their skills, but helping – and helping researchers within our group build their skills. We’re interested in them learning about how to teach other people about research. So, I think we’re trying to build the empowering and enabling and teaching ethos into the folks that come into our group that makes it easier for them to interact with our partner departments. It makes it easier for them to have useful conversations across the design research community of practice. So, I think that from a mission angle is why that’s been a central part of how I view my work. I also just fundamentally believe we can all continuously become better researchers and that one of the ways to do that is focusing on skills building as a continuous improvement approach, not as a sort of set it and forget it professional development activity.
Steve: There’s often service design happening, of various kinds, without there being research happening. Even though we should all clutch our pearls at that idea, it still happens. So, at some point research is given a title, given a mandate. Someone like you is involved. Can you talk a little about sort of your role and your own trajectory? Where it came from and how it got to where it is today?
Colin: We’re relatively new and relatively young. I think we are blessed to have a dedicated research group and research leadership, given our size and our age. So, how did we get there? You know as – when CDS was a young and scrappy startup or a handful of people, I think the roles were not as clearly defined and people did whatever they could do to help the partnerships move forward. And luckily one of the skills in that mix was research. There were people there who thought that design research was important and who liked going out and talking to people about services. And so from the very beginning that was a part of who we did our work. And so as we grew, and then we had to kind of start dividing into more discrete communities of practice, and being a little clearer about what people’s roles were, I don’t think there was ever a question that research was kind of an important dedicated skill. I think it’s very related to design and so I think that we’ve gone through some iterations of figuring out kind of exactly how we relate to design. But right now we’re a parallel community at the same level as design or product management or engineering. And I think the more we exist that way, the more we like it because researchers need different kinds of support and coaching than other folks do, right? Research is a really different skillset then designing, at a service or an interaction or even a visual level, right? And so I think our researchers are really happy to have a dedicated group where they can get feedback on their craft and have managers that are rewarded and selected by not just their expertise sort of in general design, but in doing research in this world. So, I think we’ve been happy with how that’s shaken out. And I think we’re just – we’re lucky that research was kind of part of CDS’s way from the very beginning, down to the people who were founding the team.
Steve: Did you come into CDS in the role that you have now?
Colin: I did. So, I came to CDS with experience at a similar organization in the U.S. federal government. And when I arrived it was to lead the design research team with a recognition that that was a distinct team with sort of distinct support needs and a need for a manager that knew research and knew about doing research in government. So, I know that that’s not always true and so I feel very blessed to have ended up in a situation like that.
Steve: So, there was a team, or a nascent team, and that team needed leadership?
Colin: Yes. It was a small team at that point. I think there were three of us when I arrived, and we’ve grown substantially. I think we’re now 10 people or so, and continue to be a part of the growth plans for the org.
Steve: So those 10 people – you sort of described early on about how they would be working closely with a specific department or team that they were kind of part of.
Colin: Yeah.
Steve: Maybe you can talk about what that cycle looks like or sort of how projects or jobs or roles and researchers are matched up over time. What does that look like?
Colin: Sure. Yeah, so researchers are embedded on interdisciplinary teams. We call them product teams. And those product teams work with partners through some phases. So, the first phase is a discovery phase and that is a phase where researchers really lead in open-ended research about the nature of the problem that the department is trying to solve. We usually get set up with departments who know they have an issue and are interested in kind of our different approach to things, or they have a goal and they’re interested in us helping them achieve it. But researchers lead their teams in unpacking what that means, particularly for the humans involved, right. There’s important technical discovery that happens in parallel, but we really try to get the whole team involved in talking with both the members of the public implicated by the service, but also the staff involved in delivering a service. You know government services are this wonderful like sociotechnical network of people and we try to understand how those all fit together in discovery. It’s very rare that we would just talk to the public. I think we would spend a lot of time doing that. We really emphasize that, but we also really try hard to see the other side of the service and how all those people work together to create the experience the public sees. So, that’s discovery, right. Getting the lay of the land and understanding perhaps some of the possible roots of the problem.
And then a team transitions into alpha and beta phases of building something. What that is varies tremendously based on what the problem is. I think one of our product teams with Immigration, Refugees and Citizenship Canada did a bunch of work on a letter that was involved in a rescheduling process. That was sort of classic paper content design, tested by a researcher. Happened in conjunction with some digital work they did, but was just as important. So, that could be part of an alpha or a beta, or it could be about building a new digital service like we did with Veterans Affairs Canada building a new directory of benefits for veterans. Those products take different shapes, but regardless the researcher is trying to bring the voice of all of the people we talk to in discovery back into the product development cycle. and so that can mean usability testing, or content testing. As our products get more mature, we also use quantitative methods. We run randomized control trials. We try to use analytics on things that are out in the wild. They use all of the methods at their disposal to try to keep bringing the voice of the people we’re serving into the product process. And I think what that looks like, as I say, just varies tremendously and we kind of like it that way. I think that’s one of the cool parts of being a researcher at CDS is we say well you have this team, they have their needs, you need to serve them and make sure they have the right information about people to make good product decisions, but there are lots of different ways we can get at those answers and we agilely – probably to overuse that word – assemble the methods and the timelines accordingly.
Steve: So, a researcher working on, for example, the Veterans Affairs, for the duration of that program that’s the thing that they’re working on?
Colin: Exactly. Exactly. So, we’ve been really fortunate to basically have one researcher per product team and to be able to keep them on the same team. Which – and sometimes that’s not possible, but you know research, as hard as we try to not do this, sometimes becomes a practice of implicit expertise, right? I think researchers who have a long history in an area, or with a product, kind of know things about how people will respond to the service that are hard to articulate or sort of systematically articulate in reports or the other ways that we try to codify that knowledge. And so we see huge benefit to people having some longevity in the product teams and thus in the domain that they’re working in. Not always possible, and I understand not possible everywhere, but for us I think researchers really enjoy kind of building a deep expertise that comes with doing 10, 15 or 20 studies for a particular product in a particular area.
Steve: Depending on the type of organization, the carryover from one product team to another – I mean in my mind government is sort of an example of a category where there’s a lot of different things that you’re doing.
Colin: Yeah.
Steve: And obviously there’s cultural things and organizational things. Whereas if you’re working in some commercial enterprise that maybe makes a lot of different products and maybe serves different customers, the breadth might be less.
Colin: Yeah.
Steve: So the value, I guess, if I’m doing the math in my muttering – the value of that sort of hard won knowledge is maybe necessary to preserve or cherish it in a different way, just given the breadth of what you’re doing.
Colin: I think you’re right. And one of the fun things about getting to support these folks is that I can have conversations on a given day that range from how the Royal Canadian Mounted Police want to handle cyber crime to how Canada Revenue Agency processes tax returns for low income people to how the Employment and Social Development Canada issues benefits, right. And those are incredibly different, both business processes and missions, but also involve very different people and very different kinds of concerns on the parts of those people. So, it’s pretty neat to get to hear about all of that, but it also creates a challenge in that when we start on a new team, or a new researcher comes to a new team, I think there’s some sort of basic government knowledge they’ll bring with them, but often they’re trying to come up to speed on a pretty complicated area, pretty quickly.
Steve: When you described earlier your role is to – you know in that kind of coaching relationship you have with the different researchers – but it sounds like you’re the one that has the overview or has the window into these different product teams and what they’re doing. Are there researchers – what kind of interaction do they have that isn’t through you necessarily about what each is working on or what kind of challenges they’re facing?
Colin: Yeah, absolutely. I am – one of the things I learned very early on was that I was not the best conduit for all of that information between them and each other. And so I increasingly view my role as creating opportunity for them to share directly with each other and across the org. So, what that boils down to – there are first of all researcher standups, where we talk about what research we have done, every week. And we try to focus on not just kind of what we did, but what we’re learning. And that’s usually not enough to create understanding, but it’s enough to create a hook, right. It’s enough for one person to say, “oh, that person is really talking about the experience of submitting a form that’s an application for benefits and that’s actually kind of similar to what we’re doing, so maybe we should go have a more deep conversation, or I should ask for some documentation.”
We also have rotating, dedicated research critique groups. So, researchers form groups of 3 or 4. They meet weekly to give each other feedback on their work. It’s a little trickier to give critique on research than it is on design artifacts, but just as important. So, they’ll go through research plans with each other. They’ll go through recordings of interviews and sort of talk about the approach that the researcher took and different ones other people might have took. They’ll go through reports and deliverables. And although the kind of stated purpose of those sessions is to help people grow and share skills with one another, I’ve observed that a real common output is better cross product knowledge between them, right. If you’ve spent the time really thinking about the pros and cons of someone’s research approach you tend to understand their product a bit better. So we have critique groups.
And then one of the other kind of structures we have at CDS is these research community meetings that are actually open to everyone from the organization. So, every other week we host these 45 minute kind of brown-bag, lunch style meetings where researchers give talks. They give talks on either their product work, when they’ve recently finished it up, or on their – or on new or interesting kind of methodological or government logistical things that they’re working on. So, we try to create lots of channels for that information to flow. I think that we’re still kind of a size where some of those meeting and interaction driven approaches work. As we grow, I suspect we’ll have to get more creative.
Steve: Right, critique circles for ten is different than doing that for 80, or brown-bags. Which is crazy that I would throw that number around and everyone would nod, like yeah, that’s the size of research groups in some organizations now.
Colin: Sure, sure.
Steve: It’s not that long ago that that was an absolutely ridiculous idea.
Colin: Absolutely.
Steve: So, at ten, you can have some kind of communal knowledge just based on – well, you’re putting formal things in place for sort of semi-formal knowledge exchange.
Colin: Yeah. Yeah, that’s right. I think that we’re certainly beyond the totally informal everyone talks over lunch about what’s happening size. I think we are at the edge of what works well for critique groups and sort of community meeting exchanges. But again, it’s a little trickier for us because our domains are so different, right. And so it’s harder to say we’re building a common understanding around the user of a particular product that we all study. Often, we’re studying very different things and there are things to learn across them, but there’s also real differences between them.
Steve: I’m going to switch gears a little bit. I think it builds on some of what we’re talking about, but as you talk about the team having grown, and maybe growing into the future, what do you look for? What makes for a good researcher for CDS?
Colin: So, we focus broadly on a couple of things. I think the first is craft, for lack of a better word. So, when we hire people, and when we’re going through interviews, we really dig into the details of their previous work. You know what – how did you construct the research questions? Why did you pick those questions? How did you pick the methods that you used to answer those questions? Why did you pick those methods? We spend cumulative hours working through that with folks because we really believe that the basic skills are so important because they’re hard to – if you don’t have them down, they’re
hard to maintain in our challenging context, right? So, you really have to be a great baseline researcher and know the basics really well to succeed here because – and this gets to the kind of second factor – it’s hard to do research within government and on product teams. And I think it’s hard to research everywhere, but there are a number of challenges that folks need to be ready to meet. One of those that we look for and also spend a lot of time talking about is ability to recruit research participants and build relationships and structures around doing that. So, because we’re changing domains and we’re going into new areas, it’s not uncommon for us to start a product and put a researcher on that product that has a very specific kind of group of people that we’re trying to talk to, and would be very hard to pay a commercial recruiter to access. And so then they have to get creative. They have to go make friends in the advocacy worlds related to that department. Or they have to work with that department to use administrative data to find people. All of these things are much harder than hiring a recruiter to do the work for you. That’s not to say we don’t use recruiters. Sometimes we can, but a lot of what we do is so narrow that we need to find people who don’t just love the craft of doing research, but love the craft of recruiting. I think over time we continue to look at ways to make less of the job, but it frankly just still is a reality given how many contexts we operate in and what people have to do. So, recruiting is one piece of it.
And I think the other piece of it is broadly what we call bureaucracy hacking and that is being able to navigate a bureaucratic process to get your work done within the time that we need it to get done and with enough cognitive flexibility to kind of think through what the really important pieces are, where there might be alternative routes to the really well trodden one, and ultimately get a result despite a complicated, multi-person, multi-process situation. So, we look for people who are excited to do that and who have some demonstrated skill navigating situations like that.
Steve: Is the public opinion research story you talked about earlier, is that an example of bureaucracy hacking?
Colin: I think it’s an example of institutional level bureaucracy hacking. I think that product teams themselves and people who are working on those teams often have to do the same thing, but kind of at a lower level. So, they’re working with a partner who says, “oh, like we have a departmental process around talking to this group of people. We need to go through that process in order to do this research. That process usually takes nine months. Your research is supposed to take 3 weeks, how do we make that happen?” And it’s kind of an interesting skillset, right, that’s required to navigate those situations. It’s not just sort of knowing what good research is. It’s also your ability to think analytically about a process and understand the reasons for the pieces of those processes and then look more broadly. I will say, as we grow, we try to do less of that at a departmental level and more of that at kind of a broader institutional level, but that’s still a work in process.
Steve: It’s interesting, and maybe just coincidental, but you talked about looking for recruiting skills, very specific kinds of skills, in talking to researchers, and then this bureaucracy hacking example that you gave is around the logistics of recruiting participants.
Colin: Yeah, yeah.
Steve: You know and – I mean I’m glad we’re highlighting recruiting. I think it’s sort of a neglected – just get people and talk to them sort of is sometimes the belief in research. You’re talking in some cases about getting to maybe harder to find groups of people, or groups where there’s a specific relationship. And we also talk a lot, in research in general, about operationalizing some of those things. We haven’t talked about that. What’s your view on – whether it’s the recruiting part or just in general, in the context that you’re in, how does that idea play out?
Colin: Yeah. So – I mean I think that it’s not surprising and certainly important that the industry at large is increasingly focused on operationalizing processes like recruiting and thinking about I think ways to devise that work and to scale it more efficiently. That seems totally reasonable given the size of research teams that are now part of the modern organization. I will say, at CDS I approach it with some caution, and I’m worried I’m going to sound a little like old and cantankerous, despite not being much of either…
Steve: You have me though on the call. So, just compare to me. I can be old and cantankerous. I’ll take the heat on that.
Colin: I appreciate that, Steve. No, I – for a lot of our work, the work of doing the recruiting is part of the research itself, right. So, actually going out and making connections with community organizations or professional boards, or senior centers, or all of the places where we do our research, that’s part of how we understand who we’re trying to serve and the social structures that surround them. And so I – when I think about how we’ll scale, I try not to think about ways that we would totally take that out of researchers’ hands because I think they would lose part of the picture of who they’re trying to study if they were to do recruiting in its own right. If they were to sort of separate that into a different role. I think that – the other thing that strikes me is that so little – like when I look at the substance of our research, at the substance of what we do, some of it would I think be possible to kind of mechanistically do faster, right? Like if we had better templates and better knowledge stores and sort of better processes to string those things together, like it could speed things up a little bit. But often the idiosyncrasies in people’s research and the sort of surprises that don’t fit within the template are the most important things that our researchers find. And you know perhaps that’s a sign of our organization’s maturity, but I get worried about kind of whisking all of it into a well-oiled, sleek process for – you know I wonder about the sort of edge of whiteboard, straggly sticky-note on the margins insights that we miss. And those – and for us, in our work, those are so often key, right. Those are – we’re still building an understanding of the space and so often those things that don’t fit well within your predetermined framework are the most interesting parts for us. And I worry about losing that. That said, making the logistics of scheduling people and where they go easier, I am all for it and we continue to look for ways to do that. But I think we have to be thoughtful about what we automate in our industry, just like all knowledge workers should be, I suppose.
Steve: You made a comment a few minutes ago when I asked – when you just said research in general – you said, “it’s hard.”
Colin: Yeah.
Steve: I just want to – I want to go back to that. I don’t know if I agree or disagree. Can we just reflect on that notion of research is something that’s hard? Like what about it is hard? Should it be hard? Is that a bug or a feature?
Colin: Yeah. As it came out of my mouth, I’m like oh that’s kind of a complicated statement, I wonder what you meant by that – to myself? I would say this. In some ways research isn’t and shouldn’t be hard. I think one of the things I love about this work is that I can sit down with someone from any one of our teams and explain some basics and they can go out and start learnings things in a more systematic way with some good pointers from a good 30 minute discussion. And that’s great. And so I don’t want anything I’m about to say to be read as kind of discouraging making research accessible to everyone. We say everyone at CDS can be a researcher and I really believe that. I will say that research is also a – one of those things that’s very easy to do mediocrely. I think that there’s a lot of subtleties to how you make people comfortable in research sessions. There’s subtlety to how you digest lots of qualitative information. There’s subtlety to how you arrange your research, so it influences your product in the most responsible, but impactful way. That’s all – that’s all – I think there is subtlety and trickiness to that and so I think it’s important to have an appreciation for people who are really good at doing those things and to – I don’t necessarily count myself as one of them for all of them – and to recognize that that is a real skill and that is a skill that we should – that we should celebrate in kind of our broader industry community. That is, I think we can do that without also saying hey, there are some basic things that people can do that are easy and quick and help make more people gather more data to make better decisions. I think those are sometimes placed in a false binary in the Twitter sphere, for example, and I am not as interested in that. At CDS, and I think my comment was somewhat related to the Canadian Digital Service in particular, there are some things that are quite – I won’t say uniquely hard about research, but are specific to our context. You know one of those is the sort of incredible domain switching that we expect researchers to do every 3 to 9 to 12 months. And that’s, I think, not unusual in consulting, but it is somewhat more unusual in a product driven organization like ours. There is also, I think, the hard work of figuring out how to fit your research into a interdisciplinary team. You know we don’t just write reports and make presentations and give them and sort of hand them off to designers. We don’t let people off the hook at that phase. They’re kind of in the trenches with all those other people as the product is being developed. And we expect the way that teams run their agile development process to reflect research and for researchers to be a voice in that. And I think there are lots of organizations that have that expectation. I think we’re trying to do that in combination with domain switching and with the third part of it which is, again, we’re trying to build skills. Not just in ourselves, but in our departmental partners. So, you’re trying to learn a new domain, fit your research skills into this rapidly evolving, interdisciplinary team, and help a partner learn the basics of research and appreciate research. I think that’s hard. I think it’s fair to say that’s a difficult thing to do.
Steve: I want to switch a little bit. I would love to hear if there was a point in your own personal, professional path when research was a thing that you identified with? Like I want to do that, I am that, that’s for me? I don’t know, is there a moment or a stage at which you connected with the field that you are in now?
Colin: Yeah. There was. I conducted my first usability test when I was in 7th grade. I was – this was pretty early in the days of such things. But I was working on the school’s website. That was sort of my hobby. I helped the computer teacher with that. And I was reading Jakob Neilsen’s Homepage Usability book. There was this beautiful book with multiple sort of printed out homepages and he talked about his methodology in the back and I read that. I was like oh, this usability test is sort of something we could do. So, I got someone into the computer lab and tried it out and there was this moment of like wow, when you ask other people to try something, and when you ask them questions about what they’re thinking and what their goals are, you challenge yourself in ways that you – that I didn’t expect. And it’s also incredibly rewarding. It’s an incredible high. I don’t know how else to describe it. And it continues to drive me and the folks on my team. So, from that moment on I knew that I wanted to do this kind of research in some way professionally. There were lots of steps between there and here, but I knew pretty early in life that I really loved learning about how people used computers and services and understanding their approach. And, you know, I’m really blessed to work alongside lots of other people who are similarly jazzed to learn those things about people.
Steve: And that’s an astonishing story both – to me – in the early point in your life at which this happened and the specificity when it happened. I think often these stories are about oh I went to a farm and then I realized I wanted to be a marine biologist. Like the connections are more diffuse. But you, at a very young age, were doing exactly, or pretty close to exactly the thing. I mean it’s not even a metaphorical discovery.
Colin: Yeah.
Steve: You literally discovered the work.
Colin: It’s odd to me too. And I do look at friends’ and family’s career trajectories and I’m like hah, I didn’t think that’s how it would work out for me, but I really – I just loved it. And I looked into other things, but kept coming back to this, this being what I really just enjoy doing.
Steve: Are there any points, whether it’s 7th grade or older or younger – what kinds of things can you look to in your earlier parts of your life that were, I don’t know, maybe weak signals or kind of nascent superpowers that connect to what you’re passionate about and what you’re spending time in now? Does anything exist earlier in your life?
Colin: You know, one thing that has always been somewhat odd about me, and I do think relates to my passion for this work, I have always loved seeing the metaphorical or physical backroom of the process. So, like when I’m flying, I’m always like hah, like what does the computer system that the, you know, gate agent used look like? And like what do they do and when do they do it and why? And I’ve always – I’ve always been like that to a point of I think some well natured teasing from my family about my desire to understand the details of how the parking ticket machine dispenser system works and what the numbers on that thing mean. So, I think, especially to do this work in government, you have to kind of have a love of uncovering process and humans and how they interact with that process. And the trick is you don’t have to love like process, but you have to love kind of post modern process. Right? You have to love the fact that like it is different things to different people and kind of is this great mirror game of clarity and unclarity all at the same time. That’s something from an early age that I’ve always really enjoyed. I also would say that I – so early in my career I worked for the U.S. National Park Service and one of the things that I did there was being a park guide. So, be out on the trail talking to people and listening to them and answering their questions and seeing how they interact with a place. And there – it’s real joy – I feel real joy to seeing people in an environment and both helping them discover things and see how they discover things. And often I feel like my work today isn’t that much different, right. I think I’m more open ended than park interpreters are, but it’s still here is this new land and this new world that you are encountering, and I get to be with you as you’re doing that and help you think through what you’re doing. And so I think the kind of odd areas of joy I found in that work certainly kind of echo through to what I’m doing now.
Steve: Is there any example of a notable success? Something that, you know, maybe that your researchers have done and kind of completed that you feel proud of? Or just kind of a notable example to share?
Colin: Yeah. I think that – it’s funny. You know we’ve done a lot of research that I’m proud of and I’m proud of our researchers for doing at CDS. But the things that make me most excited are the moments when I see our researchers teaching and coaching people and departments to start doing research themselves, right. So, for example, with the Veterans Affairs Canada project they – the department identified someone who would be a researcher and sort of continue on the research with that product after we handed it back off to them. And I, you know – those moments when those people were working together and getting to support them through that, I think those probably make me feel prouder than any particular deliverable or product that we’ve gotten out the door.
I will say the other exception there is I’ve looked – I was looking at our sort of internal database of research before this interview and I’ll say the other thing I’m particularly proud of our team for doing is becoming more and more diverse and creative in the types of research it does, right. I think when we started, we were trying to make the basics happen and so we did lots of interviews and we did lots of usability tests. And those are great, but there are other things to do too. I think we’ve become not just qualitative, but mixed methods. I think that we have embraced the complications of trying to kind of test services in more live ways, right. So, not sort of just usability test prototypes, but wire something up so that someone can complete an actual transaction with it and see the back end experience and orchestrate kind of a Wizard of Oz experience for them. Those, those increases in our skill and our ability to mix and match tools responsibly, that’s incredibly rewarding as a researcher at heart.
Steve: Is there anything that I should have asked you about that I didn’t, and you want to make sure we talk about?
Colin: No. I think your questions were well rounded. Nothing comes to mind from my list.
Steve: Do you have any questions for me?
Colin: What’s the most surprising thing you’ve heard during one of these interviews?
Steve: Well, this is going to sound like I’m pandering, but the fact that you did a usability test in 7th grade is – I’ve just never come across that in my career, let alone in one of these interviews. But actually – I will say – I said I would take the weight of being the old curmudgeonly person. So, here we go. I don’t like the what’s the most surprising thing question ‘cuz it presupposes that the person – and I realize this is not exactly research since we’re doing a podcast, but as an interviewer in general, I’m not trying to be surprised. I’m trying to be focused and interesting and sort of the most benign sort of factual thing should also be exciting and triggering of interpretations. So, you know what I mean? Like the surprise is me bringing my judgement into it. So, of course I gave a concrete example. I was surprised by something that you said, but I also…
Colin: Yeah.
Steve: People ask researchers that, right? When you did this study what’s the most surprising thing? And I think sometimes that question can be good if it provokes what are we learning, or where are our assumptions being challenged? But, how do we – I also want there to be space for like hey I didn’t come in here with assumptions. I came in here with just curiosity.
Colin: Yeah.
Steve: So, I’m kind of talking out of both sides of my face here, I guess. I do that.
Colin: Well, I agree. And I do think there’s a real risk to boiling research into a set of surprising and non-intuitive insights, you know. Like the idea is that we should be manufacturing surprising little blobs of knowledge that can be passed off into the ether. And I worry about that. I worry about the work getting boiled into that. And so I hear you. And I guess I agree with your curmudgeonly instincts.
Steve: Aren’t I a nice person? Any questions for me? I mean I’m going to slap you for the first one that you asked me. So, I’m just living up to the curmudgeonly mantle I chose.
Colin: I expect nothing less. Well, that’s good. No, I would say I am curious as to what your views on the burgeoning research ops movement are? As someone who I think has been in this world for many of its evolutions, what’s your reaction to the increased operations emphasis?
Steve: I mean I have – learning about research ops provoked a uninformed critical reaction. And you said it really well when you talked about the value that the recruiting can bring to the researcher. And, you know, we hear researchers talking about the value that they get out of doing their own transcripts. Personally that’s not a thing that I could deal with, but immersing yourself in the subject matter and the people in the community and the data and the conversations is super labor intensive and the idea that we’re going to fix that by kind of operationalizing things scares me because it takes away the quality and – you talked about subtlety, I think. But if you look a little deeper into what research ops people are championing, they’re not championing that. They’re – that’s my chauvinistic misinterpretation. And I think just by creating that word, I think it invites other people to also do that and maybe even implement that. But the people who are championing it are thinking about process and thinking about how does this exist? And, you know, having spent my entire career as a consultant, so not embedded in the organization, not building these processes – maybe advising on them, or kind of giving feedback, it’s sort of an interesting thing to watch from the outside. It comes up in a lot of groups. I talk to people who are kind of early on in their organizational maturity. You know I read something recently – oh, some research ops people were sort of declaring that the first researcher hire should be an ops person which, you know, from the outside seems like well of course you would say that. That’s sort of a discipline that you are evangelizing for. But I found myself working with a group where I just mentioned that idea as an idea that exists. I wasn’t in a position to say they should do this, they should not do it. But I said, given what you’re talking about your barrier here is ops, is infrastructure, is recruiting, is not reinventing the wheel. And I said, you know – they had headcount to hire one person and I said well what proportion, from zero to 100, is that person going to be dedicated towards this? You know, if the same way that you are – that you talked in the beginning, about your organizational change efforts, you know convening people, finding processes, sharing best practices. I think a research ops person can also be doing that, you know at a different level and using kind of different processes than you’re talking about, but…
Colin: Yup.
Steve: I mean that is championing the growth of research as a thing that we have skills to do at various levels efficiently. That’s what you’re – that’s one of the things you’re doing. That’s one of the things a research ops person could be doing. Taking away the hard, labor intensive, inefficient tasks that lead to the subtlety that makes research great, that you described -that’s not a thing I want to do, but I – it doesn’t have to be what ops is, so.
Yeah. If there’s nothing else, then maybe we’re at the end. Really great conversation. We covered so much interesting stuff and I learned a lot and I’m really so happy that we had the chance to speak today.
Colin: Likewise. Thanks so much for your interest. It’s been a pleasure.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit portigal.com/podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 27. Colin MacArthur of the Canadian Digital Service first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I chat with Colin MacArthur, the Head of Design Research at the Canadian Digital Service. We talk about bureaucracy hacking, spreading the gospel of research throughout government, and embedding researchers in complex domains.
Often the idiosyncrasies in people’s research and the sort of surprises that don’t fit within the template are the most important things that our researchers find. – Colin MacArthur
Show Links* It Choose You * Miranda July * The Future * PennySaver * Advancing Research conference * Colin on LinkedIn * Colin on Twitter * Canadian Digital Service * Treasury Board of Canada * Scott Brison, Canada’s First Minister of Digital Government * Public opinion research * Snowball sampling * Randomized controlled trial * Stand-up meeting * Homepage Usability by Jakob Nielsen and Marie Tahir * U.S. National Park Service * Wizard of Oz * ResearchOps
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
I just read the 2011 book “It Chooses You” by filmmaker and artist Miranda July. It’s one of the best books about ethnographic research that isn’t really actually about ethnographic research. In the book she describes a period of her life where she was creatively stalled in finishing the screenplay for her film “The Future.” As a way to either get unblocked or just avoid what she should be working on, she develops another project, to call people who have placed ads in the free classified newspaper the PennySaver, and arrange to come to their homes and interview them.
She reports on each of the interviews, including excerpts of the transcripts, and some amazing photographs. The interviews are sort of about the thing being sold, but because she’s getting outside of her cultural bubble, she takes a wider view, asking people about a period in their life when they were happy and whether or not they used a computer (since even in 2011 a newspaper with classified ads was a relic of a previous era).
These interviews are confounding, hilarious, disturbing, touching – everything you’d hope. And July is honest about what makes her uncomfortable, about her own failures to properly exhibit empathy when it’s needed, or her challenge in exercising caution in some dodgy situations while still being open to connecting with strangers. She incorporates her feelings about her own life as she hears from people about their hopes and their reflections back on their lives, lived well or not so well. She articulates her own judgements about the people she met and how that informs her current thinking about her own life and her aspirations for her future.
In one chapter she meets Beverly, a woman with Bengal leopard babies and birds and sheep and dogs. Beverly was clearly excited for Miranda’s visit, and prepared an enormous amount of fruit-and-marshmallow salad which neither July nor her crew want, but accept out of politeness, eager to get away from Beverly and her home, but then head straight to a gas station and throw the marshmallow salad in the trash, covering it up with newspaper in case Beverly stops by. Reading it, I felt my own judgement of Miranda for her judgement of Beverly, but I can imagine doing the exact same thing in a similar circumstance, and I appreciate July’s ability to observe her own judgment and infuse it with compassion at the same time. Ultimately, she views her struggles to connect as her own personal failure, saying “the fullness of Beverly’s life was menacing to me – there was no room for invention, no place for the kind of fictional conjuring that makes me feel useful, or feel anything at all. She wanted me to just actually be there and eat fruit with her.” In articulating something so nuanced and personal, we learn an awful lot about Miranda July as well as all the people, like Beverly, that she meets.
I can’t believe it took me this long to finally read this book, and I can’t recommend it highly enough.
Through Dollars to Donuts, I’m gathering stories about maturing research teams, looking for best practices, emergent approaches, and insights about organizational culture. This is of course highly related to the services I offer as a consultant. In addition to leading research studies in collaboration with my clients, I also help organizations plan and strategize about how to improve the impact research is having. Whether that’s working as a coach for individuals or teams, or running workshops, or advising on best practices, or leading training sessions, there’s a number of different ways you can engage with me. Please get in touch and let’s explore what the ideal collaboration would look like.
You can email me about this, or with feedback about the podcast, at donuts@portigal.com.
Coming up soon is the first Advancing Research conference. Rosenfeld Media is putting on this conference March 30th through April 1st 2020, at the Museum of the Moving Image in Queens New York. I’ve been working on the curation team helping to assemble the program and it’s looking like a really fantastic event. I’ll put the URL for the conference in the show notes. You can use my discount code Portigal dash ar10 to save 10 percent on your registration. I hope to see you there!
All right, on to my interview with Colin MacArthur, the Head of Design Research at the Canadian Digital Service.
Well, thanks so much for being on the podcast. I’m really happy to get to speak with you.
Colin MacArthur: My pleasure. Thanks so much for having me.
Steve: All right, let’s just start with who you are? What role do you have? What organization do you work for? I’ll just throw all the launching stuff onto you and we’ll go from there.
Colin: Absolutely. My name is Colin MacArthur. I’m the Head of Design Research at the Canadian Digital Service. We are an office of the Treasury Board of Canada that works with departments across the Canadian federal government to improve the way they serve their constituents. We do that on my team using design research and by helping people inside the government get closer to the folks who they’re trying to service. We do that in partnership with designers and engineers and product managers and policy experts on interdisciplinary teams, but the perspective that the researchers bring is a hard and fast focus on the people who we’re trying to serve and what their experience with our services is like.
Steve: Why is this under Treasury?
Colin: (laughs) Good question. The Canadian federal government, or government of Canada is organized with departments and then with a number of central agencies. And Treasury Board is one of those central agencies. Its name is a little odd in that it’s not actually the department of finance. It’s not the Ministry of Finance. It plays a central role in sort of managing and consulting with other departments about how they run themselves. It’s a management board of government. And so we are in kind of an interesting place because from Treasury Board we have a view of lots of interesting things happening across government. We’re positioned kind of naturally to give advice and also learn from departments that we work with and then to work across lots of different departments in a way that would be a little more unusual if we were nestled inside a dedicated department itself. So, Treasury Board is sort of one of the central agencies of the government and that’s why we’ve ended up where we are.
Steve: Is that where the Canadian Digital Service originated?
Colin: That’s exactly right. So, we’re relatively new. Founded just a couple of years ago and we started right in the same place we still are, inside Treasury Board, reporting to Canada’s first Minister for Digital Government who was also the President of the Treasury Board.
Steve: What’s the relationship between digital service and, I don’t know, what would you call like regular service. Like the things that government does. Because you included policy in kind of the mix of people that you collaborate with. So, everyone else you listed seemed – design, engineering, research, PM – seems very – yeah, this is how sort of software is made. How digital services are made. But policy – and this is from someone outside government, so maybe it’s a naïve question, but policy just sort of begs the question for me, like oh what’s digital versus just services?
Colin: Yeah. What a good question. I think the way we choose to look at it is we’re interested in improving services, period. So that means the elements of those services that are online, but also the elements that are offline and drift into paper processes and drift into things that are more typical policy problems. But in this day and age it’s pretty hard to have a meaningful discussion about service in general without talking about the digital pieces of those services. So, when we put together teams to work with departments we absolutely come with a digital emphasis. That’s one of the strengths that we can bring. That’s one of the pools of expertise that we have. But we’re just as interested in looking at the non-digital sides of that service. And in reality, they all fit together and attempts to kind of separate them out into the website and the paper part are often pretty hard and don’t end very well for the people we’re trying to serve. So, we kind of view ourselves as tied up in both and we try to staff our teams with expertise that allows us to do that. That said, our name is the Canadian Digital Service and I think often our entrée is our digital skills. But we try to be more than that. We don’t think digital problems can really be solved by just looking at the technology piece.
Steve: That seems to me analogous with so many efforts to do transformation or introduce new services or new products – you know private and public sector around the world where – I mean you’re kind of hinting at the power of the like the construct of digital. That it invites maybe a different mindset in approaching a problem that can exceed beyond the boundaries of what is digital. Or, as you said, they’re intertwined.
Colin: Yeah, absolutely. I think we view the computer part as only part of the digital mindset and approach that we try to bring. I think departments appreciate that. I think that we are often able to look at a problem from different angles because we’re not just looking at the IT systems involved. We’re looking at all the pieces related to the problem they have.
Steve: So, what is your role specifically?
Colin: My role as head of design research is to first of all lead and coach the researchers on our team. So, I help them as they’re embedded in product teams chart the direction they want to take with their research, improve the quality and the quantity of their research. Make sure their research is kind of fitting into the product lifecycle in the ways that we hope. And also make sure they’re developing as professionals. We are happy that folks don’t just come here to build services. They come here to build skills and we do that with our researchers, like with all of our other disciplines. So, helping and supporting the researchers is one big piece of my job.
Another is to convene the government wide community practice around design research. So, we know that design research happens in places other than CDS. We know that research is a tool that lots of other departments are using. But what I noticed when I first arrived was that there were very few conversations between people doing research in different departments, or even within a particularly large department. So, I use my role and my limited additional time to convene folks across government of Canada who do this kind of work and talk about the challenges of doing it and talk about ways that we’ve worked through some of those challenges. The joy of that is getting to see all of the sometimes hard to spot places where people are doing interesting things. You know taking interesting methodological approaches, having new discussions that aren’t visible if you don’t get them all in a room and get them to start talking. So, convening that government wide community of practice is another key part of the job. And related to that is trying to create some tools and some policy change that helps that whole community move forward. We try not just to talk about what the problems are and swap stories on the solutions. We try to learn from that community and then use our position and knowledge to nudge forward policy changes or government sort of practice changes that can help researchers across government be more effective.
Steve: Do you have an example of a policy change or process change?
Colin: Sure. So, one of the central pieces of research regulation in the government of Canada is something called the public opinion research rules and guidance, known as POR. I’ll try not to use that acronym, but public opinion research is this kind of class of research that the government has a process for managing – very thoughtfully and deliberately over a relatively – I don’t want to say slow, but certainly a longer timescale than most user research or design research would happen on. And so one of the key areas of confusion we saw when we started doing design research with partner departments was wait, can you really do this without going through the whole process for public opinion research? Isn’t what you’re doing just public opinion research? Why are these things different? Why should we believe you that they’re different?
And so we went to our colleagues in another part of Treasury Board who actually own the policy on public opinion research, and we said to them, “look, these are the kinds of things we do. These are the kinds of questions we ask. If these are the kinds of questions involved is this really public opinion research?” And their response was, “if that’s really all it is then probably not.” And what was great was that we could then work with them to write a clarification to the guidance around the policy which really just meant like updating a webpage to have a section that talks about POR relates to user or design research and kind of further explain to departments that they could be doing user design research that didn’t need to fall within this typical public opinion research cycle. And that kind of guidance is so important to loosening structural challenges to doing research across government, right? That kind of guidance from the center is what helps people trying to do research in the edge of their departments make the argument to their manager that this isn’t quite as risky as they thought it might be.
Steve: The perceived risk is in doing the research? Is that what you mean?
Colin: Exactly. And I would say risk is probably a strong word. People say “well we have these rules for doing public opinion research. I don’t really know about any other kinds of research, so you better follow these rules.” And I think that what we’re trying to do is say “well there are actually some other kinds of research that are useful and not quite the same thing”. And think about those and realize that maybe what you’re trying to do falls within those umbrellas and thus doesn’t need to go through the same process and make management and the executive layer of departments a little more comfortable with that fact.
Steve: I think implicit, and maybe you said this explicitly, the kind of processes and procedures and things that would be required to do “design research” are less onerous than to do public opinion research. Is that correct?
Colin: That’s right. I think that public opinion research is – is I think often interested in policy and government level questions about public opinions, right? And design research is often focused on questions like what’s it like to interact with this department or this service? And those are different things that require different methods and different rules of evidence and so what we try to do is keep people from getting them mixed up in one another. Now I think if my colleagues from other parts of Treasure Board were here, they would remind us that there are some forms of design research that can kind of verge into typical public opinion research and then it’s important to engage the public opinion research process. But there are also lots of things like usability testing of a new service interface that are clearly not in the realm of public opinion research and therefore we’re really happy to encourage departments to do.
Steve: So, in this convening that you describe, sort of finding these, you also mentioned that there are maybe hard to find – I may be putting a word in here – surprising…
Colin: Yeah.
Steve: …sort of areas of research. How have you come to find those people and that work?
Colin: Well you know it’s funny Steve. I think as we were setting up the community of practice, I realized that recruiting participants for research was kind of the best practice I could have for recruiting members of a government design research community of practice. So, like when you’re recruiting people for research, you put out a call for the kinds of folks you’re interested in, but you also – you snowball, right? So, we started with people that we knew and we said hey, do you know anyone else that does this kind of work, or is interested in these kinds of questions and do they know anyone else and do they know anyone else. Often the third or fourth degree from there, you get to folks who are like, “oh, I didn’t even realize what I was doing was design research, but it is and I’m excited to find this group of people who’s all trying to do something similar.”
Steve: So, by bringing these people together, some of whom wouldn’t even have identified with the labels that we would put on what you’re doing and creating a chance to share and improve practices, this seems like it spreads far beyond what design research within Canadian Digital Service would be involved, right?
Colin: Right.
Steve: The reach is much broader.
Colin: That’s absolutely right and that’s the reason why we did it, right. So, our mandate isn’t just to build services with departments. It’s to continue to improve how the government delivers service more generally. And one of the ways we do that is through finding the folks involved in doing what we do and trying to enable them, right? Give them more tools, whether they’re policy tools or methodological tools. Whether it’s – give them a space to vent or give them a space to celebrate. One of the hard things, I think, about pushing something like design research within government is it’s – it can be hard to find a place where there’s a group of people like you who are really excited about the kind of work you’re doing. And so I think there’s some practical benefits of the community and there’s also some sort of emotional support that happens in the community of practice that’s really heartwarming.
Steve: Are there ways to find – this is maybe the counter example, or the other part of the set – so that teams or departments or groups that should be doing this, that maybe don’t know that it exists, or don’t know that it’s accessible to them, that it’s reasonable or feasible, but would help them in their efforts to improve the way that they serve the people that they’re serving? How do they fit into – I mean it’s kind of a boil the ocean question since you’re already finding everyone that is doing it, but sort of opportunities to build the practice, whether you all are providing that, or you’re enabling them the way you’re enabling the people that you’ve convened? How do you think about that?
Colin: Yeah. It seems like your question is really like what about the people that aren’t doing research and should? And they are also an important group. So, I’ll say a couple of things about them. I think that most major departments do have some groups of people trying to do research by some name, right? So, they’re trying to do client experience improvement, or your digital transformation group has a group of user centric specialists, right? So, there are people trying to do something sort of related to design research, at the least. I think what we try to do is find those folks and then we try to introduce them to the body of knowledge that’s specific to conducting research. Like we believe research is a craft in and of itself and it’s hard. And it takes some work, but it’s also accessible and something many folks can learn. So, with that in mind we try to locate the folks who are working in the generally related areas and inspire and equip them to do this kind of work, or continue to improve how they do this kind of work. You know we talk about the challenges to not only design research, but sort of digital best practice, at multiple levels. One of them is the policy or sort of structural level and we – things like our POR guidance clarification, the public opinion research clarifications, those certainly help folks, but they’re not enough, right? They – folks also need the skill to do the work and they need opportunities to learn. We try to create lots of informal opportunities to see what other people are doing. And then, you know, the bottom layer, beyond just policy and skill, is, for lack of better word, inspiration. Showing that it’s possible, right? Like showing that this work can happen within the government, within all of the unique factors of the government, and that it’s useful. And so I think we try to not just give folks kind of on the edge some policy tools, but we also try to expose them to the skillset and we frankly just try to show it’s possible and continue to cheer them on as they push it forward.
Steve: That’s good. I think we’ll probably come back to the ways that you’re working with teams and departments, but I want to go back to one of the other things you said early on. You were kind of describing the two main things that you’re involved in. We’ve talked about sort of convening the community of practice aspect here, but you also talked about leading and coaching researchers. You were really emphasizing building skills was almost a core value. Like this is a thing that you’re really thinking about being an outcome for people that work in research. Where does that come from? That is not a universally held belief I think in groups of researchers.
Colin: Yeah, well I think part of it is born out of the broader mission that we’ve been discussing, right? So, I think we are not just here to do research that helps create better services. We’re here to help the gospel of research spread around the government. And so in order to do that I think I need to care both about continuously building the skills of our staff and then taking those same tools and making them available across the government. So, for me it’s hard to figure out how I could say, “ah, broader government you should be building your skills in this way” if I wasn’t also practicing that as a leader in our own community. And I’ll say we’re not just interested in sort of helping folks build their skills, but helping – and helping researchers within our group build their skills. We’re interested in them learning about how to teach other people about research. So, I think we’re trying to build the empowering and enabling and teaching ethos into the folks that come into our group that makes it easier for them to interact with our partner departments. It makes it easier for them to have useful conversations across the design research community of practice. So, I think that from a mission angle is why that’s been a central part of how I view my work. I also just fundamentally believe we can all continuously become better researchers and that one of the ways to do that is focusing on skills building as a continuous improvement approach, not as a sort of set it and forget it professional development activity.
Steve: There’s often service design happening, of various kinds, without there being research happening. Even though we should all clutch our pearls at that idea, it still happens. So, at some point research is given a title, given a mandate. Someone like you is involved. Can you talk a little about sort of your role and your own trajectory? Where it came from and how it got to where it is today?
Colin: We’re relatively new and relatively young. I think we are blessed to have a dedicated research group and research leadership, given our size and our age. So, how did we get there? You know as – when CDS was a young and scrappy startup or a handful of people, I think the roles were not as clearly defined and people did whatever they could do to help the partnerships move forward. And luckily one of the skills in that mix was research. There were people there who thought that design research was important and who liked going out and talking to people about services. And so from the very beginning that was a part of who we did our work. And so as we grew, and then we had to kind of start dividing into more discrete communities of practice, and being a little clearer about what people’s roles were, I don’t think there was ever a question that research was kind of an important dedicated skill. I think it’s very related to design and so I think that we’ve gone through some iterations of figuring out kind of exactly how we relate to design. But right now we’re a parallel community at the same level as design or product management or engineering. And I think the more we exist that way, the more we like it because researchers need different kinds of support and coaching than other folks do, right? Research is a really different skillset then designing, at a service or an interaction or even a visual level, right? And so I think our researchers are really happy to have a dedicated group where they can get feedback on their craft and have managers that are rewarded and selected by not just their expertise sort of in general design, but in doing research in this world. So, I think we’ve been happy with how that’s shaken out. And I think we’re just – we’re lucky that research was kind of part of CDS’s way from the very beginning, down to the people who were founding the team.
Steve: Did you come into CDS in the role that you have now?
Colin: I did. So, I came to CDS with experience at a similar organization in the U.S. federal government. And when I arrived it was to lead the design research team with a recognition that that was a distinct team with sort of distinct support needs and a need for a manager that knew research and knew about doing research in government. So, I know that that’s not always true and so I feel very blessed to have ended up in a situation like that.
Steve: So, there was a team, or a nascent team, and that team needed leadership?
Colin: Yes. It was a small team at that point. I think there were three of us when I arrived, and we’ve grown substantially. I think we’re now 10 people or so, and continue to be a part of the growth plans for the org.
Steve: So those 10 people – you sort of described early on about how they would be working closely with a specific department or team that they were kind of part of.
Colin: Yeah.
Steve: Maybe you can talk about what that cycle looks like or sort of how projects or jobs or roles and researchers are matched up over time. What does that look like?
Colin: Sure. Yeah, so researchers are embedded on interdisciplinary teams. We call them product teams. And those product teams work with partners through some phases. So, the first phase is a discovery phase and that is a phase where researchers really lead in open-ended research about the nature of the problem that the department is trying to solve. We usually get set up with departments who know they have an issue and are interested in kind of our different approach to things, or they have a goal and they’re interested in us helping them achieve it. But researchers lead their teams in unpacking what that means, particularly for the humans involved, right. There’s important technical discovery that happens in parallel, but we really try to get the whole team involved in talking with both the members of the public implicated by the service, but also the staff involved in delivering a service. You know government services are this wonderful like sociotechnical network of people and we try to understand how those all fit together in discovery. It’s very rare that we would just talk to the public. I think we would spend a lot of time doing that. We really emphasize that, but we also really try hard to see the other side of the service and how all those people work together to create the experience the public sees. So, that’s discovery, right. Getting the lay of the land and understanding perhaps some of the possible roots of the problem.
And then a team transitions into alpha and beta phases of building something. What that is varies tremendously based on what the problem is. I think one of our product teams with Immigration, Refugees and Citizenship Canada did a bunch of work on a letter that was involved in a rescheduling process. That was sort of classic paper content design, tested by a researcher. Happened in conjunction with some digital work they did, but was just as important. So, that could be part of an alpha or a beta, or it could be about building a new digital service like we did with Veterans Affairs Canada building a new directory of benefits for veterans. Those products take different shapes, but regardless the researcher is trying to bring the voice of all of the people we talk to in discovery back into the product development cycle. and so that can mean usability testing, or content testing. As our products get more mature, we also use quantitative methods. We run randomized control trials. We try to use analytics on things that are out in the wild. They use all of the methods at their disposal to try to keep bringing the voice of the people we’re serving into the product process. And I think what that looks like, as I say, just varies tremendously and we kind of like it that way. I think that’s one of the cool parts of being a researcher at CDS is we say well you have this team, they have their needs, you need to serve them and make sure they have the right information about people to make good product decisions, but there are lots of different ways we can get at those answers and we agilely – probably to overuse that word – assemble the methods and the timelines accordingly.
Steve: So, a researcher working on, for example, the Veterans Affairs, for the duration of that program that’s the thing that they’re working on?
Colin: Exactly. Exactly. So, we’ve been really fortunate to basically have one researcher per product team and to be able to keep them on the same team. Which – and sometimes that’s not possible, but you know research, as hard as we try to not do this, sometimes becomes a practice of implicit expertise, right? I think researchers who have a long history in an area, or with a product, kind of know things about how people will respond to the service that are hard to articulate or sort of systematically articulate in reports or the other ways that we try to codify that knowledge. And so we see huge benefit to people having some longevity in the product teams and thus in the domain that they’re working in. Not always possible, and I understand not possible everywhere, but for us I think researchers really enjoy kind of building a deep expertise that comes with doing 10, 15 or 20 studies for a particular product in a particular area.
Steve: Depending on the type of organization, the carryover from one product team to another – I mean in my mind government is sort of an example of a category where there’s a lot of different things that you’re doing.
Colin: Yeah.
Steve: And obviously there’s cultural things and organizational things. Whereas if you’re working in some commercial enterprise that maybe makes a lot of different products and maybe serves different customers, the breadth might be less.
Colin: Yeah.
Steve: So the value, I guess, if I’m doing the math in my muttering – the value of that sort of hard won knowledge is maybe necessary to preserve or cherish it in a different way, just given the breadth of what you’re doing.
Colin: I think you’re right. And one of the fun things about getting to support these folks is that I can have conversations on a given day that range from how the Royal Canadian Mounted Police want to handle cyber crime to how Canada Revenue Agency processes tax returns for low income people to how the Employment and Social Development Canada issues benefits, right. And those are incredibly different, both business processes and missions, but also involve very different people and very different kinds of concerns on the parts of those people. So, it’s pretty neat to get to hear about all of that, but it also creates a challenge in that when we start on a new team, or a new researcher comes to a new team, I think there’s some sort of basic government knowledge they’ll bring with them, but often they’re trying to come up to speed on a pretty complicated area, pretty quickly.
Steve: When you described earlier your role is to – you know in that kind of coaching relationship you have with the different researchers – but it sounds like you’re the one that has the overview or has the window into these different product teams and what they’re doing. Are there researchers – what kind of interaction do they have that isn’t through you necessarily about what each is working on or what kind of challenges they’re facing?
Colin: Yeah, absolutely. I am – one of the things I learned very early on was that I was not the best conduit for all of that information between them and each other. And so I increasingly view my role as creating opportunity for them to share directly with each other and across the org. So, what that boils down to – there are first of all researcher standups, where we talk about what research we have done, every week. And we try to focus on not just kind of what we did, but what we’re learning. And that’s usually not enough to create understanding, but it’s enough to create a hook, right. It’s enough for one person to say, “oh, that person is really talking about the experience of submitting a form that’s an application for benefits and that’s actually kind of similar to what we’re doing, so maybe we should go have a more deep conversation, or I should ask for some documentation.”
We also have rotating, dedicated research critique groups. So, researchers form groups of 3 or 4. They meet weekly to give each other feedback on their work. It’s a little trickier to give critique on research than it is on design artifacts, but just as important. So, they’ll go through research plans with each other. They’ll go through recordings of interviews and sort of talk about the approach that the researcher took and different ones other people might have took. They’ll go through reports and deliverables. And although the kind of stated purpose of those sessions is to help people grow and share skills with one another, I’ve observed that a real common output is better cross product knowledge between them, right. If you’ve spent the time really thinking about the pros and cons of someone’s research approach you tend to understand their product a bit better. So we have critique groups.
And then one of the other kind of structures we have at CDS is these research community meetings that are actually open to everyone from the organization. So, every other week we host these 45 minute kind of brown-bag, lunch style meetings where researchers give talks. They give talks on either their product work, when they’ve recently finished it up, or on their – or on new or interesting kind of methodological or government logistical things that they’re working on. So, we try to create lots of channels for that information to flow. I think that we’re still kind of a size where some of those meeting and interaction driven approaches work. As we grow, I suspect we’ll have to get more creative.
Steve: Right, critique circles for ten is different than doing that for 80, or brown-bags. Which is crazy that I would throw that number around and everyone would nod, like yeah, that’s the size of research groups in some organizations now.
Colin: Sure, sure.
Steve: It’s not that long ago that that was an absolutely ridiculous idea.
Colin: Absolutely.
Steve: So, at ten, you can have some kind of communal knowledge just based on – well, you’re putting formal things in place for sort of semi-formal knowledge exchange.
Colin: Yeah. Yeah, that’s right. I think that we’re certainly beyond the totally informal everyone talks over lunch about what’s happening size. I think we are at the edge of what works well for critique groups and sort of community meeting exchanges. But again, it’s a little trickier for us because our domains are so different, right. And so it’s harder to say we’re building a common understanding around the user of a particular product that we all study. Often, we’re studying very different things and there are things to learn across them, but there’s also real differences between them.
Steve: I’m going to switch gears a little bit. I think it builds on some of what we’re talking about, but as you talk about the team having grown, and maybe growing into the future, what do you look for? What makes for a good researcher for CDS?
Colin: So, we focus broadly on a couple of things. I think the first is craft, for lack of a better word. So, when we hire people, and when we’re going through interviews, we really dig into the details of their previous work. You know what – how did you construct the research questions? Why did you pick those questions? How did you pick the methods that you used to answer those questions? Why did you pick those methods? We spend cumulative hours working through that with folks because we really believe that the basic skills are so important because they’re hard to – if you don’t have them down, they’re
hard to maintain in our challenging context, right? So, you really have to be a great baseline researcher and know the basics really well to succeed here because – and this gets to the kind of second factor – it’s hard to do research within government and on product teams. And I think it’s hard to research everywhere, but there are a number of challenges that folks need to be ready to meet. One of those that we look for and also spend a lot of time talking about is ability to recruit research participants and build relationships and structures around doing that. So, because we’re changing domains and we’re going into new areas, it’s not uncommon for us to start a product and put a researcher on that product that has a very specific kind of group of people that we’re trying to talk to, and would be very hard to pay a commercial recruiter to access. And so then they have to get creative. They have to go make friends in the advocacy worlds related to that department. Or they have to work with that department to use administrative data to find people. All of these things are much harder than hiring a recruiter to do the work for you. That’s not to say we don’t use recruiters. Sometimes we can, but a lot of what we do is so narrow that we need to find people who don’t just love the craft of doing research, but love the craft of recruiting. I think over time we continue to look at ways to make less of the job, but it frankly just still is a reality given how many contexts we operate in and what people have to do. So, recruiting is one piece of it.
And I think the other piece of it is broadly what we call bureaucracy hacking and that is being able to navigate a bureaucratic process to get your work done within the time that we need it to get done and with enough cognitive flexibility to kind of think through what the really important pieces are, where there might be alternative routes to the really well trodden one, and ultimately get a result despite a complicated, multi-person, multi-process situation. So, we look for people who are excited to do that and who have some demonstrated skill navigating situations like that.
Steve: Is the public opinion research story you talked about earlier, is that an example of bureaucracy hacking?
Colin: I think it’s an example of institutional level bureaucracy hacking. I think that product teams themselves and people who are working on those teams often have to do the same thing, but kind of at a lower level. So, they’re working with a partner who says, “oh, like we have a departmental process around talking to this group of people. We need to go through that process in order to do this research. That process usually takes nine months. Your research is supposed to take 3 weeks, how do we make that happen?” And it’s kind of an interesting skillset, right, that’s required to navigate those situations. It’s not just sort of knowing what good research is. It’s also your ability to think analytically about a process and understand the reasons for the pieces of those processes and then look more broadly. I will say, as we grow, we try to do less of that at a departmental level and more of that at kind of a broader institutional level, but that’s still a work in process.
Steve: It’s interesting, and maybe just coincidental, but you talked about looking for recruiting skills, very specific kinds of skills, in talking to researchers, and then this bureaucracy hacking example that you gave is around the logistics of recruiting participants.
Colin: Yeah, yeah.
Steve: You know and – I mean I’m glad we’re highlighting recruiting. I think it’s sort of a neglected – just get people and talk to them sort of is sometimes the belief in research. You’re talking in some cases about getting to maybe harder to find groups of people, or groups where there’s a specific relationship. And we also talk a lot, in research in general, about operationalizing some of those things. We haven’t talked about that. What’s your view on – whether it’s the recruiting part or just in general, in the context that you’re in, how does that idea play out?
Colin: Yeah. So – I mean I think that it’s not surprising and certainly important that the industry at large is increasingly focused on operationalizing processes like recruiting and thinking about I think ways to devise that work and to scale it more efficiently. That seems totally reasonable given the size of research teams that are now part of the modern organization. I will say, at CDS I approach it with some caution, and I’m worried I’m going to sound a little like old and cantankerous, despite not being much of either…
Steve: You have me though on the call. So, just compare to me. I can be old and cantankerous. I’ll take the heat on that.
Colin: I appreciate that, Steve. No, I – for a lot of our work, the work of doing the recruiting is part of the research itself, right. So, actually going out and making connections with community organizations or professional boards, or senior centers, or all of the places where we do our research, that’s part of how we understand who we’re trying to serve and the social structures that surround them. And so I – when I think about how we’ll scale, I try not to think about ways that we would totally take that out of researchers’ hands because I think they would lose part of the picture of who they’re trying to study if they were to do recruiting in its own right. If they were to sort of separate that into a different role. I think that – the other thing that strikes me is that so little – like when I look at the substance of our research, at the substance of what we do, some of it would I think be possible to kind of mechanistically do faster, right? Like if we had better templates and better knowledge stores and sort of better processes to string those things together, like it could speed things up a little bit. But often the idiosyncrasies in people’s research and the sort of surprises that don’t fit within the template are the most important things that our researchers find. And you know perhaps that’s a sign of our organization’s maturity, but I get worried about kind of whisking all of it into a well-oiled, sleek process for – you know I wonder about the sort of edge of whiteboard, straggly sticky-note on the margins insights that we miss. And those – and for us, in our work, those are so often key, right. Those are – we’re still building an understanding of the space and so often those things that don’t fit well within your predetermined framework are the most interesting parts for us. And I worry about losing that. That said, making the logistics of scheduling people and where they go easier, I am all for it and we continue to look for ways to do that. But I think we have to be thoughtful about what we automate in our industry, just like all knowledge workers should be, I suppose.
Steve: You made a comment a few minutes ago when I asked – when you just said research in general – you said, “it’s hard.”
Colin: Yeah.
Steve: I just want to – I want to go back to that. I don’t know if I agree or disagree. Can we just reflect on that notion of research is something that’s hard? Like what about it is hard? Should it be hard? Is that a bug or a feature?
Colin: Yeah. As it came out of my mouth, I’m like oh that’s kind of a complicated statement, I wonder what you meant by that – to myself? I would say this. In some ways research isn’t and shouldn’t be hard. I think one of the things I love about this work is that I can sit down with someone from any one of our teams and explain some basics and they can go out and start learnings things in a more systematic way with some good pointers from a good 30 minute discussion. And that’s great. And so I don’t want anything I’m about to say to be read as kind of discouraging making research accessible to everyone. We say everyone at CDS can be a researcher and I really believe that. I will say that research is also a – one of those things that’s very easy to do mediocrely. I think that there’s a lot of subtleties to how you make people comfortable in research sessions. There’s subtlety to how you digest lots of qualitative information. There’s subtlety to how you arrange your research, so it influences your product in the most responsible, but impactful way. That’s all – that’s all – I think there is subtlety and trickiness to that and so I think it’s important to have an appreciation for people who are really good at doing those things and to – I don’t necessarily count myself as one of them for all of them – and to recognize that that is a real skill and that is a skill that we should – that we should celebrate in kind of our broader industry community. That is, I think we can do that without also saying hey, there are some basic things that people can do that are easy and quick and help make more people gather more data to make better decisions. I think those are sometimes placed in a false binary in the Twitter sphere, for example, and I am not as interested in that. At CDS, and I think my comment was somewhat related to the Canadian Digital Service in particular, there are some things that are quite – I won’t say uniquely hard about research, but are specific to our context. You know one of those is the sort of incredible domain switching that we expect researchers to do every 3 to 9 to 12 months. And that’s, I think, not unusual in consulting, but it is somewhat more unusual in a product driven organization like ours. There is also, I think, the hard work of figuring out how to fit your research into a interdisciplinary team. You know we don’t just write reports and make presentations and give them and sort of hand them off to designers. We don’t let people off the hook at that phase. They’re kind of in the trenches with all those other people as the product is being developed. And we expect the way that teams run their agile development process to reflect research and for researchers to be a voice in that. And I think there are lots of organizations that have that expectation. I think we’re trying to do that in combination with domain switching and with the third part of it which is, again, we’re trying to build skills. Not just in ourselves, but in our departmental partners. So, you’re trying to learn a new domain, fit your research skills into this rapidly evolving, interdisciplinary team, and help a partner learn the basics of research and appreciate research. I think that’s hard. I think it’s fair to say that’s a difficult thing to do.
Steve: I want to switch a little bit. I would love to hear if there was a point in your own personal, professional path when research was a thing that you identified with? Like I want to do that, I am that, that’s for me? I don’t know, is there a moment or a stage at which you connected with the field that you are in now?
Colin: Yeah. There was. I conducted my first usability test when I was in 7th grade. I was – this was pretty early in the days of such things. But I was working on the school’s website. That was sort of my hobby. I helped the computer teacher with that. And I was reading Jakob Neilsen’s Homepage Usability book. There was this beautiful book with multiple sort of printed out homepages and he talked about his methodology in the back and I read that. I was like oh, this usability test is sort of something we could do. So, I got someone into the computer lab and tried it out and there was this moment of like wow, when you ask other people to try something, and when you ask them questions about what they’re thinking and what their goals are, you challenge yourself in ways that you – that I didn’t expect. And it’s also incredibly rewarding. It’s an incredible high. I don’t know how else to describe it. And it continues to drive me and the folks on my team. So, from that moment on I knew that I wanted to do this kind of research in some way professionally. There were lots of steps between there and here, but I knew pretty early in life that I really loved learning about how people used computers and services and understanding their approach. And, you know, I’m really blessed to work alongside lots of other people who are similarly jazzed to learn those things about people.
Steve: And that’s an astonishing story both – to me – in the early point in your life at which this happened and the specificity when it happened. I think often these stories are about oh I went to a farm and then I realized I wanted to be a marine biologist. Like the connections are more diffuse. But you, at a very young age, were doing exactly, or pretty close to exactly the thing. I mean it’s not even a metaphorical discovery.
Colin: Yeah.
Steve: You literally discovered the work.
Colin: It’s odd to me too. And I do look at friends’ and family’s career trajectories and I’m like hah, I didn’t think that’s how it would work out for me, but I really – I just loved it. And I looked into other things, but kept coming back to this, this being what I really just enjoy doing.
Steve: Are there any points, whether it’s 7th grade or older or younger – what kinds of things can you look to in your earlier parts of your life that were, I don’t know, maybe weak signals or kind of nascent superpowers that connect to what you’re passionate about and what you’re spending time in now? Does anything exist earlier in your life?
Colin: You know, one thing that has always been somewhat odd about me, and I do think relates to my passion for this work, I have always loved seeing the metaphorical or physical backroom of the process. So, like when I’m flying, I’m always like hah, like what does the computer system that the, you know, gate agent used look like? And like what do they do and when do they do it and why? And I’ve always – I’ve always been like that to a point of I think some well natured teasing from my family about my desire to understand the details of how the parking ticket machine dispenser system works and what the numbers on that thing mean. So, I think, especially to do this work in government, you have to kind of have a love of uncovering process and humans and how they interact with that process. And the trick is you don’t have to love like process, but you have to love kind of post modern process. Right? You have to love the fact that like it is different things to different people and kind of is this great mirror game of clarity and unclarity all at the same time. That’s something from an early age that I’ve always really enjoyed. I also would say that I – so early in my career I worked for the U.S. National Park Service and one of the things that I did there was being a park guide. So, be out on the trail talking to people and listening to them and answering their questions and seeing how they interact with a place. And there – it’s real joy – I feel real joy to seeing people in an environment and both helping them discover things and see how they discover things. And often I feel like my work today isn’t that much different, right. I think I’m more open ended than park interpreters are, but it’s still here is this new land and this new world that you are encountering, and I get to be with you as you’re doing that and help you think through what you’re doing. And so I think the kind of odd areas of joy I found in that work certainly kind of echo through to what I’m doing now.
Steve: Is there any example of a notable success? Something that, you know, maybe that your researchers have done and kind of completed that you feel proud of? Or just kind of a notable example to share?
Colin: Yeah. I think that – it’s funny. You know we’ve done a lot of research that I’m proud of and I’m proud of our researchers for doing at CDS. But the things that make me most excited are the moments when I see our researchers teaching and coaching people and departments to start doing research themselves, right. So, for example, with the Veterans Affairs Canada project they – the department identified someone who would be a researcher and sort of continue on the research with that product after we handed it back off to them. And I, you know – those moments when those people were working together and getting to support them through that, I think those probably make me feel prouder than any particular deliverable or product that we’ve gotten out the door.
I will say the other exception there is I’ve looked – I was looking at our sort of internal database of research before this interview and I’ll say the other thing I’m particularly proud of our team for doing is becoming more and more diverse and creative in the types of research it does, right. I think when we started, we were trying to make the basics happen and so we did lots of interviews and we did lots of usability tests. And those are great, but there are other things to do too. I think we’ve become not just qualitative, but mixed methods. I think that we have embraced the complications of trying to kind of test services in more live ways, right. So, not sort of just usability test prototypes, but wire something up so that someone can complete an actual transaction with it and see the back end experience and orchestrate kind of a Wizard of Oz experience for them. Those, those increases in our skill and our ability to mix and match tools responsibly, that’s incredibly rewarding as a researcher at heart.
Steve: Is there anything that I should have asked you about that I didn’t, and you want to make sure we talk about?
Colin: No. I think your questions were well rounded. Nothing comes to mind from my list.
Steve: Do you have any questions for me?
Colin: What’s the most surprising thing you’ve heard during one of these interviews?
Steve: Well, this is going to sound like I’m pandering, but the fact that you did a usability test in 7th grade is – I’ve just never come across that in my career, let alone in one of these interviews. But actually – I will say – I said I would take the weight of being the old curmudgeonly person. So, here we go. I don’t like the what’s the most surprising thing question ‘cuz it presupposes that the person – and I realize this is not exactly research since we’re doing a podcast, but as an interviewer in general, I’m not trying to be surprised. I’m trying to be focused and interesting and sort of the most benign sort of factual thing should also be exciting and triggering of interpretations. So, you know what I mean? Like the surprise is me bringing my judgement into it. So, of course I gave a concrete example. I was surprised by something that you said, but I also…
Colin: Yeah.
Steve: People ask researchers that, right? When you did this study what’s the most surprising thing? And I think sometimes that question can be good if it provokes what are we learning, or where are our assumptions being challenged? But, how do we – I also want there to be space for like hey I didn’t come in here with assumptions. I came in here with just curiosity.
Colin: Yeah.
Steve: So, I’m kind of talking out of both sides of my face here, I guess. I do that.
Colin: Well, I agree. And I do think there’s a real risk to boiling research into a set of surprising and non-intuitive insights, you know. Like the idea is that we should be manufacturing surprising little blobs of knowledge that can be passed off into the ether. And I worry about that. I worry about the work getting boiled into that. And so I hear you. And I guess I agree with your curmudgeonly instincts.
Steve: Aren’t I a nice person? Any questions for me? I mean I’m going to slap you for the first one that you asked me. So, I’m just living up to the curmudgeonly mantle I chose.
Colin: I expect nothing less. Well, that’s good. No, I would say I am curious as to what your views on the burgeoning research ops movement are? As someone who I think has been in this world for many of its evolutions, what’s your reaction to the increased operations emphasis?
Steve: I mean I have – learning about research ops provoked a uninformed critical reaction. And you said it really well when you talked about the value that the recruiting can bring to the researcher. And, you know, we hear researchers talking about the value that they get out of doing their own transcripts. Personally that’s not a thing that I could deal with, but immersing yourself in the subject matter and the people in the community and the data and the conversations is super labor intensive and the idea that we’re going to fix that by kind of operationalizing things scares me because it takes away the quality and – you talked about subtlety, I think. But if you look a little deeper into what research ops people are championing, they’re not championing that. They’re – that’s my chauvinistic misinterpretation. And I think just by creating that word, I think it invites other people to also do that and maybe even implement that. But the people who are championing it are thinking about process and thinking about how does this exist? And, you know, having spent my entire career as a consultant, so not embedded in the organization, not building these processes – maybe advising on them, or kind of giving feedback, it’s sort of an interesting thing to watch from the outside. It comes up in a lot of groups. I talk to people who are kind of early on in their organizational maturity. You know I read something recently – oh, some research ops people were sort of declaring that the first researcher hire should be an ops person which, you know, from the outside seems like well of course you would say that. That’s sort of a discipline that you are evangelizing for. But I found myself working with a group where I just mentioned that idea as an idea that exists. I wasn’t in a position to say they should do this, they should not do it. But I said, given what you’re talking about your barrier here is ops, is infrastructure, is recruiting, is not reinventing the wheel. And I said, you know – they had headcount to hire one person and I said well what proportion, from zero to 100, is that person going to be dedicated towards this? You know, if the same way that you are – that you talked in the beginning, about your organizational change efforts, you know convening people, finding processes, sharing best practices. I think a research ops person can also be doing that, you know at a different level and using kind of different processes than you’re talking about, but…
Colin: Yup.
Steve: I mean that is championing the growth of research as a thing that we have skills to do at various levels efficiently. That’s what you’re – that’s one of the things you’re doing. That’s one of the things a research ops person could be doing. Taking away the hard, labor intensive, inefficient tasks that lead to the subtlety that makes research great, that you described -that’s not a thing I want to do, but I – it doesn’t have to be what ops is, so.
Yeah. If there’s nothing else, then maybe we’re at the end. Really great conversation. We covered so much interesting stuff and I learned a lot and I’m really so happy that we had the chance to speak today.
Colin: Likewise. Thanks so much for your interest. It’s been a pleasure.
Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit portigal.com/podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.
The post 27. Colin MacArthur of the Canadian Digital Service first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I talk to Jesse Zolna, who leads the User Experience Research Team at ADP’s Innovation Lab. We talk about driving change as an experiment, exposing the organization to how customers solve problems, and engineering psychology. One of the challenges we face is getting “credit” for the work that […]
The post 26. Jesse Zolna of ADP first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I talk to Jesse Zolna, who leads the User Experience Research Team at ADP’s Innovation Lab. We talk about driving change as an experiment, exposing the organization to how customers solve problems, and engineering psychology.
One of the challenges we face is getting “credit” for the work that we’ve done. A lot of what we do is help people understand the problem space better and understand these things that their users aren’t able to do, or want to do, or whatever. And oftentimes it’s not going to be brand new. Rarely do you come up with something that nobody’s ever thought of before. A lot of times we help solidify or better articulate those problems, which then you can attack much better. – Jesse Zolna
Show Links* Northern Soul * The Day Mick Jagger and Keith Richards Met Again * ‘Friends’ Will Be There For You At Beijing’s Central Perk * Lowrider Culture Spreads to Brazil and Beyond * Russian Doll * Ronald does the wai * McDonald’s to Offer International Menu Items Including (Yes!) the Stroopwafel McFlurry * Jesse on LinkedIn * Jesse on Twitter * ADP * ADP’s Innovation Lab Drives New Ideas And Cultural Change Within The Company * Barnes & Noble * Nook * Stephen Gates on Design Thinking * 5 Whys * Tufts Department of Psychology * Adlerian psychotherapy * Georgia Tech Engineering Psychology * Airtable * OKR * DJ Anne Frankenstein
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Northern Soul was a musical and cultural movement in the UK in the late sixties and early seventies. It was all about obscure soul music from America. The movement really was a scene, with clothing and dance styles, and clubs hosting dance parties, but let’s just focus on the music. For people in the UK in the 60s it wasn’t easy to get music from the US. In fact, this difficulty figures into the origin story of the Rolling Stones, where Mick Jagger and Keith Richards reconnect during a chance meeting on a train platform, and one notices the other has possession of some rare and desirable albums from the US. Anyway, Northern Soul started in that context initially, the difficulty of getting any of this music and then went on to specifically emphasize the rarities. Remember that there was no consumer music duplication technology – you had to have the 45. Many of the songs that became Northern Soul legends were commercial failures, failed artists, failed labels. And yet, these 45s, these songs found a second life, across a time, across an ocean, across cultures.
Decades later, we have the Internet, and we have globalization, and we embrace consumer enthusiasm. So there’s a cafe in Beijing modeled after Central Perk, from the TV show “Friends.” Mexican-American lowrider culture has been taken up in Brazil and New Zealand.
I find these examples fascinating, and given the frequency that these stories appear, I’m not the only one. Given the work that I do, my interest is specifically because these stories typically involve people going around the brand. The producer makes certain products and provides them to a certain audience in a certain marketplace. Sure, it’s a statement of identity to watch “Russian Doll” on Netflix, but your effort to both discover and consume is minimal. But to buy authentic parts for a product that isn’t made any more, from another country, for example, takes a lot more effort. Even with the Internet. This kind of lead user consumption is interesting.
So that’s the consumer side, going around the default path laid out by the company. And on the producer side, larger companies seem pretty intent on managing both globalization and localization. You can go to a McDonald’s in 101 different countries, but the menu will be different. Outside McDonald’s in Thailand, Ronald is posed giving the “wai”, the traditional Thai greeting of a slight bow with hands pressed together in front of the chest. Even though the McDonald’s brand spans cultures, the Thai experience is specific and self-contained within its own environment. McDonald’s redefines itself within the boundaries of the national border and even though we know Ronald can be found everywhere, this Ronald reminds a Thai customer that he is specifically in Thailand. Ronald is trying hard not to be the tourist who awkwardly adopts the local customs in order to seem “down” but actually someone who has moved in and become part of the scenery. Of course, brands, their symbols and indeed their products change meaning when they move from one culture to another, but we can consider the bare minimum meaning, just the fact of its existence in any particular culture.
Earlier this year, McDonald’s in the US introduced a limited-time “International Menu” featuring Stroopwafel McFlurry, the Grand McExtreme Bacon Burger, the Tomato Mozzarella Chicken Sandwich and Cheesy Bacon Fries from respectively Netherlands, Spain, Canada and Australia. Linda VanGosen, McDonald’s vice president of menu innovation said in a statement that “We know our US customers are curious about McDonald’s international menu items.”
It doesn’t matter if these are any good. And these aren’t intended to be authentic representations of the cuisine of these other countries; they are presented as authentic exemplars of McDonald’s in these other countries. The point here, the fascinating thing to me, is that McDonald’s is acknowledging, at least to its American market, that seams exist, that the way you experience the brand is limited by your geography, and another variation of McDonald’s – a non-American version – is out there. This is a very contained action by McDonald’s but the way it breaks the frame by pointing to something outside what they so carefully control and design is huge.
The homework for all of us is to keep our eyes open for how and when producers acknowledge in any way what lies outside the set of things they are providing to us. I believe this will continue to change. And for those of us who are in the business of producing things to be consumed, McDonald’s is signaling here that we have more choices than we might have previously thought possible.
This is an important aspect of the work that I do, helping companies to unpack these shifts in culture and consider the ways they might respond, in terms of what is appealing to the marketplace and what is authentic to the company itself. My clients have a lot of deep knowledge of what they have been doing, but they often need help getting outside that, and I help teams to build a new shared perspective and a plan to move forward. And so, the best way to support this podcast is to support my business. Hire me to help you bring a nuanced external perspective to how you understand your current and future markets. Get in touch and let’s discuss what we might do together.
I’d also love to hear how this podcast is helping you in your work. Email me at DONUTS AT PORTIGAL DOT COM or find me on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
Let’s get to the interview with Jesse Zolna. He leads the User Experience Research Team at ADP’s Innovation Lab. Jesse, welcome to the podcast.
Jesse Zolna:Thank you, Steve. Thanks for having me.
Steve:Let’s start by having you say a little bit about who you are and what you do and we’ll go from there.
Jesse:Okay. So, I lead the User Experience Research Team at ADP’s Innovation Lab, is one of the ways I put it. ADP has a sort of complex arrangement of business units and I fall within what we call shared products. It’s the stuff that runs behind all the different business units. I end up interacting with a lot of the different front end products that ADP makes. And so, just as a short term I say the innovation lab. I started here about 5 years ago when we opened the Innovation Lab. So, ADP made the decision to invest in UX and agile and design thinking, sort of all at the same time. I think I was hired #18 or 19 in the group. So, I’m leading the UX research team here in the lab.
Steve:What are some of the business units that – your lab works across these different – for those of us outside of the organization, what things could we learn about that ADP makes that you’re kind of working to inform the design of?
Jesse:At the simplest level, ADP has sort of three business units – small, medium and large businesses. And we make essentially payroll products for those businesses. That’s our core product and then on top of that we layer what we call HCM products, so anything that’s related to HR really. And then a lot of that will feed into payroll. We have what we call core HR which is all about people’s data, about who they are, where they live, and that stuff impacts the taxes that we take out of your paycheck. We have time products within each of these business units. You’ve got to record your time and obviously that calculates your pay. We have retirement services products, so like 401(k) stuff. Medical benefits, that kind of stuff. So, if you think about the paycheck as sort of the center of our business and then all the sort of – we have lots of different products that sort of feed into that center of business.
Steve:What does HCM stand for?
Jesse:Human Capital Management. It’s the newer term for HR.
Steve:I’m old enough to remember when it was called personnel and then that got outmoded by HR and now I’m going to be a dinosaur if I say HR.
Jesse:So, back then HR were bean counters and the just a cost center – the people were just a cost center. Now, today, people and talent are a company’s most valuable asset.
Steve:Okay. So, the term reflects kind of a different mindset.
Jesse:Yup.
Steve:Having heard it from you I’m sure I’ll now be recognizing the term when I come across, but I haven’t heard that before. So, I’m wondering, you described some of the different services that are provided – you’re providing services to companies who use these products to do all the human capital management and payroll for employees. How much of these different services do the employees themselves interact with? Do you know what I mean?
Jesse:Yeah. So, the initial project that I worked on when I came here is what we call the employee and manager experience which is essentially both a desk and web app that the employees use. Until ADP decided to focus on that, that was sort of an afterthought. We were building products for the HR and payroll managers that worked at our clients and an unnecessary evil was things where the employee needed to be able to go look up their information or enter their time or whatever. You know if employees wanted to get paid they’d figure out how to use our time products, so we didn’t invest a lot in that. But then as part of the decision to focus on that they realized that actually 99% of their users were employees and managers because each company has 10 HR and payroll people and 10,000 employees, or something like that. So – and it was also sort of a push into realizing that technology was changing and people were expecting better from everything that they use on a day to day basis and getting basically complaints from our clients that their employees found our products hard to use which would then reduce productivity, for both the employee who was trying to figure out how to enter their time, or figure out what medical benefits they want rather than working. And then the HR, the payroll person, had to fix errors, or help the employee navigate the system instead of do like more strategic work that they should be doing within like talent management.
Steve:So, the customers are identifying – your customers are identifying changes they want to see in the products you’re delivering to them, or the kinds of experiences you’re creating.
Jesse:Yup.
Steve:Which is not a signal to ignore.
Jesse:Yup.
Steve:So, this is the innovation lab. So, what does that mean? Those are both sort of very loaded words. For ADP what was sort of the – what was created at this kind of transition point?
Jesse:In the beginning it really was an experiment in choosing sort of the ratio of UX to development and at the same time moving to an agile development process. So, what they did was they hired all new people from different places. I came from Barnes & Noble. Worked on like their E-commerce site and the Nook device. A designer who came in at the same time he came from a media background, so had worked for NBC and MTV in the past, on their like web properties. So, they were trying to bring people in from different industries and think differently about what consumers want? And they hired about, I think, 100 people in the early days and 20 of them were UX people, whereas at the rest of the company there were probably 10,000 developers and less than 20 UX people. So, it was an experiment and if we really invested in this like will the outcomes be different. So, we had the employee and manager experience as one of our first projects and we sort of proved that you can move faster and get better results with that new ADP innovative way of thinking. So, the idea was to be an example to the rest of the company. So, now the rest of the company has seen what we’ve done at the lab and over the last 5 years has really adopted a lot of those practices and hired a lot more UX people in each of the different business units, each of the different groups and transitioned to agile development process. Both of those have been more successful in some places than others, but everybody is trying to move that way.
Steve:It’s interesting that you frame this as an experiment. What did that feel like kind of in the early days. You were, as you said, one of the early people to come into this. What did it feel like to be something that was framed as an experiment in a company this size, and I think with a big history, or long history?
Jesse:It was exciting because there was the opportunity to jump right in and have a large impact because there are so many clients and so many people using our products, yet we could start from scratch at the same time. And that’s how they sold it to us, like a startup within a big company, and we already have scale. And the other thing was we could do whatever we wanted kind of. We thought that was the right thing to do. We could do that. You know we didn’t have to go through all the red tape that the rest of the company had to do. So, we could just, you know, spend money on recruiting. We could have a very simplified NDA. Talking about research processes, that would just make it easier for us to get our work done and not have to get approval from legal and GSO and everything, which these days we have to deal with a lot more.
Steve:So, over these 5 years, as you said, the things that only were happening in the lab are now starting to, or have started to, happen in other parts of the organization. That seems like one measure of success for the experiment. It’s kind of creating models for practices for the organization to adopt. I wonder what does that then create, or open up space for the innovation lab itself to them move into different kinds of work, sort of after having birthed some processes and have them been taken up – what’s shifted for you all?
Jesse:I would say at the beginning we were really focused on our one product and proving that the user centered design process could work and you could get rapid feedback and it didn’t slow down development, and then you would see the effects on the market. And now other teams are doing it, and as you said one metric for success. Another one is that we’ve sort of changed the organization’s thinking around UX and what user centered design really is. So, the sort of simpler application of UX research is usability testing, or more evaluative, or maybe even like concept testing, like lean UX testing. When I came in, lean UX testing was like really all they thought research was. So, we’ve established also a program of ethnographic research where we do sort of foundational stuff that doesn’t necessarily have what changes can we make to this product to make it better, but broader, like what changes can we make to this quite of products and understanding how our clients are using our products within a greater context, to look for the “white spaces” and that has really opened up the organization’s eyes in terms of how research can help strategically too. You know strategy used to be the domain of only product management. They would come up with a strategy – you know based on what our competitor did, now here’s our new strategy. By the way, the competitor probably just copied us, so there’s a lot of like circularness to that. So, we’ve been able to bring in new thoughts and new ideas which I think has helped us rethink the way that we’re approaching providing tools to our clients to manage their human capital.
Steve:And is there a connection between the success you had with these initial efforts, in terms of changing the mindset of the organization? As you say, ideas only came from these kinds of places and now – obviously we know that ethnographic research can provide the kinds of new ideas that you’re talking about, but the change that’s happened here is that – I think you’re describing that people are receptive to that? They’re like oh yeah, this is another source for us to think about what to do. So, is there a connection between that mindset which is new and then the – was that driven by the work that the lab started off with?
Jesse:Yeah, well it’s related to what I said earlier about us being able to do what we think we should do because we think that’s the right thing to do. So, in the early days we were working on payroll and, aside from the employee/manager experience, I was helping with the payroll product which again is our core product. And I realized that like we knew a lot of people – we had people who knew a lot about how the time product works and how they gather that information and then it gets sent into payroll. We knew a lot about how payroll would process that. We knew a lot about how core HR data would influence that. So, we had people who understood those silos really well. But there was no understanding of how those things all kind of worked together. So, I proposed that we do some site visits where we just go watch people during their payroll process and just like say, “teach me how to do payroll from your company.” Because also every company kind of does it differently. They all have ADP payroll, most of our clients, but then they might have our time system. They might have somebody else’s time system. They might have somebody else’s compensation management system, or ours, or whatever. So, everybody kind of does it differently. So, we went to visit a bunch of different clients and like try to find the themes across and what we came back was information that was very surprising to the people internally and actually our first – we presented it several times and the first few, their reaction basically was give the name of that client, I’m going to go show them how to do it right. And I was like well, I can give you the name of all 12 clients we visited, but then what about the other 99% that we didn’t visit, are you going to go train them too? And they like didn’t quite get – they wanted to go solve it for that client that’s like very solution oriented, rather than understanding the problem and thinking about how can we work together. And they literally didn’t believe me that clients would double and triple check the time data before they put it into payroll and then after they put it into payroll, and spent hours doing this. Like literally going through Excel spreadsheets. One of the funny things the researcher on the project came up with, she used to call it the rulers of payroll. We literally had clients that would use a ruler, like on the screen, so they could look across these really long lines of data. And somebody else would be on the other screen and they’d say whatever, 1,400 – I don’t even know what the numbers were, but they would like call and respond to make sure that the like thousands of lines of data – and they were like why would they do that? It just imports automatically. But I mean the answer really was ‘cuz one time in the last year, or whatever, or maybe 20 times, I don’t know, there was an error and that caused like somebody to not get paid. And the payroll person thinks maybe they won’t be able to pay their mortgage because they didn’t get paid. So, they want to be super sure. So, anyway, it was like this idea of understanding things across these silos, coming up with these new insights that we never would have seen if we didn’t look across those silos. And so since then we’ve done probably 8 or 9 of those projects. And really we kind of started off with here’s a big domain that we kind of want to really understand deeply and we don’t necessarily have a specific piece of the product that we’re going to improve from that. And I think it gives everybody a better understanding and it gets the silos together. We actually – that research program we have has three stated goals. The first is to get people from different silos working together and to understand our users in the same way so that when they talk they have like an equal understanding and like the same vernacular and things like that when they do try to get together and work together. Number two, is just that – we call it the foundational understanding of what are our clients doing? What are the biggest pain points? How can we help them? And number three is to include non-researchers in the research process as well. So, when we go onsite we aim to have one researcher and two or three like developers, product managers, designers as like observers and note takers. So, they get to participate in the research process, understand it, see how it works. Maybe bring some of that – not only bring that empathy back home with them, but bring some of that research process and understanding too so they can think about when might be another time to do research as well.
Steve:Lots of really fascinating things. I want to pick one thing to go back to. So, when you started to present back these kinds of behaviors that you observed, like the triple checking and so on, it was surprising. “I’m going to go fix that.” What do you do after that? What do you do to sort of bring – to help these stakeholders understand what you learned and what it means and a better way to think about it than sort of fixing each of your thousands of clients?
Jesse:Actually that initial reaction was why we decided to include the third goal which is to get them out there because I figured if these people were there and they saw it they wouldn’t not believe me. I was lucky that we had some of the people who helped to establish the innovation lab and really believed in user centered design and research and are of the mindset that they want to learn. I think that makes a difference too with researchers working with people who want to learn versus people who don’t necessarily want to learn, or just want you to prove that they’re right or whatever. That’s a whole other topic of conversation. But luckily we had a few people there who from an executive level, or air cover level, were able to – I remember literally one woman said to me – you know, I told her don’t shoot the messenger, like listen to him. Like try and understand what’s going on and try to work with that and not – you know so that was super helpful to have like some believers that could help them kind of work through it really. And it took time too. Like you know just like the initial reaction was that’s wrong, like no way, let me call them. But over time I think these people were able to realize, oh wait, maybe I should like look into that. And maybe if they did look into it they could see some evidence that helped them believe and understand it better too. Because also having gone and done this ethnographic research, you know I get a super deep understanding and I can only present so much of that data and like try to tell that story and purposely try to tell it at a high level so they can understand it all. And it’s really getting those deep details and those, almost the anecdotal evidence that really brings color and life to it, that again the reason that we came up with that third goal to get those people out there.
Steve:So, if you think about someone who – it might be someone – I mean I think your caveat about they want to learn – someone who wants to learn but maybe is inclined a little bit to say well I better intervene and fix their erroneous behavior – what changes for them when they see the behavior that we’re talking about versus they hear about it from you? Like what is – I think there’s something fundamental that’s different for them. What is that?
Jesse:I think it just becomes more real. It’s no longer a rumor, or somebody else saying this. They like see it. It’s no longer a PowerPoint slide, or even – and this is why you use quotes and videos in your presentations, right, like that makes it real too. It’s no longer words on a page. It’s a person feeling pain in front of them and it’s just like more – it just becomes more real and tactical and you know I think it just fills in all the details that you need.
Steve:It reminds me of – I guess of a minor failure a few years ago where we were – our users we were looking at were people who were “unbanked” is the phrase, or “semi-banked.” And a tech company where the team themselves, that was not their lifestyle and we went out in the field with them and they were so affected by the experience, but then they asked us to – I think we didn’t do a good job sort of understanding the request, but they wanted an edited video of an extremely tight running time to like use in part of a meeting. And I think it emerged later on that the objective was for people to have the same emotional reaction from that that they did being in the field. And I get where they’re coming from. Like this really changed us and we want the rest of the organization to have this. Eventually I realized oh, you can’t get that from a video. Even though, as you say, videos sort of bring it alive more than words on a page might, but this experience of being out there – I think you’re right about sort of the anecdotal stuff. There’s the facts, sort of the details of the narrative, but you just saw things that I think those people were very moved by what happened to them in the field.
Jesse:Yeah. I think part of it too is they might not go to all of the visits, but they see one person and then when you present to them like the theme, the summary, then they can say, “oh yeah, I saw that in Joe.” So, they just understand it sort of one level deeper. And that’s what like as a researcher we often do that, like we have these themes and we understand it at a really deep level, but it’s hard to get that deep understanding. It’s hard to express that without showing all the data, but you’re trying to do – you know, just like these people in this meeting only had a certain amount of time, you know you only have – you can’t have everybody go participate in the entire research project the entire time. So, you have to summarize for them. It’s part of what we do is we go learn things and we summarize it for people so that they don’t have to go learn it themselves. But you just can’t get all of your learnings across. It’s hard. Which is part of the reason, on another subject, why, no offense, but I personally prefer being like an internal research person and like having worked on these projects for 5 years, like I feel like stuff comes out from 3 years ago that I can go back to. And I think there’s that sort of institutional memory which I realize now it’s ironic because 5 years ago when somebody would say, “oh well this is how we’ve always done payroll” I would like want to strangle them and be like that’s the worst thing for a researcher to hear is that we already know it. Like we don’t need any new information. Now, when people come to me with questions I’m like well I already know the answer to that because we did research 3 years ago. So, I have to actually consciously make sure that I’m always open for learning too, having been here for so long. So, maybe that’s the negative of being an internal person. As a consultant, you can always come in with fresh eyes and be willing to learn something new.
Steve:Right. I mean, I don’t know, the thing I like about consulting is – I mean, I’ll say the same thing you’re saying – the thing I like about consulting is being able to come into something that I haven’t seen before and try to figure it out. It’s often usually overwhelming, but I rely so heavily on people that are, that understand the problem space, so that I’m not trying to like reinvent the wheel. And I have admiration and a little bit of jealousy for people that stick with something for a long time because the time horizons for – I mean the kinds of change you’re talking about making, it’s very long. I don’t work that long term with somebody. My rationalization is we need both. You need people that live in the problem space and facilitate new ways – that hold onto the depth and sort of advocate for that depth of insight. And you need ways to keep getting fresh insight that you have with your changes over your tenure. I think we agree and you’ve got to find the role that you can sort of thrive best in, I guess.
Jesse:Yeah, totally. Yeah I mean even as an internal person I like to go into a research project as naïve – maybe not as naïve as possible, but relatively naïve so you could naturally ask the question when – well what do you think it should be – sort of in the research process. Or, how would you want it to go? Instead of like knowing the answer to that and possibly being somewhat biased or explanatory rather than questioning.
Steve:I shy away from getting into like technique too much in these conversations because I think we’re looking at the organization, but that being said, you said something that kind of intrigued me and I just wanted to ask a little more about it. When you described these ethnographies you said going into these organizations and saying, “teach me how to do payroll.” I don’t know what you literally said in the actual interviews, but any thoughts about the framing? Like there’s lots of ways to get to learn what people are doing and I wondered if you had a point of view about “teach me” as sort of the mode of inquiry?
Jesse:Yeah, I mean in that case we were purposely trying to, um, understand – uh, figure out what we don’t know. You know you don’t know what you don’t know. And we were trying to get the user’s point of view on it. So we didn’t – in that case we did not want to kind of structure the interview very much at all in a certain way because we wanted to see sort of what came out from them. We were possibly overguarding against imposing our own sort of biases in understanding and structure in it, but I mean that was essentially why we chose that sort of line of inquiry. And I do think there were literally like 4 or 5 questions we used there. And will say that – so my – we started to talk about my background, but my first job out of undergrad was for a marketing research company and their technique was literally one question and they would do one-on-one interviews and it would just be like tell me your thoughts and feelings about how “X” impacts your life? Or tell me about your – something like that. And it was literally one question. And then you were only “allowed” to repeat a word they said, or like say – we would either ladder up or ladder down. So, it’s either – so if they’d say like, “it makes me feel tired.” “Well, what happens when you’re tired.” Those would be like laddering down. Or like what does tired mean to you, or something – I think that was laddering up. It was a long time ago now, so I’m probably butchering it a little bit. But like literally one question and then like never say a word that they didn’t say yet. So, that’s kind of the technique. That has driven sort of my discovery type research, exploratory type research ever since, is like never impose your own words or structure or anything on it and just let the person you’re talking to teach you.
Steve:I’m so self-conscious of whatever word I use now, or whatever question I ask you. The reason I’m curious about the “teach me” question is I think you could ask them – since we’re talking about what are these kinds of questions we might ask, is there a difference between how do you do payroll and teach me how to do payroll?
Jesse:Um, yeah I think there is. Um, I mean I don’t know if this is necessarily true, but if you say how do you do payroll they might say this person is from ADP, I’m going to tell them how I use ADP products. Or, I’m going to tell them the right way. Whereas if I say teach me they’re going to show me the real way in a way. You know what I mean?
Steve:Yeah.
Jesse:Like it’s almost like you go to a university course on how to do something and then you go in the real world and you try to do it. It’s not exactly like the book says. Right? So, maybe it’s how do you do payroll – they would give me the book version. And teach me how to do payroll and they’d give me like the real life version.
Steve:That’s a really nice distinction between the two. So, we’ll switch gears a little bit. So, as part of this effort to bring more people out in the field and get them kind of exposure to the lives of customers and really participate in research – I think you talked about note taking and other kinds of roles that you’re giving to them. So, they are – at one level they are performing some of the tasks of research. I guess I’m wondering what’s – is there more? Is research happening that doesn’t involve you? Are they doing some of this on their own? I don’t know who “they” is?
Jesse:So, it’s related to the idea of the innovation lab sort of teaching the organization how to do these things. And so, I mean, because we’re getting different people from different groups together too – it’s not always a group that has like a robust UX function, right. So, they might go back and if they want to continue to be – to learn, they might have to do it themselves. And yeah, I think that’s happening. I do think there’s sort of the constant debate everywhere, and certainly internal within ADP. You know we have different answers in sort of different parts of ADP is like who should do the research, or whatever. And you know my team are all like dedicated researchers and we have our design partners that we work with and the design partners certainly help shape the research and we certainly help shape the design, but we’re sort of two different people and we think that splitting the sort of lead responsibility is more efficient and we can get a lot more done. Some places, even within ADP’s organization, they say that designers should do the research. And you know I’ve observed also, sometimes some people say designers should only do like evaluative research, like prototype testing type research because that’s sort of more straightforward. You know you get a task, you ask people to do a task. It’s a lot harder to like, I don’t know, lead people, I guess. But some people might argue that they might be biased and they might interpret – if they love their design too much. Again, it goes back to are they really willing to learn, or do they just want to like collect data to show that they’re right? It’s just like different ways that different people are. So, some designers probably are great at doing that and some designers might not be. And then some people might say designers actually should be part of the exploratory research so they can understand it more deeply and really – you know like we talked about earlier, having collected the data you understand the data more deeply and really understand the pains that they’re designing for better when they get to the design process. I guess you could add those together and say designers can do any kind of research. And I think that’s probably true. And then, not only designers, but product management, could also do that stuff too. We definitely encourage everybody to get out there and talk to clients and understand what they’re doing. Sometimes I worry that if you get out there and talk to one client you might overcompensate for their unique need, or pain, or whatever, which could ruin it for the other 99,000 clients or whatever. So, we do, in that program I was talking about, try to force people to participate in more than one data collection point.
Steve:I think there’s just lots of interesting issues around this. And you’re right, this is a very common topic, or question or debate, or just issue. I mean I just wonder long term – I mean right now there are people with the title researcher who have sort of expertise in all the things that you’re talking about. You listed a bunch of different aspects of what researchers are doing and I wonder sort of where does it take us, and I don’t know whether it’s 2 years or 50 years, where we’re holding on to some part of it and we have a belief that we bring value by doing that and we’re empowering other parts of it. And we believe that that adds value. Sort of wonder what will that ratio be? Or what should it be? Or will we have a role with this title in the future? Or is it a process that then gets handled a different way? I don’t know. I don’t know if you have a perspective on the misty futures?
Jesse:Yeah, I don’t know. It’s a tough question for sure. I guess while you were asking that question it reminded of – I heard a great quote recently from, I forget his name. You probably know his name, the guy who leads the design team at InVision, his quote about design thinking was “the worst part about design thinking is the word design” because everybody should be thinking this way. Obviously empathy is the number one step in design thinking. Everybody should be thinking empathetically. And we definitely have groups within ADP too where you – where you “UX” has matured enough that everybody is thinking empathetically and they think about UX as sort of everybody’s role – development, product. You know we have the triad formation – UX, Development and Product – and they work together on things. In some places all three of those people are really thinking empathetically and about end user needs. And also in most places the UX people are thinking about is this feasible or not, right? And trying to understand the tradeoffs in terms of like can we still get to the same sort of meet the user need in a different way that might be more feasible, or easier to do, or whatever too. I mean I think you’re really being successful at “design thinking” when everybody is doing all that stuff.
Steve:It reminds me of just the evolution from human resources to human capital management. There’s a fundamental shift in the belief of sort of the value of something that we used to optimize and now we want to enable and that you’re talking about sort of how products get developed and this idea of empathy being something that everyone can and should have, not sort of one group is helping another group, but that it’s a shared value or principal. So, I wonder if there’s just opportunities – this is not ADP specific, but just the profession in general to rethink how some of this stuff is structured, to get to that kind of organizational culture, or shared value and belief. Because there’s a skill aspect, right? I mean some of what you’re doing is teaching people how to do some of the mechanics of research which is – I don’t know, I’m talking more than I’m asking. Let me try for a question. Does teaching people sort of the processes and tasks of research, is that a way to help illustrated these other principles like empathy and human centeredness?
Jesse:Yeah, I mean I think the good processes and tactics for research come again from that – I don’t know if selflessness is the right way – the right word, but like the willingness to learn and the ability to admit that I might not be right, or I might not already know what needs to be done. And again, I think everybody needs to think that way. But even going back to the example of the way that I ask questions without saying somebody else’s word, like part of that is because I can’t assume that I know what you mean by happiness, or whatever, right? And so I think teaching that sort of instinct to not make assumptions is a big part of teaching the tactics for doing research. And it’s very much related to the reason Product and Development are asked to take notes during the site visits is because in my experience Product will do just what this woman did when I tried to present the ruler thing to her and say, “oh let me show you how to do that.” Like, in response to a question of how should I – I wish I could do something rather than say why would you do that? Or, how would you like to do that? They would say “here’s how to do it.” And that’s like not going to help you learn anything. It might help them learn for that one moment, but if you can learn what’s – you know ask the 5 whys? I love the 5 whys? Ask the 5 whys and bring it back and understand that and like build towards the 4th or 5th why. I think you’re in a lot better shape.
Steve:Can you explain the 5 whys?
Jesse:So, the 5 whys is a technique of interviewing where you just basically – somebody says something and you say why? They answer that and you say why, why? Basically, it’s getting from the surface level, like tactical, like I can’t do this, to why they can’t do it, or what they would want to do? Because then you can – maybe they don’t actually need to do that. Maybe there’s something an hour ago they should have done. You know what I mean? And you can solve their problem that way. It helps broaden the potential solution space, I think, by understanding the root problem, as opposed to the surface problem. But yeah, I mean I think the 5 whys is really about understanding the root of the problem and that’s what research is about is understanding the root of the problem, especially more like say discovery or exploratory type research, where generative type research is digging deeper into the root of the problem.
Steve:And there’s a lack of presumption that you understand, I think. Right, maybe a sort of question/answer thing is ask, “what are you doing?” “Oh, it’s this.” But I think you keep coming back to this principle of – it’s the willingness to learn of us, but also the not presuming that we do understand what the people we’re interested in are doing. So, the more you ask why the more you’re – it might even just be an interesting signal to yourself, the more you say it’s okay to keep asking why the more you give yourself permission to not know something and then not know something else and not know something else.
Jesse:And I actually – I set that expectation. Like if I’m doing an interview, part of my standard sort of interview introduction is I’m probably going to ask you some stupid sounding questions, I’m just making sure that I understand what you’re saying. You might feel like you just answered that question, but I’m going to ask it anyway. Please just like forgive me in advance, bear with me on that. And so it gives me permission to ask some really dumb sounding questions, but sometimes I get surprised and it’s like the best question I’ve asked all day.
Steve:You know back to our consultant and internal person thing, one thing that a consultant can do is not – I mean I guess you can set up this way – you are likely, I’m guessing, to be from ADP a lot of these times?
Jesse:Yeah, totally. I think that’s definitely like going in naïve is a great thing because you can genuinely ask a stupid question.
Steve:Do you have to overcome this sort of ADP relationship in setting up that kind of framing, to say I’m going to be asking dumb questions?
Jesse:Yeah, I mean I think – I mean I think for me it comes kind of naturally and maybe from my training, or just the natural way that I am, but I think – and like I said, I set it up at the beginning so that it doesn’t seem weird.
Steve:So you mentioned a little bit about – you mentioned your training. Do you want to say a little more about – you mentioned this market research you started off working in, but maybe you can go back before that?
Jesse:So, I went to Tufts University because they had engineering and a good psychology program and I didn’t know which one I wanted to do. Little did I know there was such a thing as engineering psychology. And Tufts had some courses in that. I ended up just doing psychology and then – so then – the story I like to tell is I went and interviewed for like psychology jobs, like – and realized I have enough problems of my own and I don’t want to deal with somebody else’s problems. So, I ended up finding this market research firm that was rooted in what they called Adlerian psychotherapy technique which this idea of just like not ever saying anything and just letting the people talk and talk and talk and talk. So, really, like enjoyed that and around that time like there were new technologies coming out. You know I got a Palm Pilot as part of my work and it was just really hard to use and I learned about this thing called human computer interaction and Tufts had just started like a certificate program in human computer interaction. So, I took some night classes. Thought it was really interesting and like the perfect combination of my psychology and engineering interests. And obviously that was in Boston. I grew up in Syracuse. Both very cold, snowy places. So, I decided to go to Atlanta and get out of the snow for a little while. I went to Georgia Tech and studied what they call engineering psychology, or human factors, and really learned a lot about like social science and statistics and like hypothesis testing and like really like hard core academic research which I don’t necessarily use today, but like I think having that basis helps me think about how to create a research project too. Even the exploratory stuff, I still kind of like have a hypothesis testing orientation to that. Even the qualitative research where I just had people talk and talk and talk. It’s not the same as collecting like hard data like I would have in school. And then, yeah, so from there sort of got into human computer interaction. You know they have a great Master’s program. I went to the engineering/psych PhD program because they would pay me rather than me paying them. That was another sort of thing that brought me along there. And then ultimately knew that like I was going to return to New York at some point so came back up here and got into the industry. So, I mean all that – so, it’s very psychology heavy. It’s very social science and like experimentation heavy, my background.
Steve:Yeah. And the team here – what’s the team look like?
Jesse:My team are 6 researchers and we have a mix of backgrounds. So, it’s not all psychology people. You know over the years I’ve taken people from biology programs, which is also scientific, but not – you know I’ve taken people from like design programs and they’ve all been good, you know great people. I’ve had people with human factors backgrounds. You know. So, we have six researchers and basically we’re like semi-embedded in like the products within shared products. So, like we – each of them sort of has a product that they focus on and they have their design and product management partners that they like do a lot of work with and collaborate with a ton and basically they are on that product team, but we all come together every week for our team meetings, but also in addition to let each other know what we’re doing and share ideas and help – you know bounce ideas off each other. And I think, you know, we kind of have the best of both worlds within our little product of being like embedded – you know part of the product team, but also centralized where we can help each other and get support from other researchers. I’ve been a researcher of one at places too and like that’s great. You get to do everything and like nobody questions you or whatever. Or they question you, but you get to design all the studies. But I’ve found that I wish there was somebody that I could like ask advice on this stuff, like research design or something. Like what do you think you would do? Or like help with analysis, or even just like take notes and like put two heads together at the end. Like what was the biggest theme that we found here? So, we get a little bit of both. We get to be a part of the product, but we also have this sort of small research community. And then at ADP there’s another, I don’t know, probably 30 or 40 researchers actually and some of them are two within a group of 10 UX people. Some of them are larger. So, it runs the gambit. But that’s how my team sort of is structured.
Steve:And what’s your role? What’s your sort of leadership role within that? What does that look like for research?
Jesse:So, I help with prioritization of the projects across – like I said, they’re semi-embedded, right. So, if one person is way overwhelmed I might say, “hey, Joe, go help Jane on this project,” because like they’ve got a lot to do and that’s important to my boss, right? Like I help them make sure that we’re focused on the things that are more visible and more strategic for the company and the group as well. That’s not necessarily the part I like the most. The part I like the most is sort of helping people think through their research plans and projects and understand like – so, this person came to me with this problem, but like what does that mean? What should I do? For me the most fun part about research is sort of translating like the request – “oh will you do a survey on this” – to like what we actually should be doing and like designing that project. I really like – to me it’s a nice challenge to do that. So, I institute some, also like processes or whatever, like define the research goals. But it’s mostly to help me help them, but also to make sure like they’re not going into a project unclear with their stakeholders of what they’re doing. Because I’ve also been at ADP long enough that I know most of the people that they’re going to be working with and I know their strengths and their shortcomings and how to like work with those. So, I think that’s another big part is like I’ve learned the ADP culture and like what works here really well. Like establishing the research goals is huge because many times I’ve been burnt with, “well I – well, did you find anything about this?” Well no because we didn’t go looking for that. You know what I mean? “Well why not?” Now I can be like because we agreed that this is what we want to do. It’s a little bit of a CYA, but it’s also like to make sure that maybe they had that in their mind at the beginning, they just didn’t say it to me. So, like working through that with the stakeholders to make sure that I know everything that I need to and now the people on my team know everything that they need to.
Steve:What’s the process for extracting those goals for a research project?
Jesse:Yeah, I mean, so, like I said they’re semi-embedded. So, for the most part they sort of understand it naturally just like in day to day conversation, or like working together and that kind of, you know. But, I mean the process we have like a little bit of a research plan forum and there’s like 5 or 6 elements that we have to fill out. You know the research goal, the guidance is 3 to 4 bullets that explicitly state what we’re trying to find out, but it isn’t so generic – like just 3 or 4 bullets, right? So, not too specific. Also not so generic that it could have been the last project you did. You know what I mean? And then like timing is you know an important one. Kind of what are we going to do with this? So, once we answer these questions like are we going to take an action on that because if there’s no answer to that we probably shouldn’t bother doing this project. Or we can deprioritize it if we have another one that has a big action. What else goes in there? Oh, like the different kinds of materials we need. So, like if we need a designer to build a prototype, like we make that explicit and get everybody’s agreement that like there’s somebody that’s going to help us do that. You know that has the time to do that. Off the top of my head I think those are the main things. The other big sort of process that I try to – try to get everybody on the team to do which I think has been really successful at ADP is in the presentations. So, we do like a classic presentation where we show a picture or like talk about the thing we were trying to understand if it’s more exploratory and there’s not like a prototype and all the things we observed or whatever. But then at the end we summarize what we call our insights table. So, it’s all the insights restated, so it’s our summary. But it’s pretty detailed. And we have our insights framework which is the observation, which is basically the data. Like nobody can argue with that. Then we have a recommendation which usually follows really logically from the observation. And sometimes it’s like so obvious it’s kind of like why’d you even say it. But again, as a researcher, that’s why I ask dumb questions and I say dumb things just to make sure that we all sort of agree on things. And then the action, and that’s we workshop with the team at the end of the presentation to establish the action. So, the recommendation, the guidance there is it should be descriptive, not prescriptive. So, it’s not like make the button a brighter color so people can see it. It’s make the button more visible. And then like you can make it larger. You can make a brighter color. You can put it in a different spot or whatever. Like the team figures out like what’s the right way to do it, that fits in with the rest of the product and is feasible and blah, blah, blah. Because we’ve had a lot of times where like we would make a very specific recommendation and there’s some thing in the background we don’t understand that makes that impossible. So, then everybody is “can’t do that” and they’re just like “I guess we can’t fix that problem.” But like, you know that’s not a good – we’ve got to work through the right answer to that problem. So, and then we have that framework and depending on the team we’re working with, you know some teams might want the prescriptive action filled out. Like they might want us to make a specific recommendation and then talk about it. Some teams, if you make a specific recommendation, they’re going to react negatively and not really want to hear anything else. So, we leave that blank. Sometimes we fill it in, or whatever, and it helps us be – also be very explicit at the end of the presentation. Like, what are we going to do now? And then we take all those and we add them – you know everybody takes their little table and they put it into – we have an Airtable that houses all of our insights from the year and I can then report to my boss, you know we had 750 insights and 350 of them like actually had impact on the products. And that’s the kind of stuff he cares about most.
Steve:I love this separation between sort of the qualities of the solution and the specifics of the solution. These sort of last two columns. And I think I’m probably misaligned in terms of my own language versus yours because I feel like there’s a column missing, which I’m sure isn’t. But if you’re saying sort of observation, like where is interpretation or synthesis?
Jesse:Yeah. So, the observation is like this thing doesn’t fit people’s mental model.
Steve:It’s in there.
Jesse:Yeah, it’s in the interpretation.
Steve:So, that thing that you’re not – I mean there’s a rawer form of data that’s like outside that window that’s like the left most column.
Jesse:Yeah. That would be on the previous slide that has like a picture and like an arrow and like…
Steve:Here’s what’s happening and then your first column observation. Okay. So, that’s our misalignment. I would give that a different label, but I see what you’re saying.
Jesse:Yeah. Each of these observations is like a thing that we saw a few times, or like a theme maybe or something like that I would say.
Steve:I think we get so messed up on these words. What’s an insight? What’s an observation? What’s a theme?
Jesse:Language is tricky. I mean that’s also like something that you see a lot, sort of everywhere, is miscommunications based on using the same work to mean different things.
Steve:So, yeah I think it obviously makes sense within a consistent practice that you run. I’m just like picking at you to translate it so I understand what it is. Thanks for doing that. I mean I really like the breaking those pieces apart and being able to help a team if they need specific action items, or empower them to make that decision themselves that there’s a way that acts on the research and that you’re being very responsive to the way that those teams work and that your process supports a variety of different teams and kind of energies.
Jesse:Yeah, I mean we also find if the team comes up with the action together they’re much more likely to actually implement that because like they can be like that was partly my idea so I’m going to make it happen. You know what I mean? Rather than somebody told me to do this and who’s this researcher? Like they’re a peer of mine, at best. You know what I mean? Like they’re definitely not telling me what to do, so I will – that’s another thing that I see a researcher’s role is supporting other people do their job better. Actually, I’m interested to hear your opinion in terms of like – so now I’m going to ask a question.
Steve:So, we’re at that point. Okay.
Jesse:Like one of the challenges that we face is sort of getting “credit” for the work that we’ve done. I think a lot of what we do is help people understand the problem space better and understand these things that their users aren’t able to do, or want to do, or whatever. And oftentimes it’s not like going to be brand new. Like rarely do you come up with something that nobody’s ever thought of before. A lot of times we help solidify or better articulate those problems, which then you can attack much better. But in the end, the product managers like look at this great idea I had and half the time I know that research definitely inspired that idea or helped to figure out how to do that, but like it’s hard to sort of take credit for it. So, that’s part of what this insights tracker is for so we can say that this action came from this research and get a little bit of credit for it. But I sort of view a researcher’s role as helping everybody else do their job better which leaves it again their job to do these things. What do you think about that?
Steve:I mean you’re just tapping into a bunch of things that I’ve been thinking and talking about and literally had some of this conversation in the 45 minutes before I’m speaking with you now. Where I was talking with someone about sort of the facilitation activity of things are stickier if people come up with them themselves, so helping them have an idea that they think is their own, and then she said what you’re saying, “well then I don’t get credit for it.” It just seems like another, beyond our language misalignment, we also are not maybe collectively aligned around what are we here to do and then how do we measure success? If we’re here to empower people then we have to measure that. And I took some umbrage recently about – there was some – I guess there was, I think, a talk happening and some people were tweeting about it and the quote that kept coming around was, “my research has no value if someone else doesn’t take action.” And, I mean it made me a little angry, but it mostly made me sad that – I mean I think researchers work very hard and are smart and passionate and are often in situations where they maybe don’t feel as valued or feel frustrated based on their expectations about what’s going to happen. My goodness, we don’t need to take that on ourselves and say what I’m doing is not valued unless somebody else takes an action because that to me is ceding all control of success to somebody else because you can’t control if someone else takes action. I mean I say this having worked on research projects for a very long time, in a variety of contexts, and having to sort of let go, or redefine for myself what success looks like. Again, as a consultant I don’t have sort of manager driven OKRs that affect my compensation. So, I might have the liberty to think about it differently. But yeah, if somebody else’s action is how you measure your own success, that just lets go of so much power and control and there are lots of reasons why other people don’t do things. And if we’re in the business of, you know – it’s back to your example of that person who was like well let me go tell them the right way to do it. I mean if you’re – and obviously you want to have a better outcome than that, but if you’re sense of worth and the quality of your work is based on that person’s choice to interpret that information that way, that’s a lot to carry and I feel like we need to redefine some of these things about what success looks like for research and like what are we here to do, right? If we’re sort of – I mean being a supporter is not the same as being an educator, is not the same as being a facilitator. Those are all adjacent words, but they are a different sense of what we contribute. I don’t know. That’s kind of a little rant from me, or a long rant for me because it’s definitely struck a nerve. So, I hear you.
Jesse:I think you said a lot of things that resonated with me too in that rant. It’s good to hear other people are sort of thinking about that too. But I really love the idea of maybe redefining the metric. So, like I said, I report up to my boss the number of insights that we’ve had, or observations that led to an action, right? Like, there’s a million reasons they didn’t happen. Or they might have done a different action that I don’t want to record because I don’t like – so, it’s not a great measure. But yeah, I mean part of it too is like can we take shared responsibility for that success of the product, right? Rather than the product manager being like look at this great product I got, or I have, and like taking all the credit for the – some product managers won’t take all the credit. Some of them will. It’s the way some people are too, I think.
Steve:Well you were pointing loosely towards a future, earlier in the conversation, where this is a mindset, and a set of processes that are sort of equally distributed across a lot of functions. So, if that were to be the case then credit also is the case and that sort of these different skills and processes are pulling together to change a product, or change a product – aspects of it that get shipped. So, then that – you know it might be more of a nirvana state that we’re sort of playing with, but that – that sort of addresses this question about where does credit go? The credit doesn’t need to be sort of parceled out because it’s about what is achieved collectively. Sorry, if that’s like a too socialist idea.
Jesse:Well, honestly, like where – like I said earlier, I think where UX is functioning the best, the triad is taking responsibility and credit collaboratively. And I think that’s great. I think that’s sort of what you’re talking about too.
Steve:Well it’s a good future vision I think that we can kind of hold on to. Is there anything else you want to talk about today?
Jesse:No. I don’t think so. Thanks for having me. It was a fun conversation.
Steve:Thank you so much.
Jesse:Thank you.
Steve: Thanks for listening! Tell a friend about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Google Play and Spotify and wherever fine podcasts are served. Head on over to portigal.com/podcast to get all the episodes with show notes and transcripts. Special thanks to DJ Anne Frankenstein. Our theme music is by Bruce Todd.
The post 26. Jesse Zolna of ADP first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I talk to Jesse Zolna, who leads the User Experience Research Team at ADP’s Innovation Lab. We talk about driving change as an experiment, exposing the organization to how customers solve problems, and engineering psychology.
One of the challenges we face is getting “credit” for the work that we’ve done. A lot of what we do is help people understand the problem space better and understand these things that their users aren’t able to do, or want to do, or whatever. And oftentimes it’s not going to be brand new. Rarely do you come up with something that nobody’s ever thought of before. A lot of times we help solidify or better articulate those problems, which then you can attack much better. – Jesse Zolna
Show Links* Northern Soul * The Day Mick Jagger and Keith Richards Met Again * ‘Friends’ Will Be There For You At Beijing’s Central Perk * Lowrider Culture Spreads to Brazil and Beyond * Russian Doll * Ronald does the wai * McDonald’s to Offer International Menu Items Including (Yes!) the Stroopwafel McFlurry * Jesse on LinkedIn * Jesse on Twitter * ADP * ADP’s Innovation Lab Drives New Ideas And Cultural Change Within The Company * Barnes & Noble * Nook * Stephen Gates on Design Thinking * 5 Whys * Tufts Department of Psychology * Adlerian psychotherapy * Georgia Tech Engineering Psychology * Airtable * OKR * DJ Anne Frankenstein
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
Northern Soul was a musical and cultural movement in the UK in the late sixties and early seventies. It was all about obscure soul music from America. The movement really was a scene, with clothing and dance styles, and clubs hosting dance parties, but let’s just focus on the music. For people in the UK in the 60s it wasn’t easy to get music from the US. In fact, this difficulty figures into the origin story of the Rolling Stones, where Mick Jagger and Keith Richards reconnect during a chance meeting on a train platform, and one notices the other has possession of some rare and desirable albums from the US. Anyway, Northern Soul started in that context initially, the difficulty of getting any of this music and then went on to specifically emphasize the rarities. Remember that there was no consumer music duplication technology – you had to have the 45. Many of the songs that became Northern Soul legends were commercial failures, failed artists, failed labels. And yet, these 45s, these songs found a second life, across a time, across an ocean, across cultures.
Decades later, we have the Internet, and we have globalization, and we embrace consumer enthusiasm. So there’s a cafe in Beijing modeled after Central Perk, from the TV show “Friends.” Mexican-American lowrider culture has been taken up in Brazil and New Zealand.
I find these examples fascinating, and given the frequency that these stories appear, I’m not the only one. Given the work that I do, my interest is specifically because these stories typically involve people going around the brand. The producer makes certain products and provides them to a certain audience in a certain marketplace. Sure, it’s a statement of identity to watch “Russian Doll” on Netflix, but your effort to both discover and consume is minimal. But to buy authentic parts for a product that isn’t made any more, from another country, for example, takes a lot more effort. Even with the Internet. This kind of lead user consumption is interesting.
So that’s the consumer side, going around the default path laid out by the company. And on the producer side, larger companies seem pretty intent on managing both globalization and localization. You can go to a McDonald’s in 101 different countries, but the menu will be different. Outside McDonald’s in Thailand, Ronald is posed giving the “wai”, the traditional Thai greeting of a slight bow with hands pressed together in front of the chest. Even though the McDonald’s brand spans cultures, the Thai experience is specific and self-contained within its own environment. McDonald’s redefines itself within the boundaries of the national border and even though we know Ronald can be found everywhere, this Ronald reminds a Thai customer that he is specifically in Thailand. Ronald is trying hard not to be the tourist who awkwardly adopts the local customs in order to seem “down” but actually someone who has moved in and become part of the scenery. Of course, brands, their symbols and indeed their products change meaning when they move from one culture to another, but we can consider the bare minimum meaning, just the fact of its existence in any particular culture.
Earlier this year, McDonald’s in the US introduced a limited-time “International Menu” featuring Stroopwafel McFlurry, the Grand McExtreme Bacon Burger, the Tomato Mozzarella Chicken Sandwich and Cheesy Bacon Fries from respectively Netherlands, Spain, Canada and Australia. Linda VanGosen, McDonald’s vice president of menu innovation said in a statement that “We know our US customers are curious about McDonald’s international menu items.”
It doesn’t matter if these are any good. And these aren’t intended to be authentic representations of the cuisine of these other countries; they are presented as authentic exemplars of McDonald’s in these other countries. The point here, the fascinating thing to me, is that McDonald’s is acknowledging, at least to its American market, that seams exist, that the way you experience the brand is limited by your geography, and another variation of McDonald’s – a non-American version – is out there. This is a very contained action by McDonald’s but the way it breaks the frame by pointing to something outside what they so carefully control and design is huge.
The homework for all of us is to keep our eyes open for how and when producers acknowledge in any way what lies outside the set of things they are providing to us. I believe this will continue to change. And for those of us who are in the business of producing things to be consumed, McDonald’s is signaling here that we have more choices than we might have previously thought possible.
This is an important aspect of the work that I do, helping companies to unpack these shifts in culture and consider the ways they might respond, in terms of what is appealing to the marketplace and what is authentic to the company itself. My clients have a lot of deep knowledge of what they have been doing, but they often need help getting outside that, and I help teams to build a new shared perspective and a plan to move forward. And so, the best way to support this podcast is to support my business. Hire me to help you bring a nuanced external perspective to how you understand your current and future markets. Get in touch and let’s discuss what we might do together.
I’d also love to hear how this podcast is helping you in your work. Email me at DONUTS AT PORTIGAL DOT COM or find me on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
Let’s get to the interview with Jesse Zolna. He leads the User Experience Research Team at ADP’s Innovation Lab. Jesse, welcome to the podcast.
Jesse Zolna:Thank you, Steve. Thanks for having me.
Steve:Let’s start by having you say a little bit about who you are and what you do and we’ll go from there.
Jesse:Okay. So, I lead the User Experience Research Team at ADP’s Innovation Lab, is one of the ways I put it. ADP has a sort of complex arrangement of business units and I fall within what we call shared products. It’s the stuff that runs behind all the different business units. I end up interacting with a lot of the different front end products that ADP makes. And so, just as a short term I say the innovation lab. I started here about 5 years ago when we opened the Innovation Lab. So, ADP made the decision to invest in UX and agile and design thinking, sort of all at the same time. I think I was hired #18 or 19 in the group. So, I’m leading the UX research team here in the lab.
Steve:What are some of the business units that – your lab works across these different – for those of us outside of the organization, what things could we learn about that ADP makes that you’re kind of working to inform the design of?
Jesse:At the simplest level, ADP has sort of three business units – small, medium and large businesses. And we make essentially payroll products for those businesses. That’s our core product and then on top of that we layer what we call HCM products, so anything that’s related to HR really. And then a lot of that will feed into payroll. We have what we call core HR which is all about people’s data, about who they are, where they live, and that stuff impacts the taxes that we take out of your paycheck. We have time products within each of these business units. You’ve got to record your time and obviously that calculates your pay. We have retirement services products, so like 401(k) stuff. Medical benefits, that kind of stuff. So, if you think about the paycheck as sort of the center of our business and then all the sort of – we have lots of different products that sort of feed into that center of business.
Steve:What does HCM stand for?
Jesse:Human Capital Management. It’s the newer term for HR.
Steve:I’m old enough to remember when it was called personnel and then that got outmoded by HR and now I’m going to be a dinosaur if I say HR.
Jesse:So, back then HR were bean counters and the just a cost center – the people were just a cost center. Now, today, people and talent are a company’s most valuable asset.
Steve:Okay. So, the term reflects kind of a different mindset.
Jesse:Yup.
Steve:Having heard it from you I’m sure I’ll now be recognizing the term when I come across, but I haven’t heard that before. So, I’m wondering, you described some of the different services that are provided – you’re providing services to companies who use these products to do all the human capital management and payroll for employees. How much of these different services do the employees themselves interact with? Do you know what I mean?
Jesse:Yeah. So, the initial project that I worked on when I came here is what we call the employee and manager experience which is essentially both a desk and web app that the employees use. Until ADP decided to focus on that, that was sort of an afterthought. We were building products for the HR and payroll managers that worked at our clients and an unnecessary evil was things where the employee needed to be able to go look up their information or enter their time or whatever. You know if employees wanted to get paid they’d figure out how to use our time products, so we didn’t invest a lot in that. But then as part of the decision to focus on that they realized that actually 99% of their users were employees and managers because each company has 10 HR and payroll people and 10,000 employees, or something like that. So – and it was also sort of a push into realizing that technology was changing and people were expecting better from everything that they use on a day to day basis and getting basically complaints from our clients that their employees found our products hard to use which would then reduce productivity, for both the employee who was trying to figure out how to enter their time, or figure out what medical benefits they want rather than working. And then the HR, the payroll person, had to fix errors, or help the employee navigate the system instead of do like more strategic work that they should be doing within like talent management.
Steve:So, the customers are identifying – your customers are identifying changes they want to see in the products you’re delivering to them, or the kinds of experiences you’re creating.
Jesse:Yup.
Steve:Which is not a signal to ignore.
Jesse:Yup.
Steve:So, this is the innovation lab. So, what does that mean? Those are both sort of very loaded words. For ADP what was sort of the – what was created at this kind of transition point?
Jesse:In the beginning it really was an experiment in choosing sort of the ratio of UX to development and at the same time moving to an agile development process. So, what they did was they hired all new people from different places. I came from Barnes & Noble. Worked on like their E-commerce site and the Nook device. A designer who came in at the same time he came from a media background, so had worked for NBC and MTV in the past, on their like web properties. So, they were trying to bring people in from different industries and think differently about what consumers want? And they hired about, I think, 100 people in the early days and 20 of them were UX people, whereas at the rest of the company there were probably 10,000 developers and less than 20 UX people. So, it was an experiment and if we really invested in this like will the outcomes be different. So, we had the employee and manager experience as one of our first projects and we sort of proved that you can move faster and get better results with that new ADP innovative way of thinking. So, the idea was to be an example to the rest of the company. So, now the rest of the company has seen what we’ve done at the lab and over the last 5 years has really adopted a lot of those practices and hired a lot more UX people in each of the different business units, each of the different groups and transitioned to agile development process. Both of those have been more successful in some places than others, but everybody is trying to move that way.
Steve:It’s interesting that you frame this as an experiment. What did that feel like kind of in the early days. You were, as you said, one of the early people to come into this. What did it feel like to be something that was framed as an experiment in a company this size, and I think with a big history, or long history?
Jesse:It was exciting because there was the opportunity to jump right in and have a large impact because there are so many clients and so many people using our products, yet we could start from scratch at the same time. And that’s how they sold it to us, like a startup within a big company, and we already have scale. And the other thing was we could do whatever we wanted kind of. We thought that was the right thing to do. We could do that. You know we didn’t have to go through all the red tape that the rest of the company had to do. So, we could just, you know, spend money on recruiting. We could have a very simplified NDA. Talking about research processes, that would just make it easier for us to get our work done and not have to get approval from legal and GSO and everything, which these days we have to deal with a lot more.
Steve:So, over these 5 years, as you said, the things that only were happening in the lab are now starting to, or have started to, happen in other parts of the organization. That seems like one measure of success for the experiment. It’s kind of creating models for practices for the organization to adopt. I wonder what does that then create, or open up space for the innovation lab itself to them move into different kinds of work, sort of after having birthed some processes and have them been taken up – what’s shifted for you all?
Jesse:I would say at the beginning we were really focused on our one product and proving that the user centered design process could work and you could get rapid feedback and it didn’t slow down development, and then you would see the effects on the market. And now other teams are doing it, and as you said one metric for success. Another one is that we’ve sort of changed the organization’s thinking around UX and what user centered design really is. So, the sort of simpler application of UX research is usability testing, or more evaluative, or maybe even like concept testing, like lean UX testing. When I came in, lean UX testing was like really all they thought research was. So, we’ve established also a program of ethnographic research where we do sort of foundational stuff that doesn’t necessarily have what changes can we make to this product to make it better, but broader, like what changes can we make to this quite of products and understanding how our clients are using our products within a greater context, to look for the “white spaces” and that has really opened up the organization’s eyes in terms of how research can help strategically too. You know strategy used to be the domain of only product management. They would come up with a strategy – you know based on what our competitor did, now here’s our new strategy. By the way, the competitor probably just copied us, so there’s a lot of like circularness to that. So, we’ve been able to bring in new thoughts and new ideas which I think has helped us rethink the way that we’re approaching providing tools to our clients to manage their human capital.
Steve:And is there a connection between the success you had with these initial efforts, in terms of changing the mindset of the organization? As you say, ideas only came from these kinds of places and now – obviously we know that ethnographic research can provide the kinds of new ideas that you’re talking about, but the change that’s happened here is that – I think you’re describing that people are receptive to that? They’re like oh yeah, this is another source for us to think about what to do. So, is there a connection between that mindset which is new and then the – was that driven by the work that the lab started off with?
Jesse:Yeah, well it’s related to what I said earlier about us being able to do what we think we should do because we think that’s the right thing to do. So, in the early days we were working on payroll and, aside from the employee/manager experience, I was helping with the payroll product which again is our core product. And I realized that like we knew a lot of people – we had people who knew a lot about how the time product works and how they gather that information and then it gets sent into payroll. We knew a lot about how payroll would process that. We knew a lot about how core HR data would influence that. So, we had people who understood those silos really well. But there was no understanding of how those things all kind of worked together. So, I proposed that we do some site visits where we just go watch people during their payroll process and just like say, “teach me how to do payroll from your company.” Because also every company kind of does it differently. They all have ADP payroll, most of our clients, but then they might have our time system. They might have somebody else’s time system. They might have somebody else’s compensation management system, or ours, or whatever. So, everybody kind of does it differently. So, we went to visit a bunch of different clients and like try to find the themes across and what we came back was information that was very surprising to the people internally and actually our first – we presented it several times and the first few, their reaction basically was give the name of that client, I’m going to go show them how to do it right. And I was like well, I can give you the name of all 12 clients we visited, but then what about the other 99% that we didn’t visit, are you going to go train them too? And they like didn’t quite get – they wanted to go solve it for that client that’s like very solution oriented, rather than understanding the problem and thinking about how can we work together. And they literally didn’t believe me that clients would double and triple check the time data before they put it into payroll and then after they put it into payroll, and spent hours doing this. Like literally going through Excel spreadsheets. One of the funny things the researcher on the project came up with, she used to call it the rulers of payroll. We literally had clients that would use a ruler, like on the screen, so they could look across these really long lines of data. And somebody else would be on the other screen and they’d say whatever, 1,400 – I don’t even know what the numbers were, but they would like call and respond to make sure that the like thousands of lines of data – and they were like why would they do that? It just imports automatically. But I mean the answer really was ‘cuz one time in the last year, or whatever, or maybe 20 times, I don’t know, there was an error and that caused like somebody to not get paid. And the payroll person thinks maybe they won’t be able to pay their mortgage because they didn’t get paid. So, they want to be super sure. So, anyway, it was like this idea of understanding things across these silos, coming up with these new insights that we never would have seen if we didn’t look across those silos. And so since then we’ve done probably 8 or 9 of those projects. And really we kind of started off with here’s a big domain that we kind of want to really understand deeply and we don’t necessarily have a specific piece of the product that we’re going to improve from that. And I think it gives everybody a better understanding and it gets the silos together. We actually – that research program we have has three stated goals. The first is to get people from different silos working together and to understand our users in the same way so that when they talk they have like an equal understanding and like the same vernacular and things like that when they do try to get together and work together. Number two, is just that – we call it the foundational understanding of what are our clients doing? What are the biggest pain points? How can we help them? And number three is to include non-researchers in the research process as well. So, when we go onsite we aim to have one researcher and two or three like developers, product managers, designers as like observers and note takers. So, they get to participate in the research process, understand it, see how it works. Maybe bring some of that – not only bring that empathy back home with them, but bring some of that research process and understanding too so they can think about when might be another time to do research as well.
Steve:Lots of really fascinating things. I want to pick one thing to go back to. So, when you started to present back these kinds of behaviors that you observed, like the triple checking and so on, it was surprising. “I’m going to go fix that.” What do you do after that? What do you do to sort of bring – to help these stakeholders understand what you learned and what it means and a better way to think about it than sort of fixing each of your thousands of clients?
Jesse:Actually that initial reaction was why we decided to include the third goal which is to get them out there because I figured if these people were there and they saw it they wouldn’t not believe me. I was lucky that we had some of the people who helped to establish the innovation lab and really believed in user centered design and research and are of the mindset that they want to learn. I think that makes a difference too with researchers working with people who want to learn versus people who don’t necessarily want to learn, or just want you to prove that they’re right or whatever. That’s a whole other topic of conversation. But luckily we had a few people there who from an executive level, or air cover level, were able to – I remember literally one woman said to me – you know, I told her don’t shoot the messenger, like listen to him. Like try and understand what’s going on and try to work with that and not – you know so that was super helpful to have like some believers that could help them kind of work through it really. And it took time too. Like you know just like the initial reaction was that’s wrong, like no way, let me call them. But over time I think these people were able to realize, oh wait, maybe I should like look into that. And maybe if they did look into it they could see some evidence that helped them believe and understand it better too. Because also having gone and done this ethnographic research, you know I get a super deep understanding and I can only present so much of that data and like try to tell that story and purposely try to tell it at a high level so they can understand it all. And it’s really getting those deep details and those, almost the anecdotal evidence that really brings color and life to it, that again the reason that we came up with that third goal to get those people out there.
Steve:So, if you think about someone who – it might be someone – I mean I think your caveat about they want to learn – someone who wants to learn but maybe is inclined a little bit to say well I better intervene and fix their erroneous behavior – what changes for them when they see the behavior that we’re talking about versus they hear about it from you? Like what is – I think there’s something fundamental that’s different for them. What is that?
Jesse:I think it just becomes more real. It’s no longer a rumor, or somebody else saying this. They like see it. It’s no longer a PowerPoint slide, or even – and this is why you use quotes and videos in your presentations, right, like that makes it real too. It’s no longer words on a page. It’s a person feeling pain in front of them and it’s just like more – it just becomes more real and tactical and you know I think it just fills in all the details that you need.
Steve:It reminds me of – I guess of a minor failure a few years ago where we were – our users we were looking at were people who were “unbanked” is the phrase, or “semi-banked.” And a tech company where the team themselves, that was not their lifestyle and we went out in the field with them and they were so affected by the experience, but then they asked us to – I think we didn’t do a good job sort of understanding the request, but they wanted an edited video of an extremely tight running time to like use in part of a meeting. And I think it emerged later on that the objective was for people to have the same emotional reaction from that that they did being in the field. And I get where they’re coming from. Like this really changed us and we want the rest of the organization to have this. Eventually I realized oh, you can’t get that from a video. Even though, as you say, videos sort of bring it alive more than words on a page might, but this experience of being out there – I think you’re right about sort of the anecdotal stuff. There’s the facts, sort of the details of the narrative, but you just saw things that I think those people were very moved by what happened to them in the field.
Jesse:Yeah. I think part of it too is they might not go to all of the visits, but they see one person and then when you present to them like the theme, the summary, then they can say, “oh yeah, I saw that in Joe.” So, they just understand it sort of one level deeper. And that’s what like as a researcher we often do that, like we have these themes and we understand it at a really deep level, but it’s hard to get that deep understanding. It’s hard to express that without showing all the data, but you’re trying to do – you know, just like these people in this meeting only had a certain amount of time, you know you only have – you can’t have everybody go participate in the entire research project the entire time. So, you have to summarize for them. It’s part of what we do is we go learn things and we summarize it for people so that they don’t have to go learn it themselves. But you just can’t get all of your learnings across. It’s hard. Which is part of the reason, on another subject, why, no offense, but I personally prefer being like an internal research person and like having worked on these projects for 5 years, like I feel like stuff comes out from 3 years ago that I can go back to. And I think there’s that sort of institutional memory which I realize now it’s ironic because 5 years ago when somebody would say, “oh well this is how we’ve always done payroll” I would like want to strangle them and be like that’s the worst thing for a researcher to hear is that we already know it. Like we don’t need any new information. Now, when people come to me with questions I’m like well I already know the answer to that because we did research 3 years ago. So, I have to actually consciously make sure that I’m always open for learning too, having been here for so long. So, maybe that’s the negative of being an internal person. As a consultant, you can always come in with fresh eyes and be willing to learn something new.
Steve:Right. I mean, I don’t know, the thing I like about consulting is – I mean, I’ll say the same thing you’re saying – the thing I like about consulting is being able to come into something that I haven’t seen before and try to figure it out. It’s often usually overwhelming, but I rely so heavily on people that are, that understand the problem space, so that I’m not trying to like reinvent the wheel. And I have admiration and a little bit of jealousy for people that stick with something for a long time because the time horizons for – I mean the kinds of change you’re talking about making, it’s very long. I don’t work that long term with somebody. My rationalization is we need both. You need people that live in the problem space and facilitate new ways – that hold onto the depth and sort of advocate for that depth of insight. And you need ways to keep getting fresh insight that you have with your changes over your tenure. I think we agree and you’ve got to find the role that you can sort of thrive best in, I guess.
Jesse:Yeah, totally. Yeah I mean even as an internal person I like to go into a research project as naïve – maybe not as naïve as possible, but relatively naïve so you could naturally ask the question when – well what do you think it should be – sort of in the research process. Or, how would you want it to go? Instead of like knowing the answer to that and possibly being somewhat biased or explanatory rather than questioning.
Steve:I shy away from getting into like technique too much in these conversations because I think we’re looking at the organization, but that being said, you said something that kind of intrigued me and I just wanted to ask a little more about it. When you described these ethnographies you said going into these organizations and saying, “teach me how to do payroll.” I don’t know what you literally said in the actual interviews, but any thoughts about the framing? Like there’s lots of ways to get to learn what people are doing and I wondered if you had a point of view about “teach me” as sort of the mode of inquiry?
Jesse:Yeah, I mean in that case we were purposely trying to, um, understand – uh, figure out what we don’t know. You know you don’t know what you don’t know. And we were trying to get the user’s point of view on it. So we didn’t – in that case we did not want to kind of structure the interview very much at all in a certain way because we wanted to see sort of what came out from them. We were possibly overguarding against imposing our own sort of biases in understanding and structure in it, but I mean that was essentially why we chose that sort of line of inquiry. And I do think there were literally like 4 or 5 questions we used there. And will say that – so my – we started to talk about my background, but my first job out of undergrad was for a marketing research company and their technique was literally one question and they would do one-on-one interviews and it would just be like tell me your thoughts and feelings about how “X” impacts your life? Or tell me about your – something like that. And it was literally one question. And then you were only “allowed” to repeat a word they said, or like say – we would either ladder up or ladder down. So, it’s either – so if they’d say like, “it makes me feel tired.” “Well, what happens when you’re tired.” Those would be like laddering down. Or like what does tired mean to you, or something – I think that was laddering up. It was a long time ago now, so I’m probably butchering it a little bit. But like literally one question and then like never say a word that they didn’t say yet. So, that’s kind of the technique. That has driven sort of my discovery type research, exploratory type research ever since, is like never impose your own words or structure or anything on it and just let the person you’re talking to teach you.
Steve:I’m so self-conscious of whatever word I use now, or whatever question I ask you. The reason I’m curious about the “teach me” question is I think you could ask them – since we’re talking about what are these kinds of questions we might ask, is there a difference between how do you do payroll and teach me how to do payroll?
Jesse:Um, yeah I think there is. Um, I mean I don’t know if this is necessarily true, but if you say how do you do payroll they might say this person is from ADP, I’m going to tell them how I use ADP products. Or, I’m going to tell them the right way. Whereas if I say teach me they’re going to show me the real way in a way. You know what I mean?
Steve:Yeah.
Jesse:Like it’s almost like you go to a university course on how to do something and then you go in the real world and you try to do it. It’s not exactly like the book says. Right? So, maybe it’s how do you do payroll – they would give me the book version. And teach me how to do payroll and they’d give me like the real life version.
Steve:That’s a really nice distinction between the two. So, we’ll switch gears a little bit. So, as part of this effort to bring more people out in the field and get them kind of exposure to the lives of customers and really participate in research – I think you talked about note taking and other kinds of roles that you’re giving to them. So, they are – at one level they are performing some of the tasks of research. I guess I’m wondering what’s – is there more? Is research happening that doesn’t involve you? Are they doing some of this on their own? I don’t know who “they” is?
Jesse:So, it’s related to the idea of the innovation lab sort of teaching the organization how to do these things. And so, I mean, because we’re getting different people from different groups together too – it’s not always a group that has like a robust UX function, right. So, they might go back and if they want to continue to be – to learn, they might have to do it themselves. And yeah, I think that’s happening. I do think there’s sort of the constant debate everywhere, and certainly internal within ADP. You know we have different answers in sort of different parts of ADP is like who should do the research, or whatever. And you know my team are all like dedicated researchers and we have our design partners that we work with and the design partners certainly help shape the research and we certainly help shape the design, but we’re sort of two different people and we think that splitting the sort of lead responsibility is more efficient and we can get a lot more done. Some places, even within ADP’s organization, they say that designers should do the research. And you know I’ve observed also, sometimes some people say designers should only do like evaluative research, like prototype testing type research because that’s sort of more straightforward. You know you get a task, you ask people to do a task. It’s a lot harder to like, I don’t know, lead people, I guess. But some people might argue that they might be biased and they might interpret – if they love their design too much. Again, it goes back to are they really willing to learn, or do they just want to like collect data to show that they’re right? It’s just like different ways that different people are. So, some designers probably are great at doing that and some designers might not be. And then some people might say designers actually should be part of the exploratory research so they can understand it more deeply and really – you know like we talked about earlier, having collected the data you understand the data more deeply and really understand the pains that they’re designing for better when they get to the design process. I guess you could add those together and say designers can do any kind of research. And I think that’s probably true. And then, not only designers, but product management, could also do that stuff too. We definitely encourage everybody to get out there and talk to clients and understand what they’re doing. Sometimes I worry that if you get out there and talk to one client you might overcompensate for their unique need, or pain, or whatever, which could ruin it for the other 99,000 clients or whatever. So, we do, in that program I was talking about, try to force people to participate in more than one data collection point.
Steve:I think there’s just lots of interesting issues around this. And you’re right, this is a very common topic, or question or debate, or just issue. I mean I just wonder long term – I mean right now there are people with the title researcher who have sort of expertise in all the things that you’re talking about. You listed a bunch of different aspects of what researchers are doing and I wonder sort of where does it take us, and I don’t know whether it’s 2 years or 50 years, where we’re holding on to some part of it and we have a belief that we bring value by doing that and we’re empowering other parts of it. And we believe that that adds value. Sort of wonder what will that ratio be? Or what should it be? Or will we have a role with this title in the future? Or is it a process that then gets handled a different way? I don’t know. I don’t know if you have a perspective on the misty futures?
Jesse:Yeah, I don’t know. It’s a tough question for sure. I guess while you were asking that question it reminded of – I heard a great quote recently from, I forget his name. You probably know his name, the guy who leads the design team at InVision, his quote about design thinking was “the worst part about design thinking is the word design” because everybody should be thinking this way. Obviously empathy is the number one step in design thinking. Everybody should be thinking empathetically. And we definitely have groups within ADP too where you – where you “UX” has matured enough that everybody is thinking empathetically and they think about UX as sort of everybody’s role – development, product. You know we have the triad formation – UX, Development and Product – and they work together on things. In some places all three of those people are really thinking empathetically and about end user needs. And also in most places the UX people are thinking about is this feasible or not, right? And trying to understand the tradeoffs in terms of like can we still get to the same sort of meet the user need in a different way that might be more feasible, or easier to do, or whatever too. I mean I think you’re really being successful at “design thinking” when everybody is doing all that stuff.
Steve:It reminds me of just the evolution from human resources to human capital management. There’s a fundamental shift in the belief of sort of the value of something that we used to optimize and now we want to enable and that you’re talking about sort of how products get developed and this idea of empathy being something that everyone can and should have, not sort of one group is helping another group, but that it’s a shared value or principal. So, I wonder if there’s just opportunities – this is not ADP specific, but just the profession in general to rethink how some of this stuff is structured, to get to that kind of organizational culture, or shared value and belief. Because there’s a skill aspect, right? I mean some of what you’re doing is teaching people how to do some of the mechanics of research which is – I don’t know, I’m talking more than I’m asking. Let me try for a question. Does teaching people sort of the processes and tasks of research, is that a way to help illustrated these other principles like empathy and human centeredness?
Jesse:Yeah, I mean I think the good processes and tactics for research come again from that – I don’t know if selflessness is the right way – the right word, but like the willingness to learn and the ability to admit that I might not be right, or I might not already know what needs to be done. And again, I think everybody needs to think that way. But even going back to the example of the way that I ask questions without saying somebody else’s word, like part of that is because I can’t assume that I know what you mean by happiness, or whatever, right? And so I think teaching that sort of instinct to not make assumptions is a big part of teaching the tactics for doing research. And it’s very much related to the reason Product and Development are asked to take notes during the site visits is because in my experience Product will do just what this woman did when I tried to present the ruler thing to her and say, “oh let me show you how to do that.” Like, in response to a question of how should I – I wish I could do something rather than say why would you do that? Or, how would you like to do that? They would say “here’s how to do it.” And that’s like not going to help you learn anything. It might help them learn for that one moment, but if you can learn what’s – you know ask the 5 whys? I love the 5 whys? Ask the 5 whys and bring it back and understand that and like build towards the 4th or 5th why. I think you’re in a lot better shape.
Steve:Can you explain the 5 whys?
Jesse:So, the 5 whys is a technique of interviewing where you just basically – somebody says something and you say why? They answer that and you say why, why? Basically, it’s getting from the surface level, like tactical, like I can’t do this, to why they can’t do it, or what they would want to do? Because then you can – maybe they don’t actually need to do that. Maybe there’s something an hour ago they should have done. You know what I mean? And you can solve their problem that way. It helps broaden the potential solution space, I think, by understanding the root problem, as opposed to the surface problem. But yeah, I mean I think the 5 whys is really about understanding the root of the problem and that’s what research is about is understanding the root of the problem, especially more like say discovery or exploratory type research, where generative type research is digging deeper into the root of the problem.
Steve:And there’s a lack of presumption that you understand, I think. Right, maybe a sort of question/answer thing is ask, “what are you doing?” “Oh, it’s this.” But I think you keep coming back to this principle of – it’s the willingness to learn of us, but also the not presuming that we do understand what the people we’re interested in are doing. So, the more you ask why the more you’re – it might even just be an interesting signal to yourself, the more you say it’s okay to keep asking why the more you give yourself permission to not know something and then not know something else and not know something else.
Jesse:And I actually – I set that expectation. Like if I’m doing an interview, part of my standard sort of interview introduction is I’m probably going to ask you some stupid sounding questions, I’m just making sure that I understand what you’re saying. You might feel like you just answered that question, but I’m going to ask it anyway. Please just like forgive me in advance, bear with me on that. And so it gives me permission to ask some really dumb sounding questions, but sometimes I get surprised and it’s like the best question I’ve asked all day.
Steve:You know back to our consultant and internal person thing, one thing that a consultant can do is not – I mean I guess you can set up this way – you are likely, I’m guessing, to be from ADP a lot of these times?
Jesse:Yeah, totally. I think that’s definitely like going in naïve is a great thing because you can genuinely ask a stupid question.
Steve:Do you have to overcome this sort of ADP relationship in setting up that kind of framing, to say I’m going to be asking dumb questions?
Jesse:Yeah, I mean I think – I mean I think for me it comes kind of naturally and maybe from my training, or just the natural way that I am, but I think – and like I said, I set it up at the beginning so that it doesn’t seem weird.
Steve:So you mentioned a little bit about – you mentioned your training. Do you want to say a little more about – you mentioned this market research you started off working in, but maybe you can go back before that?
Jesse:So, I went to Tufts University because they had engineering and a good psychology program and I didn’t know which one I wanted to do. Little did I know there was such a thing as engineering psychology. And Tufts had some courses in that. I ended up just doing psychology and then – so then – the story I like to tell is I went and interviewed for like psychology jobs, like – and realized I have enough problems of my own and I don’t want to deal with somebody else’s problems. So, I ended up finding this market research firm that was rooted in what they called Adlerian psychotherapy technique which this idea of just like not ever saying anything and just letting the people talk and talk and talk and talk. So, really, like enjoyed that and around that time like there were new technologies coming out. You know I got a Palm Pilot as part of my work and it was just really hard to use and I learned about this thing called human computer interaction and Tufts had just started like a certificate program in human computer interaction. So, I took some night classes. Thought it was really interesting and like the perfect combination of my psychology and engineering interests. And obviously that was in Boston. I grew up in Syracuse. Both very cold, snowy places. So, I decided to go to Atlanta and get out of the snow for a little while. I went to Georgia Tech and studied what they call engineering psychology, or human factors, and really learned a lot about like social science and statistics and like hypothesis testing and like really like hard core academic research which I don’t necessarily use today, but like I think having that basis helps me think about how to create a research project too. Even the exploratory stuff, I still kind of like have a hypothesis testing orientation to that. Even the qualitative research where I just had people talk and talk and talk. It’s not the same as collecting like hard data like I would have in school. And then, yeah, so from there sort of got into human computer interaction. You know they have a great Master’s program. I went to the engineering/psych PhD program because they would pay me rather than me paying them. That was another sort of thing that brought me along there. And then ultimately knew that like I was going to return to New York at some point so came back up here and got into the industry. So, I mean all that – so, it’s very psychology heavy. It’s very social science and like experimentation heavy, my background.
Steve:Yeah. And the team here – what’s the team look like?
Jesse:My team are 6 researchers and we have a mix of backgrounds. So, it’s not all psychology people. You know over the years I’ve taken people from biology programs, which is also scientific, but not – you know I’ve taken people from like design programs and they’ve all been good, you know great people. I’ve had people with human factors backgrounds. You know. So, we have six researchers and basically we’re like semi-embedded in like the products within shared products. So, like we – each of them sort of has a product that they focus on and they have their design and product management partners that they like do a lot of work with and collaborate with a ton and basically they are on that product team, but we all come together every week for our team meetings, but also in addition to let each other know what we’re doing and share ideas and help – you know bounce ideas off each other. And I think, you know, we kind of have the best of both worlds within our little product of being like embedded – you know part of the product team, but also centralized where we can help each other and get support from other researchers. I’ve been a researcher of one at places too and like that’s great. You get to do everything and like nobody questions you or whatever. Or they question you, but you get to design all the studies. But I’ve found that I wish there was somebody that I could like ask advice on this stuff, like research design or something. Like what do you think you would do? Or like help with analysis, or even just like take notes and like put two heads together at the end. Like what was the biggest theme that we found here? So, we get a little bit of both. We get to be a part of the product, but we also have this sort of small research community. And then at ADP there’s another, I don’t know, probably 30 or 40 researchers actually and some of them are two within a group of 10 UX people. Some of them are larger. So, it runs the gambit. But that’s how my team sort of is structured.
Steve:And what’s your role? What’s your sort of leadership role within that? What does that look like for research?
Jesse:So, I help with prioritization of the projects across – like I said, they’re semi-embedded, right. So, if one person is way overwhelmed I might say, “hey, Joe, go help Jane on this project,” because like they’ve got a lot to do and that’s important to my boss, right? Like I help them make sure that we’re focused on the things that are more visible and more strategic for the company and the group as well. That’s not necessarily the part I like the most. The part I like the most is sort of helping people think through their research plans and projects and understand like – so, this person came to me with this problem, but like what does that mean? What should I do? For me the most fun part about research is sort of translating like the request – “oh will you do a survey on this” – to like what we actually should be doing and like designing that project. I really like – to me it’s a nice challenge to do that. So, I institute some, also like processes or whatever, like define the research goals. But it’s mostly to help me help them, but also to make sure like they’re not going into a project unclear with their stakeholders of what they’re doing. Because I’ve also been at ADP long enough that I know most of the people that they’re going to be working with and I know their strengths and their shortcomings and how to like work with those. So, I think that’s another big part is like I’ve learned the ADP culture and like what works here really well. Like establishing the research goals is huge because many times I’ve been burnt with, “well I – well, did you find anything about this?” Well no because we didn’t go looking for that. You know what I mean? “Well why not?” Now I can be like because we agreed that this is what we want to do. It’s a little bit of a CYA, but it’s also like to make sure that maybe they had that in their mind at the beginning, they just didn’t say it to me. So, like working through that with the stakeholders to make sure that I know everything that I need to and now the people on my team know everything that they need to.
Steve:What’s the process for extracting those goals for a research project?
Jesse:Yeah, I mean, so, like I said they’re semi-embedded. So, for the most part they sort of understand it naturally just like in day to day conversation, or like working together and that kind of, you know. But, I mean the process we have like a little bit of a research plan forum and there’s like 5 or 6 elements that we have to fill out. You know the research goal, the guidance is 3 to 4 bullets that explicitly state what we’re trying to find out, but it isn’t so generic – like just 3 or 4 bullets, right? So, not too specific. Also not so generic that it could have been the last project you did. You know what I mean? And then like timing is you know an important one. Kind of what are we going to do with this? So, once we answer these questions like are we going to take an action on that because if there’s no answer to that we probably shouldn’t bother doing this project. Or we can deprioritize it if we have another one that has a big action. What else goes in there? Oh, like the different kinds of materials we need. So, like if we need a designer to build a prototype, like we make that explicit and get everybody’s agreement that like there’s somebody that’s going to help us do that. You know that has the time to do that. Off the top of my head I think those are the main things. The other big sort of process that I try to – try to get everybody on the team to do which I think has been really successful at ADP is in the presentations. So, we do like a classic presentation where we show a picture or like talk about the thing we were trying to understand if it’s more exploratory and there’s not like a prototype and all the things we observed or whatever. But then at the end we summarize what we call our insights table. So, it’s all the insights restated, so it’s our summary. But it’s pretty detailed. And we have our insights framework which is the observation, which is basically the data. Like nobody can argue with that. Then we have a recommendation which usually follows really logically from the observation. And sometimes it’s like so obvious it’s kind of like why’d you even say it. But again, as a researcher, that’s why I ask dumb questions and I say dumb things just to make sure that we all sort of agree on things. And then the action, and that’s we workshop with the team at the end of the presentation to establish the action. So, the recommendation, the guidance there is it should be descriptive, not prescriptive. So, it’s not like make the button a brighter color so people can see it. It’s make the button more visible. And then like you can make it larger. You can make a brighter color. You can put it in a different spot or whatever. Like the team figures out like what’s the right way to do it, that fits in with the rest of the product and is feasible and blah, blah, blah. Because we’ve had a lot of times where like we would make a very specific recommendation and there’s some thing in the background we don’t understand that makes that impossible. So, then everybody is “can’t do that” and they’re just like “I guess we can’t fix that problem.” But like, you know that’s not a good – we’ve got to work through the right answer to that problem. So, and then we have that framework and depending on the team we’re working with, you know some teams might want the prescriptive action filled out. Like they might want us to make a specific recommendation and then talk about it. Some teams, if you make a specific recommendation, they’re going to react negatively and not really want to hear anything else. So, we leave that blank. Sometimes we fill it in, or whatever, and it helps us be – also be very explicit at the end of the presentation. Like, what are we going to do now? And then we take all those and we add them – you know everybody takes their little table and they put it into – we have an Airtable that houses all of our insights from the year and I can then report to my boss, you know we had 750 insights and 350 of them like actually had impact on the products. And that’s the kind of stuff he cares about most.
Steve:I love this separation between sort of the qualities of the solution and the specifics of the solution. These sort of last two columns. And I think I’m probably misaligned in terms of my own language versus yours because I feel like there’s a column missing, which I’m sure isn’t. But if you’re saying sort of observation, like where is interpretation or synthesis?
Jesse:Yeah. So, the observation is like this thing doesn’t fit people’s mental model.
Steve:It’s in there.
Jesse:Yeah, it’s in the interpretation.
Steve:So, that thing that you’re not – I mean there’s a rawer form of data that’s like outside that window that’s like the left most column.
Jesse:Yeah. That would be on the previous slide that has like a picture and like an arrow and like…
Steve:Here’s what’s happening and then your first column observation. Okay. So, that’s our misalignment. I would give that a different label, but I see what you’re saying.
Jesse:Yeah. Each of these observations is like a thing that we saw a few times, or like a theme maybe or something like that I would say.
Steve:I think we get so messed up on these words. What’s an insight? What’s an observation? What’s a theme?
Jesse:Language is tricky. I mean that’s also like something that you see a lot, sort of everywhere, is miscommunications based on using the same work to mean different things.
Steve:So, yeah I think it obviously makes sense within a consistent practice that you run. I’m just like picking at you to translate it so I understand what it is. Thanks for doing that. I mean I really like the breaking those pieces apart and being able to help a team if they need specific action items, or empower them to make that decision themselves that there’s a way that acts on the research and that you’re being very responsive to the way that those teams work and that your process supports a variety of different teams and kind of energies.
Jesse:Yeah, I mean we also find if the team comes up with the action together they’re much more likely to actually implement that because like they can be like that was partly my idea so I’m going to make it happen. You know what I mean? Rather than somebody told me to do this and who’s this researcher? Like they’re a peer of mine, at best. You know what I mean? Like they’re definitely not telling me what to do, so I will – that’s another thing that I see a researcher’s role is supporting other people do their job better. Actually, I’m interested to hear your opinion in terms of like – so now I’m going to ask a question.
Steve:So, we’re at that point. Okay.
Jesse:Like one of the challenges that we face is sort of getting “credit” for the work that we’ve done. I think a lot of what we do is help people understand the problem space better and understand these things that their users aren’t able to do, or want to do, or whatever. And oftentimes it’s not like going to be brand new. Like rarely do you come up with something that nobody’s ever thought of before. A lot of times we help solidify or better articulate those problems, which then you can attack much better. But in the end, the product managers like look at this great idea I had and half the time I know that research definitely inspired that idea or helped to figure out how to do that, but like it’s hard to sort of take credit for it. So, that’s part of what this insights tracker is for so we can say that this action came from this research and get a little bit of credit for it. But I sort of view a researcher’s role as helping everybody else do their job better which leaves it again their job to do these things. What do you think about that?
Steve:I mean you’re just tapping into a bunch of things that I’ve been thinking and talking about and literally had some of this conversation in the 45 minutes before I’m speaking with you now. Where I was talking with someone about sort of the facilitation activity of things are stickier if people come up with them themselves, so helping them have an idea that they think is their own, and then she said what you’re saying, “well then I don’t get credit for it.” It just seems like another, beyond our language misalignment, we also are not maybe collectively aligned around what are we here to do and then how do we measure success? If we’re here to empower people then we have to measure that. And I took some umbrage recently about – there was some – I guess there was, I think, a talk happening and some people were tweeting about it and the quote that kept coming around was, “my research has no value if someone else doesn’t take action.” And, I mean it made me a little angry, but it mostly made me sad that – I mean I think researchers work very hard and are smart and passionate and are often in situations where they maybe don’t feel as valued or feel frustrated based on their expectations about what’s going to happen. My goodness, we don’t need to take that on ourselves and say what I’m doing is not valued unless somebody else takes an action because that to me is ceding all control of success to somebody else because you can’t control if someone else takes action. I mean I say this having worked on research projects for a very long time, in a variety of contexts, and having to sort of let go, or redefine for myself what success looks like. Again, as a consultant I don’t have sort of manager driven OKRs that affect my compensation. So, I might have the liberty to think about it differently. But yeah, if somebody else’s action is how you measure your own success, that just lets go of so much power and control and there are lots of reasons why other people don’t do things. And if we’re in the business of, you know – it’s back to your example of that person who was like well let me go tell them the right way to do it. I mean if you’re – and obviously you want to have a better outcome than that, but if you’re sense of worth and the quality of your work is based on that person’s choice to interpret that information that way, that’s a lot to carry and I feel like we need to redefine some of these things about what success looks like for research and like what are we here to do, right? If we’re sort of – I mean being a supporter is not the same as being an educator, is not the same as being a facilitator. Those are all adjacent words, but they are a different sense of what we contribute. I don’t know. That’s kind of a little rant from me, or a long rant for me because it’s definitely struck a nerve. So, I hear you.
Jesse:I think you said a lot of things that resonated with me too in that rant. It’s good to hear other people are sort of thinking about that too. But I really love the idea of maybe redefining the metric. So, like I said, I report up to my boss the number of insights that we’ve had, or observations that led to an action, right? Like, there’s a million reasons they didn’t happen. Or they might have done a different action that I don’t want to record because I don’t like – so, it’s not a great measure. But yeah, I mean part of it too is like can we take shared responsibility for that success of the product, right? Rather than the product manager being like look at this great product I got, or I have, and like taking all the credit for the – some product managers won’t take all the credit. Some of them will. It’s the way some people are too, I think.
Steve:Well you were pointing loosely towards a future, earlier in the conversation, where this is a mindset, and a set of processes that are sort of equally distributed across a lot of functions. So, if that were to be the case then credit also is the case and that sort of these different skills and processes are pulling together to change a product, or change a product – aspects of it that get shipped. So, then that – you know it might be more of a nirvana state that we’re sort of playing with, but that – that sort of addresses this question about where does credit go? The credit doesn’t need to be sort of parceled out because it’s about what is achieved collectively. Sorry, if that’s like a too socialist idea.
Jesse:Well, honestly, like where – like I said earlier, I think where UX is functioning the best, the triad is taking responsibility and credit collaboratively. And I think that’s great. I think that’s sort of what you’re talking about too.
Steve:Well it’s a good future vision I think that we can kind of hold on to. Is there anything else you want to talk about today?
Jesse:No. I don’t think so. Thanks for having me. It was a fun conversation.
Steve:Thank you so much.
Jesse:Thank you.
Steve: Thanks for listening! Tell a friend about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Google Play and Spotify and wherever fine podcasts are served. Head on over to portigal.com/podcast to get all the episodes with show notes and transcripts. Special thanks to DJ Anne Frankenstein. Our theme music is by Bruce Todd.
The post 26. Jesse Zolna of ADP first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Juliette Melton, Director of User Insight and Strategy at The New York Times. We talk about updating the old “design research” label, user research in a journalism culture, and the role of coaching.
I think that researchers can bring a kind of brightness into a space and a kind of optimism for a team and a sense that we can learn these things together. It’s a bit intangible as a quality, but when we bring on new researchers that’s really something we look for. Like is this person someone who is excited about making connections across an organization? Excited to share what we’re doing? There’s something about bringing energy into research which I think is really important. – Juliette Melton
Show Links* Freedom From Choice cocktail * Blue Apron * Sun Basket * The BEST Turkey Burger Recipe * Saigon Chicken Wings * Juliette on LinkedIn * Juliette on Twitter * New York Times * New Products & Ventures * NYT Cooking * Crosswords * Parenting * The Daily * The Weekly * Modern Love * IDEO * Survey Monkey * Reiki * Richard Scarry’s Busy, Busy Town * The Innovation Strategist
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
There used to be a restaurant in San Francisco called Heaven’s Dog, and one of the items on their cocktail menu was called “freedom from choice.” Rather than choose a specific cocktail, you’d tell the server your preferred spirit, and a bit of direction on the flavor, and they’d make you something to fit those criteria. While once freedom from choice would only have been seen negatively, here outsourcing your choice to an expert became a positive. The menu still had all the cocktail options for people that knew what they wanted, but for patrons who didn’t know what they wanted, there was an option for them as well.
A number of years ago we started using a meal kit delivery service, first Blue Apron and then later Sun Basket. In case you aren’t familiar with services like these, they deliver everything you need, once a week, to make some number of meals, in our case three. When Blue Apron started, they offered very little choice, we could opt for vegetarian or not, but otherwise, they just sent us three meals. Eventually we could opt out of pork, or fish, for example, and then they began offering a larger number of meals we could select from about a week before.
We immediately loved Blue Apron. It eliminated the planning and deciding, something that we just weren’t good at. Or that we avoided, because we didn’t enjoy it, and so we’d just make the same set of meals based on what we had done before. Using Blue Apron reduced the shopping, but not entirely as we still needed fruit, milk, bread, juice, and so on, for the rest of our meals. More interestingly and most meaningfully, it totally changed our meal prep activity. Where previously one person would have decided and cooked, and the other have cleaned, now the whole process was something we could share. Rather than anyone “owning” or controlling the preparation process, we had this well-designed, visual recipe on it’s own peice of paper. With the Blue Apron meals, we cooked together, easily weaving between chopping and heating, each person taking responsibility for a each task but no assigning, no delegating. We would discuss ways to improve or alter the recipe to suit our tastes (less lemon juice, more red pepper flakes).
Sometimes we’d take the meal kit over to someone else’s house, friends who were also Blue Apron customers, having planned to save a common meal to prepare together, playing out the group cooking activity with a larger group.
I was mixed about the results, some meals just weren’t that tasty, some meals required a lot of work for little result, and I felt like my own cooking wasn’t improving, as following the instructions didn’t give me much insight. We looked on social media to see what Blue Apron customers were saying about a particular meal, in case that gave us insight into choices we could make. Eventually, we switched to Sun Basket for more choice and better quality meals. But most of my satisfaction came from the preparation activity, rather than the actual meal itself. That wasn’t necessarily a failure; it’s worth taking satisfaction and reward where you can find it.
As with any product or service that I use, I often wonder if the producers understand the ways it’s being experienced, and how that inform the choices they make in configuring the service. I don’t know what Sun Basket or Blue Apron do or don’t know about their customers.
Anyway, after several years, I happened to be visiting someone out of town and over several nights we had delicious turkey burgers, and then really great chicken thighs. And I was overcome with a sense of regret, that I had been settling, that I was missing out on eating like this every night. The freedom from choice had been great, but now I was missing out on freedom of choice.
So when I got back home, we changed our approach again. To address the “what to make for dinner” – the problem that these services had solved for us – we made a spreadsheet. Entries included some basic examples like “tacos” but also meals from Blue Apron that we liked (with a link to the recipe), and most of the recipes torn from magazines or printed out that were sitting in a folder. Now we had our repertoire in one place, and we could refer to that rather than thinking oh wow what to make?
We began planning out a week (or part of a week, at least), discussing what to make in front of a computer (or two), opening up the recipes and updating our online grocery list based on what the recipe called for and what we didn’t have on hand. And then putting the URL for the recipe into our shared calendar for the night we planned to prepare it.
Our shared approach to cooking the meal itself of course was no longer tied to Blue Apron or Sun Basket, and we still do that. And we update the spreadsheet with the date we made a particular recipe, and any changes we would make or if we’d even want to have it again.
I subscribed to a bunch of recipe newsletters from the New York Times and elsewhere and have regularly been adding new things to try.
So we took back a significant part of the work – the planning, and the list making, and the shopping, but the results – the meals themselves – are so much better, and the experience is just as enjoyable. Indeed, the work of planning is not a chore, it’s now tied to feelings of excitement, anticipation, challenge, and discovery.
How we eat and how we feed ourselves is enormously personal and a rich vein to mine; it’s why I often ask workshop participants to practice interviewing on this particular topic. My point here is not to offer a recommendation about how you should be managing your meals, it’s to illustrate that the way we experience products and services can be different from what is intended, or what is assumed. That meaning can be a bigger driver of behavior than say “value” or “time saving.” And that perceived benefits or limitations shift and become reframed on a regular basis.
There can be a lot of nuance to uncovering this with your customers. If this sounds like something you’d like to explore, get in touch. Supporting my business is the best way you can support this podcast. And if you have any feedback in general, you can email me at DONUTS AT PORTIGAL DOT COM or find me on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
Let’s get to the conversation with Juliette Melton. She is the Director of User Insight and Strategy at the New York Times. Thanks for being on the podcast. I’m glad to get the chance to chat with you.
Juliette Melton: Thanks. Yeah, it’s nice to see you again. It’s been a while.
Steve: It has. Why don’t we start with just an introduction?
Juliette: Great. Hi. My name is Julie Melton. I’m Juliette Melton on the Internet and yeah, I’m a design researcher here at the New York Times. My title is officially the Director of User Insight and Strategy. We have in some ways moved away from the design research label, which I think is also something to maybe talk about today. But, yeah, we’re here at the New York Times. We’re sitting in a conference room just off of the newsroom where the news gets made.
Steve: Does news get made?
Juliette: Well, the news gets typed.
Steve: Can you unpack your title a little bit? Maybe that will feed into some of the themes you’re already teasing.
Juliette: Sure. So, we purposely changed the name of our group a while back and to talk about user insight and strategy. It becomes a little more broad and a little bit more impactful than I think just calling it design research because I think – in the IDEO context, design becomes a very large word. In many other places design is a little bit smaller and speaks really towards a specific function within product. And I think it’s important that research can transcend the product team because we can offer much more in an organization and we can do much more in an organization. So, by calling it user insight and strategy it puts the focus on the people who we’re actually understanding and not the end result and I think that that’s really powerful as a reminder to us and to the organization about what we’re here for.
Steve: I’m going to just back up on part of that. What does product mean in a media organization?
Juliette: That’s a good question. So, here, it refers to essentially the websites, the apps, the newsletters, all of these digital expressions of what we’re doing here. And there is some church and state separation here between what happens on the news side and on the product side. So, within the newsroom there are people who are interactive news producers, and they have a very different job than the people who are building the apps and the websites and the newsletters. It’s essentially a separate kind of function. We think about engagement with the news and how do we package, present, share everything that we’re going. And we’ll talk a bit more about it, but my current role is not really news focused. I worked on other stuff here. So, my group is called New Products and Ventures. I’m the Director of Research for these other businesses that the Times has been building. So, that includes NYT Cooking, Crosswords and we’re very proud to talk about Parenting which is a new digital product which was just released a couple of weeks ago. So, these are separate businesses. These are places where the Times is hoping to expand its audience, expand its reach, do some new things, stretch some new muscles. So, it’s really fun from a research perspective.
Steve: Are there other – without asking you draw the complete sort of corporate org chart, but those are three separate businesses. Are there others? I’m just wondering how big is the set of businesses that comprise kind of the Times universe?
Juliette: Well, that’s kind of hard to answer. There are a lot of different brands here. You know there’s The Daily, The Weekly, which is the new TV series, and then there’s other brands that sort of become part of what we do like Modern Love, for example. And these are all things for the – the organization can spin off into these different entities.
Steve: It’s interesting from the outside. I guess it’s the same with so many different kinds of multiple touchpoints, as we like to talk about businesses, where if you’re consuming a bank, for example, you – what do they say, don’t ship your org chart? So, I guess as a consumer of The Times media I might be aware of a lot of different brands, but that doesn’t necessarily map to – I think part of what you’re saying is they could be structured as businesses, or as – or not, depending on what makes sense internally. It’s invisible to me as a reader.
Juliette: Exactly. And so the things that I’m working on, I think, would be visible as separate businesses because you could subscribe to them separately. And so literally you could just subscribe to NYT Cooking, or you could just subscribe to Crosswords. Parenting doesn’t have a paywall at the moment, but down the road that could be something that you would pay for separately as well. So, yeah, there is some experience differentiation there. As opposed to you’re a reader of Modern Love – that is part of the nytimes.com world. So, that is just part of what you’d get from a Times subscription.
Steve: I’m sorry, can you say the name of the area or the division…
Juliette: That I’m working on?
Steve: Yeah.
Juliette: It’s called New Products and Ventures (NPV).
Steve: Alright. So, some could be products and some could be ventures.
Juliette: It’s sort of a catchall, yeah. I think the interesting thing there is it’s a spot for us to be innovative in a different way. To be innovative around business models as well as what we’re doing from the content side. And I think that that’s been really good for the organization and pretty interesting as a researcher.
Steve: So within sort of nytimes.com you can provide different kinds of content, but it’s always going to be under that business model because that’s the Times subscription model.
Juliette: Right, exactly.
Steve: But if you have a separate piece – like you said, you can subscribe to the Cooking app separately, so you can have that as a separate part of the business. And so that’s a business model shift.
Juliette: Yes. Exactly.
Steve: Okay.
Juliette: I don’t know how interesting this is for people. You know one shout out here is I’m a big fan of subscription models as a way to exchange value for the web. You know I’ve worked in the past with a couple of other subscription-based services and I think there’s something really honest about it. You know like someone pays you because they get value. Right. It’s just a very direct value exchange which is different from working with an organization that makes money in somewhat – in ways that are somewhat more hidden to the consumer, whether that’s through advertising, or some other kind of data brokerage. Like, yes we do have advertising at the Times, but the subscription is really like the core of the value exchange. And yes, I just like it from like an ethical perspective. I think that it’s a really like nice way to be doing business. But then also it’s fun to be working on these other smaller businesses because you can really measure success. You can measure outcomes because people are paying you. And so as a researcher it’s just really satisfying to know, okay, we did this thing and then this resulted in this much revenue. And just that direction connection I think is really – it’s good feedback.
Steve: So, as a researcher how are you using research to support the organization?
Juliette: Well it comes at a lot of different moments in the lifecycle of one of these products. For example, I did work last year with our strategy department to understand a new vertical and whether or not there was opportunity for us in that space. Looking at that from the market size perspective, from user needs, from how much brand flexibility we had, we decided eventually not to do it, but that’s a case where research can come in on a more very, very early stage, very strategic level. As you move through the phases of the design process we can be involved in very early stages of discovery. I’d mentioned briefly about the work that I’d done in Canada. I spent several years here working with our global expansion team, understanding, around the world, what might we do to expand our reader base? And that includes everything from payment models to partnerships to hiring additional people in different countries and building our new bureaus. And so it had a lot of different kinds of ramifications around what we’re doing as an organization to meet these needs. So, that all kind of starts to feel like the beginning of a process. And that’s where a lot of the work I did at IDEO, for example, comes into play – different thinking, understanding people’s needs from more of an opportunity space perspective. What might we do in this space answering some of these bigger questions? But then, you know, we’re building and we’re shipping and so the questions continue to get a little bit more specific as you go. So, it’s, you know – how exactly do we meet this need? Right. What do we do for this audience? So, for Parenting the team learned that there was this gap in – actually I don’t want to talk about that – scratch that. To speak in broader terms, because I don’t want to get too specific for the purposes of going public here, it’s about locating an opportunity and then designing for that opportunity and then along the way checking to make sure that you are designing that properly. And so that’s basically the cycle of generative to evaluative research and our team really does all of that.
Steve: In some organizations the narrative of research is about – you hear phrases like buy-in or advocacy or evangelism where it’s a newer process that is trying to support or integrate in things that the company is doing. So, now I’ve set up a very leading question. For you, what’s sort of the appetite or understanding or kind of engagement with these different areas of the organization that you have found here?
Juliette: I think that advocacy is always there. I mean I think that that doesn’t go away. I was on a panel a few weeks ago and someone came up to me afterwards – you know because we’d been talking about advocacy on the panel – and she said when does that stop? When can you not have to do that anymore? And I said like I’ve been doing this for 20 years and – and he’s like 18 – it’s like that doesn’t stop. You’re always explaining what you’re doing and why and what the impact is, what the outcome is. But I think overall this is a place that really gets it. We’ve been doing a really good job for a long time now and I can say that as someone who came in just 5 years ago. It’s like predating me I think the research team was really strong and so we just – we have a really good track record here and this is an organization that, you know, we seek to find the truth, right. And I think that that permeates the culture here. People are very curious. They really want to know what’s going on in the world. And you know when I’ve worked with journalists it’s been amazing. You know, our methods are sometimes different and the outcomes have different purposes. But so much of it is so similar to journalism that they get it right away and love it. So, it’s been very, very easy to build advocates in the newsroom because of that. And then on the product side, once you start working with a team and they get it, they just want more and more research. And I think that that ends up being more of the challenge here than people not wanting research. It’s just like bandwidth constraints. Like there’s only so much that we can do as a team. And so for us the challenge is more working with the different teams to figure out where our work can be the most impactful and where we’re going to be the most useful.
Steve: What kind of things might someone look at to assess where they’re going to have the most impact? Or to make priorities?
Juliette: One thing I always ask is – like let’s just play this out. What if we heard “X”? What if we heard “Y”? What do you think we would do about that? What could we do about that? Because sometimes what I’ve found is that people want to do research because they’re curious, but it doesn’t necessarily mean that there’s flexibility in terms of what they will do. And if they say look, we’re going to do this regardless, then I’ll say, that’s great, I don’t think this is a place where research is going to be that helpful. So, making sure that there’s bandwidth for that team to do something about it and the will to change course based on learnings. And if those aren’t in place then it’s probably not the best use of research resources.
Steve: If bandwidth was unlimited would you have a different perspective on people that start off with sort of an intractable request?
Juliette: You know, honestly, probably not because I want for research to be seen as a strategic function that leads to better outcomes, that leads to teams that feel more secure and more enlightened. And if it were something that you could just sort of sprinkle everywhere, I think the power of it might become a little more diffuse and I want for it to be like a powerful, useful tool and not just like an oh I wonder about this or that, you know.
Steve: Yeah. So, there’s a scarcity in value – that’s not really what you’re saying. I think I’m changing what you’re saying and reflecting it back to you.
Juliette: That said, I would love if we hired 10 more researchers, yes. I think that would be great.
Steve: But that still wouldn’t give you the bandwidth though…
Juliette: To answer everything for everyone? No, it would not.
Steve: What’s the size of the team right now?
Juliette: Well, so the overall insights team – I think it’s interesting here because we really encompass what would be considered market research, what would be considered product or UX research? We all sit under the same group which I think is a very positive, good thing. I’ve worked with a bunch of organizations that have separate market research and UX research functions and that just gets really messy, right, because then it’s like who owns the truth? Who owns the insights? And you don’t want to be at odds with each other. I think that when you fly in formation it just like makes so much more sense and there are very powerful research methods that have come from both the market research side and from design research/UX research. So, being able to play in all of those sandboxes, however you need to to address your clients’ needs is pretty fantastic. So, that said, yes, our research – our overall research team encompasses all those different functions. The group that I sit in we do more on the qualitative side and probably more on the product side and there’s 5 of us. And then I’m going to kind of spitball here, I think there’s probably about 15-20 altogether and, like I said, that encompasses all these different insights functions.
Steve: So, when you say 10 more – it would be great if you got 10 more people, I just wonder sort of the context of what does that mean? It’s about a 50% overall increase.
Juliette: Yeah. I think that would be lovely, but here it’s the booming newspaper industry.
Steve: So I hear.
Juliette: It’s a good time to write things on paper and sell them.
Steve: So, you’re talking a little, maybe directly or indirectly, about where – you said you work a lot with product teams, but I think you sort of said in one way or another that there’s a potential here, or just an opportunity to work across the organization in a broader way. And I don’t know if I’m putting words in your mouth or channeling something that you said, but…
Juliette: You’re channeling. You’re doing a great job, Steve.
Steve: Good feedback, thank you. You’re doing a great job as well. Good.
Juliette: Um, yeah, no I think that’s right and the work – in the same way that we can play in the different sandboxes of market research and product research, you know I work, particularly with new products and ventures, I work with the marketing teams and the designers and the product managers. So, I work with everyone, right, because from a user perspective it’s one experience. It’s like how do you relate to this organization, whether through that they tell you, what they show you, all of that is a singular experience. And so being able to be more holistic about that I think is really powerful.
Steve: I wonder sort of what parts, and I don’t mean your organization, but just organizations in general that have research kind of lurking somewhere, maybe tied to product, but I wonder what sort of – what’s the next big thing that research will be doing beyond sort of where we’re seeing right now? I don’t know. And again, I’m not asking you to speculate about your own organization, but you’ve seen a lot of different things. Do you have a – what should we be doing?
Juliette: Here’s my hunch. And this is also an equivalent about me, I’ve been doing coaching training and have been doing some personal life coaching, a few clients, to build those skills because I think that working inside of organizations, working with people here, is something that is very natural for researchers to do and I think for us to build these coaching capabilities is really helpful. You know, understanding the people who are around us, what are your unmet needs? What is this really about? What are we doing here? Asking and answering some of these deeper questions. It doesn’t have to be just about the consumers of our products. It can also be about what we’re doing inside an organization. So yeah, I’m really inspired by that and that’s why I’ve been building my own coaching muscles and that’s been super fun. And I would love to see down the road how coaching and research start to come together as – you know does that become something that looks more like org design? I don’t know. So, that’s a place that I could really see some benefit down the road and it’s been interesting to see other researchers or product people move more into coaching and therapy roles. You know, things like that, grounding the human experience in a somewhat different way.
Steve: Yeah. And I’ve definitely seen. I mean I’ll just confirm my N of one. Like I’ve seen this as well, just with people that I know. I want to ask a couple of clarifications here. Is this something that you’re doing on your own versus as part of the role you have here?
Juliette: It’s both honestly. So, it’s partly like my own passion on my own time. The non-working time. But it’s partly something that I do bring into work as well, working with different people on my team and working with people on other teams, doing kind of informal internal coaching here. And so that’s been super rewarding, but I would love to make that more official. And I do feel like as researchers that’s something that becomes natural for us and could be more codified and more supported.
Steve: I just in the last couple of weeks have had a couple of informal conversations with people about is that something that I do. But then, just in a self-serving manner I want to ask if you can put a little definition around coaching. It’s like a term that we all think we know what it means and I’m honestly not sure I know what that means?
Juliette: Well, it can mean a lot of different things. And when people ask me, well what kind of coach – I mean it’s kind of a life coach, but I hate that term, right. So, I think that we need better terminology around it for one thing. But how I see it is it’s about people understanding where they are now, where they want to go and how they might get there. And it overlaps quite a bit with more kind of standard therapeutic approaches which tends to be more about where I came from and what does that mean about where I am. I think everyone probably needs both a coach and a therapist. They’re very complimentary. But I think particularly in organizations, to understand where I am now, where do we want to go and how we might get there? I think researchers are set up really well to be able to ask some smart questions to help people develop this understanding of their own and help support them through that process.
Steve: So in research there’s an interesting tension and I don’t – we’ll see if it’s true for coaching as well, about – put it this way, are we in the recommendation business, or not? Do you know what I mean?
Juliette: I know exactly what you mean and no, we’re not. I strongly believe that? I think that our role is to unpack relevant truths so that teams can collectively make better decisions and yeah, I think informally, sure. Like I can be part of those discussions and kind of help guide them in some way. But I don’t think that research is about recommendations. And the same way, coaching is not. I think I have a bad habit of giving advice when it’s not asked for and I do hope that by going through the coaching training that I have I do less of that, right. I want to be able to know when to ask the right questions so that people develop their own insights, their own understanding. Because that’s what sticks. That’s what works. And I don’t know – if I were your coach like I wouldn’t – I don’t know, what could I tell you about what to do with your life? Like you know your life. I don’t know your life. But I could ask you questions that could help you unpack for yourself what’s relevant for yourself, in a way that it’s kind of hard to do without someone else there with you.
Steve: So, I think this is true for researchers and I think for coaches as well, I think that the choice of our language can change the frame. I think right, you’re probably not going to say to me, Steve, you should do this. But you might – and I don’t know how this applies to coaching, but I think in research we might say other people in analogous situations have done this, right. Here’s a benchmark or here’s a different case. Or we might say when I was in a similar situation I made these choices. Or, if I was in this situation myself I would make these choices. I don’t know…
Juliette: As a coach I wouldn’t say any of that. No. If someone flat out asked me, like have you been in this situation before, then I would sort of gauge whether or not that would be appropriate at the moment. But no, I wouldn’t think I would do that. And even with analogous research, I think it would be presented more as like a hypothesis. Like it seems that this other way of solving the thing could be useful for us. Like what do you unpack from this? What do you see here? And I could offer this is what I see here, but I wouldn’t say this is what this means and this is what this means for you.
Steve: Right. I think – and I hope you are – we’re just kind of bouncing between these two things which we’ve set up as analogous here, between the researcher and their client and the coach and their client. I hear from researchers that if they don’t sort of provide more recommendation or advice then they lose out on attribution which is not about ego, it’s about – it’s about advocacy. It goes back to that advocacy thing. If you’re not seen in a tangible way as the work that you’ve done has led to these changes, right – I mean there’s something about helping people come – maybe you come up with the idea, but you help them feel like they came up with it. And then it’s going to go. It’s not going to go if you tell them – I think that’s also the limitation of advice. But then it introduces this attribution and sort of impacts advocacy and so on.
Juliette: I think that’s a perennial question for us, you know. And I want for my coaching clients, and the teams I work with, to feel like – like they own their insights. I want for them to feel like they’ve been part of a process that they have directly contributed to that has resulted in insights coming forth that help them make better choices. And I might have had in my head that that’s what they should do anyway, but like – yeah, if I told them that, like it’s different, right. You want for people to come to their own understanding. And yeah, you lose credit. And it’s something that on the research team we talk about. You know like what is – like you have to be okay with some ego loss there, right. Like research is not the thing to go into if you want like the billboard with your faced on it, right? Like ideally the things that we learn become so embedded in the DNA of how an organization works that it’s not really attributable to a person, even if you were the one who made that happen. But yeah, that becomes tricky when you’re wanting to advocate for what you’ve done, right, and ideally you just have really good working relationships with the people around you and they’re kind of aware of your magic powers. Like, when this person is in the room smart, good things happen. And I feel like I make better choices when this person’s around, right. And ideally that’s what happens.
Steve: So, that’s attribution about the relationship or about the collaboration.
Juliette: Right.
Steve: But not attribution for we took action “X” because of input “Q”.
Juliette: Sometimes it is that direct, right. I mean sometimes coming out of research, like we’ll have a very clear understanding of what to do next and then we do it and it succeeds and you can like trace it very cleanly and that’s always very satisfying when that happens. Other times it’s more of a slow burn. You know it’s like you hear something in research and people start to internalize it and then they kind of start to believe it and then they start to act on it and then it becomes like a new thing and I could point to several different mindset shifts that have happened here that originated in research, that I think that – that I know that’s what happened, but I think the folklore in the organization would just sort of describe it as oh now we think this, now we believe this.
Steve: Right. So, there’s something about taking your ego out of it.
Juliette: Yeah.
Steve: I’ve complained about this a couple of times, a series of tweets I think that were from a talk a year or so ago, that I think is not an unusual sentiment, where they were saying my research doesn’t have value unless somebody takes action. And yeah, that got me upset because I think it’s a really high bar and it sets us up to – it’s a statement of our value, or our lack of value.
Juliette: That sets us up to do very small projects, right.
Steve: So that you’re always pointing towards.
Juliette: Yeah – I mean, yes, if you strictly do product based evaluative work then you could say that yeah my research always results in something because that something is like moving this button or moving – I mean like that’s something, right, but when you’re talking about change which is more systemic, which is on a deeper level, that takes time, you know, and that kind of research – those research outcomes, like I said, like it takes a while to be embedded in what you see from an organization. So, yeah, I mean there definitely have been some significant projects here which would I say two months later, like yes, someone you know moved this thing on a roadmap because of it? No. But I can say three years later we’re a different organization because of it. And so I think that having some patience and some – I don’t want to call it maturity, but it’s just kind of like being able to see a bigger picture from both a timeline and a strategy perspective, you can really take on bigger challenges and see the impact of those within an organization.
Steve: I wonder if people that – I mean I’m self-employed so I’m – these things manifest differently for me, but for people that are – where their promotion or their title, or just their sort of general success is tied to some of these things. I mean – your point about patience – so, I think there’s a couple of levels. One is sort of how do I feel about what I’m doing? Right, am I alright? Am I confident, or am I learning or am I growing? And am I successful in kind of a corporate structure which can only measure, reward and acknowledge certain kinds of things? And so yeah, I think I could encourage the former and I have had to learn that very much myself. Like things you think are going to be the rewards for doing this work are often other ones which maybe you have to go hunting for to see that they’ve even happened.
Juliette: Yeah. And I think that’s part of why, as researchers, again, like we don’t want to just stay embedded just in product teams. Right, like we need to know executives. We need to know people across the organization, people leading an organization so that we can understand the big problems facing an organization and they can understand what we’re doing about it, right? And I think that sometimes that just comes from relationships and being able to kind of get a read on what’s going on. But yeah, it’s hard. It’s hard. You know when we can put a dollar value on it that’s amazing, but it’s hard to do that.
Steve: You know you speak in a fairly optimistic way, I think, about not even just the potential for research – potential, so that says it’s kind of in the future. I mean it’s the value that research is bringing to – in the organizational mindset, the belief structure, as well as the products and services. You know, when you look outside your own organization, you know, between optimistic and pessimistic, what’s your assessments? Maybe it’s not that axis, but what’s your assessment of kind of what’s going on?
Juliette: You know I – the design research Google group that a lot of us are on, it’s grown from two people who started it. Now it’s like 1,000 something people. That’s a lot. You know, like our funny little profession, like we keep growing and people keep finding value in it and it’s something which is in many ways new to the world. We’re building it as we go and that creates its own challenges because it’s hard to know what does a 40 year career look like in design research. Like we haven’t really done that yet. Right. So, it’s not like being a doctor or a lawyer where you know like okay, at this point in my career I can expect to be partner. You know, so it’s less secure than that, in that way, but it’s also what’s fun about it, right. I mean like we are co-creating this profession and so like why not be optimistic? Right, I mean like our job is to make things better and I think that we do that even within our communities. And part of why I’ve stuck with research for so long is that it’s such a nice community. Researchers are nice people and they’re people who I like spending time with. So, yeah, I think that bodes well for the future of the community and future of our impacts.
Steve: So, what does an 18 year career look like?
Juliette: Well, I think I’m making the path as I go, you know. And so far I’ve – you know early on research was one slice of what I did. I was a front-end developer. And then later I did more product and then I focused on research and started on the more evaluative side and then moved to the more generative end. And so I’ve really done all of it as it currently exists. So, you know, I’ve been involved in the web since it looked very, very different than it does now. And so far so good, right. I mean we’re still here. But yeah, as I think about how to move forward, I think we make the path by walking it and I have a hunch that there’s something here about researchers looking inwards at organizations – and again, I don’t know a ton about org design. It may be already kind of covered by a lot of these other practices, but at least for my own life I feel like there’s really something about using the skills and techniques of being a researcher, combined with what you can do as a coach and specific training that you get as a coach, to do something different within organizations. I don’t know what that is yet, but I think there’s something there and so that’s the path that I’m kind of figuring out as we speak.
Steve: That’s exciting. I want to go back a little bit. Because you were just giving a nice overview, I think, of where some of us have been and where the community has been, that if you’re at 18 years and it’s all about walking the path to kind of find the path. There are lots of people sort of coming into the profession, let’s just say in the last three years, that – I mean – I feel like when I started, and maybe the same thing for you when you started, there wasn’t a lot of path.
Juliette: Right.
Steve: And I’m like you. I don’t sort of know where I’m going. I mean following the path is a much more confident way of – I might say stumbling – but, it’s just interesting that the field has grown and you just look at – there’s just a number of people and the time. And so somebody that sort of chose research, or was chosen by research in 2017 is – they’re part of a different community than you were 18 years ago. So, what it means for them to sort of learn. There’s, there’s – the terrain is different because it’s populated by people like you or there’s a path that’s woven by people like you. I’m going to take your lovely metaphor and just…
Juliette: Go for it.
Steve: Mow over it. I find that interesting. Just back to sort of my question about advising, the people who are joining the field are joining a different field than the one that I joined or that you joined.
Juliette: I mean it wasn’t a field at that point, right. I mean like there were a few people who were doing some great work, but it wasn’t really a community, at least as far as I was aware of. So, I think people joining now – I think it’s very cool that there is a field for them to join and they could hop on this gigantic Google group and ask a question and have 1,000 useful people helping them out. You know for people starting this now, I think they’re in a great position. You know you can actually learn this in college. Like, you couldn’t learn this in college when I was in – at least where I was in college this is not a thing. You know I studied comparative literature in undergrad and I remember telling my comparative mystical literature professor that I was wanting to go into IT, as it was all called back then, and he was like why? Why would you do that? Like just totally confused. Like that doesn’t seem like a realistic or interesting thing to do. It wasn’t really a path. So, yeah, I think it’s great that people are joining now and I think there’s more support. You know in general for all this technical, digital roles there’s been a lot more definition in terms of the different job functions and I think that that makes it a lot easier. I mean back in the day it was about like webmaster and they did everything. There aren’t webmasters anymore, right. I mean like the jobs have become much more specific. And I think that makes it easier to train, to decide like what you want to be doing to become really excellent at what you’ve done. You know, it’s a bit of a tangent, but I think that research as a standalone profession is I think something that’s also becoming more possible to do for – I’m curious your take on this as well, but I feel like the sort of broader field of UX, you know someone is like a UX person who does design and they do research and it’s all kind of rolled up into one thing – I didn’t want to do that. I wanted to be a researcher and so when I made that choice 10/11 years ago to focus on research, it really limited my options. It meant that I could either work for an agency or for a really big company. But for smaller companies they really wanted more of a full service “UX” person. I’m curious if you have seen more growth in smaller organizations hiring specific – like researchers who specifically do that?
Steve: Who are researchers?
Juliette: Just researchers, yeah.
Steve: I mean this is anecdotal, of course, but… I think there are – there seems to be sort of a pattern of maybe the first researcher in an organization is – or maybe person zero is someone who is doing multiple – we have an intern or we’re going to take some of somebody’s time, they’re interested and they’re going to go do this. A product manager maybe who has kind of got the bug, or read something or saw something. But you know when companies start hiring someone with a research title I think sometimes that first person is very junior and then it has to build up to a point where there’s a realization, oh we actually need leadership and mentorship and being able to develop a function and a practice and a team. And I have a lot of concern about that person who is sort of brought in with no mentorship. I think about the person I had coffee with about a year ago who was – I think he was not too long out of graduate school. So, professionally junior, but I think educated a certain amount, and they were the only research person in this organization and they basically were sent to do something where they were told how to go about it and what to produce. And of course, if you don’t know what you’re doing there could be a mismatch. But they took the hit for this and they, I think, ultimately were dismissed because they were unable to produce the information that was needed from the method they were required to use. And we had this long conversation and this person was convinced it was their fault and that they were deficient as a researcher.
Juliette: Oh man.
Steve: Yeah. And to me that was a lack of leadership.
Juliette: Yeah. This company needs coaches and researchers it sounds like.
Steve: Yeah. Some executive coaches or management coaches who say here’s how to manage your staff to success. And that’s I think – because there’s sort of this generic perception of research, or that it’s just a thing and like you’re just talking to people, or if it’s a survey, just go set up a Survey Monkey, then it seems easy to have people that don’t appreciate the craft and the mastery of someone like you that’s been doing this, that chose this, can bring. I mean that’s a really sort of disappointing example, I think, but there’s the researcher who has the title and is not in a situation where they’re going to be able to thrive, but I mean – if your question is like are there researcher title jobs? I think a huge amount.
Juliette: I guess it’s more like is that increasing as a phenomenon because yeah, I don’t know. I’m just curious if as these other roles have become more specific if there’s been a greater appetite for researcher as a standalone function. But yeah, I mean I’ve been that person. I’ve been that junior single researcher and it’s really hard. And that’s why I take heart in some of the larger online communities that have formed because I hope that what we can do for each other is to help advocate for methods and for approaches and to kind of stand in as mentors to people who need that support and don’t have that person there with them. Like I would love for us to keep doing that for each other and I think we do that to an extent and that’s part of why I really advocate for researchers to know other people who do this. And it’s also part of why I like working here. You know I work with a bunch of really, really smart researchers and if I need to pull in someone who is an expert in running a conjoint analysis, I can do that. And it’s nice to have this deep bench of expertise just available here and that’s part of why I really enjoyed working at – not a huge company, but like a big enough company that has a deep bench when it comes to research vs. like me off on my own doing this startup. I feel like I can be a better researcher when there are people around me who I can lean on for their expertise as well.
Steve: So, you’re okay not being able to do a conjoint analysis yourself?
Juliette: Totally. We’re always learning, right, all of us are learning, but we work with people who have PhDs in statistics and like I think that’s great. That’s probably not something that I will do. I think it’s wonderful that we can specialize in different areas and I think that makes us all stronger.
Steve: You know it’s that depth vs. breadth tradeoff. You know you’re sort of asking about research as a specialty, but even that word, to your example, it comes with so many things.
Juliette: Yes.
Steve: I started off trying to position myself as a generalist a long time ago and the more that times go by the more things I realize I know nothing about and personally it’s terrifying. Maybe that’s too dramatic a word, but I sort of have to own it and then also deny it, that there’s this thing that I don’t know how to do and so I’m aware – what does expertise look like? And what does expertise look like for you as you proceed and there are things that you’re going to pull in that are not going to – you may never be a master at conjoint?
Juliette: That’s a great questions because I feel like there are things that I’ve definitely decided to not specialize in, you know, and one of those is prototyping, for example. At one point I did do much more design and now I am totally happy not doing that. I think it’s – knowing what you are okay letting go of I think is a very important professional skill. And I think it’s part of growing, is knowing like this is something that I’m going to bring someone else in for. And, yeah when it comes to live very, very complex data analysis, there was one point in my career where I was really focusing on that and was building a pretty big skillset there. I think actually last time I saw you at IDEO I was really focused on building a very deep quantitative understanding for myself and eventually I decided, no. Like I would rather lean on the person who has the PhD in statistics and I want to refocus on the things that I feel like I can more uniquely bring to the table. And for me that’s more about qualitative research, that’s more about design thinking. Like those are things that I can uniquely do that I think someone else couldn’t necessarily uniquely do and so yeah, I think just understanding what you want to do and what you don’t want to do I think is a really nice part about getting older in this world.
Steve: So, how should someone approach that? That is maybe in the first few years of a professional career.
Juliette: You know, I think the first few years like don’t go around saying no to stuff. You know, say yes. And then follow your intuition there. But yeah, I’d say probably don’t get too – I think in general in life, like be cautious about saying no to things that you don’t really understand. Like maybe kind of dig into it a little bit because there might be something in there for you that you hadn’t expected.
Steve: Do you, like let’s just imagine the magic wand thing. You were sort of saying that if you could hire 10 researchers you wouldn’t be opposed to that.
Juliette: Right.
Steve: So, if we were to wave the magic wand and make that happen, as you would start talking to people, what kinds of things would you – I realize that we don’t have a job description – whatever – I mean what do you start to look for when you talk to people about working with them?
Juliette: Um, I like working with people who – the first thing I would say is that it’s important to hire people who bring a good energy into a room and I know this sounds like super – I did live in California for a long time, right – but I think that researchers can bring a kind of brightness into a space and a kind of optimism for a team and a sense that we can learn these things together. It’s a bit intangible as a quality, but when we bring on new researchers that’s really something we look for. Like is this person someone who is excited about making connections across an organization? Excited to share what we’re doing? You know there’s something about bringing energy into research which I think is really important, but beyond that, yeah, I mean being able to really, really listen. Being a thoughtful, humble listener to what people are saying and I can sometimes know in the first conversation with someone whether or not I would encourage them to move ahead as a researcher based on their qualities as a listener. And then being able to think thoughtfully, strategically, big picture and being able to execute. It’s hard to – it’s so hard to hire, right, because these are all kind of a little bit intangible as qualities, but yeah, looking for someone with a curious mind who truly likes people is really important. I think that’s key.
Steve: When you say brings energy into the room and is a humble listener, part of me – a part of me feels like well is that contradictory? I’m really just thinking about myself. The one is sort of an extroverted energy and one is an introverted energy, but I don’t know.
Juliette: I’ve found that most of the researchers I’ve ever worked with are true introverts and I would put myself in that same category, but we are – we are introverts who like people, you know, and I think that almost all of the researchers I’ve known fall into that category.
Steve: Maybe this is a personal issue, but why not. The bringing the energy into the room thing is interesting to me because I’m probably more of a listen first person. I mean I pride myself on being a good listener, but that also means I sort of – I watch a lot and, you know, depending on how people come across me they might see me sort of in on stage mode where I’m saying a lot and I am telling you stuff and sort of trying to bring something. But I spend most of my time sort of holding back and just waiting for those moments and I think it’s – like I feel intimidated by that description to sort of have to make an impression.
Juliette: But I think what you’re talking about is its own kind of energy, right. So, the energy that you’re bringing into a room is I support you, right. I’m here to listen to what you’re saying and it matters to me, right. And so like that’s a very particular kind of good energy.
Steve: Okay.
Juliette: Right. It doesn’t mean that you’re coming in like you’re running for Mayor, right. But it means that you’re coming into contexts and like in a positive way. You know like you’re not glowering. You’re, you’re – you’re engaging, even if you’re not saying something. And so I think that that’s – does that make sense?
Steve: Yeah. And I think just clarifying words is so important, right, because when you said energy that made me think of extroversion and speaking and being loud and “hey,” and smiling. And you’re right, there’s a lot of different ways to manifest that energy.
Juliette: So, I’m coming in as like a Reiki person and so for me it’s about like what is the thing that you’re emanating? And what is the thing that people are picking up on about you? And what are your intentions when you’re coming into a space?
Steve: Okay. So, you listed a bunch of things that you said were at varying levels of intangibility. But what are some successful ways that people could present those qualities? Demonstrate them? Whether it’s in a sort of interviewing context, or like a job interviewing context or any other?
Juliette: Yeah, I think being a thoughtful communicator is really important. And so when you’re communicating with a potential employer just paying attention to what they’re asking you for and really showing up for those conversations. I think that that’s a big part of it, right. And being able to share examples of work that you’ve done, that you’re proud of, and being able to speak to experiences that you’ve had in a really cogent way. One of the – we haven’t really talked about this, but I think one of the biggest skills that a researcher brings is a synthesis. I mean being able to look at a set of disparate facts or impressions and pulling something meaningful out of that. And so I think if you’re – if you were in an interview to be a researcher somewhere I think that you’d want to be able to speak to work you’ve done in a way that makes sense, that’s to the point, and that communicates what you want to communicate.
Steve: So, can we maybe switch sort of the who’s communicating to who scenario. So, we’re talking about people at a certain stage that are kind of approaching organizations. And then earlier on we talked about advocacy. You eluded to being on a panel about advocacy. What are things that researchers can manifest, display, whatever the verb is, for the people that are – that we want to get excited about research, that we want to sort of demonstrate the value. What should we be doing or saying or showing them?
Juliette: This gets back to our conversation on impact. Um, which again as we talked about is tricky with research because sometimes that impact is direct and sometimes it’s more indirect. But I would love for someone to be able to speak to that. You know and they could say like look we did this project and there wasn’t a direct outcome from it because there wasn’t space on the roadmap and there wasn’t budget and there wasn’t executive buy-in, but I was able to keep bringing this into the organization in ways that eventually it became part of what we did and here’s an example of it. So, you know like that’s a story that I would love to hear from someone who wanted to be a researcher because it shows that you have an awareness of the nature of reality, but that you’re able to effectively operate within complex interpersonal spaces.
Steve: So, let me change the players in this dialogue that we’re playing around with as well. So, that’s kind of a researcher to researcher thing, but now maybe it’s you, or it’s future you, and you’re talking to someone that is interested in, or has some curiosity about, or maybe some skepticism about how what you do can bring value to their part of the organization. How do you advocate for research for them? What do you highlight?
Juliette: You know it’s interesting because I feel like I haven’t had to do that for a while. You know we haven’t had to do that here because people know about success and they want some of that. So, if you can do a great job with one team at an organization other people will know about that. And so, I think being able to effectively do one good thing within an organization will help you work across the organization. Does that answer your question?
Steve: Yeah. I mean the best way to have that conversation is to preempt the necessity for that conversation by doing work that tells the story itself. I’m just saying what you said, but I said it faster.
Juliette: Yes. Thank you for synthesizing that.
Steve: That’s podcast…just repeat it back faster.
Juliette: You’re doing a good job, Steve.
Steve: Thank you. You’re doing a great job as well. So, we’re jumping around in chronology here a little bit. I want to go back to one thing that you said. Maybe two things. What did you say to your professor when they were like huh, IT, why would you want to do that?
Juliette: That’s a good question. I think this was in like 1998. It was a long time ago.
Steve: And you don’t remember exactly what it was?
Juliette: No, I don’t. But I remember – I don’t know, I wasn’t defensive. I was just kind of like. I don’t know. I think honestly I felt kind of proud that I was doing something strange and it was like I’m an iconoclast. I’m doing my own weird thing here. So yeah, I think I was probably like – I was like a nice kid. I was probably like oh yeah, I don’t know, that’s interesting. But I think on the inside I was like yeah, I’m a rebel.
Steve: Yeah. Right. Sometimes that external perspective – you know if you’re in that sort of bubble, or that identity, that perspective that you’re breaking out of is like – you may not have felt that way until someone told you that you were rebelling.
Juliette: Yeah. I mean especially at the time. It was like the Internet was just like new and weird and juicy and I was like there is something going on over there, like I want to get into that, whatever that looks like. That is strange and that is good.
Steve: And another moment, I don’t know if you can recall any details, but you talked about sort of starting off in front end development, that eventually you got into research. And I’m wondering was there a moment or series of moments where you saw that it was a thing, or that it kind of sparked for you that – do you remember what you were doing or what you were thinking when you started to feel pulled towards it?
Juliette: Yeah. You know when I was doing front-end development, and this was back in the day when like, you know web standards were just really hitting and we basically tore apart the web and rebuilt it in a different way and it was a really like heady time, you know. Starting to use CSS in ways that hadn’t been used before. I mean the underpinnings of the web just within a couple of years like started looking very, very different. And so that was just kind of an exciting time to be doing that particular work. But then eventually I just got good at it and it was like someone could hand me a mockup of something and I could build it. Like, okay. But – I do remember a couple of times where I was building something and I thought that’s a bad idea, why are we doing that? And I realized that I just wanted a different seat at the table than that. And so that’s when I moved more into the product role which at the time I think was a little less exciting than product now. I was working for an organization that had a very long term perspective on things and we were doing things like requirements documents that would take several months to do and then a development cycle that would take several months. And it would like be a year all in, right. And so I would just be writing these like tomes and again eventually it was like, yeah, I want a different kind of seat. And I remember a moment – this was also when I was considering a move to California – was if I wanted to go more into interaction design or wanted to go more into research and could sort of see different pathways. And eventually did more on like the project/product end and then while I was at one company really pulled the research threads out of that and created a new role for myself around research. And that’s when I was like I am a researcher. So, I really built that role for myself out of more of like a project/product manager role.
Steve: Maybe it’s different – it’s definitely different now than then, but any thoughts about – like you just declared research as an identity in that part of the story and I think that’s an interesting one. We don’t talk about the identity part of it enough. We talk about sort of skills, or are you a designer, or maybe you’ve heard the phrase people who do research? I’m not even sure what my question is here. Do you have a perspective on – if I throw out the idea of research as an identity does that spark anything for you?
Juliette: If you’re familiar with Richard Scarry books, Busytown, there is like the spoof of Busytown with like the little internet creatures and there was the like innovation strategist and she was building a 2×2 framework covered in Post-its and she was wearing like a printed anthropology dress that looks basically exactly like what I’m wearing now. And I was like oh, like that’s uncanny. Maybe we do have an identity. Or like maybe there is something to this that we’ve kind of like co-created as a thing that we’re doing. So, yeah, that was the first moment where I thought huh, maybe there’s a researcher identity that’s happening. But yeah. I think we’re not as cohesive, partly because I think we’re not – we’re not that stylish. Right? I feel like I can walk into a group of people and they’re like okay those are the designers, those are the exactos. I don’t know, maybe the researchers are the ones who are like very earnestly listening and kind of like a little disheveled. Maybe those are the researchers.
Steve: Right. And you’re framing identity as an external thing which is interesting. I thought more as an internal thing, but you’re kind of saying that there’s an aesthetic maybe of certain practices and that if researcher identity is observed more by behavior than that puts it into a different space than like a designer identity which might be more about…
Juliette: Steve, you’re really good at restating things in a more coherent way. Thank you for doing that.
Steve: Thanks. You brought up Richard Scarry which I think is a very good thing as well. So, thank you for doing that.
Juliette: I’m the parent of a toddler, so, you know here we are.
Steve: Yes. Write what you know. Okay. I feel like if we’ve hit Richard Scarry we might have hit – in fact Twitter parodies of Richard Scarry might be sort of peak episode. Anything else that you think we should talk about today that I didn’t ask about or you want to get to?
Juliette: Just that I’m excited to see where we go with this.
Steve: This episode?
Juliette: No, no, with research, as researchers.
Steve: Where the community goes? Where the practice goes?
Juliette: Yeah. Like, ‘cuz no one knows. We’re co-creating this thing and I guess I would say to myself, and everyone else, like take heart. Like we will keep building this thing together and it’s going to be good and just because there’s not like a traditional career path doesn’t mean you’re not going to have like awesome things you’ll keep working on and we’ll figure it out as we go.
Steve: Do you have hopes or dreams? We don’t know, you don’t know – that’s fine – but is there a place you can fantasize about it going?
Juliette: I think what I was saying about research having like a more both strategic and emotional impact within organizations that transcend making choices about how we serve customers. I think that’s a thing that we can do.
Steve: And maybe I’ll just add to that. Maybe there’s an element where that also is explicit instead of tacit and then that wraps up some of the challenges we were talking about.
Juliette: Yes. And in fact, a plug for NYT Parenting, we’ll soon be doing a series on emotional labor. So, if you think as researchers, having a lot of the work we do become acknowledged and codified I think will be very helpful. And it’s not just the tacit, emotional labor that we currently do.
Steve: Excellent. Well, this has been a fabulous conversation. I really appreciate the chance to speak with you.
Juliette: Thank you. Thank you for coming to New York.
Steve: Okay that’s a wrap! You can find Dollars to Donuts on Spotify and Google Play and of course Apple Podcasts. On the web at Portigal dot com slash podcast you will find all the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 25. Juliette Melton of The New York Times first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Juliette Melton, Director of User Insight and Strategy at The New York Times. We talk about updating the old “design research” label, user research in a journalism culture, and the role of coaching.
I think that researchers can bring a kind of brightness into a space and a kind of optimism for a team and a sense that we can learn these things together. It’s a bit intangible as a quality, but when we bring on new researchers that’s really something we look for. Like is this person someone who is excited about making connections across an organization? Excited to share what we’re doing? There’s something about bringing energy into research which I think is really important. – Juliette Melton
Show Links* Freedom From Choice cocktail * Blue Apron * Sun Basket * The BEST Turkey Burger Recipe * Saigon Chicken Wings * Juliette on LinkedIn * Juliette on Twitter * New York Times * New Products & Ventures * NYT Cooking * Crosswords * Parenting * The Daily * The Weekly * Modern Love * IDEO * Survey Monkey * Reiki * Richard Scarry’s Busy, Busy Town * The Innovation Strategist
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.
There used to be a restaurant in San Francisco called Heaven’s Dog, and one of the items on their cocktail menu was called “freedom from choice.” Rather than choose a specific cocktail, you’d tell the server your preferred spirit, and a bit of direction on the flavor, and they’d make you something to fit those criteria. While once freedom from choice would only have been seen negatively, here outsourcing your choice to an expert became a positive. The menu still had all the cocktail options for people that knew what they wanted, but for patrons who didn’t know what they wanted, there was an option for them as well.
A number of years ago we started using a meal kit delivery service, first Blue Apron and then later Sun Basket. In case you aren’t familiar with services like these, they deliver everything you need, once a week, to make some number of meals, in our case three. When Blue Apron started, they offered very little choice, we could opt for vegetarian or not, but otherwise, they just sent us three meals. Eventually we could opt out of pork, or fish, for example, and then they began offering a larger number of meals we could select from about a week before.
We immediately loved Blue Apron. It eliminated the planning and deciding, something that we just weren’t good at. Or that we avoided, because we didn’t enjoy it, and so we’d just make the same set of meals based on what we had done before. Using Blue Apron reduced the shopping, but not entirely as we still needed fruit, milk, bread, juice, and so on, for the rest of our meals. More interestingly and most meaningfully, it totally changed our meal prep activity. Where previously one person would have decided and cooked, and the other have cleaned, now the whole process was something we could share. Rather than anyone “owning” or controlling the preparation process, we had this well-designed, visual recipe on it’s own peice of paper. With the Blue Apron meals, we cooked together, easily weaving between chopping and heating, each person taking responsibility for a each task but no assigning, no delegating. We would discuss ways to improve or alter the recipe to suit our tastes (less lemon juice, more red pepper flakes).
Sometimes we’d take the meal kit over to someone else’s house, friends who were also Blue Apron customers, having planned to save a common meal to prepare together, playing out the group cooking activity with a larger group.
I was mixed about the results, some meals just weren’t that tasty, some meals required a lot of work for little result, and I felt like my own cooking wasn’t improving, as following the instructions didn’t give me much insight. We looked on social media to see what Blue Apron customers were saying about a particular meal, in case that gave us insight into choices we could make. Eventually, we switched to Sun Basket for more choice and better quality meals. But most of my satisfaction came from the preparation activity, rather than the actual meal itself. That wasn’t necessarily a failure; it’s worth taking satisfaction and reward where you can find it.
As with any product or service that I use, I often wonder if the producers understand the ways it’s being experienced, and how that inform the choices they make in configuring the service. I don’t know what Sun Basket or Blue Apron do or don’t know about their customers.
Anyway, after several years, I happened to be visiting someone out of town and over several nights we had delicious turkey burgers, and then really great chicken thighs. And I was overcome with a sense of regret, that I had been settling, that I was missing out on eating like this every night. The freedom from choice had been great, but now I was missing out on freedom of choice.
So when I got back home, we changed our approach again. To address the “what to make for dinner” – the problem that these services had solved for us – we made a spreadsheet. Entries included some basic examples like “tacos” but also meals from Blue Apron that we liked (with a link to the recipe), and most of the recipes torn from magazines or printed out that were sitting in a folder. Now we had our repertoire in one place, and we could refer to that rather than thinking oh wow what to make?
We began planning out a week (or part of a week, at least), discussing what to make in front of a computer (or two), opening up the recipes and updating our online grocery list based on what the recipe called for and what we didn’t have on hand. And then putting the URL for the recipe into our shared calendar for the night we planned to prepare it.
Our shared approach to cooking the meal itself of course was no longer tied to Blue Apron or Sun Basket, and we still do that. And we update the spreadsheet with the date we made a particular recipe, and any changes we would make or if we’d even want to have it again.
I subscribed to a bunch of recipe newsletters from the New York Times and elsewhere and have regularly been adding new things to try.
So we took back a significant part of the work – the planning, and the list making, and the shopping, but the results – the meals themselves – are so much better, and the experience is just as enjoyable. Indeed, the work of planning is not a chore, it’s now tied to feelings of excitement, anticipation, challenge, and discovery.
How we eat and how we feed ourselves is enormously personal and a rich vein to mine; it’s why I often ask workshop participants to practice interviewing on this particular topic. My point here is not to offer a recommendation about how you should be managing your meals, it’s to illustrate that the way we experience products and services can be different from what is intended, or what is assumed. That meaning can be a bigger driver of behavior than say “value” or “time saving.” And that perceived benefits or limitations shift and become reframed on a regular basis.
There can be a lot of nuance to uncovering this with your customers. If this sounds like something you’d like to explore, get in touch. Supporting my business is the best way you can support this podcast. And if you have any feedback in general, you can email me at DONUTS AT PORTIGAL DOT COM or find me on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
Let’s get to the conversation with Juliette Melton. She is the Director of User Insight and Strategy at the New York Times. Thanks for being on the podcast. I’m glad to get the chance to chat with you.
Juliette Melton: Thanks. Yeah, it’s nice to see you again. It’s been a while.
Steve: It has. Why don’t we start with just an introduction?
Juliette: Great. Hi. My name is Julie Melton. I’m Juliette Melton on the Internet and yeah, I’m a design researcher here at the New York Times. My title is officially the Director of User Insight and Strategy. We have in some ways moved away from the design research label, which I think is also something to maybe talk about today. But, yeah, we’re here at the New York Times. We’re sitting in a conference room just off of the newsroom where the news gets made.
Steve: Does news get made?
Juliette: Well, the news gets typed.
Steve: Can you unpack your title a little bit? Maybe that will feed into some of the themes you’re already teasing.
Juliette: Sure. So, we purposely changed the name of our group a while back and to talk about user insight and strategy. It becomes a little more broad and a little bit more impactful than I think just calling it design research because I think – in the IDEO context, design becomes a very large word. In many other places design is a little bit smaller and speaks really towards a specific function within product. And I think it’s important that research can transcend the product team because we can offer much more in an organization and we can do much more in an organization. So, by calling it user insight and strategy it puts the focus on the people who we’re actually understanding and not the end result and I think that that’s really powerful as a reminder to us and to the organization about what we’re here for.
Steve: I’m going to just back up on part of that. What does product mean in a media organization?
Juliette: That’s a good question. So, here, it refers to essentially the websites, the apps, the newsletters, all of these digital expressions of what we’re doing here. And there is some church and state separation here between what happens on the news side and on the product side. So, within the newsroom there are people who are interactive news producers, and they have a very different job than the people who are building the apps and the websites and the newsletters. It’s essentially a separate kind of function. We think about engagement with the news and how do we package, present, share everything that we’re going. And we’ll talk a bit more about it, but my current role is not really news focused. I worked on other stuff here. So, my group is called New Products and Ventures. I’m the Director of Research for these other businesses that the Times has been building. So, that includes NYT Cooking, Crosswords and we’re very proud to talk about Parenting which is a new digital product which was just released a couple of weeks ago. So, these are separate businesses. These are places where the Times is hoping to expand its audience, expand its reach, do some new things, stretch some new muscles. So, it’s really fun from a research perspective.
Steve: Are there other – without asking you draw the complete sort of corporate org chart, but those are three separate businesses. Are there others? I’m just wondering how big is the set of businesses that comprise kind of the Times universe?
Juliette: Well, that’s kind of hard to answer. There are a lot of different brands here. You know there’s The Daily, The Weekly, which is the new TV series, and then there’s other brands that sort of become part of what we do like Modern Love, for example. And these are all things for the – the organization can spin off into these different entities.
Steve: It’s interesting from the outside. I guess it’s the same with so many different kinds of multiple touchpoints, as we like to talk about businesses, where if you’re consuming a bank, for example, you – what do they say, don’t ship your org chart? So, I guess as a consumer of The Times media I might be aware of a lot of different brands, but that doesn’t necessarily map to – I think part of what you’re saying is they could be structured as businesses, or as – or not, depending on what makes sense internally. It’s invisible to me as a reader.
Juliette: Exactly. And so the things that I’m working on, I think, would be visible as separate businesses because you could subscribe to them separately. And so literally you could just subscribe to NYT Cooking, or you could just subscribe to Crosswords. Parenting doesn’t have a paywall at the moment, but down the road that could be something that you would pay for separately as well. So, yeah, there is some experience differentiation there. As opposed to you’re a reader of Modern Love – that is part of the nytimes.com world. So, that is just part of what you’d get from a Times subscription.
Steve: I’m sorry, can you say the name of the area or the division…
Juliette: That I’m working on?
Steve: Yeah.
Juliette: It’s called New Products and Ventures (NPV).
Steve: Alright. So, some could be products and some could be ventures.
Juliette: It’s sort of a catchall, yeah. I think the interesting thing there is it’s a spot for us to be innovative in a different way. To be innovative around business models as well as what we’re doing from the content side. And I think that that’s been really good for the organization and pretty interesting as a researcher.
Steve: So within sort of nytimes.com you can provide different kinds of content, but it’s always going to be under that business model because that’s the Times subscription model.
Juliette: Right, exactly.
Steve: But if you have a separate piece – like you said, you can subscribe to the Cooking app separately, so you can have that as a separate part of the business. And so that’s a business model shift.
Juliette: Yes. Exactly.
Steve: Okay.
Juliette: I don’t know how interesting this is for people. You know one shout out here is I’m a big fan of subscription models as a way to exchange value for the web. You know I’ve worked in the past with a couple of other subscription-based services and I think there’s something really honest about it. You know like someone pays you because they get value. Right. It’s just a very direct value exchange which is different from working with an organization that makes money in somewhat – in ways that are somewhat more hidden to the consumer, whether that’s through advertising, or some other kind of data brokerage. Like, yes we do have advertising at the Times, but the subscription is really like the core of the value exchange. And yes, I just like it from like an ethical perspective. I think that it’s a really like nice way to be doing business. But then also it’s fun to be working on these other smaller businesses because you can really measure success. You can measure outcomes because people are paying you. And so as a researcher it’s just really satisfying to know, okay, we did this thing and then this resulted in this much revenue. And just that direction connection I think is really – it’s good feedback.
Steve: So, as a researcher how are you using research to support the organization?
Juliette: Well it comes at a lot of different moments in the lifecycle of one of these products. For example, I did work last year with our strategy department to understand a new vertical and whether or not there was opportunity for us in that space. Looking at that from the market size perspective, from user needs, from how much brand flexibility we had, we decided eventually not to do it, but that’s a case where research can come in on a more very, very early stage, very strategic level. As you move through the phases of the design process we can be involved in very early stages of discovery. I’d mentioned briefly about the work that I’d done in Canada. I spent several years here working with our global expansion team, understanding, around the world, what might we do to expand our reader base? And that includes everything from payment models to partnerships to hiring additional people in different countries and building our new bureaus. And so it had a lot of different kinds of ramifications around what we’re doing as an organization to meet these needs. So, that all kind of starts to feel like the beginning of a process. And that’s where a lot of the work I did at IDEO, for example, comes into play – different thinking, understanding people’s needs from more of an opportunity space perspective. What might we do in this space answering some of these bigger questions? But then, you know, we’re building and we’re shipping and so the questions continue to get a little bit more specific as you go. So, it’s, you know – how exactly do we meet this need? Right. What do we do for this audience? So, for Parenting the team learned that there was this gap in – actually I don’t want to talk about that – scratch that. To speak in broader terms, because I don’t want to get too specific for the purposes of going public here, it’s about locating an opportunity and then designing for that opportunity and then along the way checking to make sure that you are designing that properly. And so that’s basically the cycle of generative to evaluative research and our team really does all of that.
Steve: In some organizations the narrative of research is about – you hear phrases like buy-in or advocacy or evangelism where it’s a newer process that is trying to support or integrate in things that the company is doing. So, now I’ve set up a very leading question. For you, what’s sort of the appetite or understanding or kind of engagement with these different areas of the organization that you have found here?
Juliette: I think that advocacy is always there. I mean I think that that doesn’t go away. I was on a panel a few weeks ago and someone came up to me afterwards – you know because we’d been talking about advocacy on the panel – and she said when does that stop? When can you not have to do that anymore? And I said like I’ve been doing this for 20 years and – and he’s like 18 – it’s like that doesn’t stop. You’re always explaining what you’re doing and why and what the impact is, what the outcome is. But I think overall this is a place that really gets it. We’ve been doing a really good job for a long time now and I can say that as someone who came in just 5 years ago. It’s like predating me I think the research team was really strong and so we just – we have a really good track record here and this is an organization that, you know, we seek to find the truth, right. And I think that that permeates the culture here. People are very curious. They really want to know what’s going on in the world. And you know when I’ve worked with journalists it’s been amazing. You know, our methods are sometimes different and the outcomes have different purposes. But so much of it is so similar to journalism that they get it right away and love it. So, it’s been very, very easy to build advocates in the newsroom because of that. And then on the product side, once you start working with a team and they get it, they just want more and more research. And I think that that ends up being more of the challenge here than people not wanting research. It’s just like bandwidth constraints. Like there’s only so much that we can do as a team. And so for us the challenge is more working with the different teams to figure out where our work can be the most impactful and where we’re going to be the most useful.
Steve: What kind of things might someone look at to assess where they’re going to have the most impact? Or to make priorities?
Juliette: One thing I always ask is – like let’s just play this out. What if we heard “X”? What if we heard “Y”? What do you think we would do about that? What could we do about that? Because sometimes what I’ve found is that people want to do research because they’re curious, but it doesn’t necessarily mean that there’s flexibility in terms of what they will do. And if they say look, we’re going to do this regardless, then I’ll say, that’s great, I don’t think this is a place where research is going to be that helpful. So, making sure that there’s bandwidth for that team to do something about it and the will to change course based on learnings. And if those aren’t in place then it’s probably not the best use of research resources.
Steve: If bandwidth was unlimited would you have a different perspective on people that start off with sort of an intractable request?
Juliette: You know, honestly, probably not because I want for research to be seen as a strategic function that leads to better outcomes, that leads to teams that feel more secure and more enlightened. And if it were something that you could just sort of sprinkle everywhere, I think the power of it might become a little more diffuse and I want for it to be like a powerful, useful tool and not just like an oh I wonder about this or that, you know.
Steve: Yeah. So, there’s a scarcity in value – that’s not really what you’re saying. I think I’m changing what you’re saying and reflecting it back to you.
Juliette: That said, I would love if we hired 10 more researchers, yes. I think that would be great.
Steve: But that still wouldn’t give you the bandwidth though…
Juliette: To answer everything for everyone? No, it would not.
Steve: What’s the size of the team right now?
Juliette: Well, so the overall insights team – I think it’s interesting here because we really encompass what would be considered market research, what would be considered product or UX research? We all sit under the same group which I think is a very positive, good thing. I’ve worked with a bunch of organizations that have separate market research and UX research functions and that just gets really messy, right, because then it’s like who owns the truth? Who owns the insights? And you don’t want to be at odds with each other. I think that when you fly in formation it just like makes so much more sense and there are very powerful research methods that have come from both the market research side and from design research/UX research. So, being able to play in all of those sandboxes, however you need to to address your clients’ needs is pretty fantastic. So, that said, yes, our research – our overall research team encompasses all those different functions. The group that I sit in we do more on the qualitative side and probably more on the product side and there’s 5 of us. And then I’m going to kind of spitball here, I think there’s probably about 15-20 altogether and, like I said, that encompasses all these different insights functions.
Steve: So, when you say 10 more – it would be great if you got 10 more people, I just wonder sort of the context of what does that mean? It’s about a 50% overall increase.
Juliette: Yeah. I think that would be lovely, but here it’s the booming newspaper industry.
Steve: So I hear.
Juliette: It’s a good time to write things on paper and sell them.
Steve: So, you’re talking a little, maybe directly or indirectly, about where – you said you work a lot with product teams, but I think you sort of said in one way or another that there’s a potential here, or just an opportunity to work across the organization in a broader way. And I don’t know if I’m putting words in your mouth or channeling something that you said, but…
Juliette: You’re channeling. You’re doing a great job, Steve.
Steve: Good feedback, thank you. You’re doing a great job as well. Good.
Juliette: Um, yeah, no I think that’s right and the work – in the same way that we can play in the different sandboxes of market research and product research, you know I work, particularly with new products and ventures, I work with the marketing teams and the designers and the product managers. So, I work with everyone, right, because from a user perspective it’s one experience. It’s like how do you relate to this organization, whether through that they tell you, what they show you, all of that is a singular experience. And so being able to be more holistic about that I think is really powerful.
Steve: I wonder sort of what parts, and I don’t mean your organization, but just organizations in general that have research kind of lurking somewhere, maybe tied to product, but I wonder what sort of – what’s the next big thing that research will be doing beyond sort of where we’re seeing right now? I don’t know. And again, I’m not asking you to speculate about your own organization, but you’ve seen a lot of different things. Do you have a – what should we be doing?
Juliette: Here’s my hunch. And this is also an equivalent about me, I’ve been doing coaching training and have been doing some personal life coaching, a few clients, to build those skills because I think that working inside of organizations, working with people here, is something that is very natural for researchers to do and I think for us to build these coaching capabilities is really helpful. You know, understanding the people who are around us, what are your unmet needs? What is this really about? What are we doing here? Asking and answering some of these deeper questions. It doesn’t have to be just about the consumers of our products. It can also be about what we’re doing inside an organization. So yeah, I’m really inspired by that and that’s why I’ve been building my own coaching muscles and that’s been super fun. And I would love to see down the road how coaching and research start to come together as – you know does that become something that looks more like org design? I don’t know. So, that’s a place that I could really see some benefit down the road and it’s been interesting to see other researchers or product people move more into coaching and therapy roles. You know, things like that, grounding the human experience in a somewhat different way.
Steve: Yeah. And I’ve definitely seen. I mean I’ll just confirm my N of one. Like I’ve seen this as well, just with people that I know. I want to ask a couple of clarifications here. Is this something that you’re doing on your own versus as part of the role you have here?
Juliette: It’s both honestly. So, it’s partly like my own passion on my own time. The non-working time. But it’s partly something that I do bring into work as well, working with different people on my team and working with people on other teams, doing kind of informal internal coaching here. And so that’s been super rewarding, but I would love to make that more official. And I do feel like as researchers that’s something that becomes natural for us and could be more codified and more supported.
Steve: I just in the last couple of weeks have had a couple of informal conversations with people about is that something that I do. But then, just in a self-serving manner I want to ask if you can put a little definition around coaching. It’s like a term that we all think we know what it means and I’m honestly not sure I know what that means?
Juliette: Well, it can mean a lot of different things. And when people ask me, well what kind of coach – I mean it’s kind of a life coach, but I hate that term, right. So, I think that we need better terminology around it for one thing. But how I see it is it’s about people understanding where they are now, where they want to go and how they might get there. And it overlaps quite a bit with more kind of standard therapeutic approaches which tends to be more about where I came from and what does that mean about where I am. I think everyone probably needs both a coach and a therapist. They’re very complimentary. But I think particularly in organizations, to understand where I am now, where do we want to go and how we might get there? I think researchers are set up really well to be able to ask some smart questions to help people develop this understanding of their own and help support them through that process.
Steve: So in research there’s an interesting tension and I don’t – we’ll see if it’s true for coaching as well, about – put it this way, are we in the recommendation business, or not? Do you know what I mean?
Juliette: I know exactly what you mean and no, we’re not. I strongly believe that? I think that our role is to unpack relevant truths so that teams can collectively make better decisions and yeah, I think informally, sure. Like I can be part of those discussions and kind of help guide them in some way. But I don’t think that research is about recommendations. And the same way, coaching is not. I think I have a bad habit of giving advice when it’s not asked for and I do hope that by going through the coaching training that I have I do less of that, right. I want to be able to know when to ask the right questions so that people develop their own insights, their own understanding. Because that’s what sticks. That’s what works. And I don’t know – if I were your coach like I wouldn’t – I don’t know, what could I tell you about what to do with your life? Like you know your life. I don’t know your life. But I could ask you questions that could help you unpack for yourself what’s relevant for yourself, in a way that it’s kind of hard to do without someone else there with you.
Steve: So, I think this is true for researchers and I think for coaches as well, I think that the choice of our language can change the frame. I think right, you’re probably not going to say to me, Steve, you should do this. But you might – and I don’t know how this applies to coaching, but I think in research we might say other people in analogous situations have done this, right. Here’s a benchmark or here’s a different case. Or we might say when I was in a similar situation I made these choices. Or, if I was in this situation myself I would make these choices. I don’t know…
Juliette: As a coach I wouldn’t say any of that. No. If someone flat out asked me, like have you been in this situation before, then I would sort of gauge whether or not that would be appropriate at the moment. But no, I wouldn’t think I would do that. And even with analogous research, I think it would be presented more as like a hypothesis. Like it seems that this other way of solving the thing could be useful for us. Like what do you unpack from this? What do you see here? And I could offer this is what I see here, but I wouldn’t say this is what this means and this is what this means for you.
Steve: Right. I think – and I hope you are – we’re just kind of bouncing between these two things which we’ve set up as analogous here, between the researcher and their client and the coach and their client. I hear from researchers that if they don’t sort of provide more recommendation or advice then they lose out on attribution which is not about ego, it’s about – it’s about advocacy. It goes back to that advocacy thing. If you’re not seen in a tangible way as the work that you’ve done has led to these changes, right – I mean there’s something about helping people come – maybe you come up with the idea, but you help them feel like they came up with it. And then it’s going to go. It’s not going to go if you tell them – I think that’s also the limitation of advice. But then it introduces this attribution and sort of impacts advocacy and so on.
Juliette: I think that’s a perennial question for us, you know. And I want for my coaching clients, and the teams I work with, to feel like – like they own their insights. I want for them to feel like they’ve been part of a process that they have directly contributed to that has resulted in insights coming forth that help them make better choices. And I might have had in my head that that’s what they should do anyway, but like – yeah, if I told them that, like it’s different, right. You want for people to come to their own understanding. And yeah, you lose credit. And it’s something that on the research team we talk about. You know like what is – like you have to be okay with some ego loss there, right. Like research is not the thing to go into if you want like the billboard with your faced on it, right? Like ideally the things that we learn become so embedded in the DNA of how an organization works that it’s not really attributable to a person, even if you were the one who made that happen. But yeah, that becomes tricky when you’re wanting to advocate for what you’ve done, right, and ideally you just have really good working relationships with the people around you and they’re kind of aware of your magic powers. Like, when this person is in the room smart, good things happen. And I feel like I make better choices when this person’s around, right. And ideally that’s what happens.
Steve: So, that’s attribution about the relationship or about the collaboration.
Juliette: Right.
Steve: But not attribution for we took action “X” because of input “Q”.
Juliette: Sometimes it is that direct, right. I mean sometimes coming out of research, like we’ll have a very clear understanding of what to do next and then we do it and it succeeds and you can like trace it very cleanly and that’s always very satisfying when that happens. Other times it’s more of a slow burn. You know it’s like you hear something in research and people start to internalize it and then they kind of start to believe it and then they start to act on it and then it becomes like a new thing and I could point to several different mindset shifts that have happened here that originated in research, that I think that – that I know that’s what happened, but I think the folklore in the organization would just sort of describe it as oh now we think this, now we believe this.
Steve: Right. So, there’s something about taking your ego out of it.
Juliette: Yeah.
Steve: I’ve complained about this a couple of times, a series of tweets I think that were from a talk a year or so ago, that I think is not an unusual sentiment, where they were saying my research doesn’t have value unless somebody takes action. And yeah, that got me upset because I think it’s a really high bar and it sets us up to – it’s a statement of our value, or our lack of value.
Juliette: That sets us up to do very small projects, right.
Steve: So that you’re always pointing towards.
Juliette: Yeah – I mean, yes, if you strictly do product based evaluative work then you could say that yeah my research always results in something because that something is like moving this button or moving – I mean like that’s something, right, but when you’re talking about change which is more systemic, which is on a deeper level, that takes time, you know, and that kind of research – those research outcomes, like I said, like it takes a while to be embedded in what you see from an organization. So, yeah, I mean there definitely have been some significant projects here which would I say two months later, like yes, someone you know moved this thing on a roadmap because of it? No. But I can say three years later we’re a different organization because of it. And so I think that having some patience and some – I don’t want to call it maturity, but it’s just kind of like being able to see a bigger picture from both a timeline and a strategy perspective, you can really take on bigger challenges and see the impact of those within an organization.
Steve: I wonder if people that – I mean I’m self-employed so I’m – these things manifest differently for me, but for people that are – where their promotion or their title, or just their sort of general success is tied to some of these things. I mean – your point about patience – so, I think there’s a couple of levels. One is sort of how do I feel about what I’m doing? Right, am I alright? Am I confident, or am I learning or am I growing? And am I successful in kind of a corporate structure which can only measure, reward and acknowledge certain kinds of things? And so yeah, I think I could encourage the former and I have had to learn that very much myself. Like things you think are going to be the rewards for doing this work are often other ones which maybe you have to go hunting for to see that they’ve even happened.
Juliette: Yeah. And I think that’s part of why, as researchers, again, like we don’t want to just stay embedded just in product teams. Right, like we need to know executives. We need to know people across the organization, people leading an organization so that we can understand the big problems facing an organization and they can understand what we’re doing about it, right? And I think that sometimes that just comes from relationships and being able to kind of get a read on what’s going on. But yeah, it’s hard. It’s hard. You know when we can put a dollar value on it that’s amazing, but it’s hard to do that.
Steve: You know you speak in a fairly optimistic way, I think, about not even just the potential for research – potential, so that says it’s kind of in the future. I mean it’s the value that research is bringing to – in the organizational mindset, the belief structure, as well as the products and services. You know, when you look outside your own organization, you know, between optimistic and pessimistic, what’s your assessments? Maybe it’s not that axis, but what’s your assessment of kind of what’s going on?
Juliette: You know I – the design research Google group that a lot of us are on, it’s grown from two people who started it. Now it’s like 1,000 something people. That’s a lot. You know, like our funny little profession, like we keep growing and people keep finding value in it and it’s something which is in many ways new to the world. We’re building it as we go and that creates its own challenges because it’s hard to know what does a 40 year career look like in design research. Like we haven’t really done that yet. Right. So, it’s not like being a doctor or a lawyer where you know like okay, at this point in my career I can expect to be partner. You know, so it’s less secure than that, in that way, but it’s also what’s fun about it, right. I mean like we are co-creating this profession and so like why not be optimistic? Right, I mean like our job is to make things better and I think that we do that even within our communities. And part of why I’ve stuck with research for so long is that it’s such a nice community. Researchers are nice people and they’re people who I like spending time with. So, yeah, I think that bodes well for the future of the community and future of our impacts.
Steve: So, what does an 18 year career look like?
Juliette: Well, I think I’m making the path as I go, you know. And so far I’ve – you know early on research was one slice of what I did. I was a front-end developer. And then later I did more product and then I focused on research and started on the more evaluative side and then moved to the more generative end. And so I’ve really done all of it as it currently exists. So, you know, I’ve been involved in the web since it looked very, very different than it does now. And so far so good, right. I mean we’re still here. But yeah, as I think about how to move forward, I think we make the path by walking it and I have a hunch that there’s something here about researchers looking inwards at organizations – and again, I don’t know a ton about org design. It may be already kind of covered by a lot of these other practices, but at least for my own life I feel like there’s really something about using the skills and techniques of being a researcher, combined with what you can do as a coach and specific training that you get as a coach, to do something different within organizations. I don’t know what that is yet, but I think there’s something there and so that’s the path that I’m kind of figuring out as we speak.
Steve: That’s exciting. I want to go back a little bit. Because you were just giving a nice overview, I think, of where some of us have been and where the community has been, that if you’re at 18 years and it’s all about walking the path to kind of find the path. There are lots of people sort of coming into the profession, let’s just say in the last three years, that – I mean – I feel like when I started, and maybe the same thing for you when you started, there wasn’t a lot of path.
Juliette: Right.
Steve: And I’m like you. I don’t sort of know where I’m going. I mean following the path is a much more confident way of – I might say stumbling – but, it’s just interesting that the field has grown and you just look at – there’s just a number of people and the time. And so somebody that sort of chose research, or was chosen by research in 2017 is – they’re part of a different community than you were 18 years ago. So, what it means for them to sort of learn. There’s, there’s – the terrain is different because it’s populated by people like you or there’s a path that’s woven by people like you. I’m going to take your lovely metaphor and just…
Juliette: Go for it.
Steve: Mow over it. I find that interesting. Just back to sort of my question about advising, the people who are joining the field are joining a different field than the one that I joined or that you joined.
Juliette: I mean it wasn’t a field at that point, right. I mean like there were a few people who were doing some great work, but it wasn’t really a community, at least as far as I was aware of. So, I think people joining now – I think it’s very cool that there is a field for them to join and they could hop on this gigantic Google group and ask a question and have 1,000 useful people helping them out. You know for people starting this now, I think they’re in a great position. You know you can actually learn this in college. Like, you couldn’t learn this in college when I was in – at least where I was in college this is not a thing. You know I studied comparative literature in undergrad and I remember telling my comparative mystical literature professor that I was wanting to go into IT, as it was all called back then, and he was like why? Why would you do that? Like just totally confused. Like that doesn’t seem like a realistic or interesting thing to do. It wasn’t really a path. So, yeah, I think it’s great that people are joining now and I think there’s more support. You know in general for all this technical, digital roles there’s been a lot more definition in terms of the different job functions and I think that that makes it a lot easier. I mean back in the day it was about like webmaster and they did everything. There aren’t webmasters anymore, right. I mean like the jobs have become much more specific. And I think that makes it easier to train, to decide like what you want to be doing to become really excellent at what you’ve done. You know, it’s a bit of a tangent, but I think that research as a standalone profession is I think something that’s also becoming more possible to do for – I’m curious your take on this as well, but I feel like the sort of broader field of UX, you know someone is like a UX person who does design and they do research and it’s all kind of rolled up into one thing – I didn’t want to do that. I wanted to be a researcher and so when I made that choice 10/11 years ago to focus on research, it really limited my options. It meant that I could either work for an agency or for a really big company. But for smaller companies they really wanted more of a full service “UX” person. I’m curious if you have seen more growth in smaller organizations hiring specific – like researchers who specifically do that?
Steve: Who are researchers?
Juliette: Just researchers, yeah.
Steve: I mean this is anecdotal, of course, but… I think there are – there seems to be sort of a pattern of maybe the first researcher in an organization is – or maybe person zero is someone who is doing multiple – we have an intern or we’re going to take some of somebody’s time, they’re interested and they’re going to go do this. A product manager maybe who has kind of got the bug, or read something or saw something. But you know when companies start hiring someone with a research title I think sometimes that first person is very junior and then it has to build up to a point where there’s a realization, oh we actually need leadership and mentorship and being able to develop a function and a practice and a team. And I have a lot of concern about that person who is sort of brought in with no mentorship. I think about the person I had coffee with about a year ago who was – I think he was not too long out of graduate school. So, professionally junior, but I think educated a certain amount, and they were the only research person in this organization and they basically were sent to do something where they were told how to go about it and what to produce. And of course, if you don’t know what you’re doing there could be a mismatch. But they took the hit for this and they, I think, ultimately were dismissed because they were unable to produce the information that was needed from the method they were required to use. And we had this long conversation and this person was convinced it was their fault and that they were deficient as a researcher.
Juliette: Oh man.
Steve: Yeah. And to me that was a lack of leadership.
Juliette: Yeah. This company needs coaches and researchers it sounds like.
Steve: Yeah. Some executive coaches or management coaches who say here’s how to manage your staff to success. And that’s I think – because there’s sort of this generic perception of research, or that it’s just a thing and like you’re just talking to people, or if it’s a survey, just go set up a Survey Monkey, then it seems easy to have people that don’t appreciate the craft and the mastery of someone like you that’s been doing this, that chose this, can bring. I mean that’s a really sort of disappointing example, I think, but there’s the researcher who has the title and is not in a situation where they’re going to be able to thrive, but I mean – if your question is like are there researcher title jobs? I think a huge amount.
Juliette: I guess it’s more like is that increasing as a phenomenon because yeah, I don’t know. I’m just curious if as these other roles have become more specific if there’s been a greater appetite for researcher as a standalone function. But yeah, I mean I’ve been that person. I’ve been that junior single researcher and it’s really hard. And that’s why I take heart in some of the larger online communities that have formed because I hope that what we can do for each other is to help advocate for methods and for approaches and to kind of stand in as mentors to people who need that support and don’t have that person there with them. Like I would love for us to keep doing that for each other and I think we do that to an extent and that’s part of why I really advocate for researchers to know other people who do this. And it’s also part of why I like working here. You know I work with a bunch of really, really smart researchers and if I need to pull in someone who is an expert in running a conjoint analysis, I can do that. And it’s nice to have this deep bench of expertise just available here and that’s part of why I really enjoyed working at – not a huge company, but like a big enough company that has a deep bench when it comes to research vs. like me off on my own doing this startup. I feel like I can be a better researcher when there are people around me who I can lean on for their expertise as well.
Steve: So, you’re okay not being able to do a conjoint analysis yourself?
Juliette: Totally. We’re always learning, right, all of us are learning, but we work with people who have PhDs in statistics and like I think that’s great. That’s probably not something that I will do. I think it’s wonderful that we can specialize in different areas and I think that makes us all stronger.
Steve: You know it’s that depth vs. breadth tradeoff. You know you’re sort of asking about research as a specialty, but even that word, to your example, it comes with so many things.
Juliette: Yes.
Steve: I started off trying to position myself as a generalist a long time ago and the more that times go by the more things I realize I know nothing about and personally it’s terrifying. Maybe that’s too dramatic a word, but I sort of have to own it and then also deny it, that there’s this thing that I don’t know how to do and so I’m aware – what does expertise look like? And what does expertise look like for you as you proceed and there are things that you’re going to pull in that are not going to – you may never be a master at conjoint?
Juliette: That’s a great questions because I feel like there are things that I’ve definitely decided to not specialize in, you know, and one of those is prototyping, for example. At one point I did do much more design and now I am totally happy not doing that. I think it’s – knowing what you are okay letting go of I think is a very important professional skill. And I think it’s part of growing, is knowing like this is something that I’m going to bring someone else in for. And, yeah when it comes to live very, very complex data analysis, there was one point in my career where I was really focusing on that and was building a pretty big skillset there. I think actually last time I saw you at IDEO I was really focused on building a very deep quantitative understanding for myself and eventually I decided, no. Like I would rather lean on the person who has the PhD in statistics and I want to refocus on the things that I feel like I can more uniquely bring to the table. And for me that’s more about qualitative research, that’s more about design thinking. Like those are things that I can uniquely do that I think someone else couldn’t necessarily uniquely do and so yeah, I think just understanding what you want to do and what you don’t want to do I think is a really nice part about getting older in this world.
Steve: So, how should someone approach that? That is maybe in the first few years of a professional career.
Juliette: You know, I think the first few years like don’t go around saying no to stuff. You know, say yes. And then follow your intuition there. But yeah, I’d say probably don’t get too – I think in general in life, like be cautious about saying no to things that you don’t really understand. Like maybe kind of dig into it a little bit because there might be something in there for you that you hadn’t expected.
Steve: Do you, like let’s just imagine the magic wand thing. You were sort of saying that if you could hire 10 researchers you wouldn’t be opposed to that.
Juliette: Right.
Steve: So, if we were to wave the magic wand and make that happen, as you would start talking to people, what kinds of things would you – I realize that we don’t have a job description – whatever – I mean what do you start to look for when you talk to people about working with them?
Juliette: Um, I like working with people who – the first thing I would say is that it’s important to hire people who bring a good energy into a room and I know this sounds like super – I did live in California for a long time, right – but I think that researchers can bring a kind of brightness into a space and a kind of optimism for a team and a sense that we can learn these things together. It’s a bit intangible as a quality, but when we bring on new researchers that’s really something we look for. Like is this person someone who is excited about making connections across an organization? Excited to share what we’re doing? You know there’s something about bringing energy into research which I think is really important, but beyond that, yeah, I mean being able to really, really listen. Being a thoughtful, humble listener to what people are saying and I can sometimes know in the first conversation with someone whether or not I would encourage them to move ahead as a researcher based on their qualities as a listener. And then being able to think thoughtfully, strategically, big picture and being able to execute. It’s hard to – it’s so hard to hire, right, because these are all kind of a little bit intangible as qualities, but yeah, looking for someone with a curious mind who truly likes people is really important. I think that’s key.
Steve: When you say brings energy into the room and is a humble listener, part of me – a part of me feels like well is that contradictory? I’m really just thinking about myself. The one is sort of an extroverted energy and one is an introverted energy, but I don’t know.
Juliette: I’ve found that most of the researchers I’ve ever worked with are true introverts and I would put myself in that same category, but we are – we are introverts who like people, you know, and I think that almost all of the researchers I’ve known fall into that category.
Steve: Maybe this is a personal issue, but why not. The bringing the energy into the room thing is interesting to me because I’m probably more of a listen first person. I mean I pride myself on being a good listener, but that also means I sort of – I watch a lot and, you know, depending on how people come across me they might see me sort of in on stage mode where I’m saying a lot and I am telling you stuff and sort of trying to bring something. But I spend most of my time sort of holding back and just waiting for those moments and I think it’s – like I feel intimidated by that description to sort of have to make an impression.
Juliette: But I think what you’re talking about is its own kind of energy, right. So, the energy that you’re bringing into a room is I support you, right. I’m here to listen to what you’re saying and it matters to me, right. And so like that’s a very particular kind of good energy.
Steve: Okay.
Juliette: Right. It doesn’t mean that you’re coming in like you’re running for Mayor, right. But it means that you’re coming into contexts and like in a positive way. You know like you’re not glowering. You’re, you’re – you’re engaging, even if you’re not saying something. And so I think that that’s – does that make sense?
Steve: Yeah. And I think just clarifying words is so important, right, because when you said energy that made me think of extroversion and speaking and being loud and “hey,” and smiling. And you’re right, there’s a lot of different ways to manifest that energy.
Juliette: So, I’m coming in as like a Reiki person and so for me it’s about like what is the thing that you’re emanating? And what is the thing that people are picking up on about you? And what are your intentions when you’re coming into a space?
Steve: Okay. So, you listed a bunch of things that you said were at varying levels of intangibility. But what are some successful ways that people could present those qualities? Demonstrate them? Whether it’s in a sort of interviewing context, or like a job interviewing context or any other?
Juliette: Yeah, I think being a thoughtful communicator is really important. And so when you’re communicating with a potential employer just paying attention to what they’re asking you for and really showing up for those conversations. I think that that’s a big part of it, right. And being able to share examples of work that you’ve done, that you’re proud of, and being able to speak to experiences that you’ve had in a really cogent way. One of the – we haven’t really talked about this, but I think one of the biggest skills that a researcher brings is a synthesis. I mean being able to look at a set of disparate facts or impressions and pulling something meaningful out of that. And so I think if you’re – if you were in an interview to be a researcher somewhere I think that you’d want to be able to speak to work you’ve done in a way that makes sense, that’s to the point, and that communicates what you want to communicate.
Steve: So, can we maybe switch sort of the who’s communicating to who scenario. So, we’re talking about people at a certain stage that are kind of approaching organizations. And then earlier on we talked about advocacy. You eluded to being on a panel about advocacy. What are things that researchers can manifest, display, whatever the verb is, for the people that are – that we want to get excited about research, that we want to sort of demonstrate the value. What should we be doing or saying or showing them?
Juliette: This gets back to our conversation on impact. Um, which again as we talked about is tricky with research because sometimes that impact is direct and sometimes it’s more indirect. But I would love for someone to be able to speak to that. You know and they could say like look we did this project and there wasn’t a direct outcome from it because there wasn’t space on the roadmap and there wasn’t budget and there wasn’t executive buy-in, but I was able to keep bringing this into the organization in ways that eventually it became part of what we did and here’s an example of it. So, you know like that’s a story that I would love to hear from someone who wanted to be a researcher because it shows that you have an awareness of the nature of reality, but that you’re able to effectively operate within complex interpersonal spaces.
Steve: So, let me change the players in this dialogue that we’re playing around with as well. So, that’s kind of a researcher to researcher thing, but now maybe it’s you, or it’s future you, and you’re talking to someone that is interested in, or has some curiosity about, or maybe some skepticism about how what you do can bring value to their part of the organization. How do you advocate for research for them? What do you highlight?
Juliette: You know it’s interesting because I feel like I haven’t had to do that for a while. You know we haven’t had to do that here because people know about success and they want some of that. So, if you can do a great job with one team at an organization other people will know about that. And so, I think being able to effectively do one good thing within an organization will help you work across the organization. Does that answer your question?
Steve: Yeah. I mean the best way to have that conversation is to preempt the necessity for that conversation by doing work that tells the story itself. I’m just saying what you said, but I said it faster.
Juliette: Yes. Thank you for synthesizing that.
Steve: That’s podcast…just repeat it back faster.
Juliette: You’re doing a good job, Steve.
Steve: Thank you. You’re doing a great job as well. So, we’re jumping around in chronology here a little bit. I want to go back to one thing that you said. Maybe two things. What did you say to your professor when they were like huh, IT, why would you want to do that?
Juliette: That’s a good question. I think this was in like 1998. It was a long time ago.
Steve: And you don’t remember exactly what it was?
Juliette: No, I don’t. But I remember – I don’t know, I wasn’t defensive. I was just kind of like. I don’t know. I think honestly I felt kind of proud that I was doing something strange and it was like I’m an iconoclast. I’m doing my own weird thing here. So yeah, I think I was probably like – I was like a nice kid. I was probably like oh yeah, I don’t know, that’s interesting. But I think on the inside I was like yeah, I’m a rebel.
Steve: Yeah. Right. Sometimes that external perspective – you know if you’re in that sort of bubble, or that identity, that perspective that you’re breaking out of is like – you may not have felt that way until someone told you that you were rebelling.
Juliette: Yeah. I mean especially at the time. It was like the Internet was just like new and weird and juicy and I was like there is something going on over there, like I want to get into that, whatever that looks like. That is strange and that is good.
Steve: And another moment, I don’t know if you can recall any details, but you talked about sort of starting off in front end development, that eventually you got into research. And I’m wondering was there a moment or series of moments where you saw that it was a thing, or that it kind of sparked for you that – do you remember what you were doing or what you were thinking when you started to feel pulled towards it?
Juliette: Yeah. You know when I was doing front-end development, and this was back in the day when like, you know web standards were just really hitting and we basically tore apart the web and rebuilt it in a different way and it was a really like heady time, you know. Starting to use CSS in ways that hadn’t been used before. I mean the underpinnings of the web just within a couple of years like started looking very, very different. And so that was just kind of an exciting time to be doing that particular work. But then eventually I just got good at it and it was like someone could hand me a mockup of something and I could build it. Like, okay. But – I do remember a couple of times where I was building something and I thought that’s a bad idea, why are we doing that? And I realized that I just wanted a different seat at the table than that. And so that’s when I moved more into the product role which at the time I think was a little less exciting than product now. I was working for an organization that had a very long term perspective on things and we were doing things like requirements documents that would take several months to do and then a development cycle that would take several months. And it would like be a year all in, right. And so I would just be writing these like tomes and again eventually it was like, yeah, I want a different kind of seat. And I remember a moment – this was also when I was considering a move to California – was if I wanted to go more into interaction design or wanted to go more into research and could sort of see different pathways. And eventually did more on like the project/product end and then while I was at one company really pulled the research threads out of that and created a new role for myself around research. And that’s when I was like I am a researcher. So, I really built that role for myself out of more of like a project/product manager role.
Steve: Maybe it’s different – it’s definitely different now than then, but any thoughts about – like you just declared research as an identity in that part of the story and I think that’s an interesting one. We don’t talk about the identity part of it enough. We talk about sort of skills, or are you a designer, or maybe you’ve heard the phrase people who do research? I’m not even sure what my question is here. Do you have a perspective on – if I throw out the idea of research as an identity does that spark anything for you?
Juliette: If you’re familiar with Richard Scarry books, Busytown, there is like the spoof of Busytown with like the little internet creatures and there was the like innovation strategist and she was building a 2×2 framework covered in Post-its and she was wearing like a printed anthropology dress that looks basically exactly like what I’m wearing now. And I was like oh, like that’s uncanny. Maybe we do have an identity. Or like maybe there is something to this that we’ve kind of like co-created as a thing that we’re doing. So, yeah, that was the first moment where I thought huh, maybe there’s a researcher identity that’s happening. But yeah. I think we’re not as cohesive, partly because I think we’re not – we’re not that stylish. Right? I feel like I can walk into a group of people and they’re like okay those are the designers, those are the exactos. I don’t know, maybe the researchers are the ones who are like very earnestly listening and kind of like a little disheveled. Maybe those are the researchers.
Steve: Right. And you’re framing identity as an external thing which is interesting. I thought more as an internal thing, but you’re kind of saying that there’s an aesthetic maybe of certain practices and that if researcher identity is observed more by behavior than that puts it into a different space than like a designer identity which might be more about…
Juliette: Steve, you’re really good at restating things in a more coherent way. Thank you for doing that.
Steve: Thanks. You brought up Richard Scarry which I think is a very good thing as well. So, thank you for doing that.
Juliette: I’m the parent of a toddler, so, you know here we are.
Steve: Yes. Write what you know. Okay. I feel like if we’ve hit Richard Scarry we might have hit – in fact Twitter parodies of Richard Scarry might be sort of peak episode. Anything else that you think we should talk about today that I didn’t ask about or you want to get to?
Juliette: Just that I’m excited to see where we go with this.
Steve: This episode?
Juliette: No, no, with research, as researchers.
Steve: Where the community goes? Where the practice goes?
Juliette: Yeah. Like, ‘cuz no one knows. We’re co-creating this thing and I guess I would say to myself, and everyone else, like take heart. Like we will keep building this thing together and it’s going to be good and just because there’s not like a traditional career path doesn’t mean you’re not going to have like awesome things you’ll keep working on and we’ll figure it out as we go.
Steve: Do you have hopes or dreams? We don’t know, you don’t know – that’s fine – but is there a place you can fantasize about it going?
Juliette: I think what I was saying about research having like a more both strategic and emotional impact within organizations that transcend making choices about how we serve customers. I think that’s a thing that we can do.
Steve: And maybe I’ll just add to that. Maybe there’s an element where that also is explicit instead of tacit and then that wraps up some of the challenges we were talking about.
Juliette: Yes. And in fact, a plug for NYT Parenting, we’ll soon be doing a series on emotional labor. So, if you think as researchers, having a lot of the work we do become acknowledged and codified I think will be very helpful. And it’s not just the tacit, emotional labor that we currently do.
Steve: Excellent. Well, this has been a fabulous conversation. I really appreciate the chance to speak with you.
Juliette: Thank you. Thank you for coming to New York.
Steve: Okay that’s a wrap! You can find Dollars to Donuts on Spotify and Google Play and of course Apple Podcasts. On the web at Portigal dot com slash podcast you will find all the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 25. Juliette Melton of The New York Times first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Juliette Melton, Director of User Insight and Strategy at The New York Times. We talk about updating the old “design research” label, user research in a journalism culture, and the role of coaching. I think that researchers can bring a kind of brightness into a […]
The post 25. Juliette Melton of The New York Times first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my conversation with Ashley Graham, a design research leader at IBM. We discuss synthesis as a collaborative, co-located activity, being mission-driven, and building a process that addresses complexity.
When I look at the wonderful research community, I don’t see a ton of people that look like me and so even by talking to you today I have a hope that we’re growing and that we’ll continue to see more diverse faces, diverse ways of thinking and diverse backgrounds represented in the field. – Ashley Graham
Show Links* All Those Books You’ve Bought but Haven’t Read? There’s a Word for That * My Father’s Stack of Books * Nextdoor * Freecycle * The Year’s Best Science Fiction * The Best American series * Goodreads * Steve on Goodreads * Ashley on Twitter * IBM Design * Phil Gilbert * Ginni Rometty * Sarah Brooks * Transdisciplinary Design * Howard University – Architecture * General Assembly
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal:Hi, and here we are with another episode of Dollars to Donuts, the podcast where I talk to the people who are leading user research in their organization. In 2018, Kevin Mims wrote in the New York Times about the Japanese word tsundoku – a stack of unread books. In a New Yorker article entitled My Father’s Stack of Books, Kathryn Schulz reflects on what her family referred to as The Stack, the books that accumulated in her parent’s bedroom, especially on her father’s side of the bed. These weren’t just books to be read, but also books that were recently read that should be kept near at hand. She estimated that The Stack contained 3-400 books.
For me, I switched a few years ago to getting rid of most books, passing them onto someone who might like them, or giving them away in the community via Nextdoor or Freecycle. I felt like hanging onto every book was becoming increasingly unmanageable, and in some ways was creating a barrier to acquiring – and thus reading new books. My tsundoku serves as a last-in-first-out queue, but for me unread books go in the bedroom and books that you want to keep should be displayed on bookshelves.
I was a voracious reader of books as a kid, and at this point in my life, it’s something I need to make a deliberate effort towards. I read on the Internet all day; I read several magazines regularly. I read a print newspaper every dat. Plus I’m trying to watch a ridiculous number of television shows and movies on all the platforms. And oh yeah, podcasts, right?
As a kid, I really got into science fiction and especially sci-fi short stories. At a certain point I set that genre aside but maybe 15 years ago I came across a phone-book sized annual collection of sci fi short stories. And then the annual best American short stories series. These books have got built-in portion control – read at least one story in bed, before going to sleep. But then I’d find myself in a bookstore staring at the shelves without any clue about which ones in the series I’d read. And even if I was home, if I’d given the books away after finishing them, I couldn’t just go check my shelves to avoid repurchasing something I’d read. That’s when I found Goodreads – a website and an app where I can organize the books I own but haven’t read, the books I have read, and the books I don’t own but want to read.
There’s a whole set of other things you can do on Goodreads. You can connect with other people and get their updates in your feed. You can post progress updates as you go through a book, and you can write and read book reviews. But for me, my primary motivation was to have a single location, available to me anywhere, to see what I had already read.
My Goodreads usage stayed casual and intermittent for a couple of years. But when my local library opened a brand new building, I went in and renewed my library card and that triggered a deliberate and focused shift to reading more books. I think the stack beside the bed has not decreased much in size, but I’m fine with that. I’m making use of the library website to put books on hold, maybe it’s a book I learn about from Twitter or an article in the newspaper. I’ve also been saving books on the library website that I’ll want to put on hold later. I’ve been getting books from other libraries in California. I’ve been reading graphic novels and regular novels, both in print and on my iPad. One of the cool things about the library is that borrowing a book means that I have a deadline. It’s not just that the book is expected back, it’s that the deadline makes reading the book into a tangible accomplishment. It’s due back on a certain date, and I finished it, and I returned it. I mean, I’m also free not to finish the book, but even so, that’s something I can metaphorically cross of a list. And then this is where Goodreads comes back in, because even though I haven’t connected with many people and I don’t care too much who sees my updates, going to the site and marking the book as “read” with today’s date is another marker of closure. I am definitely doing the reading for the reading, but there are these additional rewards, other bits of satisfaction, I guess we call this gamification if that’s even still a thing. I might write a tiny review, occasionally I’ve asked a question when I didn’t understand something, but mostly it’s about making that mark when it’s done and then feeling a sense of pride or accomplishment from the accruing list of books that I’ve read. This public-ish list announces something about me and what I value, and I enjoy building that even if it’s just for me to look at and reflect on. Taking a book off the stack has a tangible satisfaction, but I find it a bit more diffuse, to move an actual book from a stack to a shelf, than to do the analogous operation on an abstracted book in a digital interface. Go figure.
But this is just how I use Goodreads. I’m sure it’s hardly unique, but it’s also one of many different ways that people could and probably do use the site. I mean, I don’t have any idea what other usage models are. But of course, we have processes and tools for finding out! And this isn’t just about Goodreads but for any product or service that has many features, different sections, different experiences, it’s essential to try to understand how people are using your product. The risk is in only considering the product as a set of separate features. Who is using this particular feature, and how? And for another feature, how is that being used, and by whom? If you are working on a product that has this many facets, you are better served in learning about the bigger picture that may weave its way through different features, as well as other tools or products that support the underlying goal. If you want to improve, or optimize, or extend the capability you are offering, you’ll want to do so based on an understanding of those goals, not just feature by feature. That means you need to think about how your teams are organized, and sometimes the way you’d organize a software team to build different features isn’t the same as how you should organize designers to design those features, and almost certainly not how you should organize researchers to inform the different decisions you’re making across features.
Of course, this is the kind of thing I can help with. Often I work with clients to help them get a handle on what their customers want to accomplish, and look at how the team can focus product and design decisions to best support the people that use their product. The teams I work with come from a variety of industries and have varying levels of experience in learning about their customers and acting on those insights. I see this podcast as an extension of that work, something that I’m able to share with you. And so, the best way to support this podcast is to support my business. Hire me to lead user research projects or to coach your own team as you talk to users. Also I run in-house training workshops to teach people how to get better at fieldwork and analysis skills. Get in touch and let’s discuss what we might do together.
I’d also love to know more about how this podcast is helping you in your work. Email me at DONUTS AT PORTIGAL DOT COM or on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
Let’s get to my conversation with Ashley Graham. She is a design research leader focusing on digital, at IBM in New York City. Thanks for being on the podcast.
Ashley Graham:Thank you for having me.
Steve:I have a very, at this point, traditional way of starting which is to ask you to introduce yourself. Who are you? What do you do?
Ashley:So, I’m Ashley Graham. I lead design research for the digital part of IBM. That means we focus on the customer journeys that our clients, or potential clients, are taking. And we bring a mixed methods approach to both qualitative research, quantitative research and generating insights that can help drive our business and drive innovation for our users.
Steve:Can you give a little context about the digital part of IBM? IBM is this huge company, that for those of us outside – like I personally don’t have a good mental model like what this company does even in 2019 and how it’s divided up and sort of where you are and what your efforts are focused on.
Ashley:Yeah. So, as you can imagine, IBM is huge. We have a large breadth and depth of a portfolio. Digital really sits across all of that, right. So, we have more legacy parts of our business. We have newer, innovative parts of our portfolio and digital really seeks to understand across, right, what are the user needs, our client needs, across the portfolio? How can we bring all the parts of IBM together, all our capabilities and expertise and bring that through a digital experience? And so that’s not something that IBM has traditionally done. It’s kind of something new and so our team, the digital part of IBM, really drives best practices around digital experience, both from a design and also a business perspective.
Steve:Can you unpack digital a little bit? Again, I’m just pulling in very old mental models of IBM as kind of back in the legacy days, you know like a hardware company.
Ashley:Yup.
Steve:So, what does digital mean now?
Ashley:That’s a good question and that’s the question we’re answering. IBM has this over 100 year history, right. We’re going into our 108th year this year. And traditionally we’ve been face to face, right. The client comes to the client representative. They tell them about their problems. They tell them about their architecture and any infrastructure needs. And then the client rep goes back to IBM and says okay, what do we have? How can we serve the client and move them forward in a way that serves the people within that business and then the business needs, right? But, if we want to scale, if we want to grow, we want to transform ourselves to really thrive in the 21st century, we have to shift that model, right. We have to think about digital ways of finding new customers, our customers finding us. We have to think about digital ways to self-serve capabilities. And then we have to think about excellent user experience, right, because that’s a critical part of a digital experience is having intuitive ways to find things, get my tasks done and move on to the rest of my day.
Steve:So, how does the idea of journeys, which you kind of mentioned right off the top, what does that mean in that context?
Ashley:Yeah, so that’s a good question. I think – if I think of a B2C company, right, or a consumer company, right, a journey might be me taking a ride, right, to California. How am I going to get there? We could break that down to me discovering you know the best way. Am I going to take a plane, or a train, or a car? What are the different things I need along my journey? How do I get there? At IBM we kind of have to zoom out because the journeys are much more complex than taking a ride as an individual person. We’re talking about leading and helping organizations achieve their key routes to value. So, we like to actually use this metaphor of the United States and the federal highway system. Right. Before the federal highway system was implemented in the U.S. if you were trying to journey across the United States it was quite a journey, right. Each state has their own way of moving people through roads. They have their own signage, own signals and best practices. So, we are really looking to create those interstates – the fewest, fastest interactions that it takes to solve a large enterprise’s needs. And so that means we have to think about the whole spectrum of technologies, people, processes. They’re our client organizations own transformation and we really have to understand a lot of different things in order to design for that. So, a journey really is like an organizational transformation for our clients. It might be helping them transform the way that the people work within the organization. It might be taking them from an on-premise hardware infrastructure to a Cloud. And so it’s a big journey that we’re talking about, at the level of IBM.
Steve:Given as complicated as it is to figure out how you’re going to go from here to California, for example, the complexity of the things that I think you’re talking about at an organizational level, it’s a little mind boggling I think to – there’s just complexity. There’s a lot of complexity.
Ashley:And I think too, when I’m traveling from New York to California I probably have less constraints than an organization that’s invested 40 years in their infrastructure and is looking to like how can they maintain and build on the investments they’ve made over time. So there are a lot of constraints we have to consider, right, within our client organizations. There are a lot of long term needs that we have to consider. And so, from a research perspective, it’s really about putting together the full picture of our users needs and being able to consider and design for that, in sort of a flexible way. And also being able to reflect that we understand those deep needs and then being able to show that we can guide, right, through complex decision making. Through long term projects. And ultimately to help our client organizations arrive at a future state.
Steve:Okay. So, I’m a big enterprise that has some technology infrastructure that’s been around for 40 years and I come to you and the rest of the IBM and say I’m interested in making certain kinds of changes and others that I don’t even know what they are. What kinds of activities do you and the rest of your organization – well, let’s talk about you and the team you work with. What kinds of activities do you undertake to start responding to that?
Ashley:Yeah. So, that’s – our team is really focused on organizing what that experience is like. And so, say I’m going through a Cloud transformation, right. Or the client is journeying to Cloud. That requires a lot of different decisions, right. So, what we do is we – my team looks across IBM and says, okay, what are all the capabilities that we have from a services perspective, right, like consulting. A technology perspective. Maybe IBM comes in and helps guide those decisions over time, or even embeds and helps in some of the architecture of development. But it really – in order for us to serve, right, those transformation needs, we have to bring together a really diverse set of capabilities that IBM really excels in.
Steve:So there’s an internal examination. You’re talking about looking inward and saying well what are the pieces we can put together here? But are you – is there – how are you assessing what their needs are to figure out what pieces you would bring to bear?
Ashley:That’s a great question. So, I think there’s that inside out, there’s the inside out view. What do we have? What capabilities does IBM have from a technology and expertise perspective? But also we have to look outside in. Right. We can’t just rely on our historical knowledge or our internal strategy. So, we do a lot of actual ethnography and talking from – talking to people that maybe haven’t been with us for 40 years. Talking to people that have been with us for 40 years and understanding the patterns of needs. And we kind of talk about it in terms of key needs, or key journeys. Like what are the most important things that are happening in the market, from an IT perspective, or Cloud and cognitive perspective. We bring those in and we kind of match ‘em up – them up to our inside strategy and see okay for 2019 what are the key journeys that we need to focus on and how can we line up our capabilities to serve those? So, it’s kind of this dual lens, right. We’re looking inside out. We’re looking outside in and the synthesis work really is understanding where do we put focus? How do we organize ourselves to be able to deliver on those keys things?
Steve:Right. There’s something I’m grappling with here to find the right kinds of questions because I think you’re talking about like doing research to plan. You’re not talking about doing ethnography to figure out what the solution looks like? You’re doing ethnography to find out what sort of this larger context, to figure out how to take on a much more complex process.
Ashley:Exactly. And really it kind of reminds me of like a systems thinking perspective, right. We have to map the full spectrum of needs – understand right – if you think of a service design model, what’s happening off stage, what’s happening front stage, as users interact with us, and then what’s happening backstage. So, we map that full spectrum and then we decide okay, what’s the best way to deliver on the needs that we see across.
Steve:Is there – so again, my model may be kind of broken here, but if it’s kind of planning to plan. Again, I hate to put my labels on top of yours, but I’m working to make sense here. You’re not – you’re setting the stage for sort of exploring and building a solution for IBM to do that. So, even in doing this, building this large map, this initial map, you are looking at needs and looking at processes and journeys.
Ashley:Um-hmm.
Steve:Does the building and deploying of these kinds of solutions that support transformation, is there activities in that process that look like what we might consider user research?
Ashley:Absolutely. So, I should say my team is really focused on journeys, right. The customer journey. There are research teams that focus more on solutions, right. And so we have to have this feedback loop between the teams that are working on products or services and that are more embedded in the day to day user research, right, generating requirements needs, evaluating the quality of the user experience. Right. Working with engineers, working with product managers, right. There are teams that are very dedicated to that and they have incredible expertise in the technologies in that context. But we also have to keep in mind, if we’re talking about these key journeys, how do those products and services add up? Right. How do they add up to really solve the big needs that we see from our clients? And so someone has to kind of be that layer on top. And the mechanism to pull together the teams, pull together the expertise and capabilities that we have within IBM, and so that’s our team. And so in some ways we’re researchers and we do user research, right. We bring that outside/in perspective. But we’re also service designers in that we’re really designing how the full system in an organization works.
Steve:I feel like there’s another facet to what you’re doing though, and maybe that just falls under the umbrella of service design, but creating this deep understanding which informs this plan – plan is kind of the word I keep throwing into this. But also, you know, serving as – being in dialogue with the sort of teams that are doing a little more – the deep requirements, it sounds like that’s a little more day to day. That’s the detailed view versus the overarching view which you’re capturing. I guess this speaks to the complexity and the scope of the kinds of projects you undertake. It’s interesting for me to think about this overarching layer of understanding that’s kind of crafted that is not about what are sort of the finishes going to be on the details of interactions, but it’s really just this big view.
Ashley:Yup.
Steve:That’s really interesting. And so – I’m curious about what I think goes with sort of the scope that you’re talking about is some time horizons that must be interesting. If there’s a certain effort to make a plan and then a certain effort to build and support and roll out things – I don’t know, is there a typical time that you’re thinking about when you work on these programs?
Ashley:That’s a great question. So, this is a fairly new effort. You know we’ve been working with this customer journey and user need centric way of thinking about the future of IBM for about let’s say 2 years, or less. And so there was a phase where we were really framing like what does this look like? How do we talk about it? What are the words? What’s the right language to use? What is the relationship between a user, a team, a journey, a product, an IBMer? We were establishing that frame, building on the practice of service design. And then there’s a phase of piloting that’s kind of the phase that we’re in now. So, you might have a great frame, but how does it get really built into the DNA of the way that the company works. We found that it’s really important to pilot work with particular business units, understand are our methods working, does the frame hold up and do we know about – enough about the way that the business works yet in order to say yes we can deliver this way of working. So, we’ve been on our own journey to do the research, the practicing to finish the frame. And then there’s a future effort to scale. Right. How do we get all of IBM working like this? And so that’s really – that’s kind of the plan. Right. And it requires a lot of influence – stakeholder engagement, a lot of consensus building and alignment. And so that is also the work that our team does.
Steve:As you talk it’s clearer to me that there’s a significant focus on the organization of IBM and how IBM does things. It sounds like that’s where a lot of the effort is right now.
Ashley:Um-hmm.
Steve:Even though ultimately this is outside focused on customers that IBM is supporting, you’re trying to – you are building new practices in terms of how IBM does that.
Ashley:Exactly.
Steve:So your emphasis right now is on, at least a significant part is, on how IBM does the work that it’s been doing in terms of building things for customers.
Ashley:Exactly and I think that’s critical – that’s a critical place to focus, right, because we could generate all the understanding of user needs that we’re able, but how do we build that into the way that everyone thinks and how do we go to market in a way that actually is accurate to our understanding? And so that really requires organizational change in some ways and this work, right, this journey work is building on the design program that Phil Gilbert has been leading over the past 5 to 6 years. So, he came to IBM and said hey, Ginni (our CEO), design is critical to the future of IBM. We need to focus on user needs. And so as his program has scaled we’ve started to see a focus, not only from designers, but from other IBMers, right – engineers, product managers, executives – in adopting design thinking as a way to a) focus on user needs, but also work better cross-functionally across our business. And so this work that we’re doing to really drive customers journeys is building on that. And so we leverage the 2,000+ designers across IBM and we’re really disseminating these methods to them, building on the design practice that we already have.
Steve:So, is that an example? Because you mentioned influenced being a major part of what this team is looking at right now. What are some practice for influence that you’ve found successful here, or that you’re exploring here?
Ashley:Yeah, I think being in a company that loves numbers, I think that that’s a reality. I think what we’ve seen is that qualitative research is really important, but it’s actually not always enough. And so I’ve been really thinking about what does a mixed methods practice look like and how do we bring a lens of both qualitative and quantitative that at an executive stakeholder level is like – stands up to snuff. Can we scale the number of data points that we have to help show the size? or impact of a problem that we see? And can we then talk about its qualitative nature to help us look for solutions to solve those issues?
Steve:So is that – I feel like I get into versions of this discussion with people all the time around influence and the kinds of things that you’re talking about. And where one thing to do is speak their language. Another thing to do is to – I hate the verb educate because I think it’s so patronizing – I don’t know, to empower people with the mindset of your own language. I mean you see really bad practices where people may do a very small ethnographic study, but they’ll – like very, very small, but they’ll quote percentages. Or they’ll say 3 out of 8 or something like that as a way to make quantitative research a peer quantitative, which I get the impulse to do that…
Ashley:Yes.
Steve:…because it’s sort of speaking somebody’s language, but is also doesn’t help that person because they’re going to misinterpret – you’re misrepresenting and it invites misinterpretation.
Ashley:Right. I think it’s much more about showing the scale of the need that we’re maybe not fulfilling yet as a company, or the scale of a problem that exists where two parts of our business are not aligned, or you know, there’s a gap in product/market fit, right. So, I think in the mixed methods practice that we’re thinking about it’s much more about like understanding from a data perspective what’s happening across the business? What patterns do we see and behavior? And then laying on top of that, or infusing that with some of the qualitative insights that we get by talking to customers, right. Talking to people that are in the space that we’re working, Cloud and cognitive capabilities, and bringing those perspectives together. So, it’s less about like okay we have a number and it’s more about we have looked across, we’ve understood from what thousands of people are doing, not just 3 to 5.
Steve:So, is that the starting point and the qualitative is the supplementary? Is that kind of the model?
Ashley:They run in parallel. So, on my team I have what we call design researchers that are really practicing user research and service design. We also have data scientists on our team. And so we are kind of running different workstreams, right. Let’s say we’re working on a journey of a client, modernizing their infrastructure to a containerized environment, right. So, we’ve taken that frame as the journey. The data scientists then go to the systems of record and they use that frame to understand all that they can. And they’re really creative. I really want to emphasize that. Right. We have a lot of historical data as a company. How can we best leverage that to inform our future? So, they’re like getting access to support data, to financial data, and really putting together a picture of how our business has run and what behaviors they see. The qualitative researchers are then talking to customers, talking to people on UserTesting.com or using Respondent to conduct moderated interviews and understand what are the needs – what are the business needs that organizations have and then we bring those together. So, that’s kind of an exciting moment to start to see like what patterns are happening across and where can we derive greater insight by bringing those two lenses together.
Steve:How do you create the conditions where these stereotypically different mindsets, different methods – you know how do you create conditions where the gestalt of what both those uncover can be kind of extracted, I guess?
Ashley:Yeah, that’s a great question. I think what we’re finding is that it’s all about making it physical or visual, right. So, we have a collaboration space where we just put everything on the walls – the war room, right. Take what the data scientists have found. Let’s talk through it. Let’s try to understand it even if we don’t have the expertise of the statistical figures. We then put the qualitative insights on the wall, right. And so we’re mapping. We’re doing a lot of mapping – systems mapping. We’re doing a lot of like pain point gathering and making it all visual, making it almost tactile. It helps us to put that picture together, the fuller picture together.
Steve:And this is both groups are doing this together?
Ashley:Right. Well, I should say that is the practice that we’re growing. I think it’s still forming, but yeah, that’s my vision is that we bring all of the insights that we have. We really work through it together and we’re stronger and better in the way that we can tell the business where to go next based on those two influences.
Steve:So there’s a lot of emergence, I guess. You are defining practices that are new to this part of IBM, or new to IBM maybe, or new to sort of the field overall of all the fields that you’re kind of pulling together?
Ashley:Um-hmm.
Steve:What’s your approach to trying to innovate and process? You’re making new things in the way that you’re working. Where’s that coming from?
Ashley:It’s all in pursuit of a customer journey that is excellent, right. We really try to keep that at the center, right. What problem are we solving? We’re solving for our client’s business needs, or our future client’s business needs. So, we keep that at the center, but we are constantly making. We are constantly trying methods, iterating on them, seeing can we get to a picture that is compelling for the people that are making really important decisions for our business’s strategy. And so it’s a constant evolution and I think what’s exciting is when we get momentum or we get resonance – we reach a point of resonance. And whenever we feel that we run with it. And so I think there’s a certain intuition to the work, right, to say okay this is compelling. Like how can we amplify an insight that we’ve had or an artifact that we’ve made and how can we start to bring that into a standard practice over time?
Steve:You’re obviously being reflective about the practice. You’re doing the work, but then you’re also looking at let’s think about the best way to do this work and let’s change that.
Ashley:Absolutely, yeah. I think for us, what we’ve done is we’ve had to make time and make space for that reflection. So, now every week we have like a huge block of time on everyone’s calendar and we all get together and talk through, work through the data that we’ve gathered, synthesize together – that’s critical, right. Synthesizing just the qualitative research in Excel on your laptop is not the best way. It’s about us all thinking together through a synthesis process and that’s how we get to those moments of resonance.
Steve:There’s a couple of things. I think you’re making time work together on the work, but you’re also making time to reflect on how you’re working.
Ashley:Yes, yes. And so – we actually work in agile sprints which I know can be challenging for research, but it helps us to have a cadence of working, doing the work, reflecting on the work and pivoting. And so I think our leader, Sarah Brooks, she leads journeys at IBM. She’s a distinguished designer that’s really leading this mission and she’s been really intentional in making sure that we know the impact of the work that we’re doing, that we’re properly socializing and influencing the rest of the organization and that also we’re really tight on the methods and practices that we employ.
Steve: Can we go back to what you were saying about doing synthesis together?
Ashley: Um-hmm.
Steve:And kind of creating those moments of resonance, I guess. You said that Excel on your laptop by yourself was not really the way to go. I mean I agree with you and I’d just love to hear you like make the case for why – for what happens when people are together? I’d love to hear more about what that feels like and how it’s valuable?
Ashley:Yeah. I think – I’ve worked on a – embedded in a product team as a research lead, but leading by myself and I think, you know, you’re going to – one researcher, one person is going to have a certain interpretation of the data that they find. But when you bring in another person you’re always going to get a slightly different take. You’re going to get a different set of assumptions. You’re just going to have a more diverse, right – even bringing a second researcher in, you’re always going to have a more diverse take on the data that you found and so I think doing synthesis together – there’s two aspects to getting out of Excel. One is that diverse set of perspectives, right. The second is that when you’re in your computer you’re not thinking in the same way that you are on a whiteboard, right. And this is the premise of design thinking, right, externalizing the data that you’ve found. Externalizing your thought process, just helps you move faster and clearer. And so I think getting out of Excel is really about those two things and that’s something that we’ve been really working on in our team.
Steve:It just reminds me of this thing – it’s a thing I remember from being a little kid. I don’t know if this is unique to me, or just general for everybody, but, like being stuck on something in the classroom and like going to stand next to the teacher’s desk to ask a question and by the time I could formulate the question I’d figured it out. Or even if I’d verbalize it I was like oh now I know what it is. It seems like the externalizing process is a sense-making activity, I think.
Ashley:Um-hmm. Even to be able to communicate, right, a data point that you found, or tell a story, right. That’s a synthesis process. That’s a cognitive process. And so I think bringing that out of a person just helps us each grow in the way that we communicate and understand what we found.
Steve:When you talk about just even adding another researcher that’s going to have a different perspective on something – I mean I agree, but I also hear the things that make people anxious about qualitative research. Like you’re sort of celebrating the – it’s an aspect of uncertainty or – right, we struggle with I think being perceived as rigorous.
Ashley:Yeah.
Steve:And when you say well if someone else is going to come in the room they’re going to see something differently than I am, but that’s a strength. So, someone might hear that and say that’s a weakness because how do we know what’s real? Or how do we know what’s true if you can’t even see the same thing as somebody else.
Ashley:Um-hmm. I think too it’s less about facts. It’s more about like what’s important in the data. I think the synthesis process is a lot about prioritization and coding or articulation of what’s important. And so I think what’s important to one researcher might be different than what’s important to another. So, it’s less about like my truth/your truth. It’s more about what’s important to our team user needs and then what’s important to the business, right. Ultimately that’s customer lifetime value and meeting our business goals. So, in getting out of Excel, working together, we can all talk more about like what’s really important and that’s only going to make the work better.
Steve:I feel like this ties a little bit to the influence part of our conversation as well ‘cuz you are running things in a way that I think – there’s definitely struggle in some teams to do this. To have more than one person. To take the time to reflect on their process. To devote the time to synthesis that it requires, as opposed to kind of sitting down at your laptop and kind of pushing out the next set of recommendations. Are there elements or components of the practice that you have been building that we’ve been talking about that you think could be even adopted in small parts by somebody in a different kind organization?
Ashley:Hmm. That’s interesting. Um, I should also say this is the practice that we aspire to execute on, right. It doesn’t always happen every study, every day. That’s where we aspire to go, but I think one of the special things about the work that we do and the way that we understand customer journeys is that we really are taking a holistic view and when you start to map the whole of what users need, how they interact with us, our relationships with them, and then also what IBMers need, right, it’s kind of hard to unsee once you see it. Once you see how two IBMers are disconnected or their incentives aren’t aligned and you see how that surfaces in the user experience you kind of can’t unsee it. And so I think although I lead the design research practice, I think there’s an element of expansion of design, to this like service design frame, or even like a business model frame, that I think we’ve gotten a lot of value out of, right. So, user research is really important, but it is a foundation to a larger understanding. And so I think other organizations could benefit from that, right, in adopting a more full picture of how user research and how user experience plays within a larger context.
Steve:And you brought it back too to this, in some ways your first customer is the organization of IBM. Even though IBM is focused on its customers as kind of core to its business, you’re serving an internal – I mean everything. You’re serving everything about IBM.
Ashley:Um-hmm.
Steve:And you know that’s different. And then you mentioned sort of the figuring out how to go from New York to California. In those kinds of organizations they talk about being customer obsessed and the researchers and designers are thinking about improvements for the customer, which is also a great thing for a business to be thinking about.
Ashley:Right.
Steve:But it’s different than can we make what we do here as a company better for us as a company versus can we make a better product for our customers. Success is going to be measured in different ways in those organizations.
Ashley:Right and so we are designing – we call it orchestrating and also measuring the whole customer journey and one question we have to ask ourselves is like where do we sit in that customer journey because we’re part of it, right? Backstage. We’re somewhere and I think we’re actually pretty far backstage, right, because our first line of user is the IBMer. It’s then the client organization, but then our client organizations also have users. So, there’s this spectrum of the definition of user and we have to figure out like who’s the priority among that set of users? And where is it best to focus and that might shift over time. Maybe initially we’re designing and establishing practices for IBMers, but once those are set how can we push further towards more of a human centered perspective? And ultimately that’s getting to deliver an experience that helps that end, end user and their needs.
Steve:I’m sure there are some visuals that go with this.
Ashley:There are some big maps. I think it’s also best to give an example of this, right. So, let’s say we’re designing a journey engine modernizing to a containerized Cloud environment, right. Why – we have to understand like why are people – why are organizations doing that? Why are they modernizing? What we found is a lot of it is about driving better – their own better customer experience, their own better user experience. A faster app, right – iPhone app. A better, right, way to search – for their users to search within their business. And so it’s funny that we’re going through our transformation, our client organizations are going through their transformation and ultimately that comes down to a person using a consumer thing. So, when I pick up my banking app and I check my savings account, right, that’s ultimately coming back to something that IBM is probably involved in. And so how do we keep that experience in mind, that very human, very personal experience of knowing how much I have saved for my future and how do we then comb through the technologies and capabilities and relationships and politics that affect that? There is a great breadth and depth to the work that we do, I guess is what I’m saying.
Steve:So, if this person with the bank account is going to have a better experience the company that – that bank needs to have processes and tools and technology to enable them to be more contemporary in how they make products and services which means someone like IBM can help them understand how to go about changing those processes, changing their tools, changing the infrastructure, so that they can then go make that app. Which means that you and your team have to, you know, build processes and practices that can help IBM help that bank build better kinds of apps to help people have better experiences.
Ashley:Right.
Steve:I just said what you said, but I started going in the other direction.
Ashley:I like that you mapped it back to the other direction and I think as we can better serve our clients, right, we can help them get to a level where they can innovate. So, ultimately, right, the work that we do is about moving towards the future, right, with shepherding new technologies into the world, shepherding new practices, advising, guiding, and ultimately in pursuit of something that’s ideally life changing for someone.
Steve:That’s great. Can I go back to something else that you said before and maybe pivot from that?
Ashley:Yeah.
Steve:You’re talking about that moment of a researcher brings another researcher in to talk about some data points, or some qualitative research. And that this diversity of thought is a – starts to add value because people look at things a different way. You know – the word diversity is being used to represent a lot of different kinds of things now and I guess I’m wondering if you can talk a little about diversity overall in the design research field? What are you seeing or thinking about right now?
Ashley:Yeah. I think that’s definitely something I’m thinking about, my boss is thinking about and IBM is thinking about. From a research perspective, I think there’s a lot to be desired in terms of having a diverse representation in who is practicing it. And I think that’s really important if you’re talking about a new technology, Cloud or cognitive capability, AI, right – we need to have people that have different perspectives in the room, right. If we’re creating a data model and it’s going to affect the way that someone’s bank account, right, or their financial service – it’s really important to have people from different economic backgrounds, or from different contexts, that can understand what are the pressures, what are the needs, what are the pain points of different types of economic situations. Right. So, that’s just one example of where I think we could grow. I think when I look at researchers, right, the wonderful research community, I don’t see a ton of people that look like me and so even by talking to you today I have a hope that we’re growing and that we’ll continue to see more diverse faces, diverse ways of thinking and diverse backgrounds represented in the field.
I think too, a second point, research has traditionally been like somewhat of an academic thing, right. It was something that PhDs did, right, and maybe other folks that didn’t have a master’s degree even were kind of shut out. But I think we’re seeing a shift there. I think we’re seeing people go through things like General Assembly, or self-taught, really thriving in the research community. I myself have a background in architecture and this crazy MFA program called transdisciplinary design. And so although I picked up research practices throughout both of those educational experiences, I think I have somewhat of an untraditional background in the fact that I’m leading, right, a team of researchers and able to really maybe bring more of a generalist perspective to the practice and bring the things that I’ve learned in architecture, from systems thinking and mapping and other practices. It just only helps enrich the field and enrich the work that we do.
Steve:I do want to ask more about your background, but let’s keep talking about what you see when you look at the community of research, or what you don’t see, I guess, more importantly. Right. Let’s just imagine someone is listening to this podcast who’s kind of on the outside of the practice, but is interested in it. Do you have suggestions for them if they also are not from the groups that are currently sort of dominating the population of research?
Ashley:Yeah. Um-hmm.
Steve:What do you tell them? Or what do I tell them? What do we tell them?
Ashley:I think one of the key things to any practice, right, or field is language. So, I think a lot of what we do is about having the right language. It’s having a mental model of people, behavior, psychology, right, cognitive processing, kind of understanding that. So, I think if you can kind of just pick up a foundation of understanding of those things. Like how do we talk about user problems? How do we talk about user needs? How do we talk about motivations? If you can kind of pick up that sphere of things, I think it’s easier to move into the space, right. So, for me methods are important, but methods can be learned. And so I think focusing on the mindset and the perspective that we bring when we sit at a table of, you know, various stakeholders, right, we have to bring that user lens and that’s the most important thing that we do. It’s not about are you the best interviewer? Are you the best person to like write the survey? Because someone can always help you with that. But it’s about do you have the right mindset and do you speak up and bring the voice of the user to whatever context that you’re working in? And if you can communicate that and start to build off of that, I think you can have a viable career.
Steve:Good.
Ashley:Would you agree with that? I’m curious.
Steve:I mean I think your point about mindset vs. methods is really – if I think about just my earliest days in the field and trying to hire people – I mean there weren’t people – it just was sort of underpopulated practice. Whereas now I think there are sort of new people coming in all the time. And I remember struggling with how do we articulate someone who would be good to join our team? I worked at this agency and we used to just talk about people who got it. I mean that might – in 2019 that might be a trigger phrase for sort of exclusionary thinking.
Ashley:Yeah.
Steve:I think what we were trying to say then is the same as mindset. I think we were thinking about that there’s a frame of reference or just a common language around how you think about what we’re here to do. I was at a product design consultancy and, you know, you’re there to sort of serve the client, but also serve the – we were doing research and trying to help the company serve their customers even though our customers were the company making the things. So, there was some advocacy and evangelism and facilitation and sort of understanding hey, here’s where people are at, so what are the opportunities to sort of help them, or empower them, or do something for them, that I don’t think we had a lot of clear language around. It was sort of new to the practice or new for us. It was a long time ago and we were relatively isolated at that point. Yeah, I think – I agree. We’ve had the conversation many, many times about you can learn methods, but there’s this other thing that’s sort of harder to teach. And so I think your advice is kind of hey, get that thing.
Ashley:Yeah.
Steve:That’s the kind of thing that – so, how does someone demonstrate that they have this thing, that they have the mindset?
Ashley:Ooh, I think write about it. Those have been some of the best moments in my career so far where I worked through a project and then I went and wrote about it. Because that’s a beautiful reflection process – what was important, how should I talk about this, right? You might have had one conversation in your head, or with your team when you were doing the work, but when you reflect back on it, like what stood out? What were the major like learnings? What were the moments that really mattered? And I think writing about maybe a project that you’ve done, or how you would like to do the work, could be a really great way to yeah, just get a stronger hold on your own practice. So, portfolio building has been – I think is a great practice. Maybe not to get a new job, right.
Steve:Right.
Ashley:Not just to get a new job, or get a promotion, but just as a reflection practice. And that’s something that we encourage here at IBM. We talk about this thing, the portfolio of experiences. So, it’s not necessarily like oh here is the beautiful app that I designed, but the portfolio of experiences talks about here are the range of problems that I worked on. Here’s what I learned along the way. Here’s how I grew to be a better researcher, designer, visual designer, facilitator. Right. Here’s how I learned to talk to executive stakeholders. Here’s how I influenced them. And that’s so important. That’s so important. And so I know it’s hard. When you’re just starting out you might not have real projects. I hear that a lot. I get that in my LinkedIn – how do I get started if I haven’t worked in the field yet? Write, talk about what maybe you would do. Make up a project. Do a General Assembly course and get some experience and reflect on it.
Steve:I like portfolio, almost as a verb the way you’re talking about it and less about the noun. So, that’s developing the mindset by – so it’s less about this artifact that you’ve created and more a way of kind of thinking and communicating. And then you have stories to share and stories to tell with people that you meet.
So, we started off this thread of the conversation by talking about – you didn’t use the phrase underrepresented, but I think that’s an element of what you’re talking about. But, I guess, as often happens when we talk about increasing accessibility of anything, like this is good advice for everybody, right – regardless of your privilege or your representativeness. I don’t know, is there anything else there for people that are not represented, or are not in the sort of majority of what the field looks like. You said speak up. That was one sort of thing that…
Ashley:Yeah, speak up, have a point of view. But I think ultimately it’s on the leadership of, you know – whoever is leading a team. The responsibility is on us. It’s on me to, when I’m hiring, saying alright I posted this in this platform, I only got these types of people applying, I need to look somewhere else. And I need to consider the full range of people, but I need to make sure that I’m looking for people that aren’t currently represented on the team, whether that’s – you know we call it URM (unrepresented minority) in the HR system. So, it’s like women, people that are black, African American, Latino, Latinx, people that have accessibility needs. That is like – that’s really critical. I mean the work that we do, accessibility is so important in, in, in this space. And so having people that have that experience, are like why wouldn’t we? Why wouldn’t we have them? And we’re working on that. And then also people that identify as LGBT or even people that are veterans, right. Those are some of the groups that you want to really look for and prioritize and there are organizations out there that cater to those groups, reach out to them. Go to those communities, right, and that’s how you’re going to find those people. There’s this like concept of a pipeline issue and sure, but right – let’s say you wanted to hire more black designers, that’s something that is really important to me. Go to historically black colleges and universities, right. Partner with the leaders in that organization. Make sure that the students in that school are getting the education that will result in skills that apply to the work that we do, right. There’s a lot of work that we can do that’s not at the hiring stage, but just kind of working a little – just a few years back, right. And we’ll radically, radically change the quality of the skills that the students have coming out of those institutions and radically change the quality of our organization. That’s just one thing you can do, right, that will be really impactful. And so I think – I see that growing at IBM and as a leader I want to be part of that. I want to do that. And it’s not that hard, you know. So, let’s do it.
Steve:So, there’s – I think about sort of these disciplines, maybe nested within each other, that you have – the tech industry and its challenges with – you’ve given very good practical advice here, but, it’s challenges with hiring underrepresented minorities. And then you have kind of UX or design within the tech industry. And then you have design research within that. I think sometimes, and maybe it’s the same thing across all, but I think sometimes we – like the research practice, when we have these messier things come up – like oh, we should be talking about ethics, that I fear sometimes that design research just says well here – defaults to the UX conversation or defaults to the tech conversation. And I’m thinking about, again, diversity and inclusion in hiring, what our community looks like, do we have specific challenges or specific opportunities, or either way, within design research that we should be thinking about? Is there something unique for us as a field that we should be thinking about?
Ashley:For sure. I think maybe you’re gesturing to the practice, like within doing the actual work.
Steve:I wasn’t, but if that’s where it goes that’s fine.
Ashley:Even in we’re going to do 30 interviews – who are we interviewing and what perspective are they bringing? If they’re talking about the challenges that a user experiences in their day to day is that different for men and women? Is that different for somebody who has full vision vs. like a low vision need? So, I think even just starting there. That’s hard. That’s challenging. It’s hard enough to get to – in IBM as researchers to get to users, our customers, right. That’s an initial challenge, but then to make sure that the people that we talk to are a diverse – represent a diverse set of experiences and needs, you know that’s something we could do better. That’s something we can prioritize and talk about and cite. And so I think that’s one way. But, maybe rearticulate your question so I could answer it more directly.
Steve:There’s a tech challenge, that tech isn’t diverse enough. Tech says we have a pipeline problem and they’re not going to historically black colleges and universities to recruit – sort of the thing that you described. If you look at what tech is doing as an industry. If you look at what design is doing – I mean because those are sort of bigger entities with a lot more voice around here’s the challenges, here’s how we’re failing. I mean design research is a smaller practice. Maybe we have – like it’s a practice that my anecdotal data says it’s more women than men. So, sometimes – I don’t mean to imply that’s sufficient diversity. I mean it just – it inverts the sort of male dominance to a lot of tech. So, maybe that feels good. As a man in the field that’s kind of like oh cool, I’m in a non-traditional kind of thing. It doesn’t’ mean that I think we’ve achieved the kind of diversity that you’re talking about. But because people in design research come from different places than say software engineers do, because the field offers certain kinds of experiences and pulls on certain kinds of creative mindsets that are different – because we do different work. Like there’s something about research that – like the work we do is about telling the stories of other people who are different than us which maybe you could use to justify not being diverse because our practice is to sort of – or maybe you would use that to say well then we especially better be diverse because of it.
Ashley:Right. I think there is something privileged to design research as a practice. I think especially for myself coming out of grad school, or going through grad school, going to get an MFA was like kind of a privileged thing, right. Become an engineer or a lawyer or a doctor, it’s kind of more certain, right. If you’re investing however many hundreds of thousands of dollars in your education, I think for certain groups, maybe socioeconomically, from certain contexts, right, you kind of want to go with something that’s certain, right. So, I think maybe there’s something to the way that design researchers are traditionally educated that’s a little inaccessible. And I see that changing, but maybe there’s something to that and we can pick up on it and amplify less traditional or expensive ways of becoming ethnographers, design researchers, UX researchers. I think there’s something there that I’ve definitely had conversations about with people. Um, and again, even coming out of grad school, my graduate program, transdisciplinary design at Parsons was 5 years old when I graduated and that was the first year that like people really got jobs. So, you know, working in messy spaces, talking about soft things like feelings, behaviors, is a less certain space to be in. The outcomes are less certain and less rigid and so I think maybe that has something to do with it. And if it is, if we find that, maybe that’s an area that we can like probe in and find a solution.
Steve:Good. These are some good signals I think to toy with. Can you talk a little bit about this program? Like how did you find it or what kind of drew you to it?
Ashley:Yeah, okay. So, I went to Howard University – the illustrious Howard University, as we call it, a historically black university – and studied architecture. I got a professional Bachelor of Architecture. It’s a 5 year degree and by the 4th year I was like absolutely not. Very cool. I loved concepting. I loved research. I loved setting contexts, like okay what is this neighborhood like? How does that influence what this building should be? I love that part of it. I love the concepting. And then the visualization, like the diagramming. And, um, but did I want to go build buildings? And like architects live really difficult lives? They work really hard, right. Um, um, so I said I love some of the things here, how can I pivot or like open up the lens of problems that I want to solve? And so I went and got this graduate degree from Parsons and I only found it because I went to go look at a fashion program and I looked down the list of programs and it was at the bottom. I was like transdisciplinary design, what is that? That sounds interesting. So I went in and met with the director of the program and I was like, oh, this is a thing. Like solving complex problems, talking to people, understanding systems and behaviors in spaces. And I was like oh yeah, I think I’m going to do this. So, that program is really built on the premise that our world is so complex, we have these wicked problems that one discipline can’t solve. And so the practice of transdisciplinary design, if it is a thing, is about bringing diverse sets of mindsets, but also methods, together, and using design thinking kind of as the framing for bringing that cross collaboration and synthesis of those mindsets. So, that’s what I went to do at IBM and it turned out to be a really useful education.
Steve:Did you go to IBM from that program?
Ashley:Yes. So, I’ve been at IBM for about 4 years now.
Steve:The way you described the program – I guess it’s no coincidence the way you describe what you’re doing at IBM and the way you describe what that program’s aims are seem to have quite a bit of coherence.
Ashley:Right, right. And finishing up that program I had no idea that I would work at IBM. I didn’t think about that. But when I went and I talked to the folks that were hiring at the time, I was like oh wow, there is a need for this and it wasn’t exactly the context that I thought, but it’s been a really great place to learn and practice and then begin to lead.
Steve:I guess hindsight has always idealized things in retrospect, but just kind of the way you described it, it seems like the experience at Howard told you some of what you didn’t want to do, but also opened you up to see this transdisciplinary design program. Like you were at a point of sort of creative self-discovery that this filled – this was a thing that you didn’t know existed that then was right for you right at that time.
Ashley:Exactly. And even while I was at Howard, I mean you could imagine it’s a place of social innovation and like you know the student body there is very much engaged in politics and systems and so that was already there, right. That was already something I was thinking about. But then to move into a program where like that was the basis, the foundation, that was the work that I would do, it was like oh, this is obvious. And so I think as you – as I have gone from place to place and learning and growing, I like to follow those threads, right. Okay, systems thinking, mapping, social innovation. Right now it’s about this customer journey, service design, right. You can kind of follow those threads from place to place and build on your interests while building on your expertise and I think that’s something else that I would tell folks that are looking to get into the field, like build on something you already have. If you were a teacher, maybe become a design researcher in the education space and that’s another way to break in and to, yeah, just build on what you already have.
Steve:Do you ever imagine a future for yourself? Think about whether it’s 5 years or 10 years – I hate it to be where do you see yourself in 10 years. I don’t mean it that way.
Ashley:Oh, terrifying.
Steve:You know on these kind of – you’ve made these interesting, I don’t know if they felt like leaps – you don’t present them as leaps, but these zigs and zags I think that sort of came at the right time. If you think about a future for yourself, what kinds of work might you be doing? Or what kinds of roles or organizations do you think are there for you?
Ashley:I think it’s all about mission for me. And that’s why I was so excited to join Sarah Brooks and the journeys team and lead research. But I think it’s all about missions, like I like the big messy, unsolvable problems and maybe that’s like nature, maybe it’s nurture. But I think it’s all about that. So, maybe it’s the criminal justice system. Maybe it’s food justice. Maybe it’s design research diversity. I think anything that’s like hard and interesting and you don’t quite know the answer is something that I want to do. And so I think, if I look 5 or 10 years out, maybe I’m at IBM still doing that. Or maybe I’ve learned enough that I am going out and I want to like focus on a problem. So, I don’t know what it looks like, but I know it’s a mission. It’s always a mission.
Steve:Anything you want to plug?
Ashley:Yeah. So, if you’re interested in IBM and practicing design, or even just learning about how we talk about design and the work that we do, you can go to IBM.com/design and that will kind of launch you into our world. You can also reach out to me. I’m @ashleyograham on Twitter. And thank you.
Steve:Well thanks for a really great conversation. We explored a lot of different nooks and crannies and I think it’s fascinating. So, thanks a lot for taking the time.
Ashley:Thanks, Steve.
Steve: Thanks for being here for this episode. You can find Dollars to Donuts on Apple Podcasts, and also Pocket Casts, Castbox, and Overcast. If it has “cast” in the name, we’re probably there. Go to the website – Portigal dot com slash podcast – to get all the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 24. Ashley Graham of IBM first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my conversation with Ashley Graham, a design research leader at IBM. We discuss synthesis as a collaborative, co-located activity, being mission-driven, and building a process that addresses complexity. When I look at the wonderful research community, I don’t see a ton of people that look like me and […]
The post 24. Ashley Graham of IBM first appeared on Portigal Consulting.
This episode of Dollars to Donuts features my conversation with Ashley Graham, a design research leader at IBM. We discuss synthesis as a collaborative, co-located activity, being mission-driven, and building a process that addresses complexity.
When I look at the wonderful research community, I don’t see a ton of people that look like me and so even by talking to you today I have a hope that we’re growing and that we’ll continue to see more diverse faces, diverse ways of thinking and diverse backgrounds represented in the field. – Ashley Graham
Show Links* All Those Books You’ve Bought but Haven’t Read? There’s a Word for That * My Father’s Stack of Books * Nextdoor * Freecycle * The Year’s Best Science Fiction * The Best American series * Goodreads * Steve on Goodreads * Ashley on Twitter * IBM Design * Phil Gilbert * Ginni Rometty * Sarah Brooks * Transdisciplinary Design * Howard University – Architecture * General Assembly
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal:Hi, and here we are with another episode of Dollars to Donuts, the podcast where I talk to the people who are leading user research in their organization. In 2018, Kevin Mims wrote in the New York Times about the Japanese word tsundoku – a stack of unread books. In a New Yorker article entitled My Father’s Stack of Books, Kathryn Schulz reflects on what her family referred to as The Stack, the books that accumulated in her parent’s bedroom, especially on her father’s side of the bed. These weren’t just books to be read, but also books that were recently read that should be kept near at hand. She estimated that The Stack contained 3-400 books.
For me, I switched a few years ago to getting rid of most books, passing them onto someone who might like them, or giving them away in the community via Nextdoor or Freecycle. I felt like hanging onto every book was becoming increasingly unmanageable, and in some ways was creating a barrier to acquiring – and thus reading new books. My tsundoku serves as a last-in-first-out queue, but for me unread books go in the bedroom and books that you want to keep should be displayed on bookshelves.
I was a voracious reader of books as a kid, and at this point in my life, it’s something I need to make a deliberate effort towards. I read on the Internet all day; I read several magazines regularly. I read a print newspaper every dat. Plus I’m trying to watch a ridiculous number of television shows and movies on all the platforms. And oh yeah, podcasts, right?
As a kid, I really got into science fiction and especially sci-fi short stories. At a certain point I set that genre aside but maybe 15 years ago I came across a phone-book sized annual collection of sci fi short stories. And then the annual best American short stories series. These books have got built-in portion control – read at least one story in bed, before going to sleep. But then I’d find myself in a bookstore staring at the shelves without any clue about which ones in the series I’d read. And even if I was home, if I’d given the books away after finishing them, I couldn’t just go check my shelves to avoid repurchasing something I’d read. That’s when I found Goodreads – a website and an app where I can organize the books I own but haven’t read, the books I have read, and the books I don’t own but want to read.
There’s a whole set of other things you can do on Goodreads. You can connect with other people and get their updates in your feed. You can post progress updates as you go through a book, and you can write and read book reviews. But for me, my primary motivation was to have a single location, available to me anywhere, to see what I had already read.
My Goodreads usage stayed casual and intermittent for a couple of years. But when my local library opened a brand new building, I went in and renewed my library card and that triggered a deliberate and focused shift to reading more books. I think the stack beside the bed has not decreased much in size, but I’m fine with that. I’m making use of the library website to put books on hold, maybe it’s a book I learn about from Twitter or an article in the newspaper. I’ve also been saving books on the library website that I’ll want to put on hold later. I’ve been getting books from other libraries in California. I’ve been reading graphic novels and regular novels, both in print and on my iPad. One of the cool things about the library is that borrowing a book means that I have a deadline. It’s not just that the book is expected back, it’s that the deadline makes reading the book into a tangible accomplishment. It’s due back on a certain date, and I finished it, and I returned it. I mean, I’m also free not to finish the book, but even so, that’s something I can metaphorically cross of a list. And then this is where Goodreads comes back in, because even though I haven’t connected with many people and I don’t care too much who sees my updates, going to the site and marking the book as “read” with today’s date is another marker of closure. I am definitely doing the reading for the reading, but there are these additional rewards, other bits of satisfaction, I guess we call this gamification if that’s even still a thing. I might write a tiny review, occasionally I’ve asked a question when I didn’t understand something, but mostly it’s about making that mark when it’s done and then feeling a sense of pride or accomplishment from the accruing list of books that I’ve read. This public-ish list announces something about me and what I value, and I enjoy building that even if it’s just for me to look at and reflect on. Taking a book off the stack has a tangible satisfaction, but I find it a bit more diffuse, to move an actual book from a stack to a shelf, than to do the analogous operation on an abstracted book in a digital interface. Go figure.
But this is just how I use Goodreads. I’m sure it’s hardly unique, but it’s also one of many different ways that people could and probably do use the site. I mean, I don’t have any idea what other usage models are. But of course, we have processes and tools for finding out! And this isn’t just about Goodreads but for any product or service that has many features, different sections, different experiences, it’s essential to try to understand how people are using your product. The risk is in only considering the product as a set of separate features. Who is using this particular feature, and how? And for another feature, how is that being used, and by whom? If you are working on a product that has this many facets, you are better served in learning about the bigger picture that may weave its way through different features, as well as other tools or products that support the underlying goal. If you want to improve, or optimize, or extend the capability you are offering, you’ll want to do so based on an understanding of those goals, not just feature by feature. That means you need to think about how your teams are organized, and sometimes the way you’d organize a software team to build different features isn’t the same as how you should organize designers to design those features, and almost certainly not how you should organize researchers to inform the different decisions you’re making across features.
Of course, this is the kind of thing I can help with. Often I work with clients to help them get a handle on what their customers want to accomplish, and look at how the team can focus product and design decisions to best support the people that use their product. The teams I work with come from a variety of industries and have varying levels of experience in learning about their customers and acting on those insights. I see this podcast as an extension of that work, something that I’m able to share with you. And so, the best way to support this podcast is to support my business. Hire me to lead user research projects or to coach your own team as you talk to users. Also I run in-house training workshops to teach people how to get better at fieldwork and analysis skills. Get in touch and let’s discuss what we might do together.
I’d also love to know more about how this podcast is helping you in your work. Email me at DONUTS AT PORTIGAL DOT COM or on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
Let’s get to my conversation with Ashley Graham. She is a design research leader focusing on digital, at IBM in New York City. Thanks for being on the podcast.
Ashley Graham:Thank you for having me.
Steve:I have a very, at this point, traditional way of starting which is to ask you to introduce yourself. Who are you? What do you do?
Ashley:So, I’m Ashley Graham. I lead design research for the digital part of IBM. That means we focus on the customer journeys that our clients, or potential clients, are taking. And we bring a mixed methods approach to both qualitative research, quantitative research and generating insights that can help drive our business and drive innovation for our users.
Steve:Can you give a little context about the digital part of IBM? IBM is this huge company, that for those of us outside – like I personally don’t have a good mental model like what this company does even in 2019 and how it’s divided up and sort of where you are and what your efforts are focused on.
Ashley:Yeah. So, as you can imagine, IBM is huge. We have a large breadth and depth of a portfolio. Digital really sits across all of that, right. So, we have more legacy parts of our business. We have newer, innovative parts of our portfolio and digital really seeks to understand across, right, what are the user needs, our client needs, across the portfolio? How can we bring all the parts of IBM together, all our capabilities and expertise and bring that through a digital experience? And so that’s not something that IBM has traditionally done. It’s kind of something new and so our team, the digital part of IBM, really drives best practices around digital experience, both from a design and also a business perspective.
Steve:Can you unpack digital a little bit? Again, I’m just pulling in very old mental models of IBM as kind of back in the legacy days, you know like a hardware company.
Ashley:Yup.
Steve:So, what does digital mean now?
Ashley:That’s a good question and that’s the question we’re answering. IBM has this over 100 year history, right. We’re going into our 108th year this year. And traditionally we’ve been face to face, right. The client comes to the client representative. They tell them about their problems. They tell them about their architecture and any infrastructure needs. And then the client rep goes back to IBM and says okay, what do we have? How can we serve the client and move them forward in a way that serves the people within that business and then the business needs, right? But, if we want to scale, if we want to grow, we want to transform ourselves to really thrive in the 21st century, we have to shift that model, right. We have to think about digital ways of finding new customers, our customers finding us. We have to think about digital ways to self-serve capabilities. And then we have to think about excellent user experience, right, because that’s a critical part of a digital experience is having intuitive ways to find things, get my tasks done and move on to the rest of my day.
Steve:So, how does the idea of journeys, which you kind of mentioned right off the top, what does that mean in that context?
Ashley:Yeah, so that’s a good question. I think – if I think of a B2C company, right, or a consumer company, right, a journey might be me taking a ride, right, to California. How am I going to get there? We could break that down to me discovering you know the best way. Am I going to take a plane, or a train, or a car? What are the different things I need along my journey? How do I get there? At IBM we kind of have to zoom out because the journeys are much more complex than taking a ride as an individual person. We’re talking about leading and helping organizations achieve their key routes to value. So, we like to actually use this metaphor of the United States and the federal highway system. Right. Before the federal highway system was implemented in the U.S. if you were trying to journey across the United States it was quite a journey, right. Each state has their own way of moving people through roads. They have their own signage, own signals and best practices. So, we are really looking to create those interstates – the fewest, fastest interactions that it takes to solve a large enterprise’s needs. And so that means we have to think about the whole spectrum of technologies, people, processes. They’re our client organizations own transformation and we really have to understand a lot of different things in order to design for that. So, a journey really is like an organizational transformation for our clients. It might be helping them transform the way that the people work within the organization. It might be taking them from an on-premise hardware infrastructure to a Cloud. And so it’s a big journey that we’re talking about, at the level of IBM.
Steve:Given as complicated as it is to figure out how you’re going to go from here to California, for example, the complexity of the things that I think you’re talking about at an organizational level, it’s a little mind boggling I think to – there’s just complexity. There’s a lot of complexity.
Ashley:And I think too, when I’m traveling from New York to California I probably have less constraints than an organization that’s invested 40 years in their infrastructure and is looking to like how can they maintain and build on the investments they’ve made over time. So there are a lot of constraints we have to consider, right, within our client organizations. There are a lot of long term needs that we have to consider. And so, from a research perspective, it’s really about putting together the full picture of our users needs and being able to consider and design for that, in sort of a flexible way. And also being able to reflect that we understand those deep needs and then being able to show that we can guide, right, through complex decision making. Through long term projects. And ultimately to help our client organizations arrive at a future state.
Steve:Okay. So, I’m a big enterprise that has some technology infrastructure that’s been around for 40 years and I come to you and the rest of the IBM and say I’m interested in making certain kinds of changes and others that I don’t even know what they are. What kinds of activities do you and the rest of your organization – well, let’s talk about you and the team you work with. What kinds of activities do you undertake to start responding to that?
Ashley:Yeah. So, that’s – our team is really focused on organizing what that experience is like. And so, say I’m going through a Cloud transformation, right. Or the client is journeying to Cloud. That requires a lot of different decisions, right. So, what we do is we – my team looks across IBM and says, okay, what are all the capabilities that we have from a services perspective, right, like consulting. A technology perspective. Maybe IBM comes in and helps guide those decisions over time, or even embeds and helps in some of the architecture of development. But it really – in order for us to serve, right, those transformation needs, we have to bring together a really diverse set of capabilities that IBM really excels in.
Steve:So there’s an internal examination. You’re talking about looking inward and saying well what are the pieces we can put together here? But are you – is there – how are you assessing what their needs are to figure out what pieces you would bring to bear?
Ashley:That’s a great question. So, I think there’s that inside out, there’s the inside out view. What do we have? What capabilities does IBM have from a technology and expertise perspective? But also we have to look outside in. Right. We can’t just rely on our historical knowledge or our internal strategy. So, we do a lot of actual ethnography and talking from – talking to people that maybe haven’t been with us for 40 years. Talking to people that have been with us for 40 years and understanding the patterns of needs. And we kind of talk about it in terms of key needs, or key journeys. Like what are the most important things that are happening in the market, from an IT perspective, or Cloud and cognitive perspective. We bring those in and we kind of match ‘em up – them up to our inside strategy and see okay for 2019 what are the key journeys that we need to focus on and how can we line up our capabilities to serve those? So, it’s kind of this dual lens, right. We’re looking inside out. We’re looking outside in and the synthesis work really is understanding where do we put focus? How do we organize ourselves to be able to deliver on those keys things?
Steve:Right. There’s something I’m grappling with here to find the right kinds of questions because I think you’re talking about like doing research to plan. You’re not talking about doing ethnography to figure out what the solution looks like? You’re doing ethnography to find out what sort of this larger context, to figure out how to take on a much more complex process.
Ashley:Exactly. And really it kind of reminds me of like a systems thinking perspective, right. We have to map the full spectrum of needs – understand right – if you think of a service design model, what’s happening off stage, what’s happening front stage, as users interact with us, and then what’s happening backstage. So, we map that full spectrum and then we decide okay, what’s the best way to deliver on the needs that we see across.
Steve:Is there – so again, my model may be kind of broken here, but if it’s kind of planning to plan. Again, I hate to put my labels on top of yours, but I’m working to make sense here. You’re not – you’re setting the stage for sort of exploring and building a solution for IBM to do that. So, even in doing this, building this large map, this initial map, you are looking at needs and looking at processes and journeys.
Ashley:Um-hmm.
Steve:Does the building and deploying of these kinds of solutions that support transformation, is there activities in that process that look like what we might consider user research?
Ashley:Absolutely. So, I should say my team is really focused on journeys, right. The customer journey. There are research teams that focus more on solutions, right. And so we have to have this feedback loop between the teams that are working on products or services and that are more embedded in the day to day user research, right, generating requirements needs, evaluating the quality of the user experience. Right. Working with engineers, working with product managers, right. There are teams that are very dedicated to that and they have incredible expertise in the technologies in that context. But we also have to keep in mind, if we’re talking about these key journeys, how do those products and services add up? Right. How do they add up to really solve the big needs that we see from our clients? And so someone has to kind of be that layer on top. And the mechanism to pull together the teams, pull together the expertise and capabilities that we have within IBM, and so that’s our team. And so in some ways we’re researchers and we do user research, right. We bring that outside/in perspective. But we’re also service designers in that we’re really designing how the full system in an organization works.
Steve:I feel like there’s another facet to what you’re doing though, and maybe that just falls under the umbrella of service design, but creating this deep understanding which informs this plan – plan is kind of the word I keep throwing into this. But also, you know, serving as – being in dialogue with the sort of teams that are doing a little more – the deep requirements, it sounds like that’s a little more day to day. That’s the detailed view versus the overarching view which you’re capturing. I guess this speaks to the complexity and the scope of the kinds of projects you undertake. It’s interesting for me to think about this overarching layer of understanding that’s kind of crafted that is not about what are sort of the finishes going to be on the details of interactions, but it’s really just this big view.
Ashley:Yup.
Steve:That’s really interesting. And so – I’m curious about what I think goes with sort of the scope that you’re talking about is some time horizons that must be interesting. If there’s a certain effort to make a plan and then a certain effort to build and support and roll out things – I don’t know, is there a typical time that you’re thinking about when you work on these programs?
Ashley:That’s a great question. So, this is a fairly new effort. You know we’ve been working with this customer journey and user need centric way of thinking about the future of IBM for about let’s say 2 years, or less. And so there was a phase where we were really framing like what does this look like? How do we talk about it? What are the words? What’s the right language to use? What is the relationship between a user, a team, a journey, a product, an IBMer? We were establishing that frame, building on the practice of service design. And then there’s a phase of piloting that’s kind of the phase that we’re in now. So, you might have a great frame, but how does it get really built into the DNA of the way that the company works. We found that it’s really important to pilot work with particular business units, understand are our methods working, does the frame hold up and do we know about – enough about the way that the business works yet in order to say yes we can deliver this way of working. So, we’ve been on our own journey to do the research, the practicing to finish the frame. And then there’s a future effort to scale. Right. How do we get all of IBM working like this? And so that’s really – that’s kind of the plan. Right. And it requires a lot of influence – stakeholder engagement, a lot of consensus building and alignment. And so that is also the work that our team does.
Steve:As you talk it’s clearer to me that there’s a significant focus on the organization of IBM and how IBM does things. It sounds like that’s where a lot of the effort is right now.
Ashley:Um-hmm.
Steve:Even though ultimately this is outside focused on customers that IBM is supporting, you’re trying to – you are building new practices in terms of how IBM does that.
Ashley:Exactly.
Steve:So your emphasis right now is on, at least a significant part is, on how IBM does the work that it’s been doing in terms of building things for customers.
Ashley:Exactly and I think that’s critical – that’s a critical place to focus, right, because we could generate all the understanding of user needs that we’re able, but how do we build that into the way that everyone thinks and how do we go to market in a way that actually is accurate to our understanding? And so that really requires organizational change in some ways and this work, right, this journey work is building on the design program that Phil Gilbert has been leading over the past 5 to 6 years. So, he came to IBM and said hey, Ginni (our CEO), design is critical to the future of IBM. We need to focus on user needs. And so as his program has scaled we’ve started to see a focus, not only from designers, but from other IBMers, right – engineers, product managers, executives – in adopting design thinking as a way to a) focus on user needs, but also work better cross-functionally across our business. And so this work that we’re doing to really drive customers journeys is building on that. And so we leverage the 2,000+ designers across IBM and we’re really disseminating these methods to them, building on the design practice that we already have.
Steve:So, is that an example? Because you mentioned influenced being a major part of what this team is looking at right now. What are some practice for influence that you’ve found successful here, or that you’re exploring here?
Ashley:Yeah, I think being in a company that loves numbers, I think that that’s a reality. I think what we’ve seen is that qualitative research is really important, but it’s actually not always enough. And so I’ve been really thinking about what does a mixed methods practice look like and how do we bring a lens of both qualitative and quantitative that at an executive stakeholder level is like – stands up to snuff. Can we scale the number of data points that we have to help show the size? or impact of a problem that we see? And can we then talk about its qualitative nature to help us look for solutions to solve those issues?
Steve:So is that – I feel like I get into versions of this discussion with people all the time around influence and the kinds of things that you’re talking about. And where one thing to do is speak their language. Another thing to do is to – I hate the verb educate because I think it’s so patronizing – I don’t know, to empower people with the mindset of your own language. I mean you see really bad practices where people may do a very small ethnographic study, but they’ll – like very, very small, but they’ll quote percentages. Or they’ll say 3 out of 8 or something like that as a way to make quantitative research a peer quantitative, which I get the impulse to do that…
Ashley:Yes.
Steve:…because it’s sort of speaking somebody’s language, but is also doesn’t help that person because they’re going to misinterpret – you’re misrepresenting and it invites misinterpretation.
Ashley:Right. I think it’s much more about showing the scale of the need that we’re maybe not fulfilling yet as a company, or the scale of a problem that exists where two parts of our business are not aligned, or you know, there’s a gap in product/market fit, right. So, I think in the mixed methods practice that we’re thinking about it’s much more about like understanding from a data perspective what’s happening across the business? What patterns do we see and behavior? And then laying on top of that, or infusing that with some of the qualitative insights that we get by talking to customers, right. Talking to people that are in the space that we’re working, Cloud and cognitive capabilities, and bringing those perspectives together. So, it’s less about like okay we have a number and it’s more about we have looked across, we’ve understood from what thousands of people are doing, not just 3 to 5.
Steve:So, is that the starting point and the qualitative is the supplementary? Is that kind of the model?
Ashley:They run in parallel. So, on my team I have what we call design researchers that are really practicing user research and service design. We also have data scientists on our team. And so we are kind of running different workstreams, right. Let’s say we’re working on a journey of a client, modernizing their infrastructure to a containerized environment, right. So, we’ve taken that frame as the journey. The data scientists then go to the systems of record and they use that frame to understand all that they can. And they’re really creative. I really want to emphasize that. Right. We have a lot of historical data as a company. How can we best leverage that to inform our future? So, they’re like getting access to support data, to financial data, and really putting together a picture of how our business has run and what behaviors they see. The qualitative researchers are then talking to customers, talking to people on UserTesting.com or using Respondent to conduct moderated interviews and understand what are the needs – what are the business needs that organizations have and then we bring those together. So, that’s kind of an exciting moment to start to see like what patterns are happening across and where can we derive greater insight by bringing those two lenses together.
Steve:How do you create the conditions where these stereotypically different mindsets, different methods – you know how do you create conditions where the gestalt of what both those uncover can be kind of extracted, I guess?
Ashley:Yeah, that’s a great question. I think what we’re finding is that it’s all about making it physical or visual, right. So, we have a collaboration space where we just put everything on the walls – the war room, right. Take what the data scientists have found. Let’s talk through it. Let’s try to understand it even if we don’t have the expertise of the statistical figures. We then put the qualitative insights on the wall, right. And so we’re mapping. We’re doing a lot of mapping – systems mapping. We’re doing a lot of like pain point gathering and making it all visual, making it almost tactile. It helps us to put that picture together, the fuller picture together.
Steve:And this is both groups are doing this together?
Ashley:Right. Well, I should say that is the practice that we’re growing. I think it’s still forming, but yeah, that’s my vision is that we bring all of the insights that we have. We really work through it together and we’re stronger and better in the way that we can tell the business where to go next based on those two influences.
Steve:So there’s a lot of emergence, I guess. You are defining practices that are new to this part of IBM, or new to IBM maybe, or new to sort of the field overall of all the fields that you’re kind of pulling together?
Ashley:Um-hmm.
Steve:What’s your approach to trying to innovate and process? You’re making new things in the way that you’re working. Where’s that coming from?
Ashley:It’s all in pursuit of a customer journey that is excellent, right. We really try to keep that at the center, right. What problem are we solving? We’re solving for our client’s business needs, or our future client’s business needs. So, we keep that at the center, but we are constantly making. We are constantly trying methods, iterating on them, seeing can we get to a picture that is compelling for the people that are making really important decisions for our business’s strategy. And so it’s a constant evolution and I think what’s exciting is when we get momentum or we get resonance – we reach a point of resonance. And whenever we feel that we run with it. And so I think there’s a certain intuition to the work, right, to say okay this is compelling. Like how can we amplify an insight that we’ve had or an artifact that we’ve made and how can we start to bring that into a standard practice over time?
Steve:You’re obviously being reflective about the practice. You’re doing the work, but then you’re also looking at let’s think about the best way to do this work and let’s change that.
Ashley:Absolutely, yeah. I think for us, what we’ve done is we’ve had to make time and make space for that reflection. So, now every week we have like a huge block of time on everyone’s calendar and we all get together and talk through, work through the data that we’ve gathered, synthesize together – that’s critical, right. Synthesizing just the qualitative research in Excel on your laptop is not the best way. It’s about us all thinking together through a synthesis process and that’s how we get to those moments of resonance.
Steve:There’s a couple of things. I think you’re making time work together on the work, but you’re also making time to reflect on how you’re working.
Ashley:Yes, yes. And so – we actually work in agile sprints which I know can be challenging for research, but it helps us to have a cadence of working, doing the work, reflecting on the work and pivoting. And so I think our leader, Sarah Brooks, she leads journeys at IBM. She’s a distinguished designer that’s really leading this mission and she’s been really intentional in making sure that we know the impact of the work that we’re doing, that we’re properly socializing and influencing the rest of the organization and that also we’re really tight on the methods and practices that we employ.
Steve: Can we go back to what you were saying about doing synthesis together?
Ashley: Um-hmm.
Steve:And kind of creating those moments of resonance, I guess. You said that Excel on your laptop by yourself was not really the way to go. I mean I agree with you and I’d just love to hear you like make the case for why – for what happens when people are together? I’d love to hear more about what that feels like and how it’s valuable?
Ashley:Yeah. I think – I’ve worked on a – embedded in a product team as a research lead, but leading by myself and I think, you know, you’re going to – one researcher, one person is going to have a certain interpretation of the data that they find. But when you bring in another person you’re always going to get a slightly different take. You’re going to get a different set of assumptions. You’re just going to have a more diverse, right – even bringing a second researcher in, you’re always going to have a more diverse take on the data that you found and so I think doing synthesis together – there’s two aspects to getting out of Excel. One is that diverse set of perspectives, right. The second is that when you’re in your computer you’re not thinking in the same way that you are on a whiteboard, right. And this is the premise of design thinking, right, externalizing the data that you’ve found. Externalizing your thought process, just helps you move faster and clearer. And so I think getting out of Excel is really about those two things and that’s something that we’ve been really working on in our team.
Steve:It just reminds me of this thing – it’s a thing I remember from being a little kid. I don’t know if this is unique to me, or just general for everybody, but, like being stuck on something in the classroom and like going to stand next to the teacher’s desk to ask a question and by the time I could formulate the question I’d figured it out. Or even if I’d verbalize it I was like oh now I know what it is. It seems like the externalizing process is a sense-making activity, I think.
Ashley:Um-hmm. Even to be able to communicate, right, a data point that you found, or tell a story, right. That’s a synthesis process. That’s a cognitive process. And so I think bringing that out of a person just helps us each grow in the way that we communicate and understand what we found.
Steve:When you talk about just even adding another researcher that’s going to have a different perspective on something – I mean I agree, but I also hear the things that make people anxious about qualitative research. Like you’re sort of celebrating the – it’s an aspect of uncertainty or – right, we struggle with I think being perceived as rigorous.
Ashley:Yeah.
Steve:And when you say well if someone else is going to come in the room they’re going to see something differently than I am, but that’s a strength. So, someone might hear that and say that’s a weakness because how do we know what’s real? Or how do we know what’s true if you can’t even see the same thing as somebody else.
Ashley:Um-hmm. I think too it’s less about facts. It’s more about like what’s important in the data. I think the synthesis process is a lot about prioritization and coding or articulation of what’s important. And so I think what’s important to one researcher might be different than what’s important to another. So, it’s less about like my truth/your truth. It’s more about what’s important to our team user needs and then what’s important to the business, right. Ultimately that’s customer lifetime value and meeting our business goals. So, in getting out of Excel, working together, we can all talk more about like what’s really important and that’s only going to make the work better.
Steve:I feel like this ties a little bit to the influence part of our conversation as well ‘cuz you are running things in a way that I think – there’s definitely struggle in some teams to do this. To have more than one person. To take the time to reflect on their process. To devote the time to synthesis that it requires, as opposed to kind of sitting down at your laptop and kind of pushing out the next set of recommendations. Are there elements or components of the practice that you have been building that we’ve been talking about that you think could be even adopted in small parts by somebody in a different kind organization?
Ashley:Hmm. That’s interesting. Um, I should also say this is the practice that we aspire to execute on, right. It doesn’t always happen every study, every day. That’s where we aspire to go, but I think one of the special things about the work that we do and the way that we understand customer journeys is that we really are taking a holistic view and when you start to map the whole of what users need, how they interact with us, our relationships with them, and then also what IBMers need, right, it’s kind of hard to unsee once you see it. Once you see how two IBMers are disconnected or their incentives aren’t aligned and you see how that surfaces in the user experience you kind of can’t unsee it. And so I think although I lead the design research practice, I think there’s an element of expansion of design, to this like service design frame, or even like a business model frame, that I think we’ve gotten a lot of value out of, right. So, user research is really important, but it is a foundation to a larger understanding. And so I think other organizations could benefit from that, right, in adopting a more full picture of how user research and how user experience plays within a larger context.
Steve:And you brought it back too to this, in some ways your first customer is the organization of IBM. Even though IBM is focused on its customers as kind of core to its business, you’re serving an internal – I mean everything. You’re serving everything about IBM.
Ashley:Um-hmm.
Steve:And you know that’s different. And then you mentioned sort of the figuring out how to go from New York to California. In those kinds of organizations they talk about being customer obsessed and the researchers and designers are thinking about improvements for the customer, which is also a great thing for a business to be thinking about.
Ashley:Right.
Steve:But it’s different than can we make what we do here as a company better for us as a company versus can we make a better product for our customers. Success is going to be measured in different ways in those organizations.
Ashley:Right and so we are designing – we call it orchestrating and also measuring the whole customer journey and one question we have to ask ourselves is like where do we sit in that customer journey because we’re part of it, right? Backstage. We’re somewhere and I think we’re actually pretty far backstage, right, because our first line of user is the IBMer. It’s then the client organization, but then our client organizations also have users. So, there’s this spectrum of the definition of user and we have to figure out like who’s the priority among that set of users? And where is it best to focus and that might shift over time. Maybe initially we’re designing and establishing practices for IBMers, but once those are set how can we push further towards more of a human centered perspective? And ultimately that’s getting to deliver an experience that helps that end, end user and their needs.
Steve:I’m sure there are some visuals that go with this.
Ashley:There are some big maps. I think it’s also best to give an example of this, right. So, let’s say we’re designing a journey engine modernizing to a containerized Cloud environment, right. Why – we have to understand like why are people – why are organizations doing that? Why are they modernizing? What we found is a lot of it is about driving better – their own better customer experience, their own better user experience. A faster app, right – iPhone app. A better, right, way to search – for their users to search within their business. And so it’s funny that we’re going through our transformation, our client organizations are going through their transformation and ultimately that comes down to a person using a consumer thing. So, when I pick up my banking app and I check my savings account, right, that’s ultimately coming back to something that IBM is probably involved in. And so how do we keep that experience in mind, that very human, very personal experience of knowing how much I have saved for my future and how do we then comb through the technologies and capabilities and relationships and politics that affect that? There is a great breadth and depth to the work that we do, I guess is what I’m saying.
Steve:So, if this person with the bank account is going to have a better experience the company that – that bank needs to have processes and tools and technology to enable them to be more contemporary in how they make products and services which means someone like IBM can help them understand how to go about changing those processes, changing their tools, changing the infrastructure, so that they can then go make that app. Which means that you and your team have to, you know, build processes and practices that can help IBM help that bank build better kinds of apps to help people have better experiences.
Ashley:Right.
Steve:I just said what you said, but I started going in the other direction.
Ashley:I like that you mapped it back to the other direction and I think as we can better serve our clients, right, we can help them get to a level where they can innovate. So, ultimately, right, the work that we do is about moving towards the future, right, with shepherding new technologies into the world, shepherding new practices, advising, guiding, and ultimately in pursuit of something that’s ideally life changing for someone.
Steve:That’s great. Can I go back to something else that you said before and maybe pivot from that?
Ashley:Yeah.
Steve:You’re talking about that moment of a researcher brings another researcher in to talk about some data points, or some qualitative research. And that this diversity of thought is a – starts to add value because people look at things a different way. You know – the word diversity is being used to represent a lot of different kinds of things now and I guess I’m wondering if you can talk a little about diversity overall in the design research field? What are you seeing or thinking about right now?
Ashley:Yeah. I think that’s definitely something I’m thinking about, my boss is thinking about and IBM is thinking about. From a research perspective, I think there’s a lot to be desired in terms of having a diverse representation in who is practicing it. And I think that’s really important if you’re talking about a new technology, Cloud or cognitive capability, AI, right – we need to have people that have different perspectives in the room, right. If we’re creating a data model and it’s going to affect the way that someone’s bank account, right, or their financial service – it’s really important to have people from different economic backgrounds, or from different contexts, that can understand what are the pressures, what are the needs, what are the pain points of different types of economic situations. Right. So, that’s just one example of where I think we could grow. I think when I look at researchers, right, the wonderful research community, I don’t see a ton of people that look like me and so even by talking to you today I have a hope that we’re growing and that we’ll continue to see more diverse faces, diverse ways of thinking and diverse backgrounds represented in the field.
I think too, a second point, research has traditionally been like somewhat of an academic thing, right. It was something that PhDs did, right, and maybe other folks that didn’t have a master’s degree even were kind of shut out. But I think we’re seeing a shift there. I think we’re seeing people go through things like General Assembly, or self-taught, really thriving in the research community. I myself have a background in architecture and this crazy MFA program called transdisciplinary design. And so although I picked up research practices throughout both of those educational experiences, I think I have somewhat of an untraditional background in the fact that I’m leading, right, a team of researchers and able to really maybe bring more of a generalist perspective to the practice and bring the things that I’ve learned in architecture, from systems thinking and mapping and other practices. It just only helps enrich the field and enrich the work that we do.
Steve:I do want to ask more about your background, but let’s keep talking about what you see when you look at the community of research, or what you don’t see, I guess, more importantly. Right. Let’s just imagine someone is listening to this podcast who’s kind of on the outside of the practice, but is interested in it. Do you have suggestions for them if they also are not from the groups that are currently sort of dominating the population of research?
Ashley:Yeah. Um-hmm.
Steve:What do you tell them? Or what do I tell them? What do we tell them?
Ashley:I think one of the key things to any practice, right, or field is language. So, I think a lot of what we do is about having the right language. It’s having a mental model of people, behavior, psychology, right, cognitive processing, kind of understanding that. So, I think if you can kind of just pick up a foundation of understanding of those things. Like how do we talk about user problems? How do we talk about user needs? How do we talk about motivations? If you can kind of pick up that sphere of things, I think it’s easier to move into the space, right. So, for me methods are important, but methods can be learned. And so I think focusing on the mindset and the perspective that we bring when we sit at a table of, you know, various stakeholders, right, we have to bring that user lens and that’s the most important thing that we do. It’s not about are you the best interviewer? Are you the best person to like write the survey? Because someone can always help you with that. But it’s about do you have the right mindset and do you speak up and bring the voice of the user to whatever context that you’re working in? And if you can communicate that and start to build off of that, I think you can have a viable career.
Steve:Good.
Ashley:Would you agree with that? I’m curious.
Steve:I mean I think your point about mindset vs. methods is really – if I think about just my earliest days in the field and trying to hire people – I mean there weren’t people – it just was sort of underpopulated practice. Whereas now I think there are sort of new people coming in all the time. And I remember struggling with how do we articulate someone who would be good to join our team? I worked at this agency and we used to just talk about people who got it. I mean that might – in 2019 that might be a trigger phrase for sort of exclusionary thinking.
Ashley:Yeah.
Steve:I think what we were trying to say then is the same as mindset. I think we were thinking about that there’s a frame of reference or just a common language around how you think about what we’re here to do. I was at a product design consultancy and, you know, you’re there to sort of serve the client, but also serve the – we were doing research and trying to help the company serve their customers even though our customers were the company making the things. So, there was some advocacy and evangelism and facilitation and sort of understanding hey, here’s where people are at, so what are the opportunities to sort of help them, or empower them, or do something for them, that I don’t think we had a lot of clear language around. It was sort of new to the practice or new for us. It was a long time ago and we were relatively isolated at that point. Yeah, I think – I agree. We’ve had the conversation many, many times about you can learn methods, but there’s this other thing that’s sort of harder to teach. And so I think your advice is kind of hey, get that thing.
Ashley:Yeah.
Steve:That’s the kind of thing that – so, how does someone demonstrate that they have this thing, that they have the mindset?
Ashley:Ooh, I think write about it. Those have been some of the best moments in my career so far where I worked through a project and then I went and wrote about it. Because that’s a beautiful reflection process – what was important, how should I talk about this, right? You might have had one conversation in your head, or with your team when you were doing the work, but when you reflect back on it, like what stood out? What were the major like learnings? What were the moments that really mattered? And I think writing about maybe a project that you’ve done, or how you would like to do the work, could be a really great way to yeah, just get a stronger hold on your own practice. So, portfolio building has been – I think is a great practice. Maybe not to get a new job, right.
Steve:Right.
Ashley:Not just to get a new job, or get a promotion, but just as a reflection practice. And that’s something that we encourage here at IBM. We talk about this thing, the portfolio of experiences. So, it’s not necessarily like oh here is the beautiful app that I designed, but the portfolio of experiences talks about here are the range of problems that I worked on. Here’s what I learned along the way. Here’s how I grew to be a better researcher, designer, visual designer, facilitator. Right. Here’s how I learned to talk to executive stakeholders. Here’s how I influenced them. And that’s so important. That’s so important. And so I know it’s hard. When you’re just starting out you might not have real projects. I hear that a lot. I get that in my LinkedIn – how do I get started if I haven’t worked in the field yet? Write, talk about what maybe you would do. Make up a project. Do a General Assembly course and get some experience and reflect on it.
Steve:I like portfolio, almost as a verb the way you’re talking about it and less about the noun. So, that’s developing the mindset by – so it’s less about this artifact that you’ve created and more a way of kind of thinking and communicating. And then you have stories to share and stories to tell with people that you meet.
So, we started off this thread of the conversation by talking about – you didn’t use the phrase underrepresented, but I think that’s an element of what you’re talking about. But, I guess, as often happens when we talk about increasing accessibility of anything, like this is good advice for everybody, right – regardless of your privilege or your representativeness. I don’t know, is there anything else there for people that are not represented, or are not in the sort of majority of what the field looks like. You said speak up. That was one sort of thing that…
Ashley:Yeah, speak up, have a point of view. But I think ultimately it’s on the leadership of, you know – whoever is leading a team. The responsibility is on us. It’s on me to, when I’m hiring, saying alright I posted this in this platform, I only got these types of people applying, I need to look somewhere else. And I need to consider the full range of people, but I need to make sure that I’m looking for people that aren’t currently represented on the team, whether that’s – you know we call it URM (unrepresented minority) in the HR system. So, it’s like women, people that are black, African American, Latino, Latinx, people that have accessibility needs. That is like – that’s really critical. I mean the work that we do, accessibility is so important in, in, in this space. And so having people that have that experience, are like why wouldn’t we? Why wouldn’t we have them? And we’re working on that. And then also people that identify as LGBT or even people that are veterans, right. Those are some of the groups that you want to really look for and prioritize and there are organizations out there that cater to those groups, reach out to them. Go to those communities, right, and that’s how you’re going to find those people. There’s this like concept of a pipeline issue and sure, but right – let’s say you wanted to hire more black designers, that’s something that is really important to me. Go to historically black colleges and universities, right. Partner with the leaders in that organization. Make sure that the students in that school are getting the education that will result in skills that apply to the work that we do, right. There’s a lot of work that we can do that’s not at the hiring stage, but just kind of working a little – just a few years back, right. And we’ll radically, radically change the quality of the skills that the students have coming out of those institutions and radically change the quality of our organization. That’s just one thing you can do, right, that will be really impactful. And so I think – I see that growing at IBM and as a leader I want to be part of that. I want to do that. And it’s not that hard, you know. So, let’s do it.
Steve:So, there’s – I think about sort of these disciplines, maybe nested within each other, that you have – the tech industry and its challenges with – you’ve given very good practical advice here, but, it’s challenges with hiring underrepresented minorities. And then you have kind of UX or design within the tech industry. And then you have design research within that. I think sometimes, and maybe it’s the same thing across all, but I think sometimes we – like the research practice, when we have these messier things come up – like oh, we should be talking about ethics, that I fear sometimes that design research just says well here – defaults to the UX conversation or defaults to the tech conversation. And I’m thinking about, again, diversity and inclusion in hiring, what our community looks like, do we have specific challenges or specific opportunities, or either way, within design research that we should be thinking about? Is there something unique for us as a field that we should be thinking about?
Ashley:For sure. I think maybe you’re gesturing to the practice, like within doing the actual work.
Steve:I wasn’t, but if that’s where it goes that’s fine.
Ashley:Even in we’re going to do 30 interviews – who are we interviewing and what perspective are they bringing? If they’re talking about the challenges that a user experiences in their day to day is that different for men and women? Is that different for somebody who has full vision vs. like a low vision need? So, I think even just starting there. That’s hard. That’s challenging. It’s hard enough to get to – in IBM as researchers to get to users, our customers, right. That’s an initial challenge, but then to make sure that the people that we talk to are a diverse – represent a diverse set of experiences and needs, you know that’s something we could do better. That’s something we can prioritize and talk about and cite. And so I think that’s one way. But, maybe rearticulate your question so I could answer it more directly.
Steve:There’s a tech challenge, that tech isn’t diverse enough. Tech says we have a pipeline problem and they’re not going to historically black colleges and universities to recruit – sort of the thing that you described. If you look at what tech is doing as an industry. If you look at what design is doing – I mean because those are sort of bigger entities with a lot more voice around here’s the challenges, here’s how we’re failing. I mean design research is a smaller practice. Maybe we have – like it’s a practice that my anecdotal data says it’s more women than men. So, sometimes – I don’t mean to imply that’s sufficient diversity. I mean it just – it inverts the sort of male dominance to a lot of tech. So, maybe that feels good. As a man in the field that’s kind of like oh cool, I’m in a non-traditional kind of thing. It doesn’t’ mean that I think we’ve achieved the kind of diversity that you’re talking about. But because people in design research come from different places than say software engineers do, because the field offers certain kinds of experiences and pulls on certain kinds of creative mindsets that are different – because we do different work. Like there’s something about research that – like the work we do is about telling the stories of other people who are different than us which maybe you could use to justify not being diverse because our practice is to sort of – or maybe you would use that to say well then we especially better be diverse because of it.
Ashley:Right. I think there is something privileged to design research as a practice. I think especially for myself coming out of grad school, or going through grad school, going to get an MFA was like kind of a privileged thing, right. Become an engineer or a lawyer or a doctor, it’s kind of more certain, right. If you’re investing however many hundreds of thousands of dollars in your education, I think for certain groups, maybe socioeconomically, from certain contexts, right, you kind of want to go with something that’s certain, right. So, I think maybe there’s something to the way that design researchers are traditionally educated that’s a little inaccessible. And I see that changing, but maybe there’s something to that and we can pick up on it and amplify less traditional or expensive ways of becoming ethnographers, design researchers, UX researchers. I think there’s something there that I’ve definitely had conversations about with people. Um, and again, even coming out of grad school, my graduate program, transdisciplinary design at Parsons was 5 years old when I graduated and that was the first year that like people really got jobs. So, you know, working in messy spaces, talking about soft things like feelings, behaviors, is a less certain space to be in. The outcomes are less certain and less rigid and so I think maybe that has something to do with it. And if it is, if we find that, maybe that’s an area that we can like probe in and find a solution.
Steve:Good. These are some good signals I think to toy with. Can you talk a little bit about this program? Like how did you find it or what kind of drew you to it?
Ashley:Yeah, okay. So, I went to Howard University – the illustrious Howard University, as we call it, a historically black university – and studied architecture. I got a professional Bachelor of Architecture. It’s a 5 year degree and by the 4th year I was like absolutely not. Very cool. I loved concepting. I loved research. I loved setting contexts, like okay what is this neighborhood like? How does that influence what this building should be? I love that part of it. I love the concepting. And then the visualization, like the diagramming. And, um, but did I want to go build buildings? And like architects live really difficult lives? They work really hard, right. Um, um, so I said I love some of the things here, how can I pivot or like open up the lens of problems that I want to solve? And so I went and got this graduate degree from Parsons and I only found it because I went to go look at a fashion program and I looked down the list of programs and it was at the bottom. I was like transdisciplinary design, what is that? That sounds interesting. So I went in and met with the director of the program and I was like, oh, this is a thing. Like solving complex problems, talking to people, understanding systems and behaviors in spaces. And I was like oh yeah, I think I’m going to do this. So, that program is really built on the premise that our world is so complex, we have these wicked problems that one discipline can’t solve. And so the practice of transdisciplinary design, if it is a thing, is about bringing diverse sets of mindsets, but also methods, together, and using design thinking kind of as the framing for bringing that cross collaboration and synthesis of those mindsets. So, that’s what I went to do at IBM and it turned out to be a really useful education.
Steve:Did you go to IBM from that program?
Ashley:Yes. So, I’ve been at IBM for about 4 years now.
Steve:The way you described the program – I guess it’s no coincidence the way you describe what you’re doing at IBM and the way you describe what that program’s aims are seem to have quite a bit of coherence.
Ashley:Right, right. And finishing up that program I had no idea that I would work at IBM. I didn’t think about that. But when I went and I talked to the folks that were hiring at the time, I was like oh wow, there is a need for this and it wasn’t exactly the context that I thought, but it’s been a really great place to learn and practice and then begin to lead.
Steve:I guess hindsight has always idealized things in retrospect, but just kind of the way you described it, it seems like the experience at Howard told you some of what you didn’t want to do, but also opened you up to see this transdisciplinary design program. Like you were at a point of sort of creative self-discovery that this filled – this was a thing that you didn’t know existed that then was right for you right at that time.
Ashley:Exactly. And even while I was at Howard, I mean you could imagine it’s a place of social innovation and like you know the student body there is very much engaged in politics and systems and so that was already there, right. That was already something I was thinking about. But then to move into a program where like that was the basis, the foundation, that was the work that I would do, it was like oh, this is obvious. And so I think as you – as I have gone from place to place and learning and growing, I like to follow those threads, right. Okay, systems thinking, mapping, social innovation. Right now it’s about this customer journey, service design, right. You can kind of follow those threads from place to place and build on your interests while building on your expertise and I think that’s something else that I would tell folks that are looking to get into the field, like build on something you already have. If you were a teacher, maybe become a design researcher in the education space and that’s another way to break in and to, yeah, just build on what you already have.
Steve:Do you ever imagine a future for yourself? Think about whether it’s 5 years or 10 years – I hate it to be where do you see yourself in 10 years. I don’t mean it that way.
Ashley:Oh, terrifying.
Steve:You know on these kind of – you’ve made these interesting, I don’t know if they felt like leaps – you don’t present them as leaps, but these zigs and zags I think that sort of came at the right time. If you think about a future for yourself, what kinds of work might you be doing? Or what kinds of roles or organizations do you think are there for you?
Ashley:I think it’s all about mission for me. And that’s why I was so excited to join Sarah Brooks and the journeys team and lead research. But I think it’s all about missions, like I like the big messy, unsolvable problems and maybe that’s like nature, maybe it’s nurture. But I think it’s all about that. So, maybe it’s the criminal justice system. Maybe it’s food justice. Maybe it’s design research diversity. I think anything that’s like hard and interesting and you don’t quite know the answer is something that I want to do. And so I think, if I look 5 or 10 years out, maybe I’m at IBM still doing that. Or maybe I’ve learned enough that I am going out and I want to like focus on a problem. So, I don’t know what it looks like, but I know it’s a mission. It’s always a mission.
Steve:Anything you want to plug?
Ashley:Yeah. So, if you’re interested in IBM and practicing design, or even just learning about how we talk about design and the work that we do, you can go to IBM.com/design and that will kind of launch you into our world. You can also reach out to me. I’m @ashleyograham on Twitter. And thank you.
Steve:Well thanks for a really great conversation. We explored a lot of different nooks and crannies and I think it’s fascinating. So, thanks a lot for taking the time.
Ashley:Thanks, Steve.
Steve: Thanks for being here for this episode. You can find Dollars to Donuts on Apple Podcasts, and also Pocket Casts, Castbox, and Overcast. If it has “cast” in the name, we’re probably there. Go to the website – Portigal dot com slash podcast – to get all the episodes with show notes and transcripts. Our theme music is by Bruce Todd.
The post 24. Ashley Graham of IBM first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Michele Marut who leads user experience research at CBRE Build. We discuss the curation of research repositories, using research to go beyond fixing things, and building processes and tools that can be used by researchers and people who do research.
The philosophy is that the trained researchers should be taking on the most critical, the most risky projects. That’s where they can add the most value. Are they going to lose a lot of money? Is this a totally new workflow? Is this really new to the world? So, really focus trained researchers in that area. – Michele Marut
Show Links* Medical Storytelling at the VA Bridges Gap Between Patients And Caregivers * My Life, My Story: Advancing the Veteran Experience * Michele on LinkedIn * Michele on Twitter * CBRE * CBRE Build * WTF is Proptech? * Zillow * Airtable * Clubhouse * Tomer Sharon on Dollars to Donuts * Democratizing UX (and Polaris) * WeWork * Optimal Sort * M.S. in Human Environment Relations, Cornell * Kohler Company * Consumer Products Safety Commission * OXO Good Grips * MVP (Minimum Viable Product) * UXPA (User Experience Professionals Association) * IxDA (Interaction Design Association)
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Thanks for joining me on Dollars to Donuts, the podcast where I talk with people who lead user research in their organization.
As a consultant, I find myself collaborating with very different types of organizations in terms of the amount of experience they have in doing user research, or learning from user research, or acting on what we learn from doing user research. There might be strong leadership in user experience design, or product design, or service design…or that might be a completely foreign concept. That’s a significant challenge in my work as a consultant, ensuring that I’m in a position to assess and respond to that diversity. It’s also something I really enjoy, and I see working on this podcast as part of that journey, something that I’m able to share with you. And so, the best way to support this podcast is to support my own consulting business. You can hire me to lead user research projects or to coach your own team as you talk to users. I help organizations put together research roadmaps so they can prioritize their limited resources. And I run training workshops to level up fieldwork and analysis skills. Please get in touch to see what we might do together.
Otherwise, if you have feedback about this podcast, email me at DONUTS AT PORTIGAL DOT COM or write me on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
I was intrigued by a story on NPR last month about program created at a VA hospital in Madison, Wisconsin, called “My Life, My Story.” In this program, staff and trained volunteers conduct open-ended interviews with veterans about their lives, letting the patient decide what they want to share about themselves. The interview is turned into a thousand-word biography that the patient reviews and revises, and then is included in their medical record. Patients also can share these stories with friends and family. This particular VA hospital has been training other hospitals in the VA system over the past few years. The article describes the benefit to the patient, and the different caregivers, both in the act of telling the story, and in having this kind of softer information available for review in their medical record. The article also explains the evolution of the “My Life, My Story” program, and how they “tried having patients fill out surveys, which were useful but still left the team wanting more. Next, they tried getting patients to write down their life stories themselves, but not many people really wanted to. Finally, an epiphany: Hire a writer to interview the patients, and put what they learned on paper.”
This is a primordial form of user research; there’s no sense-making of patterns across groups of users; the data gathered from a user of the system only has value when applied directly to that user’s specific experience. I mean, maybe there’s a cultural shift that comes from making this kind of information available in the medical record itself, maybe it invites providers to be hungry for this kind of data, maybe it changes the conversations internally around patients as people with rich and messy lives beyond their medical conditions. And I want to dismiss it as unscaleable – this idea of interviewing every single user, rather than a sample, but – look at the implementation here – it’s absolutely about operationalizing it so that this service, asking open-ended questions about a patient’s life, is a component of the experience for each patient.
In some ways, it’s a bit threatening to user researchers, that the simple act of just asking people about themselves can benefit the delivery of services so significantly, but on the other hand it also suggests that the potential value researchers can bring is well beyond what we usually say it is. If you’re looking towards the future of user research, this may be an early signal for possible directions.
Let’s get to the interview, with Michele Marut, who is leading user experience research for CBRE Build in New York city. Thank you for being on the podcast.
Michele Marut: Happy to be here, Steve.
Steve: Let’s start with an introduction. Who are you? What do you do?
Michele: My name is Michele Marut. I am currently a lead UX researcher at CBRE and I’m working to build up a research center of excellence in the sales office.
Steve: What kind of business is CBRE?
Michele: CBRE does commercial real estate. You’ve probably seen their names on buildings, especially if you’re in bigger cities. But if you’re doing retail or industrial and logistics types of work you might see them. And if you’re searching for office space you might have encountered them from a broker and advisory perspective.
Steve: What part of CBRE are you working in?
Michele: I’m working with a group called CBRE Build and it sprung from an acquisition that happened approximately 2 years ago when they acquired a company named Floored that specialized in 2D and 3D visualization tools that were really helping brokers to help sell their buildings and spaces. And so those tools are still in existence, but since then my manager has been tasked with creating more of an innovation group and they’ve created other tools that complement the original 2D and 3D tools, as well as tools that help people calculate and anticipate space that they might need. Tools that help work with test fits and that can be used by people beyond brokers and landlords and more for helping you really figure out the right space for your needs. It’s still focused on commercial real estate.
Steve: So, is this kind of a product and tools part of a larger real estate organization?
Michele: It’s a digital and technology section of a larger company that’s focused on commercial real estate. A fun word to talk about is proptech, or property tech. We are very much in the property and proptech space if you see that. Some people might think it’s a buzzword, but it’s what’s happening with other parts of the industry as you’ve seen other groups – financial went through this, health care is going through this, even consumer – commercial real estate, as well as other parts, even in your home and consumer real estate is finally getting more interactive. It’s more tech savvy. And so, some of these things are finally coming to commercial real estate.
Steve: I think the New York Times keeps writing about how companies like Zillow will now buy your house as well as tell you what it’s worth and there’s some technology that makes that happen.
Michele: Yes, yes.
Steve: So, is that proptech?
Michele: That’s proptech. Proptech can be consumer, if you’re looking for a house or an apartment. And it’s also in commercial real estate. So, it’s all different things. Anything that helps take data, technology and information and make it more innovative and probably disrupting the status quo of what you’ve probably been buying and selling real estate for hundreds of years the same way. We’re kind of at that inflection point for tech now, for real estate.
Steve: Okay. That’s great context. And then you mentioned center of excellence? Is that right?
Michele: Yeah.
Steve: What does that mean?
Michele: So, some of the centers of excellence are business terms that have been around. Here what we’re trying to do is one, build a group so there’s more researchers. There was only, I think, a researcher who had come from support. Before I joined there was one or two people in the other parts. We’re certainly adding more trained and skilled researchers as part 1, literally building the team. Part 2 we are creating templates and processes so that anybody, whether they’re a trained researcher or a person who does research (PwDR) can actually leverage some of our tools. They can get started very quickly. They can find the right participants. They can use our templates. They can use interview guides. And they can probably execute a basic research plan on their own. And then the third component is understanding who else in the company is doing research, making sure that we’re connecting them and building this community so that we can continue to grow our research center of practice, but also to share like if we’re actually talking to brokers and the same people across the company. We can share what we’ve learned and we can start to share journey maps. We can start to share new insights and then try to really understand does it differ by geography? Does it differ by region, or something else? So, that we start building on a shared platform of insights and not redoing work that somebody else already did.
Steve: What’s the process? When you come into an organization you want to find – I don’t know, these sort of pockets or different areas – how do you go about doing that?
Michele: So, somehow I was able to connect with the person in Seattle who introduced me to somebody in Dallas and we initially just said we’re going to have essentially a meeting. That meeting that I thought was only going to be 3 people is now been a monthly meeting that’s grown to 5 – probably 5 to 8 with a few people – we’ve now had designers who are basically doing research. We have a product manager who is actually doing his own research and is a huge advocate for research, who is attending that. And so, based on those meetings and others they’ve invited us – in one case we spoke at a lunch and learn. In another case we were a guest – we were kind of a guest speaker, or guess attendee, at other meetings and people have started to find us and either say, “oh we have 73 personas on this topic.” Or, “we don’t know. We think we have 73 personas. We can’t find them, but you should talk to this person.” It’s been, I would say, very grassroots, but we’ve established kind of this monthly meetup and we at least know a handful of people that are here and who to reach out to and we’re getting the word out. People sometimes reach out to us.
Steve: So, does that step outside of – originally you were sort of describing where CBRE Build is, sort of what its mission is. Are you now making these connections outside of that?
Michele: Yeah. So, I’m making the connections – outside of CBRE Build my mission is definitely to establish the New York City office as a center of excellence for user research, but also in so doing, because we’re under the same umbrella, the digital and tech team umbrella, that part of that is just elevating everybody else. And also, to do our job, so it makes sense for us to talk to people in other locations and other parts of the business because of the overlaps.
Steve: What was going on at the organization where there was – in the time when you were talking about being part of CBRE Build there’s obviously some desire, or hunger or some sense that something’s – there’s more that we could be doing. I know that’s sort of before you were here, but like I’m wondering what are the conditions that set up for a willingness to drive the kind of work that you’re doing for them now?
Michele: I think that the company has always embraced user centered design and understanding the users. All the product managers that I interviewed with and spoke to, they were doing the best that they could with user research. Sometimes they certainly hired outside contractors to execute research. So, there was already that interest in user research. People were doing their own jobs to be done. They were ready. Again, doing their own interviews. And some people have taken UX classes, or had some background in user research and similarly there were designers here that were also doing what we would call user research. And so, there was already I think a good platform having product managers and designers who were trying to make their products better for the user, not just a technology play. And so, I think that that was really exciting. It was also – what attracted me is that it’s not – it wasn’t simply the end side of a project that they wanted. They also wanted help starting to develop MVPs and being more experimental. Like is the – we have some insights, can we make something? Is this really the right fit? Okay, great. You know it is, or it’s not quite, we need to go back to the drawing board and tweak that a little bit and then get into building it right. So, the fact that they were talking a lot about that and doing more innovative processes and were open to experimentation, that made me want to come here.
Steve: That makes me think about like what’s – you know when researchers are looking at organizations to see if they want to join them, some of the things that you mentioned were about – were not research specific, but how are products being made.
Michele: Right, right.
Steve: So, you’re looking for sort of the context that’s going to make the research that you want to do…
Michele: Yeah, yeah. So, were there designers here and actually at the same time they are since hiring – they are going to be hiring a new design director, but there was already the foundation of the designers in this office. And then there was – I had a sense of there might be other designs in other pockets of the organization. I didn’t know to what extent, but the fact that they – I’m not sure when I heard about it, but I definitely understood they were trying to improve product development on all levels research, from formal design as well as product management. So, they had those tools.
Steve: You mentioned one of sort of the three different things you’re doing was around building tools for people who do research (PwDR). Do you have a perspective on – I think this question of who does research and what do we do as researchers to support that – is there a philosophy or a goal kind of beneath that for you that you think about?
Michele: The philosophy I think is that the trained researchers should be taking on the most critical, you know places, the most risky projects. And that’s where they can really also add the most value. Are they going to lose a lot of money? Is this a totally new workflow? Is this really new to the world? So, really focus trained researchers in that area. I believe that once people are more familiar with some of the usability basics and they under – they really – if they really embrace – like we really want to understand which concepts are resonating better, which aspects of the concepts, and they do it with an open mindset that they can start to get feedback on a concept. One of my favorite stories involves a designer where originally I think I’d led several of the sessions, so he kind of understood my script for one point and then he was able – we set him up. So, he was going to lead the sessions with these other participants who were recruited. They already met the criteria. And I remember him coming back and he said he just showing them the concepts. And I think it was probably 3 or 4 and then he said they just weren’t getting it. And then he kind of – he commuted on the bus so they went home on the bus and he like redid something based on all those things and he redid the concept based on the feedback, so I think – and then eventually it worked. It tested better. But the fact that he, even on his own, exposed 3 or 4 people without me saying like can you do you this, can you do this? And like 3 or 4 people couldn’t and he finally understood there was some issue with the concept and it wasn’t about him and it wasn’t about them, but that he needed to take out information and redesign it. Like that’s my favorite story because we set it up, he had my original script, he didn’t have to find the people, but he got that insight and redesigned it. So, that’s my favorite story is someone maybe who isn’t trained, but being in the right circumstances and then being able to act on it. Was able to do that.
Steve: That’s great. I mean that is a good example of sort of setting people up to be successful. And you didn’t need to be there for that, so you could do something else.
Michele: Right. But he was also very aware and the fact that he was then motivated to be like oh wow, it really, really isn’t working. You know, let me see how I can rework this? And that he was humble enough to say – not have the ego or anything, but to really say okay, something doesn’t make sense. You know.
Steve: Right. Because you can experience those research sessions in different ways. Like even that phrase, they aren’t getting it starts to point outward.
Michele: Right. But I think it was that he had to do – the fact that he also persisted and did all the sessions and then realized okay there’s some – and maybe – and I don’t know. Again, this is several years ago, but I remember the essence of that and so I think that if you can help set that up, I believe in helping find participants. I’ve done this in other cases where we’re going to say it is easy to find participants. Sometimes you might have a dedicated recruiter. Sometimes you have a research ops person. Or where I do believe there’s still skill involved in getting the right people, but that having some of that infrastructure around can help say, okay, everybody doesn’t have to worry about that, but I don’t think that other people should be necessarily taking on – there’s a lot of skill that is involved in recruiting, so I think research or research ops can say we’re going to help find the people, we are also going to help you write your script and if do this – and essentially this is what you’re looking for. And also having enough people in the room. I also believe in watching, having people that are going to do it – if they’ve watched professional researchers do this, if they’ve been in enough sessions, they understand essentially, hopefully what they’re looking for and what they’re not. I wouldn’t throw somebody in without any training. They also can reuse scripts. Like in some cases I’ve done a lot of automated usability testing. And so, again, usually designers and product managers love that because they can watch the people do things on a real screen. They’re not even having to worry about writing their prompts. They’re maybe reusing or rewriting a prompt that I wrote or another researcher wrote and then they can go back and watch that. And usually those are also really insightful for the designers to watch that and also to watch it – what I feel – this is making me think is another thing where it’s very focused. I think that other people doing research, some automated research is really good, or even their own, if it’s super focused. It’s very ABC. Could they get from A to B to C? No, they got lost. Okay. That is easy to identify. Where it gets really complex, where I think there’s a lot of interactions, that’s where you probably need a team. You need more dedicated researchers. But I definitely think that things like that can be – people can be trained to do that.
Steve: So, what are some of the most impactful tools or processes, or things that you’ve set up to help these folks?
Michele: So, we’re still in the process of setting up – we’re calling it kind of like a research Wiki. Off that research Wiki we start – we’re having templates. We have past surveys and details that you can reuse – or surveys with the details of like sample questions. If this is an automated study, if it’s a survey, here’s what those questions look like so that it’s starting to be a little bit more self-serviced. We’re starting to link any journey maps, any personas that were done for projects that actually this might relate to your project. So, we’re trying – we’re making sure – we’re not calling it by the project name or a code name, but calling it here’s this type of person, making sure that anybody else can find that. So, we’re setting – basically hacking Google Docs to make a series of mini-Wikis, but it seems to be working. And then on one project we actually have gone so far as, we’re using Airtable and making Airtable to kind of capture some of the feedback from key elements. That’s something that’s still, I guess, under development, but the product manager actually loved Airtable and she had set something before and because Airtable is really like a giant spreadsheet I was able to say well here’s where I would put in a few recordings, here’s some clips, here’s some evidence that I would also add as I find it and we were doing something that lasted over several months and we got into the habit of at least trying to add one or two lines with maybe keywords that we could do – that we could put into Airtable. And what a really nice outcome of that is we’ve now been able to link those Airtable nuggets and pieces of information to Clubhouse tickets which is where there are design and engineering tickets and so we’ve been able to re-look at what’s in Airtable and say oh these things all relate to a certain topic, or a theme, we can create a link and then when an engineer and designer is working on that we can send them that link that it literally has all the evidence, at least at a high level, and if they ever wanted to know more they can go back to here’s the entire survey, here’s this entire research report or something like that, but it’s all off that link. And so, I think that that was exciting for me because of the impact of finally having a process, at least on one project, where things – sometimes in the past they’ve gone into repositories, but they weren’t always linked at the time to a design or an engineering ticket. Sometimes there was a lot of rewriting and so this is something that’s in progress. We hope we do a blog post and share it out, but it’s working for this one project. If it’s scalable we’d love to share some best practices and learn from that.
Steve: You mentioned repository and that seems to be sort of a trigger word in the user research field right now.
Michele: Yeah. So, I think a research repository is definitely a trigger word. I had the opportunity to work with Tomer Sharon who started Polaris at WeWork and I was one of the people that did not create Polaris as much as tried using it as a dedicated researcher and providing tweaks for some of the early information that was in there. Some of you may or may not know is that Polaris started as an Airtable base before we created – at WeWork that we created a homemade tool. And so, I believe that some of the lessons that I learned from working on Airtable there and learning from working on the homemade tool Polaris, I was able to apply here. What I took from that was still making sure there are single line items that can stand on their own as kind of a summary or key word. Like, you know, the participant did not understand this word. They used this word instead of that word. Things like that. Things that are very self-explanatory to anybody reading. And then if there is evidence, is it a screenshot? Is it a record – is it a link to a recording, even with a timestamp. Sometimes we can put in a little bit of a clip of like hey, the user really had trouble doing that. But that went into Airtable. And then at least being able to search by keywords. Sometimes I was prioritizing it by impact to user. Was it critical? It’s really going to be a showstopper? Is it moderate? They’re going to kind of muddle through, you know, but it’s not a show stopper. And then minor, it’s going to impact them and irritate them. And so, putting in, at the very least. And then also what the designers and the product manager loved is making sure there was an area for notes or ideas because sometimes people have ideas and I think that you should be able to capture that at the time. Maybe it’s – you know we really, we don’t have this in help, and not that you want to rely on help, but in some cases it could be like we want to make sure that they’re looking for this keyword in help because they all use this keyword. So, make sure it’s there. Or, we really – they use this other competitor tool, here’s the ideas. We should really – or they used a consumer tool and the consumer tool reminded them of this tool, we should look into that. So, making sure we can capture all that. I think it’s not – because of how it is it hasn’t been, for me it’s been a challenge. We have some verbatims from surveys that was great to add into Airtable, but I also had to add high level summaries of surveys or items that were maybe not as conducive to Airtable, so essentially hacking Airtable to make this repository and for that reason that’s why I think it’s not complete. I still kind of have a research Wiki which is really like oh you’re looking for the personas for the project. You’re looking for – oh, here were some key things that we did. I did a biofuture type of workshop once with some participants to really help us prioritize other features. That didn’t lend itself well to Airtable, but we have the people recorded saying hey this is valuable or it isn’t. So, I feel like the research repositories bog down because there isn’t a good – the way – in order for them to be useful, 1) you have to fill it and 2) you’re not always getting the same types of input. Sometimes it really is observing people and even if you’ve got the transcript you still need the context of what feature were they trying to buy, to add like really, really meaningful things, if I was to do that into Airtable, and for it to make sense at some other point. So, I think like there’s room for growth in terms of research repositories, but at least I have a link to say hey we did this. You know if anyone is considering these 10 features go back and at least look at some of these notes. We have people prioritizing that. So, I think that there’s so many different types of research and the repositories, to make it perfect, I don’t know if it’s possible.
Steve: Are there processes that you need to put in place, either on the – I don’t know, is it fair to call it production and consumption – the sort of different users of these repositories.
Michele: So, I think, yeah, a lot of it was tasked with, again, definitely the product manager who already left Airtable and myself trying to be – what I wound up doing is making sure if I had verbatims that would add value that at least those verbatims got into that. I know that it’s challenging. I think what will help is making a form for if people do have other things. We do have kind of our – some of our engineering and production teams who are catching bugs and catching other feedback, or even something that doesn’t seem right. So, there’s a way to create a form to add that and that’s been really good too. I think also – I know that other product managers are using Airtable, but I’m not sure how they’re doing that and I think that also – I know it’s time consuming, even if you – you know, like I said, even if you get the transcript you still have to translate it into something that’s meaningful. So, I think, you know making sure that it’s useful. It’s not going to weigh you down as you do your projects. So, having every product manager enter it, I don’t know if that’s reasonable. As a researcher I certainly can’t enter everything either and that’s what I’m saying. Like I’m linking, hey we did this awesome biofuture priority exercise but I did not – we have like spreadsheets, we have videos. I didn’t have time certainly to do that. And also, we’d moved on. I think that’s the nature of some of the repositories. Like, oh great, we did that week, we prioritized, we learned from it, there’s some high-level takeaways. But to go back and put in all these nuggets, it’s frustrating to me because I’m sure there’s nuggets that could be used in like 5 months, but it’s also not time pressing for me to do. I need to help like the project teams move on. So, I think that that’s been a – that’s a challenge. So, I’m pretty sure time is a challenge. And then also just making sure – I think that everyone, like you said, can access it. We – like other teams we have Google Docs. We have Outlook. We have Airtable. We have tons and tons of tools and so staying on top of that. The reason I wanted to put the feedback into Airtable is because the product manager was already using Airtable and I linked the – I add the links that have my research evidence in Airtable to Clubhouse because the designers and the engineers are using Clubhouse. And so, what I’ve found in the past is you have to link it to wherever the teams are. And so that’s – in some cases I’ve used Wiki – Wikis and Confluence and other tools to do that, but I think that that was a challenge that I had. We definitely had that sometimes with Polaris. Like people could find information when it was either kind of given to them, but it wasn’t directly linked potentially to where – when they started those tickets, or they were aware of it. So, I think this is kind of step forward that as a researcher I’m actively helping to put that in. I know other colleagues do that. They put things into Jira. They put things into the Wikis, wherever designers and engineers are going to need to be acting on it.
Steve: Is there anything that – what do you have to do to create conditions where people are going to say, oh I have this question, let me go see if there’s any research that speaks to it?
Michele: So, I think it’s two – I think the first stop would be probably talking to a researcher and saying are you even aware of that and a researcher might say, yeah, I know we did some projects like that. Or talk to so and so, I think that they did that. I think creating – what I’m trying to do is create the research Wiki so that hopefully it could at least be by – if those topics emerge or the personas that – again, because we’re in commercial real estate there’s a lot that deal with brokers, but helping you find that right one, at least maybe – at least there’s going to be here’s the place to start and then here’s people you can talk to. I don’t think sending someone to an Airtable database is the right way. I think that there’s some type of – we’re working – we also have something in the works that we’re going to be creating essentially a curated repository and this is something that the other researchers and other people wanted where it is – here’s journey maps, here’s personas, here’s information that we’ve all agreed is okay to standalone. So, someone who wasn’t on the project can use and find and then that we’re going to put that on more of a public – not a public, but an internet site, essentially linked to user research. And so – but it will definitely be curated and I think they might start there. So, that’s one more place.
Steve: So, even people being willing to say oh I have a question, I should approach either researchers or internal researchers, as opposed to, I don’t know, doing nothing.
Michele: Yeah. I think the first thing was is there a researcher or someone on your team, or product team, that’s probably been doing that? They probably know, or they can probably be a resource to get in touch with other people because again, we do monthly meetups. We’re in touch – even designers might know if a project has been done, or if you focus again on what was the user type and not the project, and the code names, then you might actually say well 3 years ago we did this type of work and you might find value in that. And that’s where eventually we want to do with almost creating these – as I said, talking to the other designers and research people that there would be an area that’s curated is really important to us because I think that there’s a thought that you could find all these random usability studies, you can find draft reports, but 1) they may not have value to other people – again, 5 months ago, even the usability test I ran last week, the menu that we were testing has already changed in a week. It’s almost not – you know the insights aren’t relevant and that’s fine because we actually changed the menu and that’s great, but I don’t know that anybody needs to be able to access that forever. The project – it’s lagged on the project team, but putting that into a research repository or something, accessible by anybody, I don’t think is going to add value at this time.
Steve: I like the verb curation kind of attached to research repository. It starts to make sense given the complexity of the context you’re talking about.
Michele: Yeah.
Steve: Can you talk a little bit about how the team has evolved in your time here?
Michele: Sure. So, in terms of the broader team, so when I started in November of 2018 there was someone who was working part-time. She had been – as a researcher – she’d originally been in support. She’d been here for 2 ½ years and she’d kind of learned user research and she was awesome, but it was also very self-taught, but it was great that she had a user mindset. I asked her to give me kind of like just a background on what was happening and what tools we had and I asked her – and we started with well what tools is everybody using? Even like – even identify oh there’s Airtable. Oh, one group was using Optimal Sort. We have licenses and information. So, okay, again, this kind of confirmed people are doing things for user research, but maybe it’s not organized or something. So, we started to 1) create that list so we understood what tools do we even have access to? Do I need to buy tools? And then at the same time they started recruiting – or they’d always been recruiting for that design director, as I said, who will start later in the summer. So, that was new that there would actually be again a design director. Designers had been reporting to a lead or a group product manager, but that was a change. So, um and then also working more closely, I think, with Seattle, instead of me just being focused on building out New York as a center of excellence, really trying to work – there was a lead researcher who had been in Seattle who has since gone on to become a UX manager. He hired another research there so there’s – for, um – you know for a few days we had, I think, three or four people that were actually dedicated researchers, people who had titles that were different changing and I’ve been – just last week we had another researcher start here in New York City. So, depending on how you’re counting it there’s probably four or five people. Some people are part time in different levels, but there’s – within this Build group we kind of have finally the start of an actual user research team. And I say team loosely because we’re not necessarily all reporting into the same people. We’re reporting up to the same SVP, but that’s why I also see us more of a center of excellence. We’re working to – we have now some Wikis. We have shared tools. We have a Slack group. When someone actually got a demo last week about a new tool we were considering, she was like oh I’m going to put this right into this tools folder and I was so excited because we had that and we didn’t have anything like that six months ago. And the other thing is it’s been really exciting to see the product managers take interest, that I’ve worked with like some of them have been doing jobs to be done. They’ve been doing different types of projects. But to see what they’ve been doing with their research – and in one case we were able to also help create a journey map that one product manager saw and was like I can take this to my clients. So, I think the impact on the team is like starting to actually say here’s what research can do and in ways that you didn’t anticipate that maybe we could do that. So that was – so, we’re starting to change just the mindset potentially.
Steve: Whether it’s here or in other roles that you had, I’m wondering about when you look at people that are potential to join your team, what kind of things strike you about perspective – about researchers that might be perspective new colleagues?
Michele: Um, I think you have to be super inquisitive and analytical. So, are you going to just take – hey these are the top three problems and is that a given? I think that researchers and UX people tend to say well no that’s not the – you know we’re going to poke holes in that. So, I think, that’s number one, being inquisitive and analytical. I also want to understand if they have – you know do they have that user mindset? Are they able to understand different perspectives? Have they done different types of projects where it’s not just following somebody home, but it’s like literally spending the time working with them. I’ve had the fortune to work across different types of product industries. In one case, working for a medical company, I was able to – I spent all day driving along with an oxygen delivery person and just knowing that you’ve had that chance to do field studies and that you haven’t just been in a lab is something that’s really important to me that you have – ideally you have wider experience than sitting behind a desk.
Steve: How do you – like what are sort of cues of like the user mindset? How does that exhibit.
Michele: Well I usually – I definitely like to understand, if they’re presenting their portfolio or a case study, what was – you know what were you really – what were you asked to do? What was the actual request and then what did you decide to do? How did you interpret the problem? And then what was – who did you decide – like how did you decide to approach this? But then who – like what were really those key insights that really – what was the impact to that user beyond moving – again, not from a usability perspective, but from probably a higher level of like here’s really where they were having these issues. They couldn’t figure something out; they were struggling to do that. Like did they really then advocate as that key change kind of thing. Something’s not right. We saw that pain point and then that became ideally one of the key insights that they hopefully informed a project about. And so maybe they had that big impact. So, that’s what I’m trying to say. It’s like could you see – can you see some of the challenges? Or even if it was like something that a user loved that you were able to take it from another place and say we’re going to impact this project or projects with that because that’s so valuable. They get really excited about that. We don’t want to ruin it. We want to make sure they can keep doing that or do that better. And did they have that user mindset, that user insight, that they were then able to be that champion to take it through the project.
Steve: You know that’s such a good point. I think we all, and myself, certainly get caught up in we’re about finding problems and fixing them. But your point, we find things that people love, those are huge opportunities to either protect, or celebrate or upstand.
Michele: Right, right, right.
Steve: Yeah. It’s easy to get lost in the fixing – we’re here to fix stuff.
Michele: I think so and I think like – and then also having that freedom is suddenly sometimes different. It’s like oh, we don’t have to only just fix stuff. We can make this – they love it, let’s give them more. You know that might not be a bad idea. Like can we make it so they can spend more time – I think in this case, like for a lot of the knowledge workers, let them do more time analyzing. Like if they’re really good at coming up with these insights, they want to find the best office for their clients, well like they don’t want to be just entering the data. They want to be reviewing the images that we’re providing, the analysis, and let them do that. And that might be something that they love, that type of things.
Steve: Right. It reminds me of projects where the assumption going in was we were going to kind of remove steps and save time and remove effort and that you find that there’s parts of problem solving, or all sorts of weird parts of our lives, that people find satisfaction or even like joy and delight in.
Michele: Yeah, right.
Steve: And that our job isn’t to just eliminate stuff. Like you said, there’s parts of the work that are…
Michele: Right, right. I remember interviewing someone when I was working on like a – I think it was a small business. And she was like well I love paying the employees. Like it took a lot of time, but at the end of the week like she felt successful. She was happy to pay her employees. So even doing – like understanding that step, like when she’s using our product, but she’s happy. It’s not a chore. It’s – you know it’s not a chore. Like she’s happy to pay her employees and she wants to reward them. So, like I think having some of those insights and saying like oh, this is a potential delighter, how do we build on that, is potentially something to keep in mind.
Steve: Right. That seems like a potentially big reframe from small businesses don’t want to spend money. Every loss is a loss. Every outlay of money is a loss. And this person is finding meaning in that.
Michele: Yeah.
Steve: Because of what it symbolizes to them and what accomplishment there is.
Michele: Yeah.
Steve: That’s great. I love those things in research.
Michele: Right.
Steve: Because you can understand what you’re really providing.
Michele: Yeah.
Steve: So, you’ve mentioned a couple of different contexts like delivering oxygen and so on. Can you talk about some of the other roles, or maybe even just how you got into this work? Maybe we’ll start back there.
Michele: My degree, I got a Masters of Science from Cornell. It’s like in human environment relations with a concentration in human factors and ergonomics. And so, when I started the Internet was just getting started. I think I did my thesis which related to actually some household tasks and products. And it was more about physical design, but I remember like I hand coded my survey because HTML was just kind of starting out and I was literally – they didn’t have Survey Monkey or anything. So, I was doing things like that. My first job after grad school was working for Kohler Company. They do kitchens and bathtubs, but I had the chance to work with a lot of really talented industrial designers. Eventually my career included working for the Consumer Products Safety Commission and that’s where I learned a lot about safety and some hazardous issues that could have been prevented because if usability and design had been better – and the work there was more focused on standards writing and reviewing and it was really informative for me, but I wanted to go back to product development.
Steve: What is the Consumer Product Safety Commission?
Michele: This is a federal agency. You might know them because they do all of the recalls. They also are the people that tend to put those warning labels on different things that come out. There are a lot of voluntary standards that you might – they happen to appear a lot on consumer products. Maybe baby products in particular, but they affect a lot of different products. I remember looking at a gas grill that I think you bought on an infomercial and it kept kind of tipping over and the human factors and the usability perspective of that was that it looked like you could light it in one place, but that actually where you could light it. Again, this is probably a really cheap grill, but the reality is, from a human factors and design perspective, why wouldn’t you light it in this one place? And when you lit it there it actually caused accidents. And so that had to be recalled. But that was a really different, I think, experience than a lot of my colleagues have had. And so that – I was grateful for that experience, but I wanted to get back into product design. I eventually made my way to a medical products company which was exciting because it was starting to blend the physical, like products like sleep apnea and oxygen machines which also had some digital screens and different – you know just different types of products that I think today we take for granted, but back in the day it seemed like things were either all digital and very, maybe archaic, or they were almost all physical with just some lights or something. And so that was where I think it preceded the mobile phones, but I had a sense of designing with kind of physical and digital at the same time, what we would call hardware and software today, but it wasn’t always called like that. I was working with industrial design and product designers who were physical product designers at the time.
Steve: How did you find your program – your school program? How did you find your way to that?
Michele: That I think was a fun story. I did my undergrad – I wanted originally to study architecture. I spent a little bit of time with kind of the intro to architecture classes. I decided that actually I didn’t want to be an architect. I wanted to study the psychology and I think I somehow understood more about designing for the people, but I didn’t know anything about what it was called. And so, I finished my undergraduate degree in psychology and eventually I started just talking to people. I think the OXO Good Grips that come out – I think I did an interview – I think I did like an informational interview to say what is this? This seems really interesting. I think I called somebody and I think I got some leads and eventually I remember just talking to everybody until somebody finally said we think this is your field and I think that there was a program – one was at Cornell and one was in Wisconsin. And eventually I think I just found these programs that said this sounds like what you’re doing and I applied and I was lucky enough to get in and then I found that program and that’s how I did that. It was definitely a researcher doing research to do that, but also really using your network and just saying I think there’s a way to make all these things easier. It seems like people are doing this. I know there’s something related to psychology and design. I don’t know what it’s called, but I think there’s something here. People seem to be doing this type of thing.
Steve: If you – hindsight is, of course, 20/20, but would you – is there a different path – given where you are now, is there a different path you might have taken?
Michele: I don’t know. I mean – yeah, from psychology, I mean in terms of – I think, like I said, I wanted to apply it. Definitely I could have gone to law school. Like could have done really, really different things. But I don’t think – I think it would have been great if the UX program had been more – if I’d been more aware of the UX programs if things today had existed. But I think I kind of had to make my way through the career and I’m happy that I got a chance to, again, like work on physical and digital and mobile products and things that I think it’s – because it was sort of me trying to figure it out and maybe that’s not as different from other UX people that oh this seems interesting, maybe I can go work in this area, or there’s ways to apply this. I think it worked out okay.
Steve: I’m sure there’s some analysis we could do of kind of when you were in school or when you were sort of looking for your jobs what you were aware of in terms of what kind of work there was, and what kind of educational options there were. Because, right, maybe if you’re starting now you have access to certain choices. If you started when you started, or when I started, you and I were doing different versions of finding our way with some serendipity and some sort of opportunistic research, but the context was so different.
Michele: Right. I think I would also add, I think the assumption was I had this undergrad degree in psychology. I think I just got a job – like I wasn’t doing anything in the field. I was working in a publishing agency, which was great, but it wasn’t – I was really doing that to pay the bills essentially. I didn’t – had I been able to be an apprentice, had I found something else, maybe I wouldn’t have gone to grad school. I think the assumption at the time was there’s something out there. I think I probably assumed you had to get training and go to grad school. And so that’s the path I took. But potentially, in this day and age, you might do more of an apprentice, or you might join a bigger company and see if you can get training on the job in that sense.
Steve: Just hearing you talk reminds me that there’s a history of this profession. And relatively speaking we’re still talking about recent times, but it seems like the field goes back longer than we might – maybe be collectively forget the history.
Michele: Yeah. Definitely, I mean, I think that’s an unknown right, that it goes back to people working on the planes and the pilots – you know that the controls were in the wrong places and then they had to standardize, but that was really kind of the field originally or human factors. I think a lot of the work they were also doing, as I understand it, probably with the early computers, even with NASA and things in the 60s, that was definitely pioneering the work of human computer interaction which sounds today so goofy to say human computer interaction, or human systems interaction. But it definitely goes back decades, way before us, and generations.
Steve: Right. You have a human factors degree. I have an HCI degree, you know human computer interaction. And those seem like rare terms. I don’t find people with that as their background as much. We didn’t have the phrase user experience, or UX, or user research, or any of those things.
Michele: No, no.
Steve: So, we were finding our way.
Michele: I don’t think I had the user research title until about 10 years ago. I think that’s also when some of the industry changed a little bit, but that people were doing task analysis. They were trying to understand the system. They were interviewing people. They were testing, whether they used the work usability testing or prototype testing or concept testing, they were essentially going through all of the things that we do today. They just might have called it something different.
Steve: I mean and some of those sort of classic examples of human factors research and some of the things that these examples, there’s just a lot of detail to sort of the decisions that are being made.
Michele: Right.
Steve: A few minutes ago, we were talking about reframing our understanding of what the problem is. So, there’s a – even that’s the – there’s other axis we consider, but just among that sort of detail vs. big picture it seems like there’s a big range of what researchers do.
Michele: Yes. I think researchers have a really key role to play at both parts of the double diamond process, if you think of the traditional, you’re trying to design the right thing and can you inform that? What are the key insights? Are we going to build the right thing? Maybe researchers are helping to take those insights into a high level, MVP, is this even the right fit? But once you’ve decided to build the right thing are you building it right? Can you test multiple concepts? Are there three or four ways that you can actually figure that out? If it’s totally new to the world, even if it’s for the future, maybe there’s two or three different ways you can do a new feature and then can you test that and iterate that? And so sometimes it gets into really detailed design, all the way down to the icons, the placement on the page, the labeling and all those other aspects that really go into refining a product and making a great product.
Steve: And you like that part of it?
Michele: I like both. You know, I think – I love the idea of really working at the upfront doing really generative and informative research and going out and saying we’ve interviewed all these people, what’s really working, what’s not. What’s their workflow? What are like – what are things that nobody’s found before and how could we maybe take this into something new? I love doing like rapid experiments to say maybe there’s something we could go based on that and taking that into what you might call an early MVP, but of course yeah, once we do it I want to make sure things make sense and that it’s really easy to use and so that’s all dear to my heart at the end of the day.
Steve: When you think about the user research profession, or the practice, whatever we kind of are collectively, do you have any hopes or goals in the sort of medium term for this field?
Michele: I hope that we work more closely with all the different, I guess you would say, factions of user experience and design. On one hand I love that there’s a zillion different meetups and I can go on Eventbrite and see anything related to UX and research. On the other hand, I get bummed out that I’m like oh actually they should have just talked to that other group. Like there’s a lot of groups – there’s UXPA. There was IxDA, I don’t know if they’re still active? But there are so many groups that have either been active or could help some of these, what I’m seeing as random meetups. And I would love if some things got a little more organized, or if there was just way to say you don’t need a new Slack channel. You can talk to all these people. There are communities. And letting people know. And maybe that’s, at a very high level, like just formalizing it. Yeah, I’ve been in the field for a while. I don’t like that we still go in circles about what the title is and so I’m okay not having the same titles and things, but I would love people to know like there is definitely a very long history of people doing this work and making sure that people have access to that and the resources.
Steve: Yeah, it seems like we – as people and certainly the technology profession, we have a short memory. We’re attracted to kind of the shiny. I mean this was already a long time ago, but I remember someone telling me a story about – I think that they were at an agency and they went in to talk to somebody about doing research and the person said to them, alright, what’s new, what do you got? And like even that idea that a thing that you’re going to be doing with research should be new. I mean obviously contexts change.
Michele: Right.
Steve: And we adapt to them, but just the idea that new itself was sort of – was a value, seems to sort of fly in the face of what you’re saying, there’s a whole history here that we’re leveraging. We don’t need to reinvent it.
Michele: Right, yeah. I’d probably be happy just to let – if there was like a blog post or a tweet that said hi, there’s an entire history, here’s fields if you’re new and getting started – here’s three or four fields that you might want to check out, related fields. Here’s – you know there’s tons of templates and resources that people have been putting out over the years, here’s 10 of them, or something like that. And here’s – if you’re anywhere in the world there probably is a meetup that’s related to what you’re looking for. It just might be called UX or HCI or research or design research, or something else.
Steve: Right. All the variations of all the terms.
Michele: Yeah.
Steve: Is there anything else you think we should talk about today? Anything I didn’t ask you about?
Michele: I think this has been really positive. I’m looking to chat with anyone else if you’re also trying to establish research as more of a center of excellence and building up the processes and the infrastructure that go beyond one team or a series of projects. I’d love for you to reach out.
Steve: What’s the best platform, or whatever, for people to connect you on? We’ll put it in the show notes for the episode.
Michele: Twitter is great.
Steve: Okay. Well, thanks very much, Michele, for a great conversation. I really appreciate your time.
Michele: Thank you very much, Steve.
Steve: Well, there we go! Thanks for being here for this episode. You can find Dollars to Donuts wherever you do your podcast listening. Didn’t we used to call that podcatching? I don’t know. You know you want to give us five stars on Apple Podcasts am I right?. Visit Portigal dot com slash podcast for all the episodes including show notes and transcripts. Our theme music is by Bruce Todd.
The post 23. Michele Marut of CBRE Build first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Michele Marut who leads user experience research at CBRE Build. We discuss the curation of research repositories, using research to go beyond fixing things, and building processes and tools that can be used by researchers and people who do research. The philosophy is that the […]
The post 23. Michele Marut of CBRE Build first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Michele Marut who leads user experience research at CBRE Build. We discuss the curation of research repositories, using research to go beyond fixing things, and building processes and tools that can be used by researchers and people who do research.
The philosophy is that the trained researchers should be taking on the most critical, the most risky projects. That’s where they can add the most value. Are they going to lose a lot of money? Is this a totally new workflow? Is this really new to the world? So, really focus trained researchers in that area. – Michele Marut
Show Links* Medical Storytelling at the VA Bridges Gap Between Patients And Caregivers * My Life, My Story: Advancing the Veteran Experience * Michele on LinkedIn * Michele on Twitter * CBRE * CBRE Build * WTF is Proptech? * Zillow * Airtable * Clubhouse * Tomer Sharon on Dollars to Donuts * Democratizing UX (and Polaris) * WeWork * Optimal Sort * M.S. in Human Environment Relations, Cornell * Kohler Company * Consumer Products Safety Commission * OXO Good Grips * MVP (Minimum Viable Product) * UXPA (User Experience Professionals Association) * IxDA (Interaction Design Association)
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Thanks for joining me on Dollars to Donuts, the podcast where I talk with people who lead user research in their organization.
As a consultant, I find myself collaborating with very different types of organizations in terms of the amount of experience they have in doing user research, or learning from user research, or acting on what we learn from doing user research. There might be strong leadership in user experience design, or product design, or service design…or that might be a completely foreign concept. That’s a significant challenge in my work as a consultant, ensuring that I’m in a position to assess and respond to that diversity. It’s also something I really enjoy, and I see working on this podcast as part of that journey, something that I’m able to share with you. And so, the best way to support this podcast is to support my own consulting business. You can hire me to lead user research projects or to coach your own team as you talk to users. I help organizations put together research roadmaps so they can prioritize their limited resources. And I run training workshops to level up fieldwork and analysis skills. Please get in touch to see what we might do together.
Otherwise, if you have feedback about this podcast, email me at DONUTS AT PORTIGAL DOT COM or write me on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
I was intrigued by a story on NPR last month about program created at a VA hospital in Madison, Wisconsin, called “My Life, My Story.” In this program, staff and trained volunteers conduct open-ended interviews with veterans about their lives, letting the patient decide what they want to share about themselves. The interview is turned into a thousand-word biography that the patient reviews and revises, and then is included in their medical record. Patients also can share these stories with friends and family. This particular VA hospital has been training other hospitals in the VA system over the past few years. The article describes the benefit to the patient, and the different caregivers, both in the act of telling the story, and in having this kind of softer information available for review in their medical record. The article also explains the evolution of the “My Life, My Story” program, and how they “tried having patients fill out surveys, which were useful but still left the team wanting more. Next, they tried getting patients to write down their life stories themselves, but not many people really wanted to. Finally, an epiphany: Hire a writer to interview the patients, and put what they learned on paper.”
This is a primordial form of user research; there’s no sense-making of patterns across groups of users; the data gathered from a user of the system only has value when applied directly to that user’s specific experience. I mean, maybe there’s a cultural shift that comes from making this kind of information available in the medical record itself, maybe it invites providers to be hungry for this kind of data, maybe it changes the conversations internally around patients as people with rich and messy lives beyond their medical conditions. And I want to dismiss it as unscaleable – this idea of interviewing every single user, rather than a sample, but – look at the implementation here – it’s absolutely about operationalizing it so that this service, asking open-ended questions about a patient’s life, is a component of the experience for each patient.
In some ways, it’s a bit threatening to user researchers, that the simple act of just asking people about themselves can benefit the delivery of services so significantly, but on the other hand it also suggests that the potential value researchers can bring is well beyond what we usually say it is. If you’re looking towards the future of user research, this may be an early signal for possible directions.
Let’s get to the interview, with Michele Marut, who is leading user experience research for CBRE Build in New York city. Thank you for being on the podcast.
Michele Marut: Happy to be here, Steve.
Steve: Let’s start with an introduction. Who are you? What do you do?
Michele: My name is Michele Marut. I am currently a lead UX researcher at CBRE and I’m working to build up a research center of excellence in the sales office.
Steve: What kind of business is CBRE?
Michele: CBRE does commercial real estate. You’ve probably seen their names on buildings, especially if you’re in bigger cities. But if you’re doing retail or industrial and logistics types of work you might see them. And if you’re searching for office space you might have encountered them from a broker and advisory perspective.
Steve: What part of CBRE are you working in?
Michele: I’m working with a group called CBRE Build and it sprung from an acquisition that happened approximately 2 years ago when they acquired a company named Floored that specialized in 2D and 3D visualization tools that were really helping brokers to help sell their buildings and spaces. And so those tools are still in existence, but since then my manager has been tasked with creating more of an innovation group and they’ve created other tools that complement the original 2D and 3D tools, as well as tools that help people calculate and anticipate space that they might need. Tools that help work with test fits and that can be used by people beyond brokers and landlords and more for helping you really figure out the right space for your needs. It’s still focused on commercial real estate.
Steve: So, is this kind of a product and tools part of a larger real estate organization?
Michele: It’s a digital and technology section of a larger company that’s focused on commercial real estate. A fun word to talk about is proptech, or property tech. We are very much in the property and proptech space if you see that. Some people might think it’s a buzzword, but it’s what’s happening with other parts of the industry as you’ve seen other groups – financial went through this, health care is going through this, even consumer – commercial real estate, as well as other parts, even in your home and consumer real estate is finally getting more interactive. It’s more tech savvy. And so, some of these things are finally coming to commercial real estate.
Steve: I think the New York Times keeps writing about how companies like Zillow will now buy your house as well as tell you what it’s worth and there’s some technology that makes that happen.
Michele: Yes, yes.
Steve: So, is that proptech?
Michele: That’s proptech. Proptech can be consumer, if you’re looking for a house or an apartment. And it’s also in commercial real estate. So, it’s all different things. Anything that helps take data, technology and information and make it more innovative and probably disrupting the status quo of what you’ve probably been buying and selling real estate for hundreds of years the same way. We’re kind of at that inflection point for tech now, for real estate.
Steve: Okay. That’s great context. And then you mentioned center of excellence? Is that right?
Michele: Yeah.
Steve: What does that mean?
Michele: So, some of the centers of excellence are business terms that have been around. Here what we’re trying to do is one, build a group so there’s more researchers. There was only, I think, a researcher who had come from support. Before I joined there was one or two people in the other parts. We’re certainly adding more trained and skilled researchers as part 1, literally building the team. Part 2 we are creating templates and processes so that anybody, whether they’re a trained researcher or a person who does research (PwDR) can actually leverage some of our tools. They can get started very quickly. They can find the right participants. They can use our templates. They can use interview guides. And they can probably execute a basic research plan on their own. And then the third component is understanding who else in the company is doing research, making sure that we’re connecting them and building this community so that we can continue to grow our research center of practice, but also to share like if we’re actually talking to brokers and the same people across the company. We can share what we’ve learned and we can start to share journey maps. We can start to share new insights and then try to really understand does it differ by geography? Does it differ by region, or something else? So, that we start building on a shared platform of insights and not redoing work that somebody else already did.
Steve: What’s the process? When you come into an organization you want to find – I don’t know, these sort of pockets or different areas – how do you go about doing that?
Michele: So, somehow I was able to connect with the person in Seattle who introduced me to somebody in Dallas and we initially just said we’re going to have essentially a meeting. That meeting that I thought was only going to be 3 people is now been a monthly meeting that’s grown to 5 – probably 5 to 8 with a few people – we’ve now had designers who are basically doing research. We have a product manager who is actually doing his own research and is a huge advocate for research, who is attending that. And so, based on those meetings and others they’ve invited us – in one case we spoke at a lunch and learn. In another case we were a guest – we were kind of a guest speaker, or guess attendee, at other meetings and people have started to find us and either say, “oh we have 73 personas on this topic.” Or, “we don’t know. We think we have 73 personas. We can’t find them, but you should talk to this person.” It’s been, I would say, very grassroots, but we’ve established kind of this monthly meetup and we at least know a handful of people that are here and who to reach out to and we’re getting the word out. People sometimes reach out to us.
Steve: So, does that step outside of – originally you were sort of describing where CBRE Build is, sort of what its mission is. Are you now making these connections outside of that?
Michele: Yeah. So, I’m making the connections – outside of CBRE Build my mission is definitely to establish the New York City office as a center of excellence for user research, but also in so doing, because we’re under the same umbrella, the digital and tech team umbrella, that part of that is just elevating everybody else. And also, to do our job, so it makes sense for us to talk to people in other locations and other parts of the business because of the overlaps.
Steve: What was going on at the organization where there was – in the time when you were talking about being part of CBRE Build there’s obviously some desire, or hunger or some sense that something’s – there’s more that we could be doing. I know that’s sort of before you were here, but like I’m wondering what are the conditions that set up for a willingness to drive the kind of work that you’re doing for them now?
Michele: I think that the company has always embraced user centered design and understanding the users. All the product managers that I interviewed with and spoke to, they were doing the best that they could with user research. Sometimes they certainly hired outside contractors to execute research. So, there was already that interest in user research. People were doing their own jobs to be done. They were ready. Again, doing their own interviews. And some people have taken UX classes, or had some background in user research and similarly there were designers here that were also doing what we would call user research. And so, there was already I think a good platform having product managers and designers who were trying to make their products better for the user, not just a technology play. And so, I think that that was really exciting. It was also – what attracted me is that it’s not – it wasn’t simply the end side of a project that they wanted. They also wanted help starting to develop MVPs and being more experimental. Like is the – we have some insights, can we make something? Is this really the right fit? Okay, great. You know it is, or it’s not quite, we need to go back to the drawing board and tweak that a little bit and then get into building it right. So, the fact that they were talking a lot about that and doing more innovative processes and were open to experimentation, that made me want to come here.
Steve: That makes me think about like what’s – you know when researchers are looking at organizations to see if they want to join them, some of the things that you mentioned were about – were not research specific, but how are products being made.
Michele: Right, right.
Steve: So, you’re looking for sort of the context that’s going to make the research that you want to do…
Michele: Yeah, yeah. So, were there designers here and actually at the same time they are since hiring – they are going to be hiring a new design director, but there was already the foundation of the designers in this office. And then there was – I had a sense of there might be other designs in other pockets of the organization. I didn’t know to what extent, but the fact that they – I’m not sure when I heard about it, but I definitely understood they were trying to improve product development on all levels research, from formal design as well as product management. So, they had those tools.
Steve: You mentioned one of sort of the three different things you’re doing was around building tools for people who do research (PwDR). Do you have a perspective on – I think this question of who does research and what do we do as researchers to support that – is there a philosophy or a goal kind of beneath that for you that you think about?
Michele: The philosophy I think is that the trained researchers should be taking on the most critical, you know places, the most risky projects. And that’s where they can really also add the most value. Are they going to lose a lot of money? Is this a totally new workflow? Is this really new to the world? So, really focus trained researchers in that area. I believe that once people are more familiar with some of the usability basics and they under – they really – if they really embrace – like we really want to understand which concepts are resonating better, which aspects of the concepts, and they do it with an open mindset that they can start to get feedback on a concept. One of my favorite stories involves a designer where originally I think I’d led several of the sessions, so he kind of understood my script for one point and then he was able – we set him up. So, he was going to lead the sessions with these other participants who were recruited. They already met the criteria. And I remember him coming back and he said he just showing them the concepts. And I think it was probably 3 or 4 and then he said they just weren’t getting it. And then he kind of – he commuted on the bus so they went home on the bus and he like redid something based on all those things and he redid the concept based on the feedback, so I think – and then eventually it worked. It tested better. But the fact that he, even on his own, exposed 3 or 4 people without me saying like can you do you this, can you do this? And like 3 or 4 people couldn’t and he finally understood there was some issue with the concept and it wasn’t about him and it wasn’t about them, but that he needed to take out information and redesign it. Like that’s my favorite story because we set it up, he had my original script, he didn’t have to find the people, but he got that insight and redesigned it. So, that’s my favorite story is someone maybe who isn’t trained, but being in the right circumstances and then being able to act on it. Was able to do that.
Steve: That’s great. I mean that is a good example of sort of setting people up to be successful. And you didn’t need to be there for that, so you could do something else.
Michele: Right. But he was also very aware and the fact that he was then motivated to be like oh wow, it really, really isn’t working. You know, let me see how I can rework this? And that he was humble enough to say – not have the ego or anything, but to really say okay, something doesn’t make sense. You know.
Steve: Right. Because you can experience those research sessions in different ways. Like even that phrase, they aren’t getting it starts to point outward.
Michele: Right. But I think it was that he had to do – the fact that he also persisted and did all the sessions and then realized okay there’s some – and maybe – and I don’t know. Again, this is several years ago, but I remember the essence of that and so I think that if you can help set that up, I believe in helping find participants. I’ve done this in other cases where we’re going to say it is easy to find participants. Sometimes you might have a dedicated recruiter. Sometimes you have a research ops person. Or where I do believe there’s still skill involved in getting the right people, but that having some of that infrastructure around can help say, okay, everybody doesn’t have to worry about that, but I don’t think that other people should be necessarily taking on – there’s a lot of skill that is involved in recruiting, so I think research or research ops can say we’re going to help find the people, we are also going to help you write your script and if do this – and essentially this is what you’re looking for. And also having enough people in the room. I also believe in watching, having people that are going to do it – if they’ve watched professional researchers do this, if they’ve been in enough sessions, they understand essentially, hopefully what they’re looking for and what they’re not. I wouldn’t throw somebody in without any training. They also can reuse scripts. Like in some cases I’ve done a lot of automated usability testing. And so, again, usually designers and product managers love that because they can watch the people do things on a real screen. They’re not even having to worry about writing their prompts. They’re maybe reusing or rewriting a prompt that I wrote or another researcher wrote and then they can go back and watch that. And usually those are also really insightful for the designers to watch that and also to watch it – what I feel – this is making me think is another thing where it’s very focused. I think that other people doing research, some automated research is really good, or even their own, if it’s super focused. It’s very ABC. Could they get from A to B to C? No, they got lost. Okay. That is easy to identify. Where it gets really complex, where I think there’s a lot of interactions, that’s where you probably need a team. You need more dedicated researchers. But I definitely think that things like that can be – people can be trained to do that.
Steve: So, what are some of the most impactful tools or processes, or things that you’ve set up to help these folks?
Michele: So, we’re still in the process of setting up – we’re calling it kind of like a research Wiki. Off that research Wiki we start – we’re having templates. We have past surveys and details that you can reuse – or surveys with the details of like sample questions. If this is an automated study, if it’s a survey, here’s what those questions look like so that it’s starting to be a little bit more self-serviced. We’re starting to link any journey maps, any personas that were done for projects that actually this might relate to your project. So, we’re trying – we’re making sure – we’re not calling it by the project name or a code name, but calling it here’s this type of person, making sure that anybody else can find that. So, we’re setting – basically hacking Google Docs to make a series of mini-Wikis, but it seems to be working. And then on one project we actually have gone so far as, we’re using Airtable and making Airtable to kind of capture some of the feedback from key elements. That’s something that’s still, I guess, under development, but the product manager actually loved Airtable and she had set something before and because Airtable is really like a giant spreadsheet I was able to say well here’s where I would put in a few recordings, here’s some clips, here’s some evidence that I would also add as I find it and we were doing something that lasted over several months and we got into the habit of at least trying to add one or two lines with maybe keywords that we could do – that we could put into Airtable. And what a really nice outcome of that is we’ve now been able to link those Airtable nuggets and pieces of information to Clubhouse tickets which is where there are design and engineering tickets and so we’ve been able to re-look at what’s in Airtable and say oh these things all relate to a certain topic, or a theme, we can create a link and then when an engineer and designer is working on that we can send them that link that it literally has all the evidence, at least at a high level, and if they ever wanted to know more they can go back to here’s the entire survey, here’s this entire research report or something like that, but it’s all off that link. And so, I think that that was exciting for me because of the impact of finally having a process, at least on one project, where things – sometimes in the past they’ve gone into repositories, but they weren’t always linked at the time to a design or an engineering ticket. Sometimes there was a lot of rewriting and so this is something that’s in progress. We hope we do a blog post and share it out, but it’s working for this one project. If it’s scalable we’d love to share some best practices and learn from that.
Steve: You mentioned repository and that seems to be sort of a trigger word in the user research field right now.
Michele: Yeah. So, I think a research repository is definitely a trigger word. I had the opportunity to work with Tomer Sharon who started Polaris at WeWork and I was one of the people that did not create Polaris as much as tried using it as a dedicated researcher and providing tweaks for some of the early information that was in there. Some of you may or may not know is that Polaris started as an Airtable base before we created – at WeWork that we created a homemade tool. And so, I believe that some of the lessons that I learned from working on Airtable there and learning from working on the homemade tool Polaris, I was able to apply here. What I took from that was still making sure there are single line items that can stand on their own as kind of a summary or key word. Like, you know, the participant did not understand this word. They used this word instead of that word. Things like that. Things that are very self-explanatory to anybody reading. And then if there is evidence, is it a screenshot? Is it a record – is it a link to a recording, even with a timestamp. Sometimes we can put in a little bit of a clip of like hey, the user really had trouble doing that. But that went into Airtable. And then at least being able to search by keywords. Sometimes I was prioritizing it by impact to user. Was it critical? It’s really going to be a showstopper? Is it moderate? They’re going to kind of muddle through, you know, but it’s not a show stopper. And then minor, it’s going to impact them and irritate them. And so, putting in, at the very least. And then also what the designers and the product manager loved is making sure there was an area for notes or ideas because sometimes people have ideas and I think that you should be able to capture that at the time. Maybe it’s – you know we really, we don’t have this in help, and not that you want to rely on help, but in some cases it could be like we want to make sure that they’re looking for this keyword in help because they all use this keyword. So, make sure it’s there. Or, we really – they use this other competitor tool, here’s the ideas. We should really – or they used a consumer tool and the consumer tool reminded them of this tool, we should look into that. So, making sure we can capture all that. I think it’s not – because of how it is it hasn’t been, for me it’s been a challenge. We have some verbatims from surveys that was great to add into Airtable, but I also had to add high level summaries of surveys or items that were maybe not as conducive to Airtable, so essentially hacking Airtable to make this repository and for that reason that’s why I think it’s not complete. I still kind of have a research Wiki which is really like oh you’re looking for the personas for the project. You’re looking for – oh, here were some key things that we did. I did a biofuture type of workshop once with some participants to really help us prioritize other features. That didn’t lend itself well to Airtable, but we have the people recorded saying hey this is valuable or it isn’t. So, I feel like the research repositories bog down because there isn’t a good – the way – in order for them to be useful, 1) you have to fill it and 2) you’re not always getting the same types of input. Sometimes it really is observing people and even if you’ve got the transcript you still need the context of what feature were they trying to buy, to add like really, really meaningful things, if I was to do that into Airtable, and for it to make sense at some other point. So, I think like there’s room for growth in terms of research repositories, but at least I have a link to say hey we did this. You know if anyone is considering these 10 features go back and at least look at some of these notes. We have people prioritizing that. So, I think that there’s so many different types of research and the repositories, to make it perfect, I don’t know if it’s possible.
Steve: Are there processes that you need to put in place, either on the – I don’t know, is it fair to call it production and consumption – the sort of different users of these repositories.
Michele: So, I think, yeah, a lot of it was tasked with, again, definitely the product manager who already left Airtable and myself trying to be – what I wound up doing is making sure if I had verbatims that would add value that at least those verbatims got into that. I know that it’s challenging. I think what will help is making a form for if people do have other things. We do have kind of our – some of our engineering and production teams who are catching bugs and catching other feedback, or even something that doesn’t seem right. So, there’s a way to create a form to add that and that’s been really good too. I think also – I know that other product managers are using Airtable, but I’m not sure how they’re doing that and I think that also – I know it’s time consuming, even if you – you know, like I said, even if you get the transcript you still have to translate it into something that’s meaningful. So, I think, you know making sure that it’s useful. It’s not going to weigh you down as you do your projects. So, having every product manager enter it, I don’t know if that’s reasonable. As a researcher I certainly can’t enter everything either and that’s what I’m saying. Like I’m linking, hey we did this awesome biofuture priority exercise but I did not – we have like spreadsheets, we have videos. I didn’t have time certainly to do that. And also, we’d moved on. I think that’s the nature of some of the repositories. Like, oh great, we did that week, we prioritized, we learned from it, there’s some high-level takeaways. But to go back and put in all these nuggets, it’s frustrating to me because I’m sure there’s nuggets that could be used in like 5 months, but it’s also not time pressing for me to do. I need to help like the project teams move on. So, I think that that’s been a – that’s a challenge. So, I’m pretty sure time is a challenge. And then also just making sure – I think that everyone, like you said, can access it. We – like other teams we have Google Docs. We have Outlook. We have Airtable. We have tons and tons of tools and so staying on top of that. The reason I wanted to put the feedback into Airtable is because the product manager was already using Airtable and I linked the – I add the links that have my research evidence in Airtable to Clubhouse because the designers and the engineers are using Clubhouse. And so, what I’ve found in the past is you have to link it to wherever the teams are. And so that’s – in some cases I’ve used Wiki – Wikis and Confluence and other tools to do that, but I think that that was a challenge that I had. We definitely had that sometimes with Polaris. Like people could find information when it was either kind of given to them, but it wasn’t directly linked potentially to where – when they started those tickets, or they were aware of it. So, I think this is kind of step forward that as a researcher I’m actively helping to put that in. I know other colleagues do that. They put things into Jira. They put things into the Wikis, wherever designers and engineers are going to need to be acting on it.
Steve: Is there anything that – what do you have to do to create conditions where people are going to say, oh I have this question, let me go see if there’s any research that speaks to it?
Michele: So, I think it’s two – I think the first stop would be probably talking to a researcher and saying are you even aware of that and a researcher might say, yeah, I know we did some projects like that. Or talk to so and so, I think that they did that. I think creating – what I’m trying to do is create the research Wiki so that hopefully it could at least be by – if those topics emerge or the personas that – again, because we’re in commercial real estate there’s a lot that deal with brokers, but helping you find that right one, at least maybe – at least there’s going to be here’s the place to start and then here’s people you can talk to. I don’t think sending someone to an Airtable database is the right way. I think that there’s some type of – we’re working – we also have something in the works that we’re going to be creating essentially a curated repository and this is something that the other researchers and other people wanted where it is – here’s journey maps, here’s personas, here’s information that we’ve all agreed is okay to standalone. So, someone who wasn’t on the project can use and find and then that we’re going to put that on more of a public – not a public, but an internet site, essentially linked to user research. And so – but it will definitely be curated and I think they might start there. So, that’s one more place.
Steve: So, even people being willing to say oh I have a question, I should approach either researchers or internal researchers, as opposed to, I don’t know, doing nothing.
Michele: Yeah. I think the first thing was is there a researcher or someone on your team, or product team, that’s probably been doing that? They probably know, or they can probably be a resource to get in touch with other people because again, we do monthly meetups. We’re in touch – even designers might know if a project has been done, or if you focus again on what was the user type and not the project, and the code names, then you might actually say well 3 years ago we did this type of work and you might find value in that. And that’s where eventually we want to do with almost creating these – as I said, talking to the other designers and research people that there would be an area that’s curated is really important to us because I think that there’s a thought that you could find all these random usability studies, you can find draft reports, but 1) they may not have value to other people – again, 5 months ago, even the usability test I ran last week, the menu that we were testing has already changed in a week. It’s almost not – you know the insights aren’t relevant and that’s fine because we actually changed the menu and that’s great, but I don’t know that anybody needs to be able to access that forever. The project – it’s lagged on the project team, but putting that into a research repository or something, accessible by anybody, I don’t think is going to add value at this time.
Steve: I like the verb curation kind of attached to research repository. It starts to make sense given the complexity of the context you’re talking about.
Michele: Yeah.
Steve: Can you talk a little bit about how the team has evolved in your time here?
Michele: Sure. So, in terms of the broader team, so when I started in November of 2018 there was someone who was working part-time. She had been – as a researcher – she’d originally been in support. She’d been here for 2 ½ years and she’d kind of learned user research and she was awesome, but it was also very self-taught, but it was great that she had a user mindset. I asked her to give me kind of like just a background on what was happening and what tools we had and I asked her – and we started with well what tools is everybody using? Even like – even identify oh there’s Airtable. Oh, one group was using Optimal Sort. We have licenses and information. So, okay, again, this kind of confirmed people are doing things for user research, but maybe it’s not organized or something. So, we started to 1) create that list so we understood what tools do we even have access to? Do I need to buy tools? And then at the same time they started recruiting – or they’d always been recruiting for that design director, as I said, who will start later in the summer. So, that was new that there would actually be again a design director. Designers had been reporting to a lead or a group product manager, but that was a change. So, um and then also working more closely, I think, with Seattle, instead of me just being focused on building out New York as a center of excellence, really trying to work – there was a lead researcher who had been in Seattle who has since gone on to become a UX manager. He hired another research there so there’s – for, um – you know for a few days we had, I think, three or four people that were actually dedicated researchers, people who had titles that were different changing and I’ve been – just last week we had another researcher start here in New York City. So, depending on how you’re counting it there’s probably four or five people. Some people are part time in different levels, but there’s – within this Build group we kind of have finally the start of an actual user research team. And I say team loosely because we’re not necessarily all reporting into the same people. We’re reporting up to the same SVP, but that’s why I also see us more of a center of excellence. We’re working to – we have now some Wikis. We have shared tools. We have a Slack group. When someone actually got a demo last week about a new tool we were considering, she was like oh I’m going to put this right into this tools folder and I was so excited because we had that and we didn’t have anything like that six months ago. And the other thing is it’s been really exciting to see the product managers take interest, that I’ve worked with like some of them have been doing jobs to be done. They’ve been doing different types of projects. But to see what they’ve been doing with their research – and in one case we were able to also help create a journey map that one product manager saw and was like I can take this to my clients. So, I think the impact on the team is like starting to actually say here’s what research can do and in ways that you didn’t anticipate that maybe we could do that. So that was – so, we’re starting to change just the mindset potentially.
Steve: Whether it’s here or in other roles that you had, I’m wondering about when you look at people that are potential to join your team, what kind of things strike you about perspective – about researchers that might be perspective new colleagues?
Michele: Um, I think you have to be super inquisitive and analytical. So, are you going to just take – hey these are the top three problems and is that a given? I think that researchers and UX people tend to say well no that’s not the – you know we’re going to poke holes in that. So, I think, that’s number one, being inquisitive and analytical. I also want to understand if they have – you know do they have that user mindset? Are they able to understand different perspectives? Have they done different types of projects where it’s not just following somebody home, but it’s like literally spending the time working with them. I’ve had the fortune to work across different types of product industries. In one case, working for a medical company, I was able to – I spent all day driving along with an oxygen delivery person and just knowing that you’ve had that chance to do field studies and that you haven’t just been in a lab is something that’s really important to me that you have – ideally you have wider experience than sitting behind a desk.
Steve: How do you – like what are sort of cues of like the user mindset? How does that exhibit.
Michele: Well I usually – I definitely like to understand, if they’re presenting their portfolio or a case study, what was – you know what were you really – what were you asked to do? What was the actual request and then what did you decide to do? How did you interpret the problem? And then what was – who did you decide – like how did you decide to approach this? But then who – like what were really those key insights that really – what was the impact to that user beyond moving – again, not from a usability perspective, but from probably a higher level of like here’s really where they were having these issues. They couldn’t figure something out; they were struggling to do that. Like did they really then advocate as that key change kind of thing. Something’s not right. We saw that pain point and then that became ideally one of the key insights that they hopefully informed a project about. And so maybe they had that big impact. So, that’s what I’m trying to say. It’s like could you see – can you see some of the challenges? Or even if it was like something that a user loved that you were able to take it from another place and say we’re going to impact this project or projects with that because that’s so valuable. They get really excited about that. We don’t want to ruin it. We want to make sure they can keep doing that or do that better. And did they have that user mindset, that user insight, that they were then able to be that champion to take it through the project.
Steve: You know that’s such a good point. I think we all, and myself, certainly get caught up in we’re about finding problems and fixing them. But your point, we find things that people love, those are huge opportunities to either protect, or celebrate or upstand.
Michele: Right, right, right.
Steve: Yeah. It’s easy to get lost in the fixing – we’re here to fix stuff.
Michele: I think so and I think like – and then also having that freedom is suddenly sometimes different. It’s like oh, we don’t have to only just fix stuff. We can make this – they love it, let’s give them more. You know that might not be a bad idea. Like can we make it so they can spend more time – I think in this case, like for a lot of the knowledge workers, let them do more time analyzing. Like if they’re really good at coming up with these insights, they want to find the best office for their clients, well like they don’t want to be just entering the data. They want to be reviewing the images that we’re providing, the analysis, and let them do that. And that might be something that they love, that type of things.
Steve: Right. It reminds me of projects where the assumption going in was we were going to kind of remove steps and save time and remove effort and that you find that there’s parts of problem solving, or all sorts of weird parts of our lives, that people find satisfaction or even like joy and delight in.
Michele: Yeah, right.
Steve: And that our job isn’t to just eliminate stuff. Like you said, there’s parts of the work that are…
Michele: Right, right. I remember interviewing someone when I was working on like a – I think it was a small business. And she was like well I love paying the employees. Like it took a lot of time, but at the end of the week like she felt successful. She was happy to pay her employees. So even doing – like understanding that step, like when she’s using our product, but she’s happy. It’s not a chore. It’s – you know it’s not a chore. Like she’s happy to pay her employees and she wants to reward them. So, like I think having some of those insights and saying like oh, this is a potential delighter, how do we build on that, is potentially something to keep in mind.
Steve: Right. That seems like a potentially big reframe from small businesses don’t want to spend money. Every loss is a loss. Every outlay of money is a loss. And this person is finding meaning in that.
Michele: Yeah.
Steve: Because of what it symbolizes to them and what accomplishment there is.
Michele: Yeah.
Steve: That’s great. I love those things in research.
Michele: Right.
Steve: Because you can understand what you’re really providing.
Michele: Yeah.
Steve: So, you’ve mentioned a couple of different contexts like delivering oxygen and so on. Can you talk about some of the other roles, or maybe even just how you got into this work? Maybe we’ll start back there.
Michele: My degree, I got a Masters of Science from Cornell. It’s like in human environment relations with a concentration in human factors and ergonomics. And so, when I started the Internet was just getting started. I think I did my thesis which related to actually some household tasks and products. And it was more about physical design, but I remember like I hand coded my survey because HTML was just kind of starting out and I was literally – they didn’t have Survey Monkey or anything. So, I was doing things like that. My first job after grad school was working for Kohler Company. They do kitchens and bathtubs, but I had the chance to work with a lot of really talented industrial designers. Eventually my career included working for the Consumer Products Safety Commission and that’s where I learned a lot about safety and some hazardous issues that could have been prevented because if usability and design had been better – and the work there was more focused on standards writing and reviewing and it was really informative for me, but I wanted to go back to product development.
Steve: What is the Consumer Product Safety Commission?
Michele: This is a federal agency. You might know them because they do all of the recalls. They also are the people that tend to put those warning labels on different things that come out. There are a lot of voluntary standards that you might – they happen to appear a lot on consumer products. Maybe baby products in particular, but they affect a lot of different products. I remember looking at a gas grill that I think you bought on an infomercial and it kept kind of tipping over and the human factors and the usability perspective of that was that it looked like you could light it in one place, but that actually where you could light it. Again, this is probably a really cheap grill, but the reality is, from a human factors and design perspective, why wouldn’t you light it in this one place? And when you lit it there it actually caused accidents. And so that had to be recalled. But that was a really different, I think, experience than a lot of my colleagues have had. And so that – I was grateful for that experience, but I wanted to get back into product design. I eventually made my way to a medical products company which was exciting because it was starting to blend the physical, like products like sleep apnea and oxygen machines which also had some digital screens and different – you know just different types of products that I think today we take for granted, but back in the day it seemed like things were either all digital and very, maybe archaic, or they were almost all physical with just some lights or something. And so that was where I think it preceded the mobile phones, but I had a sense of designing with kind of physical and digital at the same time, what we would call hardware and software today, but it wasn’t always called like that. I was working with industrial design and product designers who were physical product designers at the time.
Steve: How did you find your program – your school program? How did you find your way to that?
Michele: That I think was a fun story. I did my undergrad – I wanted originally to study architecture. I spent a little bit of time with kind of the intro to architecture classes. I decided that actually I didn’t want to be an architect. I wanted to study the psychology and I think I somehow understood more about designing for the people, but I didn’t know anything about what it was called. And so, I finished my undergraduate degree in psychology and eventually I started just talking to people. I think the OXO Good Grips that come out – I think I did an interview – I think I did like an informational interview to say what is this? This seems really interesting. I think I called somebody and I think I got some leads and eventually I remember just talking to everybody until somebody finally said we think this is your field and I think that there was a program – one was at Cornell and one was in Wisconsin. And eventually I think I just found these programs that said this sounds like what you’re doing and I applied and I was lucky enough to get in and then I found that program and that’s how I did that. It was definitely a researcher doing research to do that, but also really using your network and just saying I think there’s a way to make all these things easier. It seems like people are doing this. I know there’s something related to psychology and design. I don’t know what it’s called, but I think there’s something here. People seem to be doing this type of thing.
Steve: If you – hindsight is, of course, 20/20, but would you – is there a different path – given where you are now, is there a different path you might have taken?
Michele: I don’t know. I mean – yeah, from psychology, I mean in terms of – I think, like I said, I wanted to apply it. Definitely I could have gone to law school. Like could have done really, really different things. But I don’t think – I think it would have been great if the UX program had been more – if I’d been more aware of the UX programs if things today had existed. But I think I kind of had to make my way through the career and I’m happy that I got a chance to, again, like work on physical and digital and mobile products and things that I think it’s – because it was sort of me trying to figure it out and maybe that’s not as different from other UX people that oh this seems interesting, maybe I can go work in this area, or there’s ways to apply this. I think it worked out okay.
Steve: I’m sure there’s some analysis we could do of kind of when you were in school or when you were sort of looking for your jobs what you were aware of in terms of what kind of work there was, and what kind of educational options there were. Because, right, maybe if you’re starting now you have access to certain choices. If you started when you started, or when I started, you and I were doing different versions of finding our way with some serendipity and some sort of opportunistic research, but the context was so different.
Michele: Right. I think I would also add, I think the assumption was I had this undergrad degree in psychology. I think I just got a job – like I wasn’t doing anything in the field. I was working in a publishing agency, which was great, but it wasn’t – I was really doing that to pay the bills essentially. I didn’t – had I been able to be an apprentice, had I found something else, maybe I wouldn’t have gone to grad school. I think the assumption at the time was there’s something out there. I think I probably assumed you had to get training and go to grad school. And so that’s the path I took. But potentially, in this day and age, you might do more of an apprentice, or you might join a bigger company and see if you can get training on the job in that sense.
Steve: Just hearing you talk reminds me that there’s a history of this profession. And relatively speaking we’re still talking about recent times, but it seems like the field goes back longer than we might – maybe be collectively forget the history.
Michele: Yeah. Definitely, I mean, I think that’s an unknown right, that it goes back to people working on the planes and the pilots – you know that the controls were in the wrong places and then they had to standardize, but that was really kind of the field originally or human factors. I think a lot of the work they were also doing, as I understand it, probably with the early computers, even with NASA and things in the 60s, that was definitely pioneering the work of human computer interaction which sounds today so goofy to say human computer interaction, or human systems interaction. But it definitely goes back decades, way before us, and generations.
Steve: Right. You have a human factors degree. I have an HCI degree, you know human computer interaction. And those seem like rare terms. I don’t find people with that as their background as much. We didn’t have the phrase user experience, or UX, or user research, or any of those things.
Michele: No, no.
Steve: So, we were finding our way.
Michele: I don’t think I had the user research title until about 10 years ago. I think that’s also when some of the industry changed a little bit, but that people were doing task analysis. They were trying to understand the system. They were interviewing people. They were testing, whether they used the work usability testing or prototype testing or concept testing, they were essentially going through all of the things that we do today. They just might have called it something different.
Steve: I mean and some of those sort of classic examples of human factors research and some of the things that these examples, there’s just a lot of detail to sort of the decisions that are being made.
Michele: Right.
Steve: A few minutes ago, we were talking about reframing our understanding of what the problem is. So, there’s a – even that’s the – there’s other axis we consider, but just among that sort of detail vs. big picture it seems like there’s a big range of what researchers do.
Michele: Yes. I think researchers have a really key role to play at both parts of the double diamond process, if you think of the traditional, you’re trying to design the right thing and can you inform that? What are the key insights? Are we going to build the right thing? Maybe researchers are helping to take those insights into a high level, MVP, is this even the right fit? But once you’ve decided to build the right thing are you building it right? Can you test multiple concepts? Are there three or four ways that you can actually figure that out? If it’s totally new to the world, even if it’s for the future, maybe there’s two or three different ways you can do a new feature and then can you test that and iterate that? And so sometimes it gets into really detailed design, all the way down to the icons, the placement on the page, the labeling and all those other aspects that really go into refining a product and making a great product.
Steve: And you like that part of it?
Michele: I like both. You know, I think – I love the idea of really working at the upfront doing really generative and informative research and going out and saying we’ve interviewed all these people, what’s really working, what’s not. What’s their workflow? What are like – what are things that nobody’s found before and how could we maybe take this into something new? I love doing like rapid experiments to say maybe there’s something we could go based on that and taking that into what you might call an early MVP, but of course yeah, once we do it I want to make sure things make sense and that it’s really easy to use and so that’s all dear to my heart at the end of the day.
Steve: When you think about the user research profession, or the practice, whatever we kind of are collectively, do you have any hopes or goals in the sort of medium term for this field?
Michele: I hope that we work more closely with all the different, I guess you would say, factions of user experience and design. On one hand I love that there’s a zillion different meetups and I can go on Eventbrite and see anything related to UX and research. On the other hand, I get bummed out that I’m like oh actually they should have just talked to that other group. Like there’s a lot of groups – there’s UXPA. There was IxDA, I don’t know if they’re still active? But there are so many groups that have either been active or could help some of these, what I’m seeing as random meetups. And I would love if some things got a little more organized, or if there was just way to say you don’t need a new Slack channel. You can talk to all these people. There are communities. And letting people know. And maybe that’s, at a very high level, like just formalizing it. Yeah, I’ve been in the field for a while. I don’t like that we still go in circles about what the title is and so I’m okay not having the same titles and things, but I would love people to know like there is definitely a very long history of people doing this work and making sure that people have access to that and the resources.
Steve: Yeah, it seems like we – as people and certainly the technology profession, we have a short memory. We’re attracted to kind of the shiny. I mean this was already a long time ago, but I remember someone telling me a story about – I think that they were at an agency and they went in to talk to somebody about doing research and the person said to them, alright, what’s new, what do you got? And like even that idea that a thing that you’re going to be doing with research should be new. I mean obviously contexts change.
Michele: Right.
Steve: And we adapt to them, but just the idea that new itself was sort of – was a value, seems to sort of fly in the face of what you’re saying, there’s a whole history here that we’re leveraging. We don’t need to reinvent it.
Michele: Right, yeah. I’d probably be happy just to let – if there was like a blog post or a tweet that said hi, there’s an entire history, here’s fields if you’re new and getting started – here’s three or four fields that you might want to check out, related fields. Here’s – you know there’s tons of templates and resources that people have been putting out over the years, here’s 10 of them, or something like that. And here’s – if you’re anywhere in the world there probably is a meetup that’s related to what you’re looking for. It just might be called UX or HCI or research or design research, or something else.
Steve: Right. All the variations of all the terms.
Michele: Yeah.
Steve: Is there anything else you think we should talk about today? Anything I didn’t ask you about?
Michele: I think this has been really positive. I’m looking to chat with anyone else if you’re also trying to establish research as more of a center of excellence and building up the processes and the infrastructure that go beyond one team or a series of projects. I’d love for you to reach out.
Steve: What’s the best platform, or whatever, for people to connect you on? We’ll put it in the show notes for the episode.
Michele: Twitter is great.
Steve: Okay. Well, thanks very much, Michele, for a great conversation. I really appreciate your time.
Michele: Thank you very much, Steve.
Steve: Well, there we go! Thanks for being here for this episode. You can find Dollars to Donuts wherever you do your podcast listening. Didn’t we used to call that podcatching? I don’t know. You know you want to give us five stars on Apple Podcasts am I right?. Visit Portigal dot com slash podcast for all the episodes including show notes and transcripts. Our theme music is by Bruce Todd.
The post 23. Michele Marut of CBRE Build first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Vicki Tollemache, the Director of UX Research at Grubhub. We discuss how to manage incoming research requests, running a weekly research session for testing designs, and why candidates should come into job interviews with a point of view about the company’s product. To me, researchers […]
The post 22. Vicki Tollemache of Grubhub first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Vicki Tollemache, the Director of UX Research at Grubhub. We discuss how to manage incoming research requests, running a weekly research session for testing designs, and why candidates should come into job interviews with a point of view about the company’s product.
To me, researchers are educators. They’re there to translate and educate the organization that they work with about who their users are, what they’re experiencing, where their pain points are, what they care about, what their motivations are. There’s a number of ways you could communicate that and you can educate. Experience is probably one of the best, but due to time constraints, not everyone can come into the field with us and experience that. If you just rely on reports and communicating from the researchers there’s something that’s left out in the details. There’s a richness that’s not there that I think even researchers realize. – Vicki Tollemache
Show Links* Mind the Product * Newton Minow’s “Vast Wasteland” Speech * 500-channel universe * Golden Age of Television * Roku * Roku Guide * Vicki on LinkedIn * Grubhub * Seamless * userinterviews.com * Sam Hall * John le Carré * Anthony Bourdain: Parts Unknown * Usability Sciences * Ron Johnson and J.C. Penney * Motley crew * Mötley Crüe * Chuck Klosterman – But What If We’re Wrong?
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Thanks for joining me on Dollars to Donuts, the podcast where I talk with people who lead user research in their organization.
I’ve mentioned a public workshop happening in San Francisco; it looks like that has fallen through but I will be speaking at the Mind the Product Conference in San Francisco next month. I’m also doing in-house training workshops so let’s talk if that’s something your team might want to pursue.
In my consulting practice I have the opportunity to work with different organizations with varying levels of investment in research, varying levels of maturity in their research and product practices, and so on. I started this podcast as an extension of that, as a way to highlight the emergent practice of user research leadership. So supporting me and my business is the best way for you to support this podcast. My consulting work informs the podcast. It also pays for this podcast. If you have thoughts about the Dollars to Donuts, email me at DONUTS AT PORTIGAL DOT COM or write me on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
In 1961 the new chairman of the FCC gave his first speech, addressing the National Association of Broadcasters. He saw the potential of television, but complained that watching the programming currently available revealed a “vast wasteland.” That phrase echoed down the decades. In 1992 when cable TV executive John Malone coined the phrase “the 500 channel universe.” While that might have signified opportunity to the industry, but to the viewing public it felt like the vastness was simply increasing in scope. Yet somehow we made it today where 500 channels seems almost quaint, and critics herald the Golden Age of Television also known as Peak TV. Well, we purchased a Roku TV set recently. Roku basically provides an operating system for the television where you can add apps, just like you’d add apps to a phone. So we added all the usual suspects like Netflix, YouTube, Hulu, NBC, Vimeo, Amazon, and some others that we discovered like Pluto which is a free streaming channel with a ton of programming. But what I didn’t realize was that Roku is a somewhat open platform that allows interested parties to add content. Looking into how to do that, it doesn’t seem that much harder than putting out a podcast. And while the choice isn’t as overwhelming broad as with podcasts, there are an astonishing number of Roku channels. I found a blog that every week or two updates with the latest channels, not all of them, just the ones that they’ve reviewed! Here’s some of their latest update reviews.
Funny TV Network – A collection of funny clips from Family Feud hosted by Steve Harvey
Know Your Tools – Tool reviews, safety tips, recommendations and innovations
The Home Depot Channel – Tips and tutorials for home remodeling, home maintenance, and tool use
Louisiana Cajun Recipes – A Cajun cooking show hosted by a self-taught cook
Smoky Ribs BBQ – Essential grilling recipes
Stories of the Century – A 1950s Western TV series about a railroad detective who roams the west, tracking down outlaws and bandits who are preying on the railroad
SOS Coast Guard – A 12-chapter 1937 film serial starring Bela Lugosi and Ralph Byrd
Sci-Fi Zone – 19 vintage Sci-Fi movies from the 1940s and 50s
DramaTV – Vintage public domain dramas from the 1940s and 50s
Scary TV – 20 vintage horror movies from the 1930s, 40s and 50s
Shemaroo Yoga – Yoga tutorials from Anushka Malao
Aircraft Channel – Aircraft accidents and near accidents reconstructed with analysis of what actually happened
Rockwell Off Road – Mud-bogging and proving grounds videos
It’s not particularly different from what’s already online on some platforms, but I was surprised at how the bar for “television” had got, I certainly expect a lot of crap from TV in general but this is barely curated Internet detritus that mingles with TV channels from established media players. I’m not saying all this content is necessarily garbage, or isn’t of interest to some people, perhaps many people, but that my mental model for changing channels on a television set, even if I can’t pick anything over the air where I live and haven’t had cable for many years, that’s still an entrenched mental model, so to find that this new television lets me watch NBC and the Opossum Saga at the same navigation level in the menu is just surprising. I think they’ve got some way of streamlining the experience so searching on the platform will more likely reveal big brands that have more traffic or that perhaps have paid slotting fees. You won’t come across The Lawnmower Channel unless you know to look for it, I think. And just like I’ve mentioned in previous episodes these shifts in mental models, in how producers expect something to be used versus how consumers expect something to be used, these are fantastic things to explore in user research, especially as systems grow in complexity and scale, like this 5 zillion channel rokuverse.
Well, I think it is time to get to the interview. I spoke with Vicki Tollemache who is the Director of UX Research at Grubhub. All right, well, thank you for being on the podcast.
Vicki Tollemache: Thank you for having me.
Steve: Why don’t you start by introducing yourself? Who are you? What do you do?
Vicki: I am Victoria Tollemache, but everyone calls me Vicki. I am the Director of UX Research here at Grubhub. Essentially, I manage a team of researchers across our ecosystem and my job is really to make sure that we are working in a way, doing research that’s going to impact our organization. So, it’s a lot of strategy. I’m working with product leads and our design VP to make sure that we are positioned correctly and chasing the right questions.
Steve: Hmm, chasing the right questions. What does that mean?
Vicki: That means I mean – my research team – I think generally research teams are much smaller than the design organizations and product organizations they’re supporting. We have a million questions coming at us of all shapes and sizes and it’s determining which questions make the most sense for where we are as a business and which questions, if we get answers to, will the business be able to respond to and we’ll be able to have impact? So, making sure that we’re positioning in the right space.
Steve: Right. You mentioned impact as well right off. So, what does impact look like?
Vicki: Impact to me is, in the scheme of things maybe it’s identifying needs within our users ecosystem that we could create solutions for, to better solve for pain points they’re experiencing. Or it is – maybe we’re creating a new experience and we’re not quite certain if what we’ve created matches the user’s mental mindset. So, doing research at that point that our product and design teams can respond to and make changes to.
Steve: Does that fall under the label of evaluation? It feels like there’s something else to it.
Vicki: It can be. What do you mean?
Steve: I mean maybe you’re just describing the way I’d like to see evaluation done where it’s not just thumbs up/thumbs down on…
Vicki: Oh no, no, it’s iterating right through the process and we’re working with design to be like what can we change about it to make it a better experience? And then potentially testing again.
Steve: And you’re framing it around mental mindset which is sort of the underpinnings of a concept.
Vicki: Sure, absolutely.
Steve: Not the implementation of a concept. I think that’s why evaluative to me is like – evaluative is maybe about the details of the design?
Vicki: Sure.
Steve: And so I feel like maybe there’s another word that describes what you’re doing which is looking at the value proposition or the construct, or the mental mindset.
Vicki: Sometimes we call it an experience audit here, but I feel like that’s a term we’ve made up ourselves. I don’t know if I’ve ever heard that in the field or in the wild. Because we were trying to describe some of the things that we’re doing and we’re like are we auditing the experience?
Steve: Not that you asked my opinion, but audit seems more passive than what you’re talking at. To get at that gap between a mental mindset of a consumer of something and producer of something is more – that’s more – you have to extract that. You have to synthesize that.
Vicki: Sure. I think I’m speaking to two different types of research, right. One is when we’re going out in to the field and trying to understand that environment that our users live in, and especially where they meet, because we tend to find that that’s where a lot of the friction points are. And then working with design and product to be like how do we solve for some of these problems that we’re finding? Are these problems that we knew existed even? And then once we’ve gotten to that place where we’ve come up with solutions, then pairing with our users to determine have we solved for this in a way that actually makes sense for the users?
Steve: When you say “where they meet” in that first part?
Vicki: So, I mean Grubhub is an ecosystem, right? We have what we consider four distinct types of users. We have drivers. We have our restaurant partners. We have our diners – these are the people who are consumer facing. They’re ordering from us. And then we have internal employees. I mean the easiest, most basic way to explain that is we have an internal care provider that provides help to all our different users and those users don’t exist by themselves. They’re constantly interacting with each other, right.
Steve: Yeah.
Vicki: And usually they’re interacting with each other when there’s problems. And so either we can help them solve those problems, or sometimes we can maybe be a hindrance in how they solve those problems and they have to try to work around us. So, those are sometimes some of the biggest opportunities for us.
Steve: Okay. So, where there’s interactions or intersections between those different…
Vicki: Types of users.
Steve: In the ecosystem.
Vicki: Yeah.
Steve: Is opportunity.
Vicki: Opportunity, yeah, for us to – sometimes it’s about providing them more independence and building trust, right. And then there’s understanding why is there trust breakdown and why do they feel like they don’t have autonomy to kind of control the – the situation that they’re in.
Steve: Maybe we could step back.
Vicki: Sure, we did kind of jump right into it.
Steve: That’s me, not you. So, what is Grubhub? It’s an ecosystem of these things, but what is – how do they combine?
Vicki: I think Grubhub considers themselves a marketplace that is providing restaurants the opportunity to compete for consumers in regards to ordering food for delivery or pickup. So, that’s what we provide.
Steve: In what parts of the world are you running services?
Vicki: The U.S. and then we have a little bit of presence in London, but that’s generally on the corporate aspect of it, right. So, we also have a corporate platform where companies can partner with us and provide their employees a credit and then the employees kind of – it keeps them at work if they can order lunch at work and kind of eat through a meeting kind of thing. So, we have a little bit of presence in the UK with that. But otherwise, I think we’re in 300 markets in the U.S.
Steve: Is that a lot?
Vicki: I mean I think we have a very good presence in the United States. I think if you went into any major city, or even minor city, we would be there, right. So, we’ll be in Little Rock, Arkansas. We might not be in Jasper, Texas. And I’m using references from the South because I am from the South, so it’s easier references.
Steve: Okay. Alright. That’s good context. So, let’s – so, you’re sort of describing some of the things that you’re working on. And you made a point that research teams are often smaller than design teams.
Vicki: Yeah, product and design teams, right.
Steve: Yeah. And that’s the case here.
Vicki: That is the case here. I think – there are 7 of us currently, including myself. We have an open position, so that will be an 8th person. And then we also swell in the summertime because we usually bring in one intern. And last year we were fortunate enough to hire an intern, so it’s a good opportunity for interns as well.
Steve: What does a user research intern do at Grubhub?
Vicki: Well in the days since I’ve been here I always want to win best internship. That’s always my goal. I want that intern to get as much experience as possible. Usually they come in and they tell me they would feel more comfortable getting moderating experience, being able to run projects on their own. Last year we had the intern own – it felt like the easiest place for them to start was kind of doing more smaller research questions, so more evaluative research. So, they took over – I think we called it Research Day – and essentially ran that for the entire summer. But then we also took them out into the field with us for one project. So, they got to go on a trip and experience what it’s like to be within the restaurants, shadowing the drivers, us going into the diner’s home and learning about food. So, they’re owning one thing, which is kind of nice right, that they get to own something. They’re getting experience moderating. They’re putting projects together. They’re working with product. They’re working with design. And because Research Day goes across the entire ecosystem they’re getting exposed to everything. So, it’s like a really good place for them to start and then they will get to partner with some of the researchers. Kind of like a more mentor/mentee and go out into the field on some of our bigger research projects.
Steve: What is Research Day?
Vicki: Well we called it – at first it started as Diner Day. It’s just a standing day, once a week users come into the lab. The week before the designers can pitch ideas they have. We go through, based off how thought out, flush out the ideas, priority based against our organization’s needs. What are we going to be testing? Do we have access to it? Can we get the right users in? So, we go through and we kind of have a pitch day. We choose the idea and then 5 days later we test that idea in a lab. So, it’s supposed to just be really, really quick – 24 hour turnaround for findings. Just a bulleted list of findings that the designers can then go take action off of.
Steve: And that runs?
Vicki: Every week. It’s really frequent cadence. I found in past organizations sometimes people can push off doing research because they say it takes too long. It takes too long to go through scoping. It takes too long finding people to recruit. So, I felt like if we had this standing day setup for diners – just bringing 5 diners in once a week – it meant that designers could kind of come in last minute sometimes and we can get research done for them. And they can also experience that and watch that because we’re in a lab based situation. Or we can also go on the streets of New York. But in the lab base it’s kind of nice because we can also do remote interviews and reach out to people outside of New York, because New York is kind of an outlier compared to the rest of the country in regards to what our platform looks like. Also, people are very experienced ordering food from Grubhub or Seamless here and sometimes we’re looking for a split of like maybe less professional food orderers.
Steve: Seamless is another platform.
Vicki: Seamless and Grubhub are the same thing, just different brand. Eat24 is part of our brand as well. But they’re all the same. There’s just a different brand logo in the top left hand corner.
Steve: When you do Diner Day or Research Day, so a tactical question, you don’t have a lot of lead time.
Vicki: Um-mm. It’s quick and dirty.
Steve: So do you already know kind of what people are coming in before the idea is pitched? What’s the dependency there?
Vicki: Generally it’s a mix of diners, right. It’s starting to get harder and harder to find truly new diners to our platform, but it’s a mix of new or returning diners and we can sometimes switch what that ratio is within the 5 days we have from doing the pitch to actually doing the recruiting. So, we just know we’re going to bring 5 diners in. Some of them are going to be new to us, never ordered from us. A percentage of them are going to be returning to us, or we may shift that where it’s all returning based on what the project is.
Steve: So, you have kind of infrastructure in place such that you can quickly change.
Vicki: Yeah. That’s the nice thing about food, right. I mean I worked – I came from AT&T before this and I worked on the B2B side and we had to recruit network engineers and it was really difficult. It’s expensive. They don’t have much time. It’s hard to get them in. But so many people order food. It’s just an easier recruit, right. So, within 5 days we can get those people. We used a platform called userinterviews.com. Have you heard of it?
Steve: Yeah. I haven’t used it, but I’ve heard of it.
Vicki: It makes it very easy. And they’ll recruit anywhere for us in the U.S., so it’s super nice and they turn it around very, very fast.
Steve: So, once you understand what that flow is going to be…
Vicki: We have that relationship and they know that we’re doing that on certain days, so it’s just like an ongoing recruit that we have going.
Steve: Okay. Just coming back to one of the things you said, you mentioned a couple of times kind of aligning the research activities you do, whether it’s at that level or other things, with the business questions, the goals of the organization. And I think maybe you mentioned this, but maybe you could talk a little bit about how you come to an understanding of what those goals and questions are for the organization so that you can choose the research activities that support that. How is that input for you?
Vicki: I think when I first came – I mean I’ll be honest about this too. When I first came in it seemed like – and we still have this to a certain extent and I think a lot of companies deal with this – ongoing reprioritization of everything. We begin to work on something and then it will be like, no our priorities have changed. So, we had like a lot of like stop/start projects. And at the time that didn’t make sense to me so I wanted our research team to prioritize things based off what we knew our users needs and identifying those. And then we had little bit of leadership change and we had a new CPO come in, Sam Hall, he’s from ClassPass, he’s great. And he’s actually – I work pretty closely with him to understand like what the business priorities are. And I work pretty closely with our design VP to understand some of the design challenges that we’re facing as well. And even for something like Research Day we have an ongoing backlog of research questions that’s coming from the designers and/or product, but there’s some weeks where we don’t have anyone come to pitch and then my team has questions that we will go answer. So, we don’t not do a research day because product and design aren’t asking us questions, but we have questions as well based off things we’ve seen in the field and some of those can actually be themes that come up later. So, in the end, my answer is it varies. Working closely with product helps us from a business perspective. Working closely with our design leadership helps me because they have questions that they are also separately going after based off kind of some of the design questions that they have. And then I just work to prioritize those. We do a scoring. Does that answer the question?
Steve: A scoring?
Vicki: Yeah. So, certain people get a weighted score and in the end it’s also kind of what my team thinks is going to be the most valuable for us, and also where we’ll have impact. I mean I worked at organizations where they’re like every year they set a goal for the research team to do more research, like more projects. And that doesn’t make any sense to me. It should be about doing projects that actually impact the organization, right, that people actually make changes from. You can do a million research projects. If no one does anything with that research it’s like dealing into the abyss.
Steve: So, if I come to you and say I have a question about how people go through this flow in the app, how would you score that?
Vicki: Who are you? Are you a designer? Are you within the leadership team? And then what is that flow? Like what part of the experience is that? Is that something that we have as an initiative that we’re heavily focused on that we’re invested in as a business to like make a better experience? So, I’d have to ask some of those questions and then I’d also look at do we already have previous research that might be able to answer some of your questions? So, there’s a number of factors.
Steve: And sorry if I’m getting like obsessively detailed, but like are you – when you say score to me that means there’s a bunch of questions and you kind of apply a number to those and total them.
Vicki: Sure. Yeah. And it does work like that. It depends on who’s asking for it? Where we are in the product lifecycle? Like if we do this research for you will you be able to make changes from it within this release? Or are you telling us you might be able to make changes six months from now. So, we’re asking – do we have impact? Who is asking for this? How does it align to our business priorities? How does it align to our research and design priorities because we’re part of the design organization? And then there are certain scores that go with each of those answers.
Steve: Okay. I think I’ve heard those questions as kind of an intake or prioritization approach, but I haven’t heard it labeled as scoring which I think makes so much sense when you say it.
Vicki: And then I usually – because in the end our design organization is within our product organization and I usually also work fairly closely with our CPO to be like this is what I’m getting from the design organization. This is what I’m getting from your product leads. This is how we’re prioritizing. Just to make sure. Like it’s a gut check, like does this seem right?
Steve: So, I mean, there’s the example where if nothing comes in for Research Day you have your own things.
Vicki: Yup.
Steve: Are there situations where you’re proposing research to these kind of leaders that you’re working with?
Vicki: Absolutely. Last summer we had a new VP of design come in and one of the things he really wanted to push is quality of our product, right. And then the question came to me, like how do we measure quality? So, there’s a number of ways that we could go about measuring quality. I work with the data science team to go after certain metrics, or to understand how we might be able to measure that within some of our data. But then from my background I was like we could also do benchmarking. Like that’s something we haven’t done before and we rely really heavily on A/B testing which sometimes is like building things very piecemeal. So, I proposed that we go through and do a twice a year benchmark after some major releases to measure like what is this experience for users in our current product. And then if you look at the definition of quality you can only truly measure quality by comparing it to something of greater or lesser quality. So, the suggestion was where we can can we also do a comparative where we’re benchmarking against one of our competitors. And they were completely on board with that. And they allowed us to go to an outside vendor and go through the benchmarking process.
Steve: I thought you were going to say you’re benchmarking against yourself over time.
Vicki: You can do that – we’re doing that as well, but also against our competitors. When we can. We can’t do that obviously on the business side, but we can do that on the diner’s side.
Steve: You can’t do it in the business side because you can’t get access…
Vicki: We don’t have access.
Steve: …access to that part of it.
Vicki: No.
Steve: Okay. I see.
Vicki: And that’s always one of the things – I’ve heard of researchers who will ask – generally our restaurant partners or driver partners, they’re also using our competitors, but from an ethical perspective I don’t like to ask those types of users to share that information. I just don’t think it’s appropriate.
Steve: How do you benchmark – you know when you’re doing competitive benchmarking, how do you compare apples to apples?
Vicki: Our product teams are broken up into initiatives. On the diner side we have two initiatives. One of them is more about like the core product, right. And then benchmark is really about is our core product usable. Like where are there friction points. So, I worked with our product lead and our designers to identify like what do we consider core tasks? And those are based off like what are common like user flows, user journeys, through the app? But it also was based off where do we also know that we see CPO, so care calls taking place, because we also are always trying to lower those, right. So, from that we’ve mapped out like what our tasks were and then we took another one of our competitors, who essentially is going to have the same, similar tasks and we just did a benchmark against those tasks for those competitors. So like, you know task timing, success, ease of use. We also created this quality metric which is kind of based off of reliability. And then also just qualitative feedback, right. But there’s other things as well. There are things sometimes we hear from our users that are taking place that maybe we weren’t aware of and we definitely will go to product and design and be like hey we think that we need to do a deeper dive here. We’re hearing a number of things. There’s friction taking place that we weren’t aware of. Maybe we’ve made changes somewhere – and that’s the thing with an ecosystem, right. You might think you’ve made an improvement in one space and you’ve actually created two problems somewhere else and we’re definitely given the freedom to then go explore that and suggest solutions for those additional problems. Or maybe roll back the first solution to begin with.
Steve: You brought up solution as part of the research process…
Vicki: Yeah.
Steve: … a couple of times and I wonder if you could just say more about what’s a researcher’s role in – what is…
Vicki: A researcher’s role in solutioning?
Steve: Yes.
Vicki: Um, well, at Grubhub, again because we’re a small team, I think in some ways I wish that we could be more hands on than we are. Maybe we’ll identify a need and our product team has also had a hypothesis that that’s a need as well, so they’ll be like how do we solve for this thing that we’ve identified? It’s an opportunity for business. We see that it’s an unfulfilled need for users. Perhaps we’ll put together – we’ll define what the problem is. We’ll put together a workshop. A researcher will usually be involved in that workshop, but sometimes they can’t be so then maybe one of us come in as like a speaker, to speak to like – see the work and solutioning that they do and comment on it or consult on it. But it usually begins there and then we’ll go through rounds of iterating on the design as we test it. And a lot of that will take place in Research Day. Does this make sense?
Steve: It makes sense. Maybe you have thoughts about sort of the philosophy.
Vicki: I think my team would like to be more hands on with what takes place in those workshops, but we just have a limitation of time.
Steve: Yeah. Why do you think that’s something that they want? If you look at researchers around the world, what – I feel like some are very interested in the product, the details, the solutions.
Vicki: Absolutely, right.
Steve: Some are – some maybe don’t see that as their role.
Vicki: I think there’s always like, just from a human perspective, a desire to control. So, I mean, from that basic level it makes sense to me that people want to be more involved in solutioning, but I also think – you know researchers, they’re out in the field. They’re witnessing that. Maybe they write a report. Maybe they communicate their findings, but sometimes they have to be fairly like theme, high-level findings and there’s a lot in the details that can get left out of reporting and I think they feel like they provide value to be in those solutioning sessions because they can speak to some of that nuance that maybe wasn’t communicated within a report, right.
Steve: Yeah. Well even in the example you gave where there’s just no bandwidth, that maybe you guys are working in a consulting role.
Vicki: Yeah. I mean when you have a smaller team, unfortunately it becomes that way, right.
Steve: And whether you’re consulting, or in those sessions, you’re still there to – it sounds to me – so, tease apart – I think what you mean by solutioning is a collaboration where solutions are generated, but it’s not that researchers are there to do design.
Vicki: No, yeah.
Steve: But they’re there to bring that extra detail to inform those design decisions.
Vicki: Yeah, I mean – I think to me researchers are educators, right. Like they’re kind of there to translate and educate the organization that they work with about who their users are, what they’re experiencing, where their pain points are, what they care about, what their motivations are. And there’s a number of ways you could communicate that and you can educate. Experience is probably one of the best, but due to time constraints of everybody, not everyone can come into the field with us and experience that and I think there’s something to be said about – outside of not having the experience yourself with users in the field, if you just rely on reports and communicating from the researchers you kind of – there’s something that’s left out in the details, right. There’s like a richness that’s not there that I think even researchers realize. Like if my presence is there I can speak to some of those one-off situations that maybe we didn’t cover in the report, but might get brought up within solutioning. Does that make sense?
Steve: Yeah, yeah. I think we’re sort of unpacking what solutioning looks like. That it’s not – that there’s a dialogue and a bunch of different perspectives that can come forward and that the researcher isn’t necessarily there to say oh I think we should start the task here instead of over here.
Vicki: Yeah. Yeah. It’s more about bringing that richness of like, if we start the task here this might actually happen to some of the users. Whereas if we started it here maybe that won’t happen, if that’s a good or bad thing. Does that make sense?
Steve: Yes.
Vicki: Just that deeper understanding. I really do believe that researchers tend to just be educators.
Steve: That’s a good metaphor, isn’t it?
Vicki: Yeah. It’s what it feels like to me. I remember – well I was reading this book recently, John le Carré – do you know who he is? He wrote Tinker, Tailor, Soldier, Spy.
Steve: Um-hmm.
Vicki: And he was talking about the role of a war journalist and it’s really not about reporting on just what’s taking place in the field, but it’s also about like building empathy and trying to encourage people to care. I kind of feel like in some ways we’re kind of like war reporters, based off his description. We’re out in the trenches.
Steve: Right. You want to bring it back in a way – I mean it’s back to your impact point early on.
Vicki: But there is a lot left out when you communicate through reports and findings, right. Sometimes you’ll have a user who’s extremely insightful, but do you just report to that one thing, or is that like a one-off? But then you’re in solutioning session and you’re like oh, well it’s actually useful here. This is where I should share it.
Steve: So, yeah, if we took away the constraint of limited resources or just the bandwidth that the folks on the team have, what are the ways that they could be educating?
Vicki: Well – I mean that’s the thing – not just taking constraints from my team, but I think one of the things realistically, I wish that we could take – or I wish the product and designers – product team members and designers could come into the field more and have that experience more, right. I think by them being better versed in what that experience is like within different markets, within different types of restaurants, with different types of users, they have a better kind of base to make decisions from. They’re more informed. They have a deeper understanding. They understand what some of the more nuanced situations might be and therefore they can make informed decisions. But unfortunately with meeting schedules everyone is like strapped for time, right. Like sometimes the reality is they just can’t come out into the field.
We have another program here at Grubhub called Parts Unknown. I’ve branded all of our research programs. We also do – it’s not all within programs, but Parts Unknown, started by – we were inspired by Anthony Bourdain, obviously – this was before, unfortunately, he committed suicide, but bringing people out into different markets and taking a deep look at what our ecosystem looks like outside of New York and Chicago, because we are kind of outliers, and understanding like what food culture is in these cities and how people think about food and what types of food they order and how that impacts our ecosystem and what our ecosystem looks like there, and we were able to bring – and we still do this – product and designers into the field and I feel like that has been some of the more like enriching, like those people come out with a much, much more informed – they’re much more informed and I think they’re able to make better decisions as a result of it.
Steve: That sounds like it’s kind of an immersion experience.
Vicki: Yeah, yeah. And sometimes we were partnering with food bloggers too to have a night with them, just to understand like their take on the market. It was pretty interesting. So, we still do that, but we’ve also kind of evolved that program as well. It’s not just about looking at different markets. It’s also about looking at different parts of an experience that maybe we haven’t looked at in a while and trying to determine we think it’s this, is it actually this for our users?
Steve: Right. Is this a category where you look at the different people in your ecosystem – is there – how rapidly do these things evolve?
Vicki: What do you mean?
Steve: I mean like how static are the things you know about different people in the ecosystem?
Vicki: Well I think as our industry changes and people are competing against each other within our industry that there are things that are happening that are changing the ecosystem on a pretty regular basis. Does that make sense?
Steve: Yeah. So, hence the need to – if there’s areas of the ecosystem that you haven’t looked at in a while, you have to go back…
Vicki: We have to revisit. Absolutely. And that’s kind of – this is again where I get nervous about if I should speak to this or not? So, some of the things – we have just certain areas that we haven’t looked at in a long time. I mean our company is 20 years old and we’ll be like oh this is our workflow and then we’ll have our CPO be like I want us to take a look at this because I have questions about what this workflow really is. So, we’ll go through and do this thing as an audit. The researchers will sign up as users and go through an audit with that experience, what it’s actually like as a user. And then we can pinpoint like this is what we thought it was, this is what it actually is. We think it takes 10 days. It actually takes 30 days and this is why it takes 30 days. That kind of stuff.
Steve: Yeah. So, if there’s so many different facets to the experience in the ecosystem and there’s change happening.
Vicki: Yeah. And you have – within our – I mean it’s not just our product organization, right. Like we’re working with an ops team, a sales team, we have a marketing team and sometimes different teams are building different parts of a flow and they aren’t coming together to check like does that all make sense. Isn’t it always like the idea of like your product is a reflection of your organization and how well they work together?
Steve: Right. What’s the cliché about don’t ship your org chart. I don’t know who to attribute that too.
Vicki: Yeah, I think that is a cliché, yes.
Steve: Okay. Can we talk about your team a little bit?
Vicki: Sure, what would you like to know?
Steve: What kind of people have found their way to your team?
Vicki: Well I inherited the majority of my team and they kind of have a pretty diverse background. We have a couple of – I think my team is comprised of a designer/comedian, an industrial designer, someone with a sociology background, someone with a psychology background and then two researchers who actually have like human factors degrees. And they come from a variety – most – only one of them, our intern from last summer, this is her first job. The rest of them come from a variety of other industries to get here. And one of our researchers, our associate principal, he and I have worked at like 3 to 4 other jobs together. I’ve known him for a very long time at this point.
Steve: I feel like I would be remiss if I don’t ask about /comedian because that’s just there to be asked about.
Vicki: He does like a whole bunch of live storytelling, but I think it makes him a very good presenter. He kind of has like charisma on the stage. I’ve actually had a – when he presents I have people reach out to me to be like you should hire more comedians. Because it makes him very engaging. He’s very good at telling stories, which is why I mention that he’s a designer/comedian background. But he’s worked in research for like, I think, the last 8 years.
Steve: I’ve definitely come across people in research who have either actual, or sort of – they have actual education in theater or some related field.
Vicki: Well you know communication is such a huge aspect of it and if you have that charisma and ease with yourself I think it just gets the message across better sometimes. People are more engaged and feel maybe less threatened by the message. Does that make sense?
Steve: Yeah.
Vicki: There’s something to it. He definitely knows how to charm an audience. I think it also makes sometimes the research a bit more stickier because people are more engaged with presenters like that.
Steve: I think – right, people that find their way into research eventually discover that so much of the work is about working with your colleagues and not…
Vicki: Oh, absolutely. It’s about relationships, right.
Steve: Far less than it is about fieldwork or research…
Vicki: I think that’s anything, right. Like in the end it’s all very relationship based, right. But I think there’s something too, I know the previous director here, he had us do an improv workshop and there is something about – my sister I told you is a comedian. There is something about like comedians in the end generally are introverts who kind of force themselves to be extroverts, which I feel like there’s parallels with that with research. And then there’s this idea like, with especially in improv, where instead of saying no to things it’s kind of that yes/and and building off of each other which also tends to work better with relationships as well. So, maybe it kind of facilitates that. I don’t know. But yeah, in the end it is all relationship based.
Steve: When you talk to people as prospective hires – I don’t know who you hire, but whether it’s the interns or other people, what kind of things are you looking at that kind of strike you as this person might be worth talking to?
Vicki: I definitely want someone who is very analytical. I mean we have – I’ve had candidates come into interview who, when you ask like what do you think of our platform, they just say it’s great. I want someone who has actually maybe taken a look at it from a critical lens and has identified things that they think might be opportunities. Even better if they showed friends of theirs, or family members, and kind of did a small usability test on it and came back with a critique. My team is very collaborative. We get along very well. So, huge egos aren’t a big thing. Like I want people who are going to come in – kind of say everyone brings something to the table, everyone offers something, and I feel like for researchers they learn a lot from working together as well. So, I definitely want someone who is going to fit into that culture and is excited to collaborate with other researchers.
Steve: But how do you look for that in a…
Vicki: An interview process?
Steve: Or whatever – the scouting or any part of that process before you actually are working with somebody?
Vicki: I mean for me, we have a recruiting group, right, who does all the recruiting and they bring in resumes. We talk about kind of what I’m looking for in the resumes. I mean a lot of it I’m sussing out when I’m doing phone interviews for the initial screen, right. And I guess a lot of it is me just trying to get to the core of who they are and how honest or comfortable I feel like they are with describing who they are. The intern that we have right now, I keep referring to her, she was probably one of the best interviews I’ve ever done and she just told me a very honest story about a research project she did, that they presented it, that the company rejected the idea that they came up with, and I was like what did you do? And she was like I drank a glass of wine, it was 6 weeks of work. And then she told me about like, she’s like but then the next day we went back to the drawing board and we decided to do – and it was just like a very honest – she exposed vulnerability. She was very much herself. She was funny. She could speak very clearly to herself. And I’m kind of looking for that. When I interview someone I want to feel like I’m having a drink with someone, that they’re comfortable describing who they are, which I also think is a sense of maturity where people aren’t – I mean interview process is kind of nervous, right, so you’re coming in maybe with a little bit more of a sense of yourself and a little bit more comfortable about who you are and being honest with who you are. I think a lot of people make the mistake of hiding themselves in an interview and I think you want to be very, very open and honest about who you are because if you aren’t it may not be the right job for you to begin with, right. So, I think I’m kind of looking for that. Just some who can speak to what they’ve done and who they are in a very clear and honest way. Does that make sense?
Steve: Do you think that’s a research specific way of looking at candidates?
Vicki: That’s human specific.
Steve: That’s just hiring someone you want to hire.
Vicki: Yeah, that’s hiring – but then on top of that then I’m looking for like, I want to hear examples of projects that they’ve done in the past and how they’ve approached research? I definitely like to hear about how they’ve handled failure in the past and what they’ve learned from it? Situations that they’ve had to work – environments they’ve had to work in that were not comfortable for them, and why? I think generally the most basic questions anyone asks in an interview though. But it’s definitely about how comfortable and honest and open I feel that they are in the interview that is the thing that probably sways me. But also that they have to be analytical. I do have an issue when people come in and they haven’t done any research about Grubhub, or looked at our product or had any thoughts on it at all. Because if you’re a researcher you should be asking questions, right, and doing some research.
Steve: I want to pull apart some of what you’re saying.
Vicki: Sure.
Steve: Because you’re saying analytical, but the example you’re giving is them having looked at Grubhub. Those seem like different things to me.
Vicki: Well, I want them to have – I mean when you go – when anyone goes in for an interview, haven’t you done a little bit of research on the company that you’re going to work for? You’re curious about the product, like what is this product? Like what kind of research will I be doing? Let me take a look at it. Are there a huge amount of problems that I see? Is this going to be like a battle, or is this like oh, I can see like some big – I want someone who has like thought through like what am I going to be working on? What seemed to be the problems here? And I want them to speak to like problems that they’ve identified. Does that make sense?
Steve: Yeah.
Vicki: It’s kind of like they should have been a little heuristic, or maybe even like shown it to a couple of their friends and have an idea of like these are some of the things I think that you might be dealing with. And even speak to like how they might solve some of those problems. I mean we usually also have them go through a practice where we use a site that is not Grubhub to understand like how they look at something? What problems they identify? And then what kind of research solution they would put together for that? But that’s after the phone screen.
Steve: Okay. So, I understand a little more what you’re referring to when you say analytical. There’s a bit there of initiative, but also thinking through something before you kind of get into it.
Vicki: Yeah. And to me, if they can speak to that – like because one of the concerns I have is the communication aspect, especially when you’re doing any sort of like generative, ethnographic work, I mean that’s – I used to have a researcher who called it the big messy and it’s – you go into the field – there’s a lot of stuff that you can focus on, like what do they – how do they hone into what’s important, right. And I kind of feel like I get that from talking to them about like what they’ve seen within our own product. If I have someone speak to the fact that UberEats changed their app logo color and they didn’t like it, I don’t know if I think that that is necessarily a big important finding. Whereas, if I have someone talk about problems they experienced within our search results and like how that might not be conducive to someone looking for food, that makes more sense to me. So, what are they honing in on? What do they consider a finding? And how do they communicate that to me? That speaks a lot to how they might do that for the organization.
Steve: I’ve worked as a consultant my whole career, pretty much, and many years as an agency and I think – maybe the reason you’re hearing me sort of check in on this approach is that I think in the agency lifetime that I had, and certainly my own practice, there’s kind of a consultancy hubris of like coming in and like let me tell you about your thing. So, when you say you want your prospective hires to tell you about your thing.
Vicki: I want them to have an opinion, right.
Steve: Yeah.
Vicki: And I want them to be able to communicate that opinion. And I want to understand what that opinion is based on. And if they don’t that’s concerning to me.
Steve: Yeah. So, I think there’s something here about the process more than the substance of their opinion, but the process – because they don’t have access – like I never want to be the person, personally, just to come in and say look I don’t know what strategy decisions you have, but here’s what I think you should do.
Vicki: I’m not wanting them to lead that, but I’m wanting them to have looked at our product and been like here’s some things I think might be weaknesses, right. Like I’ve considered what your offering is. I’ve looked at it and I’ve noticed these things. I mean that’s the least you can do when you go in for an interview, right? Whenever I go in for an interview I review whatever I can access of the product, depending on what part of the business I’m interviewing for, and I make sure like, if someone asks me what do I think of it that I have like feedback based off some sort of heuristic I’ve done and I can tie that back to some sort of best practice of some kind, right. And so I kind of expect that as well from researchers coming in. If they haven’t looked at the product at all they kind of haven’t prepped themselves for the interview.
Steve: Yeah.
Vicki: And then that concerns me. And also researchers should always be asking questions, right. Like you should want to know like who would I be dealing with here? I’ve had companies reach out to me where I’m like oh my God that taxonomy would be crazy. It’s so much, like how do they deal with that, right. Like I can look at that and then I’ll have those questions for like how do you guys handle that. You can tell a lot by the questions people ask as well.
Steve: Right, that’s another aspect. What kind of questions are you looking researchers to bring you in the hiring process?
Vicki: We definitely want them to ask questions. I’m always shocked at how many people have no questions for me. I mean I expect them to ask about typically how we do research. I know right now we’re hiring for kind of an entry level person. So, a lot of times I’m trying to understand have they approached certain methodologies? How are they like wanting to expand or grow as a researcher themselves? I want to understand from them – like where they want to work, what experience they want to gain? And then also like how we handle research at Grubhub, kind of similar to all the same questions that you asked me at the beginning of this interview. I kind of expect them to be asking those same questions, right.
Steve: Now they’re just going to listen to this.
Vicki: I do think sometimes on Fridays – I call it 30 minutes with like a college grad, because I get a lot of college kids reaching out to me through LinkedIn, asking me how to get jobs. And I’ll talk to them for the first 30 minutes of my Friday and just give them feedback about what I think is a good interview and we’ll go through that. It’s very similar to all the questions that you’re asking right now.
Steve: I’m just the college kid, right!
Vicki: So, from the agency side – I came from an agency background as well. I worked for a company at the beginning of my career called Usability Sciences. They’re based in Dallas. They’ve been around like 30 years. It’s always interesting to me the agency side because I feel like even though agencies want to come and be like this is what you should do, there’s always kind of like this sphere of like overstepping that with the client, right. There’s that like kind of thin boundary and I know talking to researchers who are just from an agency background, they sometimes feel like they have no voice. Or they can’t push back hard enough because they’re afraid they’ll lose the client. And so therefore sometimes they’re not quite – especially if they’ve only had agency experience – I think that’s one of the things they talk about experience they want to get is like being able to work closely with product and scope and push back when they feel like they are more empowered to do so. And that’s something that they lack from agency.
Steve: What about for you? So what are the key things you’ve seen as a difference when you’ve worked in-house or worked as a consultant?
Vicki: Well I started – I left consultancy in 2013 and for me I’d started in 2007/2008 and I just saw a change in the industry, right. 2007/2008 sometimes we’d be working with someone from marketing. They would come in. They would want us to do like just the usability test. That same client would come back year after year, same usability test, no changes. Same usability test, no changes. And then as time was progressing I would see that companies where all of a sudden you had like a research director. Or you had a researcher and they were our go to person. And I could see that people were staffing up inside and that I think the industry was swaying maybe from like agency, or definitely what my consultancy offered, to that offering taking place within the business. So, yeah, I wanted to go see what it was like. Like why are people not making changes and coming back to us year after year? What is the political landscape that they’re in that maybe is leading to something like that? How do you maybe have more control or more say when you’re internal vs. some of the limitations you feel like when you’re consulting? And just to be more embedded, right. To get to work closely with designers and to work – I mean we would come in and do like maybe 3 weeks with a company. We wouldn’t do like a long, long embedded process. So, it was just that understanding of like what is it like to be embedded and work with product and design and what are the limitations they’re facing? Because I could see those within the projects that I was working on, like that was happening, but I didn’t understand why that was happening?
Steve: What was the first job that you got post-agency?
Vicki: JCPenney. That was one of our clients and I jumped over to them. But probably after – I don’t know if you know anything about JCPenney? They went through a big shift where they hired this guy, Ron Johnson, from Apple. And he decided to kill all of their rewards and sales programs. And if you know anything about the JCPenney customer, that’s – customer shops sales first at JCPenney. And their loyal customers are very much versed in that whole points/rewards program, so by getting rid of that they alienated their core customer base, the company kind of took a dive that I don’t know if they’ve really come out of it. I think it’s also like retail has changed, right, and that was taking place in the midst of that change of retail because of online sales. So, I went there. It was interesting, but also the company was doing massive layoffs, constantly. So, then I went to AT&T after that.
Steve: What were you doing at AT&T?
Vicki: AT&T was a closer commute. At the end JCPenney was a 2 hour commute for me each way which is bananas. I was in Dallas at this time. AT&T, on the business side, essentially they had a portal of portals and we came into a product organization that’s first question to me was we don’t know who are users are or what they do. So, there it was just like starting at the very bottom and trying to help inform product who was just desperate for information. So, that was a lot of talking with network engineers. And I have mixed feelings about personas, but that was actually one of the best scenarios I’ve seen for why you build personas. Like if someone is like we don’t know who our users are, maybe you build some personas to explain who your users are. They didn’t have much access to data either. They had little knowledge of who was using the platform and how.
Steve: So they didn’t know anything?
Vicki: Yeah. It’s interesting. There’s like a spectrum of like not knowing anything to like then working in a place where people assume they are the user themselves and that they know everything, right. I mean both come with their own challenges and problems. It’s probably better to be somewhere in the middle.
Steve: And from AT&T?
Vicki: Came here.
Steve: What’s the role that you came into?
Vicki: Manager of the New York – so I manage the New York team which is all the diner facing. And we had a director that oversaw everything. He left, I guess about a year ago now. And then they gave me 6 months to progress into his space.
Steve: So, is this your first…
Vicki: Director position?
Steve: I was going to say leadership in general around research?
Vicki: I’ve been a manager since JCPenney. But this is first director. But I don’t have a manager beneath me. So, I’m still managing as well, right. Everyone direct reports to me.
Steve: Is that the difference between a manager and a director? Can you tell I don’t have a job inside an organization?
Vicki: I think it’s different for every company, but within our org structure most directors have managers and I do not. So, it’s me and six, soon to be seven, direct reports.
Steve: Can we go back then before this agency – what was it that led you into that kind of work?
Vicki: I started my career in marketing research at a small ad agency in Dallas who handled all of McDonald’s local DMA markets. McDonalds provides an allotment of money for national coverage and then depending on the size of the DMAs, the DMAs are given money to do local advertising and then sometimes the DMAs themselves fund additional advertising. So, my agency handled all the local advertising for markets.
Steve: Is DMA direct mail?
Vicki: Designated marketing area. And through that I started doing – they started investing in like creating microsites for like ad campaigns and we would track those through Google Analytics. And what – I was in charge of helping tag those and I’d have to come up with like what’s the user’s journey? Where should we tag? What do we consider success? What are we measuring? And then I was at a Christmas party and I met someone and explained to them what I did and he worked at Usability Sciences and was like oh, you should come work for us. It sounds like you’re just doing like heuristics. ‘Cuz oftentimes when I went through tagging I would discover like there’s no clear path how to get from here to here and that’s one of like the core paths that users would try to accomplish on this microsite. And so from a Christmas party I got hired at Usability Sciences. Yeah, but it was just – I went to school with a business degree. I came out and worked within marketing research. I did a lot of – I essentially was like a data analyst if I’m honest. And then just through the fact that they had certain jobs that they didn’t know who to give to and they gave it to me. I started working with Google Analytics and Omniture and tagging and then somehow that led to me getting hired as a researcher. I had not even considered that this was a career when I was in college that I could – I mean I was in school in the 90s. No one talked about it. This was not like a career path I’d ever heard of.
Steve: Right.
Vicki: But I did enjoy research. And also I had – part of my degree was business, computer information systems, and so I had a pretty good understanding of databases and database structure from that which helped a lot with doing data.
Steve: So, when you think about your team now, like there’s probably a few different ways that people have come up – come into research. Like what you just described. You said some people have come through different kinds of designs, some human factors trained people – like what’s the…
Vicki: A successful background? I don’t know if – is that what you were going to ask?
Steve: No, I wasn’t, but you can answer a different question.
Vicki: I don’t think there is one. I think it kind of comes down to the person, right. I don’t – I don’t know – I have a weird view of education. I kind of – my mom is English, my Dad is South African and generally there school is considered as a way to make you a more well-rounded person and then once you leave school maybe you specialize in something and then you pick up those skills somewhere else. And I kind of feel like, just from all the different researchers I’ve worked with and the different backgrounds that they have, that I don’t know if having like a degree in human factors actually means you’re going to be the most successful researcher. I’ve worked with researchers who have nutrition degrees who are amazing. So, I think it just comes down to the individual.
Steve: I feel like there was a period of time where most – I think most researchers, some came from a very heavy social science background, but most in industry came from just ad hoc backgrounds like what you’re talking about, or my own background.
Vicki: But that’s changed, right? Or it is changing.
Steve: Yes. And, right, so what does that mean – we’re probably at the early stage of that change, but…
Vicki: I don’t know if we are. ‘Cuz I have a question. Like a couple of years ago when I, like maybe like 2012/13, I was considering a move to the Northwest. And at that time I felt that it was very difficult to have an undergrad degree and experience I had, which was a decent amount of experience, like 6 years, right, and find a company in the Northwest that was willing to hire someone who hadn’t specialized and done a post-grad, or a PhD. Now they might hire someone with a PhD who had no experience in the field over hiring someone who has 6 years of experience, but not PhD. And then I feel like since then that attitude has changed, but I’m not quite certain why that attitude has changed. It seems like people are more open to hiring people from various backgrounds. Is that just my experience? Have you experienced that?
Steve: I mean my sense with research is that demand exceeds supply.
Vicki: There’s a low supply, right.
Steve: So, doesn’t that mean we have to revisit sort of who gets to play?
Vicki: I didn’t know – I thought maybe – I’ve worked with a number of PhD candidates, or people with PhDs and they are very strict and maybe one-minded about how research has to be done. Whereas I think sometimes, especially within the world of business, you have to be able to maybe make compromises, or like maybe take shortcuts, but feel comfortable with those shortcuts, and sometimes I find that people with the academic background are less comfortable taking those. Have you had that experience? And I didn’t know if that was actually what was impacting? Like maybe the application of research within the business world, maybe academics, it was a hard blend. Does that make sense?
Steve: Yeah. I mean my perspective is anecdotal at best.
Vicki: Yeah, mine too. I’m just like guessing. Like I’m trying to understand like is that what happened? Or like maybe your point. Maybe it’s just there’s not enough supply so they’re like we’ll take anyone.
Steve: Right. But is – and I feel like early on I met academics who were sort of early in the industry and who represented some of that mindset that you’re talking about and that just constantly I meet people with high levels of education who are so excited and hungry to extend what they knew how to do.
Vicki: Sure.
Steve: Kind of in the kind of work that we’re talking about. So, what I don’t know is how much did sort of academia shape and maybe limit their mindset. But then I’m also excited about teams like yours where you have different backgrounds.
Vicki: Yeah. I think you want the blend, right? Like everyone should kind of have a different mindset, right. Because they’re all bringing a different lens. That’s one of the reasons I also like designers and product people in the field. Everyone is seeing something different through their lens because of their background and their experience. And so it’s much more insightful when you have a pretty good mix, a motley crew as I like to call it.
Steve: Yeah. That’s a great phrase. And maybe if someone you know, for example, comes from academia and has kind of a here’s how I was trained to look at problems, that person is going to do well among a motley crew where they’re going to be kind of creatively elbowed once in a while.
Vicki: Actually, at AT&T we had a researcher I loved and her background was anthropology and the rest of us were maybe a little more – had come to it a different path, and she definitely pushed the rest of the team in a very good way. It was a good balance.
Steve: Yeah. If you look at research as a team sport which is kind of where you started off.
Vicki: I think it is, yeah. I think it is a team sport.
Steve: Then you’re casting a team to have this great thing together.
Vicki: Yeah. That’s one of the things – a lot of the people I interview are coming from backgrounds where they are maybe one – a solo researcher in an organization, or they’re one of two researchers and they don’t get to work together because they’re only one of two and their desire is to work with a team of researchers and I actually do believe – in my consulting life we always worked as a pair and I actually think it just strengthened you because you were learning from another researcher and I feel like you picked things up and learned just from seeing what that other approach was doing or talking about – like understanding how people communicated things. Like there’s something to be said about research as a team is much stronger. I always feel for people who are doing it – they’re on the path alone, by themselves.
Steve: What do you think about researchers and research in like 2029? Who is it going to be made of? How do we kind of…
Vicki: I don’t know.
Steve: What’s the desired future? How do we get there?
Vicki: I mean I kind of go back and forth on this. I do feel like if companies – I think researchers are always – if research is a team they’re always doing to be less than there are of design and product. And in some worlds design and product need to take on aspects of research, right. And so, then is it that people will just be – we’ll have design and product or maybe jobs will begin to blend together more and there won’t necessarily be a dedicated researcher? But I can go back and forth on that where I’m like no, I kind of thing you’ll always probably want at least a research team who can oversee and maybe help teach people how to do their own research, that maybe don’t necessarily do all the research on their own, or it’s not its own kind of thing, because I do think the more involved product and design are in research, the more that they take on, the better they become. So, I don’t know. The future is – I’m wondering if design and product and research will all kind of blend together and people will begin to own all three of those skills which might be a lot to ask from someone.
Steve: But even think about your motley crew metaphor. You’re talking about sort of motley crew – it’s hard to say without sounding like I’m saying the name of the band.
Vicki: Well it is the name of the band.
Steve: I know, but I feel like the emphasis is different. Motley crew, I’m trying to just use the original phrase, but it’s coming out like the band. If research as a team, the way you have it now, is built up of a motley crew, imagine – what you’re talking about makes me think about like if you’d sort of change the departmental structure there’s a different make up of a motley crew that includes the skills of design and the skills of product and the skills of research in a diverse way.
Vicki: Sure.
Steve: And so what is that? Is that a new thing that’s like a hybrid?
Vicki: I don’t know the answer. I mean my tendency is to not be that extreme. I think that design and product and research will have to blend because they talk about like embedding design and research and product together, but then at some point people are going to be like, well can’t the designer also do part of the research, or maybe own the product? So, it will blend, but to the extent I think depends on the company and where they are in that evolutionary process. And in some ways, I mean design, even research, like to take that on – I feel like researchers themselves wear a lot of hats and then designers doing researchers, that’s a lot more hats. And then mixing product – I don’t know if a human is possible of like being that multi-faceted, right? So, I don’t know what it will look like, but I do imagine that they are going to have to blend together more. I don’t know exactly what that will be. Having a research team, especially if these research teams working more in like a consultancy basis because they’re so much smaller, and knowing that you build a better product and design organization by them being more involved in research just lends me to believe that there will be more blending, but I don’t quite know what that will look like and to the extent of it, and I’m sure it will be different on a basis by basis situation, depending on how evolved that company is.
Steve: That’s a very good, specific, non-conclusive…
Vicki: Did I just dance around that?
Steve: Well, we’re talking about what the future is going to be and how to get there. So, it’s not like you have the magic answer.
Vicki: Yeah, but I don’t want to be like it will be this, ‘cuz the future – do you ever read Chuck Klosterman? Are you a Klosterman fan?
Steve: Yes.
Vicki: Did you read his last book about where he’s like essentially everything we know now might be wrong? And it’s like trying to think through what we know now
and how that might be wrong and therefore what multiple instances of the future might be based off us being wrong now is the only thing that we know.
Steve: Yeah.
Vicki: So, that is leading me to my roundabout way of answering that question. It’s a very good book if you haven’t read it. It’s really, really good.
Steve: That’s a nice pivot from my Motley Crue reference.
Vicki: To Klosterman. It melds well.
Steve: Okay. It seems like maybe that’s where we should look to wrap up. Is there anything else that we should talk about?
Vicki: I don’t know. Have I answered all of your questions?
Steve: It’s not possible to answer all of anyone’s questions.
Vicki: That’s true. I mean my greatest fear for this is that I would be your most boring interview.
Steve: Well, my eyes were just fluttering. It’s for the listeners of this to decide if they’re bored or not. If they made it this far.
Vicki: They’re engaged in some aspect.
Steve: Some aspect. Whether it’s just rage listening, or, I don’t know.
Vicki: Rage listening!
Steve: Is that a thing?
Vicki: I’m assuming it must be for someone.
Steve: Alright, well thanks very much for being a guest. It was great chatting with you.
Vicki: Great chatting with you as well.
Steve: All right. That’s the wrap on another episode! Subscribe to Dollars to Donuts wherever you get podcasts. If you’re using Apple Podcasts, how’s about giving the podcast a rating and even a short review. This helps other people find out about the podcast. Portigal dot com slash podcast has transcripts, show notes, and all of the episodes. Follow the podcast on Twitter, and buy my books Interviewing Users and Doorbells Danger and Dead Batteries from Rosenfeld Media or Amazon. Our theme music is by Bruce Todd.
The post 22. Vicki Tollemache of Grubhub first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Vicki Tollemache, the Director of UX Research at Grubhub. We discuss how to manage incoming research requests, running a weekly research session for testing designs, and why candidates should come into job interviews with a point of view about the company’s product.
To me, researchers are educators. They’re there to translate and educate the organization that they work with about who their users are, what they’re experiencing, where their pain points are, what they care about, what their motivations are. There’s a number of ways you could communicate that and you can educate. Experience is probably one of the best, but due to time constraints, not everyone can come into the field with us and experience that. If you just rely on reports and communicating from the researchers there’s something that’s left out in the details. There’s a richness that’s not there that I think even researchers realize. – Vicki Tollemache
Show Links* Mind the Product * Newton Minow’s “Vast Wasteland” Speech * 500-channel universe * Golden Age of Television * Roku * Roku Guide * Vicki on LinkedIn * Grubhub * Seamless * userinterviews.com * Sam Hall * John le Carré * Anthony Bourdain: Parts Unknown * Usability Sciences * Ron Johnson and J.C. Penney * Motley crew * Mötley Crüe * Chuck Klosterman – But What If We’re Wrong?
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Thanks for joining me on Dollars to Donuts, the podcast where I talk with people who lead user research in their organization.
I’ve mentioned a public workshop happening in San Francisco; it looks like that has fallen through but I will be speaking at the Mind the Product Conference in San Francisco next month. I’m also doing in-house training workshops so let’s talk if that’s something your team might want to pursue.
In my consulting practice I have the opportunity to work with different organizations with varying levels of investment in research, varying levels of maturity in their research and product practices, and so on. I started this podcast as an extension of that, as a way to highlight the emergent practice of user research leadership. So supporting me and my business is the best way for you to support this podcast. My consulting work informs the podcast. It also pays for this podcast. If you have thoughts about the Dollars to Donuts, email me at DONUTS AT PORTIGAL DOT COM or write me on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
In 1961 the new chairman of the FCC gave his first speech, addressing the National Association of Broadcasters. He saw the potential of television, but complained that watching the programming currently available revealed a “vast wasteland.” That phrase echoed down the decades. In 1992 when cable TV executive John Malone coined the phrase “the 500 channel universe.” While that might have signified opportunity to the industry, but to the viewing public it felt like the vastness was simply increasing in scope. Yet somehow we made it today where 500 channels seems almost quaint, and critics herald the Golden Age of Television also known as Peak TV. Well, we purchased a Roku TV set recently. Roku basically provides an operating system for the television where you can add apps, just like you’d add apps to a phone. So we added all the usual suspects like Netflix, YouTube, Hulu, NBC, Vimeo, Amazon, and some others that we discovered like Pluto which is a free streaming channel with a ton of programming. But what I didn’t realize was that Roku is a somewhat open platform that allows interested parties to add content. Looking into how to do that, it doesn’t seem that much harder than putting out a podcast. And while the choice isn’t as overwhelming broad as with podcasts, there are an astonishing number of Roku channels. I found a blog that every week or two updates with the latest channels, not all of them, just the ones that they’ve reviewed! Here’s some of their latest update reviews.
Funny TV Network – A collection of funny clips from Family Feud hosted by Steve Harvey
Know Your Tools – Tool reviews, safety tips, recommendations and innovations
The Home Depot Channel – Tips and tutorials for home remodeling, home maintenance, and tool use
Louisiana Cajun Recipes – A Cajun cooking show hosted by a self-taught cook
Smoky Ribs BBQ – Essential grilling recipes
Stories of the Century – A 1950s Western TV series about a railroad detective who roams the west, tracking down outlaws and bandits who are preying on the railroad
SOS Coast Guard – A 12-chapter 1937 film serial starring Bela Lugosi and Ralph Byrd
Sci-Fi Zone – 19 vintage Sci-Fi movies from the 1940s and 50s
DramaTV – Vintage public domain dramas from the 1940s and 50s
Scary TV – 20 vintage horror movies from the 1930s, 40s and 50s
Shemaroo Yoga – Yoga tutorials from Anushka Malao
Aircraft Channel – Aircraft accidents and near accidents reconstructed with analysis of what actually happened
Rockwell Off Road – Mud-bogging and proving grounds videos
It’s not particularly different from what’s already online on some platforms, but I was surprised at how the bar for “television” had got, I certainly expect a lot of crap from TV in general but this is barely curated Internet detritus that mingles with TV channels from established media players. I’m not saying all this content is necessarily garbage, or isn’t of interest to some people, perhaps many people, but that my mental model for changing channels on a television set, even if I can’t pick anything over the air where I live and haven’t had cable for many years, that’s still an entrenched mental model, so to find that this new television lets me watch NBC and the Opossum Saga at the same navigation level in the menu is just surprising. I think they’ve got some way of streamlining the experience so searching on the platform will more likely reveal big brands that have more traffic or that perhaps have paid slotting fees. You won’t come across The Lawnmower Channel unless you know to look for it, I think. And just like I’ve mentioned in previous episodes these shifts in mental models, in how producers expect something to be used versus how consumers expect something to be used, these are fantastic things to explore in user research, especially as systems grow in complexity and scale, like this 5 zillion channel rokuverse.
Well, I think it is time to get to the interview. I spoke with Vicki Tollemache who is the Director of UX Research at Grubhub. All right, well, thank you for being on the podcast.
Vicki Tollemache: Thank you for having me.
Steve: Why don’t you start by introducing yourself? Who are you? What do you do?
Vicki: I am Victoria Tollemache, but everyone calls me Vicki. I am the Director of UX Research here at Grubhub. Essentially, I manage a team of researchers across our ecosystem and my job is really to make sure that we are working in a way, doing research that’s going to impact our organization. So, it’s a lot of strategy. I’m working with product leads and our design VP to make sure that we are positioned correctly and chasing the right questions.
Steve: Hmm, chasing the right questions. What does that mean?
Vicki: That means I mean – my research team – I think generally research teams are much smaller than the design organizations and product organizations they’re supporting. We have a million questions coming at us of all shapes and sizes and it’s determining which questions make the most sense for where we are as a business and which questions, if we get answers to, will the business be able to respond to and we’ll be able to have impact? So, making sure that we’re positioning in the right space.
Steve: Right. You mentioned impact as well right off. So, what does impact look like?
Vicki: Impact to me is, in the scheme of things maybe it’s identifying needs within our users ecosystem that we could create solutions for, to better solve for pain points they’re experiencing. Or it is – maybe we’re creating a new experience and we’re not quite certain if what we’ve created matches the user’s mental mindset. So, doing research at that point that our product and design teams can respond to and make changes to.
Steve: Does that fall under the label of evaluation? It feels like there’s something else to it.
Vicki: It can be. What do you mean?
Steve: I mean maybe you’re just describing the way I’d like to see evaluation done where it’s not just thumbs up/thumbs down on…
Vicki: Oh no, no, it’s iterating right through the process and we’re working with design to be like what can we change about it to make it a better experience? And then potentially testing again.
Steve: And you’re framing it around mental mindset which is sort of the underpinnings of a concept.
Vicki: Sure, absolutely.
Steve: Not the implementation of a concept. I think that’s why evaluative to me is like – evaluative is maybe about the details of the design?
Vicki: Sure.
Steve: And so I feel like maybe there’s another word that describes what you’re doing which is looking at the value proposition or the construct, or the mental mindset.
Vicki: Sometimes we call it an experience audit here, but I feel like that’s a term we’ve made up ourselves. I don’t know if I’ve ever heard that in the field or in the wild. Because we were trying to describe some of the things that we’re doing and we’re like are we auditing the experience?
Steve: Not that you asked my opinion, but audit seems more passive than what you’re talking at. To get at that gap between a mental mindset of a consumer of something and producer of something is more – that’s more – you have to extract that. You have to synthesize that.
Vicki: Sure. I think I’m speaking to two different types of research, right. One is when we’re going out in to the field and trying to understand that environment that our users live in, and especially where they meet, because we tend to find that that’s where a lot of the friction points are. And then working with design and product to be like how do we solve for some of these problems that we’re finding? Are these problems that we knew existed even? And then once we’ve gotten to that place where we’ve come up with solutions, then pairing with our users to determine have we solved for this in a way that actually makes sense for the users?
Steve: When you say “where they meet” in that first part?
Vicki: So, I mean Grubhub is an ecosystem, right? We have what we consider four distinct types of users. We have drivers. We have our restaurant partners. We have our diners – these are the people who are consumer facing. They’re ordering from us. And then we have internal employees. I mean the easiest, most basic way to explain that is we have an internal care provider that provides help to all our different users and those users don’t exist by themselves. They’re constantly interacting with each other, right.
Steve: Yeah.
Vicki: And usually they’re interacting with each other when there’s problems. And so either we can help them solve those problems, or sometimes we can maybe be a hindrance in how they solve those problems and they have to try to work around us. So, those are sometimes some of the biggest opportunities for us.
Steve: Okay. So, where there’s interactions or intersections between those different…
Vicki: Types of users.
Steve: In the ecosystem.
Vicki: Yeah.
Steve: Is opportunity.
Vicki: Opportunity, yeah, for us to – sometimes it’s about providing them more independence and building trust, right. And then there’s understanding why is there trust breakdown and why do they feel like they don’t have autonomy to kind of control the – the situation that they’re in.
Steve: Maybe we could step back.
Vicki: Sure, we did kind of jump right into it.
Steve: That’s me, not you. So, what is Grubhub? It’s an ecosystem of these things, but what is – how do they combine?
Vicki: I think Grubhub considers themselves a marketplace that is providing restaurants the opportunity to compete for consumers in regards to ordering food for delivery or pickup. So, that’s what we provide.
Steve: In what parts of the world are you running services?
Vicki: The U.S. and then we have a little bit of presence in London, but that’s generally on the corporate aspect of it, right. So, we also have a corporate platform where companies can partner with us and provide their employees a credit and then the employees kind of – it keeps them at work if they can order lunch at work and kind of eat through a meeting kind of thing. So, we have a little bit of presence in the UK with that. But otherwise, I think we’re in 300 markets in the U.S.
Steve: Is that a lot?
Vicki: I mean I think we have a very good presence in the United States. I think if you went into any major city, or even minor city, we would be there, right. So, we’ll be in Little Rock, Arkansas. We might not be in Jasper, Texas. And I’m using references from the South because I am from the South, so it’s easier references.
Steve: Okay. Alright. That’s good context. So, let’s – so, you’re sort of describing some of the things that you’re working on. And you made a point that research teams are often smaller than design teams.
Vicki: Yeah, product and design teams, right.
Steve: Yeah. And that’s the case here.
Vicki: That is the case here. I think – there are 7 of us currently, including myself. We have an open position, so that will be an 8th person. And then we also swell in the summertime because we usually bring in one intern. And last year we were fortunate enough to hire an intern, so it’s a good opportunity for interns as well.
Steve: What does a user research intern do at Grubhub?
Vicki: Well in the days since I’ve been here I always want to win best internship. That’s always my goal. I want that intern to get as much experience as possible. Usually they come in and they tell me they would feel more comfortable getting moderating experience, being able to run projects on their own. Last year we had the intern own – it felt like the easiest place for them to start was kind of doing more smaller research questions, so more evaluative research. So, they took over – I think we called it Research Day – and essentially ran that for the entire summer. But then we also took them out into the field with us for one project. So, they got to go on a trip and experience what it’s like to be within the restaurants, shadowing the drivers, us going into the diner’s home and learning about food. So, they’re owning one thing, which is kind of nice right, that they get to own something. They’re getting experience moderating. They’re putting projects together. They’re working with product. They’re working with design. And because Research Day goes across the entire ecosystem they’re getting exposed to everything. So, it’s like a really good place for them to start and then they will get to partner with some of the researchers. Kind of like a more mentor/mentee and go out into the field on some of our bigger research projects.
Steve: What is Research Day?
Vicki: Well we called it – at first it started as Diner Day. It’s just a standing day, once a week users come into the lab. The week before the designers can pitch ideas they have. We go through, based off how thought out, flush out the ideas, priority based against our organization’s needs. What are we going to be testing? Do we have access to it? Can we get the right users in? So, we go through and we kind of have a pitch day. We choose the idea and then 5 days later we test that idea in a lab. So, it’s supposed to just be really, really quick – 24 hour turnaround for findings. Just a bulleted list of findings that the designers can then go take action off of.
Steve: And that runs?
Vicki: Every week. It’s really frequent cadence. I found in past organizations sometimes people can push off doing research because they say it takes too long. It takes too long to go through scoping. It takes too long finding people to recruit. So, I felt like if we had this standing day setup for diners – just bringing 5 diners in once a week – it meant that designers could kind of come in last minute sometimes and we can get research done for them. And they can also experience that and watch that because we’re in a lab based situation. Or we can also go on the streets of New York. But in the lab base it’s kind of nice because we can also do remote interviews and reach out to people outside of New York, because New York is kind of an outlier compared to the rest of the country in regards to what our platform looks like. Also, people are very experienced ordering food from Grubhub or Seamless here and sometimes we’re looking for a split of like maybe less professional food orderers.
Steve: Seamless is another platform.
Vicki: Seamless and Grubhub are the same thing, just different brand. Eat24 is part of our brand as well. But they’re all the same. There’s just a different brand logo in the top left hand corner.
Steve: When you do Diner Day or Research Day, so a tactical question, you don’t have a lot of lead time.
Vicki: Um-mm. It’s quick and dirty.
Steve: So do you already know kind of what people are coming in before the idea is pitched? What’s the dependency there?
Vicki: Generally it’s a mix of diners, right. It’s starting to get harder and harder to find truly new diners to our platform, but it’s a mix of new or returning diners and we can sometimes switch what that ratio is within the 5 days we have from doing the pitch to actually doing the recruiting. So, we just know we’re going to bring 5 diners in. Some of them are going to be new to us, never ordered from us. A percentage of them are going to be returning to us, or we may shift that where it’s all returning based on what the project is.
Steve: So, you have kind of infrastructure in place such that you can quickly change.
Vicki: Yeah. That’s the nice thing about food, right. I mean I worked – I came from AT&T before this and I worked on the B2B side and we had to recruit network engineers and it was really difficult. It’s expensive. They don’t have much time. It’s hard to get them in. But so many people order food. It’s just an easier recruit, right. So, within 5 days we can get those people. We used a platform called userinterviews.com. Have you heard of it?
Steve: Yeah. I haven’t used it, but I’ve heard of it.
Vicki: It makes it very easy. And they’ll recruit anywhere for us in the U.S., so it’s super nice and they turn it around very, very fast.
Steve: So, once you understand what that flow is going to be…
Vicki: We have that relationship and they know that we’re doing that on certain days, so it’s just like an ongoing recruit that we have going.
Steve: Okay. Just coming back to one of the things you said, you mentioned a couple of times kind of aligning the research activities you do, whether it’s at that level or other things, with the business questions, the goals of the organization. And I think maybe you mentioned this, but maybe you could talk a little bit about how you come to an understanding of what those goals and questions are for the organization so that you can choose the research activities that support that. How is that input for you?
Vicki: I think when I first came – I mean I’ll be honest about this too. When I first came in it seemed like – and we still have this to a certain extent and I think a lot of companies deal with this – ongoing reprioritization of everything. We begin to work on something and then it will be like, no our priorities have changed. So, we had like a lot of like stop/start projects. And at the time that didn’t make sense to me so I wanted our research team to prioritize things based off what we knew our users needs and identifying those. And then we had little bit of leadership change and we had a new CPO come in, Sam Hall, he’s from ClassPass, he’s great. And he’s actually – I work pretty closely with him to understand like what the business priorities are. And I work pretty closely with our design VP to understand some of the design challenges that we’re facing as well. And even for something like Research Day we have an ongoing backlog of research questions that’s coming from the designers and/or product, but there’s some weeks where we don’t have anyone come to pitch and then my team has questions that we will go answer. So, we don’t not do a research day because product and design aren’t asking us questions, but we have questions as well based off things we’ve seen in the field and some of those can actually be themes that come up later. So, in the end, my answer is it varies. Working closely with product helps us from a business perspective. Working closely with our design leadership helps me because they have questions that they are also separately going after based off kind of some of the design questions that they have. And then I just work to prioritize those. We do a scoring. Does that answer the question?
Steve: A scoring?
Vicki: Yeah. So, certain people get a weighted score and in the end it’s also kind of what my team thinks is going to be the most valuable for us, and also where we’ll have impact. I mean I worked at organizations where they’re like every year they set a goal for the research team to do more research, like more projects. And that doesn’t make any sense to me. It should be about doing projects that actually impact the organization, right, that people actually make changes from. You can do a million research projects. If no one does anything with that research it’s like dealing into the abyss.
Steve: So, if I come to you and say I have a question about how people go through this flow in the app, how would you score that?
Vicki: Who are you? Are you a designer? Are you within the leadership team? And then what is that flow? Like what part of the experience is that? Is that something that we have as an initiative that we’re heavily focused on that we’re invested in as a business to like make a better experience? So, I’d have to ask some of those questions and then I’d also look at do we already have previous research that might be able to answer some of your questions? So, there’s a number of factors.
Steve: And sorry if I’m getting like obsessively detailed, but like are you – when you say score to me that means there’s a bunch of questions and you kind of apply a number to those and total them.
Vicki: Sure. Yeah. And it does work like that. It depends on who’s asking for it? Where we are in the product lifecycle? Like if we do this research for you will you be able to make changes from it within this release? Or are you telling us you might be able to make changes six months from now. So, we’re asking – do we have impact? Who is asking for this? How does it align to our business priorities? How does it align to our research and design priorities because we’re part of the design organization? And then there are certain scores that go with each of those answers.
Steve: Okay. I think I’ve heard those questions as kind of an intake or prioritization approach, but I haven’t heard it labeled as scoring which I think makes so much sense when you say it.
Vicki: And then I usually – because in the end our design organization is within our product organization and I usually also work fairly closely with our CPO to be like this is what I’m getting from the design organization. This is what I’m getting from your product leads. This is how we’re prioritizing. Just to make sure. Like it’s a gut check, like does this seem right?
Steve: So, I mean, there’s the example where if nothing comes in for Research Day you have your own things.
Vicki: Yup.
Steve: Are there situations where you’re proposing research to these kind of leaders that you’re working with?
Vicki: Absolutely. Last summer we had a new VP of design come in and one of the things he really wanted to push is quality of our product, right. And then the question came to me, like how do we measure quality? So, there’s a number of ways that we could go about measuring quality. I work with the data science team to go after certain metrics, or to understand how we might be able to measure that within some of our data. But then from my background I was like we could also do benchmarking. Like that’s something we haven’t done before and we rely really heavily on A/B testing which sometimes is like building things very piecemeal. So, I proposed that we go through and do a twice a year benchmark after some major releases to measure like what is this experience for users in our current product. And then if you look at the definition of quality you can only truly measure quality by comparing it to something of greater or lesser quality. So, the suggestion was where we can can we also do a comparative where we’re benchmarking against one of our competitors. And they were completely on board with that. And they allowed us to go to an outside vendor and go through the benchmarking process.
Steve: I thought you were going to say you’re benchmarking against yourself over time.
Vicki: You can do that – we’re doing that as well, but also against our competitors. When we can. We can’t do that obviously on the business side, but we can do that on the diner’s side.
Steve: You can’t do it in the business side because you can’t get access…
Vicki: We don’t have access.
Steve: …access to that part of it.
Vicki: No.
Steve: Okay. I see.
Vicki: And that’s always one of the things – I’ve heard of researchers who will ask – generally our restaurant partners or driver partners, they’re also using our competitors, but from an ethical perspective I don’t like to ask those types of users to share that information. I just don’t think it’s appropriate.
Steve: How do you benchmark – you know when you’re doing competitive benchmarking, how do you compare apples to apples?
Vicki: Our product teams are broken up into initiatives. On the diner side we have two initiatives. One of them is more about like the core product, right. And then benchmark is really about is our core product usable. Like where are there friction points. So, I worked with our product lead and our designers to identify like what do we consider core tasks? And those are based off like what are common like user flows, user journeys, through the app? But it also was based off where do we also know that we see CPO, so care calls taking place, because we also are always trying to lower those, right. So, from that we’ve mapped out like what our tasks were and then we took another one of our competitors, who essentially is going to have the same, similar tasks and we just did a benchmark against those tasks for those competitors. So like, you know task timing, success, ease of use. We also created this quality metric which is kind of based off of reliability. And then also just qualitative feedback, right. But there’s other things as well. There are things sometimes we hear from our users that are taking place that maybe we weren’t aware of and we definitely will go to product and design and be like hey we think that we need to do a deeper dive here. We’re hearing a number of things. There’s friction taking place that we weren’t aware of. Maybe we’ve made changes somewhere – and that’s the thing with an ecosystem, right. You might think you’ve made an improvement in one space and you’ve actually created two problems somewhere else and we’re definitely given the freedom to then go explore that and suggest solutions for those additional problems. Or maybe roll back the first solution to begin with.
Steve: You brought up solution as part of the research process…
Vicki: Yeah.
Steve: … a couple of times and I wonder if you could just say more about what’s a researcher’s role in – what is…
Vicki: A researcher’s role in solutioning?
Steve: Yes.
Vicki: Um, well, at Grubhub, again because we’re a small team, I think in some ways I wish that we could be more hands on than we are. Maybe we’ll identify a need and our product team has also had a hypothesis that that’s a need as well, so they’ll be like how do we solve for this thing that we’ve identified? It’s an opportunity for business. We see that it’s an unfulfilled need for users. Perhaps we’ll put together – we’ll define what the problem is. We’ll put together a workshop. A researcher will usually be involved in that workshop, but sometimes they can’t be so then maybe one of us come in as like a speaker, to speak to like – see the work and solutioning that they do and comment on it or consult on it. But it usually begins there and then we’ll go through rounds of iterating on the design as we test it. And a lot of that will take place in Research Day. Does this make sense?
Steve: It makes sense. Maybe you have thoughts about sort of the philosophy.
Vicki: I think my team would like to be more hands on with what takes place in those workshops, but we just have a limitation of time.
Steve: Yeah. Why do you think that’s something that they want? If you look at researchers around the world, what – I feel like some are very interested in the product, the details, the solutions.
Vicki: Absolutely, right.
Steve: Some are – some maybe don’t see that as their role.
Vicki: I think there’s always like, just from a human perspective, a desire to control. So, I mean, from that basic level it makes sense to me that people want to be more involved in solutioning, but I also think – you know researchers, they’re out in the field. They’re witnessing that. Maybe they write a report. Maybe they communicate their findings, but sometimes they have to be fairly like theme, high-level findings and there’s a lot in the details that can get left out of reporting and I think they feel like they provide value to be in those solutioning sessions because they can speak to some of that nuance that maybe wasn’t communicated within a report, right.
Steve: Yeah. Well even in the example you gave where there’s just no bandwidth, that maybe you guys are working in a consulting role.
Vicki: Yeah. I mean when you have a smaller team, unfortunately it becomes that way, right.
Steve: And whether you’re consulting, or in those sessions, you’re still there to – it sounds to me – so, tease apart – I think what you mean by solutioning is a collaboration where solutions are generated, but it’s not that researchers are there to do design.
Vicki: No, yeah.
Steve: But they’re there to bring that extra detail to inform those design decisions.
Vicki: Yeah, I mean – I think to me researchers are educators, right. Like they’re kind of there to translate and educate the organization that they work with about who their users are, what they’re experiencing, where their pain points are, what they care about, what their motivations are. And there’s a number of ways you could communicate that and you can educate. Experience is probably one of the best, but due to time constraints of everybody, not everyone can come into the field with us and experience that and I think there’s something to be said about – outside of not having the experience yourself with users in the field, if you just rely on reports and communicating from the researchers you kind of – there’s something that’s left out in the details, right. There’s like a richness that’s not there that I think even researchers realize. Like if my presence is there I can speak to some of those one-off situations that maybe we didn’t cover in the report, but might get brought up within solutioning. Does that make sense?
Steve: Yeah, yeah. I think we’re sort of unpacking what solutioning looks like. That it’s not – that there’s a dialogue and a bunch of different perspectives that can come forward and that the researcher isn’t necessarily there to say oh I think we should start the task here instead of over here.
Vicki: Yeah. Yeah. It’s more about bringing that richness of like, if we start the task here this might actually happen to some of the users. Whereas if we started it here maybe that won’t happen, if that’s a good or bad thing. Does that make sense?
Steve: Yes.
Vicki: Just that deeper understanding. I really do believe that researchers tend to just be educators.
Steve: That’s a good metaphor, isn’t it?
Vicki: Yeah. It’s what it feels like to me. I remember – well I was reading this book recently, John le Carré – do you know who he is? He wrote Tinker, Tailor, Soldier, Spy.
Steve: Um-hmm.
Vicki: And he was talking about the role of a war journalist and it’s really not about reporting on just what’s taking place in the field, but it’s also about like building empathy and trying to encourage people to care. I kind of feel like in some ways we’re kind of like war reporters, based off his description. We’re out in the trenches.
Steve: Right. You want to bring it back in a way – I mean it’s back to your impact point early on.
Vicki: But there is a lot left out when you communicate through reports and findings, right. Sometimes you’ll have a user who’s extremely insightful, but do you just report to that one thing, or is that like a one-off? But then you’re in solutioning session and you’re like oh, well it’s actually useful here. This is where I should share it.
Steve: So, yeah, if we took away the constraint of limited resources or just the bandwidth that the folks on the team have, what are the ways that they could be educating?
Vicki: Well – I mean that’s the thing – not just taking constraints from my team, but I think one of the things realistically, I wish that we could take – or I wish the product and designers – product team members and designers could come into the field more and have that experience more, right. I think by them being better versed in what that experience is like within different markets, within different types of restaurants, with different types of users, they have a better kind of base to make decisions from. They’re more informed. They have a deeper understanding. They understand what some of the more nuanced situations might be and therefore they can make informed decisions. But unfortunately with meeting schedules everyone is like strapped for time, right. Like sometimes the reality is they just can’t come out into the field.
We have another program here at Grubhub called Parts Unknown. I’ve branded all of our research programs. We also do – it’s not all within programs, but Parts Unknown, started by – we were inspired by Anthony Bourdain, obviously – this was before, unfortunately, he committed suicide, but bringing people out into different markets and taking a deep look at what our ecosystem looks like outside of New York and Chicago, because we are kind of outliers, and understanding like what food culture is in these cities and how people think about food and what types of food they order and how that impacts our ecosystem and what our ecosystem looks like there, and we were able to bring – and we still do this – product and designers into the field and I feel like that has been some of the more like enriching, like those people come out with a much, much more informed – they’re much more informed and I think they’re able to make better decisions as a result of it.
Steve: That sounds like it’s kind of an immersion experience.
Vicki: Yeah, yeah. And sometimes we were partnering with food bloggers too to have a night with them, just to understand like their take on the market. It was pretty interesting. So, we still do that, but we’ve also kind of evolved that program as well. It’s not just about looking at different markets. It’s also about looking at different parts of an experience that maybe we haven’t looked at in a while and trying to determine we think it’s this, is it actually this for our users?
Steve: Right. Is this a category where you look at the different people in your ecosystem – is there – how rapidly do these things evolve?
Vicki: What do you mean?
Steve: I mean like how static are the things you know about different people in the ecosystem?
Vicki: Well I think as our industry changes and people are competing against each other within our industry that there are things that are happening that are changing the ecosystem on a pretty regular basis. Does that make sense?
Steve: Yeah. So, hence the need to – if there’s areas of the ecosystem that you haven’t looked at in a while, you have to go back…
Vicki: We have to revisit. Absolutely. And that’s kind of – this is again where I get nervous about if I should speak to this or not? So, some of the things – we have just certain areas that we haven’t looked at in a long time. I mean our company is 20 years old and we’ll be like oh this is our workflow and then we’ll have our CPO be like I want us to take a look at this because I have questions about what this workflow really is. So, we’ll go through and do this thing as an audit. The researchers will sign up as users and go through an audit with that experience, what it’s actually like as a user. And then we can pinpoint like this is what we thought it was, this is what it actually is. We think it takes 10 days. It actually takes 30 days and this is why it takes 30 days. That kind of stuff.
Steve: Yeah. So, if there’s so many different facets to the experience in the ecosystem and there’s change happening.
Vicki: Yeah. And you have – within our – I mean it’s not just our product organization, right. Like we’re working with an ops team, a sales team, we have a marketing team and sometimes different teams are building different parts of a flow and they aren’t coming together to check like does that all make sense. Isn’t it always like the idea of like your product is a reflection of your organization and how well they work together?
Steve: Right. What’s the cliché about don’t ship your org chart. I don’t know who to attribute that too.
Vicki: Yeah, I think that is a cliché, yes.
Steve: Okay. Can we talk about your team a little bit?
Vicki: Sure, what would you like to know?
Steve: What kind of people have found their way to your team?
Vicki: Well I inherited the majority of my team and they kind of have a pretty diverse background. We have a couple of – I think my team is comprised of a designer/comedian, an industrial designer, someone with a sociology background, someone with a psychology background and then two researchers who actually have like human factors degrees. And they come from a variety – most – only one of them, our intern from last summer, this is her first job. The rest of them come from a variety of other industries to get here. And one of our researchers, our associate principal, he and I have worked at like 3 to 4 other jobs together. I’ve known him for a very long time at this point.
Steve: I feel like I would be remiss if I don’t ask about /comedian because that’s just there to be asked about.
Vicki: He does like a whole bunch of live storytelling, but I think it makes him a very good presenter. He kind of has like charisma on the stage. I’ve actually had a – when he presents I have people reach out to me to be like you should hire more comedians. Because it makes him very engaging. He’s very good at telling stories, which is why I mention that he’s a designer/comedian background. But he’s worked in research for like, I think, the last 8 years.
Steve: I’ve definitely come across people in research who have either actual, or sort of – they have actual education in theater or some related field.
Vicki: Well you know communication is such a huge aspect of it and if you have that charisma and ease with yourself I think it just gets the message across better sometimes. People are more engaged and feel maybe less threatened by the message. Does that make sense?
Steve: Yeah.
Vicki: There’s something to it. He definitely knows how to charm an audience. I think it also makes sometimes the research a bit more stickier because people are more engaged with presenters like that.
Steve: I think – right, people that find their way into research eventually discover that so much of the work is about working with your colleagues and not…
Vicki: Oh, absolutely. It’s about relationships, right.
Steve: Far less than it is about fieldwork or research…
Vicki: I think that’s anything, right. Like in the end it’s all very relationship based, right. But I think there’s something too, I know the previous director here, he had us do an improv workshop and there is something about – my sister I told you is a comedian. There is something about like comedians in the end generally are introverts who kind of force themselves to be extroverts, which I feel like there’s parallels with that with research. And then there’s this idea like, with especially in improv, where instead of saying no to things it’s kind of that yes/and and building off of each other which also tends to work better with relationships as well. So, maybe it kind of facilitates that. I don’t know. But yeah, in the end it is all relationship based.
Steve: When you talk to people as prospective hires – I don’t know who you hire, but whether it’s the interns or other people, what kind of things are you looking at that kind of strike you as this person might be worth talking to?
Vicki: I definitely want someone who is very analytical. I mean we have – I’ve had candidates come into interview who, when you ask like what do you think of our platform, they just say it’s great. I want someone who has actually maybe taken a look at it from a critical lens and has identified things that they think might be opportunities. Even better if they showed friends of theirs, or family members, and kind of did a small usability test on it and came back with a critique. My team is very collaborative. We get along very well. So, huge egos aren’t a big thing. Like I want people who are going to come in – kind of say everyone brings something to the table, everyone offers something, and I feel like for researchers they learn a lot from working together as well. So, I definitely want someone who is going to fit into that culture and is excited to collaborate with other researchers.
Steve: But how do you look for that in a…
Vicki: An interview process?
Steve: Or whatever – the scouting or any part of that process before you actually are working with somebody?
Vicki: I mean for me, we have a recruiting group, right, who does all the recruiting and they bring in resumes. We talk about kind of what I’m looking for in the resumes. I mean a lot of it I’m sussing out when I’m doing phone interviews for the initial screen, right. And I guess a lot of it is me just trying to get to the core of who they are and how honest or comfortable I feel like they are with describing who they are. The intern that we have right now, I keep referring to her, she was probably one of the best interviews I’ve ever done and she just told me a very honest story about a research project she did, that they presented it, that the company rejected the idea that they came up with, and I was like what did you do? And she was like I drank a glass of wine, it was 6 weeks of work. And then she told me about like, she’s like but then the next day we went back to the drawing board and we decided to do – and it was just like a very honest – she exposed vulnerability. She was very much herself. She was funny. She could speak very clearly to herself. And I’m kind of looking for that. When I interview someone I want to feel like I’m having a drink with someone, that they’re comfortable describing who they are, which I also think is a sense of maturity where people aren’t – I mean interview process is kind of nervous, right, so you’re coming in maybe with a little bit more of a sense of yourself and a little bit more comfortable about who you are and being honest with who you are. I think a lot of people make the mistake of hiding themselves in an interview and I think you want to be very, very open and honest about who you are because if you aren’t it may not be the right job for you to begin with, right. So, I think I’m kind of looking for that. Just some who can speak to what they’ve done and who they are in a very clear and honest way. Does that make sense?
Steve: Do you think that’s a research specific way of looking at candidates?
Vicki: That’s human specific.
Steve: That’s just hiring someone you want to hire.
Vicki: Yeah, that’s hiring – but then on top of that then I’m looking for like, I want to hear examples of projects that they’ve done in the past and how they’ve approached research? I definitely like to hear about how they’ve handled failure in the past and what they’ve learned from it? Situations that they’ve had to work – environments they’ve had to work in that were not comfortable for them, and why? I think generally the most basic questions anyone asks in an interview though. But it’s definitely about how comfortable and honest and open I feel that they are in the interview that is the thing that probably sways me. But also that they have to be analytical. I do have an issue when people come in and they haven’t done any research about Grubhub, or looked at our product or had any thoughts on it at all. Because if you’re a researcher you should be asking questions, right, and doing some research.
Steve: I want to pull apart some of what you’re saying.
Vicki: Sure.
Steve: Because you’re saying analytical, but the example you’re giving is them having looked at Grubhub. Those seem like different things to me.
Vicki: Well, I want them to have – I mean when you go – when anyone goes in for an interview, haven’t you done a little bit of research on the company that you’re going to work for? You’re curious about the product, like what is this product? Like what kind of research will I be doing? Let me take a look at it. Are there a huge amount of problems that I see? Is this going to be like a battle, or is this like oh, I can see like some big – I want someone who has like thought through like what am I going to be working on? What seemed to be the problems here? And I want them to speak to like problems that they’ve identified. Does that make sense?
Steve: Yeah.
Vicki: It’s kind of like they should have been a little heuristic, or maybe even like shown it to a couple of their friends and have an idea of like these are some of the things I think that you might be dealing with. And even speak to like how they might solve some of those problems. I mean we usually also have them go through a practice where we use a site that is not Grubhub to understand like how they look at something? What problems they identify? And then what kind of research solution they would put together for that? But that’s after the phone screen.
Steve: Okay. So, I understand a little more what you’re referring to when you say analytical. There’s a bit there of initiative, but also thinking through something before you kind of get into it.
Vicki: Yeah. And to me, if they can speak to that – like because one of the concerns I have is the communication aspect, especially when you’re doing any sort of like generative, ethnographic work, I mean that’s – I used to have a researcher who called it the big messy and it’s – you go into the field – there’s a lot of stuff that you can focus on, like what do they – how do they hone into what’s important, right. And I kind of feel like I get that from talking to them about like what they’ve seen within our own product. If I have someone speak to the fact that UberEats changed their app logo color and they didn’t like it, I don’t know if I think that that is necessarily a big important finding. Whereas, if I have someone talk about problems they experienced within our search results and like how that might not be conducive to someone looking for food, that makes more sense to me. So, what are they honing in on? What do they consider a finding? And how do they communicate that to me? That speaks a lot to how they might do that for the organization.
Steve: I’ve worked as a consultant my whole career, pretty much, and many years as an agency and I think – maybe the reason you’re hearing me sort of check in on this approach is that I think in the agency lifetime that I had, and certainly my own practice, there’s kind of a consultancy hubris of like coming in and like let me tell you about your thing. So, when you say you want your prospective hires to tell you about your thing.
Vicki: I want them to have an opinion, right.
Steve: Yeah.
Vicki: And I want them to be able to communicate that opinion. And I want to understand what that opinion is based on. And if they don’t that’s concerning to me.
Steve: Yeah. So, I think there’s something here about the process more than the substance of their opinion, but the process – because they don’t have access – like I never want to be the person, personally, just to come in and say look I don’t know what strategy decisions you have, but here’s what I think you should do.
Vicki: I’m not wanting them to lead that, but I’m wanting them to have looked at our product and been like here’s some things I think might be weaknesses, right. Like I’ve considered what your offering is. I’ve looked at it and I’ve noticed these things. I mean that’s the least you can do when you go in for an interview, right? Whenever I go in for an interview I review whatever I can access of the product, depending on what part of the business I’m interviewing for, and I make sure like, if someone asks me what do I think of it that I have like feedback based off some sort of heuristic I’ve done and I can tie that back to some sort of best practice of some kind, right. And so I kind of expect that as well from researchers coming in. If they haven’t looked at the product at all they kind of haven’t prepped themselves for the interview.
Steve: Yeah.
Vicki: And then that concerns me. And also researchers should always be asking questions, right. Like you should want to know like who would I be dealing with here? I’ve had companies reach out to me where I’m like oh my God that taxonomy would be crazy. It’s so much, like how do they deal with that, right. Like I can look at that and then I’ll have those questions for like how do you guys handle that. You can tell a lot by the questions people ask as well.
Steve: Right, that’s another aspect. What kind of questions are you looking researchers to bring you in the hiring process?
Vicki: We definitely want them to ask questions. I’m always shocked at how many people have no questions for me. I mean I expect them to ask about typically how we do research. I know right now we’re hiring for kind of an entry level person. So, a lot of times I’m trying to understand have they approached certain methodologies? How are they like wanting to expand or grow as a researcher themselves? I want to understand from them – like where they want to work, what experience they want to gain? And then also like how we handle research at Grubhub, kind of similar to all the same questions that you asked me at the beginning of this interview. I kind of expect them to be asking those same questions, right.
Steve: Now they’re just going to listen to this.
Vicki: I do think sometimes on Fridays – I call it 30 minutes with like a college grad, because I get a lot of college kids reaching out to me through LinkedIn, asking me how to get jobs. And I’ll talk to them for the first 30 minutes of my Friday and just give them feedback about what I think is a good interview and we’ll go through that. It’s very similar to all the questions that you’re asking right now.
Steve: I’m just the college kid, right!
Vicki: So, from the agency side – I came from an agency background as well. I worked for a company at the beginning of my career called Usability Sciences. They’re based in Dallas. They’ve been around like 30 years. It’s always interesting to me the agency side because I feel like even though agencies want to come and be like this is what you should do, there’s always kind of like this sphere of like overstepping that with the client, right. There’s that like kind of thin boundary and I know talking to researchers who are just from an agency background, they sometimes feel like they have no voice. Or they can’t push back hard enough because they’re afraid they’ll lose the client. And so therefore sometimes they’re not quite – especially if they’ve only had agency experience – I think that’s one of the things they talk about experience they want to get is like being able to work closely with product and scope and push back when they feel like they are more empowered to do so. And that’s something that they lack from agency.
Steve: What about for you? So what are the key things you’ve seen as a difference when you’ve worked in-house or worked as a consultant?
Vicki: Well I started – I left consultancy in 2013 and for me I’d started in 2007/2008 and I just saw a change in the industry, right. 2007/2008 sometimes we’d be working with someone from marketing. They would come in. They would want us to do like just the usability test. That same client would come back year after year, same usability test, no changes. Same usability test, no changes. And then as time was progressing I would see that companies where all of a sudden you had like a research director. Or you had a researcher and they were our go to person. And I could see that people were staffing up inside and that I think the industry was swaying maybe from like agency, or definitely what my consultancy offered, to that offering taking place within the business. So, yeah, I wanted to go see what it was like. Like why are people not making changes and coming back to us year after year? What is the political landscape that they’re in that maybe is leading to something like that? How do you maybe have more control or more say when you’re internal vs. some of the limitations you feel like when you’re consulting? And just to be more embedded, right. To get to work closely with designers and to work – I mean we would come in and do like maybe 3 weeks with a company. We wouldn’t do like a long, long embedded process. So, it was just that understanding of like what is it like to be embedded and work with product and design and what are the limitations they’re facing? Because I could see those within the projects that I was working on, like that was happening, but I didn’t understand why that was happening?
Steve: What was the first job that you got post-agency?
Vicki: JCPenney. That was one of our clients and I jumped over to them. But probably after – I don’t know if you know anything about JCPenney? They went through a big shift where they hired this guy, Ron Johnson, from Apple. And he decided to kill all of their rewards and sales programs. And if you know anything about the JCPenney customer, that’s – customer shops sales first at JCPenney. And their loyal customers are very much versed in that whole points/rewards program, so by getting rid of that they alienated their core customer base, the company kind of took a dive that I don’t know if they’ve really come out of it. I think it’s also like retail has changed, right, and that was taking place in the midst of that change of retail because of online sales. So, I went there. It was interesting, but also the company was doing massive layoffs, constantly. So, then I went to AT&T after that.
Steve: What were you doing at AT&T?
Vicki: AT&T was a closer commute. At the end JCPenney was a 2 hour commute for me each way which is bananas. I was in Dallas at this time. AT&T, on the business side, essentially they had a portal of portals and we came into a product organization that’s first question to me was we don’t know who are users are or what they do. So, there it was just like starting at the very bottom and trying to help inform product who was just desperate for information. So, that was a lot of talking with network engineers. And I have mixed feelings about personas, but that was actually one of the best scenarios I’ve seen for why you build personas. Like if someone is like we don’t know who our users are, maybe you build some personas to explain who your users are. They didn’t have much access to data either. They had little knowledge of who was using the platform and how.
Steve: So they didn’t know anything?
Vicki: Yeah. It’s interesting. There’s like a spectrum of like not knowing anything to like then working in a place where people assume they are the user themselves and that they know everything, right. I mean both come with their own challenges and problems. It’s probably better to be somewhere in the middle.
Steve: And from AT&T?
Vicki: Came here.
Steve: What’s the role that you came into?
Vicki: Manager of the New York – so I manage the New York team which is all the diner facing. And we had a director that oversaw everything. He left, I guess about a year ago now. And then they gave me 6 months to progress into his space.
Steve: So, is this your first…
Vicki: Director position?
Steve: I was going to say leadership in general around research?
Vicki: I’ve been a manager since JCPenney. But this is first director. But I don’t have a manager beneath me. So, I’m still managing as well, right. Everyone direct reports to me.
Steve: Is that the difference between a manager and a director? Can you tell I don’t have a job inside an organization?
Vicki: I think it’s different for every company, but within our org structure most directors have managers and I do not. So, it’s me and six, soon to be seven, direct reports.
Steve: Can we go back then before this agency – what was it that led you into that kind of work?
Vicki: I started my career in marketing research at a small ad agency in Dallas who handled all of McDonald’s local DMA markets. McDonalds provides an allotment of money for national coverage and then depending on the size of the DMAs, the DMAs are given money to do local advertising and then sometimes the DMAs themselves fund additional advertising. So, my agency handled all the local advertising for markets.
Steve: Is DMA direct mail?
Vicki: Designated marketing area. And through that I started doing – they started investing in like creating microsites for like ad campaigns and we would track those through Google Analytics. And what – I was in charge of helping tag those and I’d have to come up with like what’s the user’s journey? Where should we tag? What do we consider success? What are we measuring? And then I was at a Christmas party and I met someone and explained to them what I did and he worked at Usability Sciences and was like oh, you should come work for us. It sounds like you’re just doing like heuristics. ‘Cuz oftentimes when I went through tagging I would discover like there’s no clear path how to get from here to here and that’s one of like the core paths that users would try to accomplish on this microsite. And so from a Christmas party I got hired at Usability Sciences. Yeah, but it was just – I went to school with a business degree. I came out and worked within marketing research. I did a lot of – I essentially was like a data analyst if I’m honest. And then just through the fact that they had certain jobs that they didn’t know who to give to and they gave it to me. I started working with Google Analytics and Omniture and tagging and then somehow that led to me getting hired as a researcher. I had not even considered that this was a career when I was in college that I could – I mean I was in school in the 90s. No one talked about it. This was not like a career path I’d ever heard of.
Steve: Right.
Vicki: But I did enjoy research. And also I had – part of my degree was business, computer information systems, and so I had a pretty good understanding of databases and database structure from that which helped a lot with doing data.
Steve: So, when you think about your team now, like there’s probably a few different ways that people have come up – come into research. Like what you just described. You said some people have come through different kinds of designs, some human factors trained people – like what’s the…
Vicki: A successful background? I don’t know if – is that what you were going to ask?
Steve: No, I wasn’t, but you can answer a different question.
Vicki: I don’t think there is one. I think it kind of comes down to the person, right. I don’t – I don’t know – I have a weird view of education. I kind of – my mom is English, my Dad is South African and generally there school is considered as a way to make you a more well-rounded person and then once you leave school maybe you specialize in something and then you pick up those skills somewhere else. And I kind of feel like, just from all the different researchers I’ve worked with and the different backgrounds that they have, that I don’t know if having like a degree in human factors actually means you’re going to be the most successful researcher. I’ve worked with researchers who have nutrition degrees who are amazing. So, I think it just comes down to the individual.
Steve: I feel like there was a period of time where most – I think most researchers, some came from a very heavy social science background, but most in industry came from just ad hoc backgrounds like what you’re talking about, or my own background.
Vicki: But that’s changed, right? Or it is changing.
Steve: Yes. And, right, so what does that mean – we’re probably at the early stage of that change, but…
Vicki: I don’t know if we are. ‘Cuz I have a question. Like a couple of years ago when I, like maybe like 2012/13, I was considering a move to the Northwest. And at that time I felt that it was very difficult to have an undergrad degree and experience I had, which was a decent amount of experience, like 6 years, right, and find a company in the Northwest that was willing to hire someone who hadn’t specialized and done a post-grad, or a PhD. Now they might hire someone with a PhD who had no experience in the field over hiring someone who has 6 years of experience, but not PhD. And then I feel like since then that attitude has changed, but I’m not quite certain why that attitude has changed. It seems like people are more open to hiring people from various backgrounds. Is that just my experience? Have you experienced that?
Steve: I mean my sense with research is that demand exceeds supply.
Vicki: There’s a low supply, right.
Steve: So, doesn’t that mean we have to revisit sort of who gets to play?
Vicki: I didn’t know – I thought maybe – I’ve worked with a number of PhD candidates, or people with PhDs and they are very strict and maybe one-minded about how research has to be done. Whereas I think sometimes, especially within the world of business, you have to be able to maybe make compromises, or like maybe take shortcuts, but feel comfortable with those shortcuts, and sometimes I find that people with the academic background are less comfortable taking those. Have you had that experience? And I didn’t know if that was actually what was impacting? Like maybe the application of research within the business world, maybe academics, it was a hard blend. Does that make sense?
Steve: Yeah. I mean my perspective is anecdotal at best.
Vicki: Yeah, mine too. I’m just like guessing. Like I’m trying to understand like is that what happened? Or like maybe your point. Maybe it’s just there’s not enough supply so they’re like we’ll take anyone.
Steve: Right. But is – and I feel like early on I met academics who were sort of early in the industry and who represented some of that mindset that you’re talking about and that just constantly I meet people with high levels of education who are so excited and hungry to extend what they knew how to do.
Vicki: Sure.
Steve: Kind of in the kind of work that we’re talking about. So, what I don’t know is how much did sort of academia shape and maybe limit their mindset. But then I’m also excited about teams like yours where you have different backgrounds.
Vicki: Yeah. I think you want the blend, right? Like everyone should kind of have a different mindset, right. Because they’re all bringing a different lens. That’s one of the reasons I also like designers and product people in the field. Everyone is seeing something different through their lens because of their background and their experience. And so it’s much more insightful when you have a pretty good mix, a motley crew as I like to call it.
Steve: Yeah. That’s a great phrase. And maybe if someone you know, for example, comes from academia and has kind of a here’s how I was trained to look at problems, that person is going to do well among a motley crew where they’re going to be kind of creatively elbowed once in a while.
Vicki: Actually, at AT&T we had a researcher I loved and her background was anthropology and the rest of us were maybe a little more – had come to it a different path, and she definitely pushed the rest of the team in a very good way. It was a good balance.
Steve: Yeah. If you look at research as a team sport which is kind of where you started off.
Vicki: I think it is, yeah. I think it is a team sport.
Steve: Then you’re casting a team to have this great thing together.
Vicki: Yeah. That’s one of the things – a lot of the people I interview are coming from backgrounds where they are maybe one – a solo researcher in an organization, or they’re one of two researchers and they don’t get to work together because they’re only one of two and their desire is to work with a team of researchers and I actually do believe – in my consulting life we always worked as a pair and I actually think it just strengthened you because you were learning from another researcher and I feel like you picked things up and learned just from seeing what that other approach was doing or talking about – like understanding how people communicated things. Like there’s something to be said about research as a team is much stronger. I always feel for people who are doing it – they’re on the path alone, by themselves.
Steve: What do you think about researchers and research in like 2029? Who is it going to be made of? How do we kind of…
Vicki: I don’t know.
Steve: What’s the desired future? How do we get there?
Vicki: I mean I kind of go back and forth on this. I do feel like if companies – I think researchers are always – if research is a team they’re always doing to be less than there are of design and product. And in some worlds design and product need to take on aspects of research, right. And so, then is it that people will just be – we’ll have design and product or maybe jobs will begin to blend together more and there won’t necessarily be a dedicated researcher? But I can go back and forth on that where I’m like no, I kind of thing you’ll always probably want at least a research team who can oversee and maybe help teach people how to do their own research, that maybe don’t necessarily do all the research on their own, or it’s not its own kind of thing, because I do think the more involved product and design are in research, the more that they take on, the better they become. So, I don’t know. The future is – I’m wondering if design and product and research will all kind of blend together and people will begin to own all three of those skills which might be a lot to ask from someone.
Steve: But even think about your motley crew metaphor. You’re talking about sort of motley crew – it’s hard to say without sounding like I’m saying the name of the band.
Vicki: Well it is the name of the band.
Steve: I know, but I feel like the emphasis is different. Motley crew, I’m trying to just use the original phrase, but it’s coming out like the band. If research as a team, the way you have it now, is built up of a motley crew, imagine – what you’re talking about makes me think about like if you’d sort of change the departmental structure there’s a different make up of a motley crew that includes the skills of design and the skills of product and the skills of research in a diverse way.
Vicki: Sure.
Steve: And so what is that? Is that a new thing that’s like a hybrid?
Vicki: I don’t know the answer. I mean my tendency is to not be that extreme. I think that design and product and research will have to blend because they talk about like embedding design and research and product together, but then at some point people are going to be like, well can’t the designer also do part of the research, or maybe own the product? So, it will blend, but to the extent I think depends on the company and where they are in that evolutionary process. And in some ways, I mean design, even research, like to take that on – I feel like researchers themselves wear a lot of hats and then designers doing researchers, that’s a lot more hats. And then mixing product – I don’t know if a human is possible of like being that multi-faceted, right? So, I don’t know what it will look like, but I do imagine that they are going to have to blend together more. I don’t know exactly what that will be. Having a research team, especially if these research teams working more in like a consultancy basis because they’re so much smaller, and knowing that you build a better product and design organization by them being more involved in research just lends me to believe that there will be more blending, but I don’t quite know what that will look like and to the extent of it, and I’m sure it will be different on a basis by basis situation, depending on how evolved that company is.
Steve: That’s a very good, specific, non-conclusive…
Vicki: Did I just dance around that?
Steve: Well, we’re talking about what the future is going to be and how to get there. So, it’s not like you have the magic answer.
Vicki: Yeah, but I don’t want to be like it will be this, ‘cuz the future – do you ever read Chuck Klosterman? Are you a Klosterman fan?
Steve: Yes.
Vicki: Did you read his last book about where he’s like essentially everything we know now might be wrong? And it’s like trying to think through what we know now
and how that might be wrong and therefore what multiple instances of the future might be based off us being wrong now is the only thing that we know.
Steve: Yeah.
Vicki: So, that is leading me to my roundabout way of answering that question. It’s a very good book if you haven’t read it. It’s really, really good.
Steve: That’s a nice pivot from my Motley Crue reference.
Vicki: To Klosterman. It melds well.
Steve: Okay. It seems like maybe that’s where we should look to wrap up. Is there anything else that we should talk about?
Vicki: I don’t know. Have I answered all of your questions?
Steve: It’s not possible to answer all of anyone’s questions.
Vicki: That’s true. I mean my greatest fear for this is that I would be your most boring interview.
Steve: Well, my eyes were just fluttering. It’s for the listeners of this to decide if they’re bored or not. If they made it this far.
Vicki: They’re engaged in some aspect.
Steve: Some aspect. Whether it’s just rage listening, or, I don’t know.
Vicki: Rage listening!
Steve: Is that a thing?
Vicki: I’m assuming it must be for someone.
Steve: Alright, well thanks very much for being a guest. It was great chatting with you.
Vicki: Great chatting with you as well.
Steve: All right. That’s the wrap on another episode! Subscribe to Dollars to Donuts wherever you get podcasts. If you’re using Apple Podcasts, how’s about giving the podcast a rating and even a short review. This helps other people find out about the podcast. Portigal dot com slash podcast has transcripts, show notes, and all of the episodes. Follow the podcast on Twitter, and buy my books Interviewing Users and Doorbells Danger and Dead Batteries from Rosenfeld Media or Amazon. Our theme music is by Bruce Todd.
The post 22. Vicki Tollemache of Grubhub first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Ruth Ellison, Head of User Research at DTA, the Digital Transformation Agency in Australia. We discuss the challenges of user research – and digital product development – in government, embedding researchers into product teams but maintaining a guild model to connect them, and how research can impact policy.
My role is really more of an enabling function, looking at how do we bring in the right people into the teams? When they’re here, how do we help mentor them? I’m connecting them to other researchers in our communities. Also trying to look at how we lift the conversation around research. Part of my role is about that strategic aspect of research. How do we do it better? How do we help enable the broad decision making of government? – Ruth Ellison
Show Links* ~~Fundamentals of Interviewing Users (SF)~~ * Ruth on LinkedIn * Ruth on Twitter * Digital Transformation Agency * DTO becomes DTA * 18F * GDS (Government Digital Service) * Leisa Reichelt * Leisa Reichelt on Dollars to Donuts (part 1) * Leisa Reichelt on Dollars to Donuts (part 2) * Canberra * Medicare * TEDxCanberra * Science and Geek themed jewellery
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Greetings and thanks for checking out this episode of Dollars to Donuts, the podcast where I talk with people who lead user research in their organization.
Coming up in San Francisco on September 13th, I’m teaching a public workshop – Fundamentals of Interviewing Users. I’ll put the link in the show notes. I bet you know someone in the San Francisco Bay Area who would get value out of this workshop and I would appreciate you recommending it. I also work with organizations directly to help them elevate their user research practices.
Of course, supporting me and my business is the best way for you to support this podcast and help me make more episodes. If you have thoughts about the podcast, reach out to me at DONUTS AT PORTIGAL DOT COM or on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
I went to a cafe in my neighborhood. I placed my order and then swiped my card in the payment terminal. They told me “We’ll call you when your order is ready” and I went and sat down. I heard a couple of orders get called, “double cappuccino, soy milk latte” etc. After a few minutes they called me: “Steve!”
I was briefly taken aback. They never asked me for my name, how did they know that order was for me? I realized that my when I paid for my order by credit card, of course they got my name. But this seemed like a new customer service behavior. I was curious so I paid attention the next time I went to Starbucks. They asked me for my name. They do this before payment. My local Starbucks is inconsistent as to whether or not they ask for my name and whether or not they call out my order by the contents of the order or by my name.
There are many regular routines that we go through that become almost scripted, so when something goes off-script, like being called by name when I was never asked for my name, it really jumps out. Eventually the script gets rewritten and we regard the change as familiar, but those moments of change are sometimes tentative moments in an experience. This cafe could have asked me for my name not so they had my name, but so that they could signal to me that they were asking for permission to address me by name in a few minutes. I’m sure there are cases where the name on the credit card doesn’t match to how someone prefers to be addressed.
Maybe I’m just too sensitive in noticing this change, to find it an abrupt surprise. But you can just imagine the well-meaning coffee shop staff feeling excited about being able to do this, to get the customer name and call out the orders in a more personalized manner than just “double americano.” They could, but did anyone stop to think if they should?
Note that I’m not complaining about my service experience, just reflecting on it to suggest that it’s an interesting moment, when things to change. Knowing that things are going to change is an opportunity to get ahead of that change and try to understand more deeply from the people who use that service what it is that they are expecting, and where there might be mismatches between what you want to do and what that change will mean for these customers.
Well, on to the interview. It was wonderful to get to speak with Ruth Ellison and I think you’re going to really like our conversation. She is Head of User Research at the Digital Transformation Agency in Australia.
Ruth, welcome to the podcast. Thank you for being here.
Ruth Ellison: Thank you, Steve. Glad to be here. Thanks for having me.
Steve: So, why don’t we begin as I often do – as we all often do, I guess – ask you for an introduction. Tell us, what do you do?
Ruth: Hi. My name is Ruth Ellison. I’m the Head of User Research at the Digital Transformation Agency. So, at DTA – we call it DTA – we love our acronyms in government. So, DTA is a small, and relatively new, government agency set up to actually help Australian government create simple and fast clear services. So, this involves improving the skill sets of people who work in the space, the digital space, and also just helping looking at projects and how we do digital transformation across government.
Steve: So, what is digital transformation?
Ruth: Oh, that’s the million-dollar question.
Steve: Do I have to pay a million dollars to get the answer?
Ruth: It seems that we’ve got a digital transformation strategy. Um, for me it’s really about how we’re transforming the way government thinks and approach problems. How do we help our citizens interact with government in ways that are better? So, we use digital, but really for me it’s not just about digital. It’s about the people. It’s about the services and how we move to much more human centered lens to problem solving and problem definition and even just the way we deliver our services.
Steve: And then who are the – I don’t know if clients is the right word, but who do you engage with within governments?
Ruth: So, the DTA has a very interesting function. We’re a centralized government agency, very small. There’s only a few hundred of us. Our clients are actually other government agencies, mostly at the federal level. Because we have three levels of government in Australia – federal, state and local. So, we work mostly at the federal level, but part of our function is also looking – we have services cross between boundaries of government and how does that work across? So, our clients are mostly at the federal level. The other government agencies, often very, very large or very small, it doesn’t matter the size. What matters is what are they doing that involves interactions with their users, which is normally citizens or businesses and organizations.
Steve: What are different ways that those relationships get initiated in government?
Ruth: This is interesting for us. A lot of work comes through, depending on the size of the project, as opposed to the size of the problem, is actually involved in investments that’s over a certain number of dollars, but we’re also interested in projects that have a social impact as well. So, if it meets a range of our internal priorities we’re keen to get involved. So, the internet really involves a way to transform the way government interacts with citizens. We’re interested in reaching out and working with people. So, it’s not just government. We also work with government and not for profits and other private organizations to work out what’s the best way to collaborate and finding ways of working. It’s really exciting.
Steve: Can you give some examples, over the last few years, what kinds of things that DTA has worked on?
Ruth: Yeah, so part of our transformation agenda is how do we uplift the skillsets required to work in these ways? I’ve been involved in a few projects back in earlier days of DTA – it’s called Digital Transformation Office. One of them was actually around looking at how people – this might sound really boring, but it actually was fascinating, about tax obligations. So, we’re looking at the space of how do people start up small businesses and what are the kind of challenges and barriers that they face when interacting with government. So, we went to the full discovery process, going through discovery. We do alpha, beta and live. So, discovery was really about what is this problem space looking like for starting a business. As a government we have a lot of assumptions around how people interact with us in our role within people’s lives, but it’s really – discovery, quickly discover that there’s a lot of other factors and lots of other interactions that happen that can be quite surprising. Based on that the team actually – we actually narrowed it down and actually looked at this particularly interesting space, like the maker movement and the people who are making a lot of jewelry, or they’re crafting beers, or they’re doing very niche kind of things where it’s kind of a hobby at the moment. They start shifting over into potentially a small business. That space is fraught with a lot of questions and uncertainty from our citizens because they’re not sure if, “what happens if I don’t say anything to the tax office?” Do I get a tax debt down the track? You don’t want to end up with a $20,000 debt. It’s very scary. How do we actually help solve that particular problem? So, through this kind of discovery and alpha process we narrow it down and actually helped, I think, three agencies, work on this particular problem and how we define what people’s obligations are to the government, and how we can make that a little bit easier?
Steve: What gets output from that process and what gets made or implemented by an agency as a result of that kind of program?
Ruth: I think people hear the word digital and they tend to assume is it a digital product? In this case it was, but some other projects were not. In this particular case we end up with like a little smart, online, questionnaire toolset, but we didn’t know that was what we were going to have. The team was quite deliberate in starting out very open to look at what is the actual problems and what’s the context of these lives that we’re trying to understand because we can’t just assume that a digital tool is going to solve that particular theme. And it’s a bit of a mind shift for government because often it’s quicker just to go down, and just – we’re going to have an app or a new website to fix this. And part of this way of working is going actually, it’s not actually true. Are there other ways we can solve this? And how do we make people’s lives easier. So, we did end up the digital thing and end with this and it actually crosses over multiple agencies which for those that don’t work in government, it’s actually a very challenging space. Agencies tend to work in silos, just for the nature of the way we fund projects and the way we work, and the way skillsets are built, that it’s very sell into one agency at a time. So, to have something that worked across multiple departments and agencies was actually a really amazing achievement and was well done to the teams for being part of this process and actually being willing to change as we went along and learn lots of new stuff about our users?
Steve: You’re talking about the teams in these agencies, that they’re open to that?
Ruth: Yes. So, I think in the earlier days of DTO or DTA people treated this as a bit of an experiment. So, what we did was we formed multidisciplinary teams. So, we didn’t have user research going out doing research and throwing it over the fence. We’re trying to shift that mindset because traditionally a lot of research happening to government is very much on the valuative side. We do a lot of usability testing. We do a report so you kind of give it back to the teams and they do stuff with it and you hope they implement some of it. This was about trying to change the way we approach research, but approach the whole problem landscape. So, the team was formed from, I think, 3 or 4 different agencies, including us. I was actually one of the researchers embedded into that particular team at that time. So, this was before my Head of Research job. And then we actually worked with the teams, with developers, business subject matter experts, taxation experts, and my favorite, a lawyer, as well on how we actually tackle this. And the reason we start looking at those kind of skillsets was because we find if we’re just doing a very IT focused problem and we don’t then look at the problem from a human lens, it comes down to a very technical solution. So, by bringing people like the business folks in and the lawyers, we start having this kind of transformation that occurs that’s very different. So, the lawyer, I was working with her and before she joined the team she was saying we have to send stuff up to her team and they’ll approve the wording, and everything go out to the users. And it was slowing down the processes of how we do design and research. So, the team asked her to join our team and she became embedded. She went from this process of “research stuff, that’s not really what I do. I’m here to be your taxation expert and your lawyer, or the legal expert” to coming on all the research and just changing her mindset on how we actually approach the way we word things. So, going from “we can’t say that because it’s not how government says things,” to “that’s not going to work for the people we saw,” was a massive mindset shift and it was fascinating.
Steve: What do you think – how did you create the conditions for her to have that kind of personal/professional transformation?
Ruth: So, that’s an interesting question. It really comes to the whole team. We were very lucky to have a team of people from different agencies who were willing to try these new ways of working. A lot of them had done our job before, but not necessarily worked where you have all these multiple skillsets in one team, embedded for a 20-week process of trying to go through a full discovery all the way up to releasing something. So, part of that was we have social contracts in place as a team and how do we create a safe environment – psychologically safe – where we can raise things and challenge each other and it’s okay to do that? It was an active work every day. You know the whole team is working on these kind of ways of working to say it’s okay to come along. It’s okay to not agree with things, but come along and let’s just try having an open mind when we’re going out to do this research. And let’s just observe and we’ll come back and as a team we do our analysis and our synthesis together. And that really helped her and the other team members transformation of their mindset because everyone was involved in the process. Not just me as a researcher bringing somebody else out with me. And I think that was a significant mindset change.
Steve: So, this group of people, 20 weeks – like how big was the team?
Ruth: The team was – I think we had about 8-10. The size of the team changes and even now in my role as Head of Research we help teams to understand that the team does change. The nature of the team changes over the course of these phases. So, that was – we kept it small for that particular team because we had to move fairly quickly. We had to deliver something. But we also had to show the teams how we could uplift the kind of capability in both research and the other kind of specialist skillsets. There was agile skillsets. There was things around development. But my job was to lead the research transformation.
Steve: When you say uplift, does this mean that people that participate in this 20 weeks come back to their agency afterwards with – they’ve gone through some professional development?
Ruth: Yeah. So, the goal is for them to go back to their home agencies or departments and actually continue the momentum. So, we learn a lot about that process. That’s a whole = how do you build momentum and continue momentum when you’ve taken the team out of their context in their daily working lives? So, they were embedded into our agency for that 20 weeks where while we create the space for them to do that. We had the walls. We had the equipment that we could take them out. It was very different when they went home. So, we learned a lot about that process and we iterate throughout our services as a result. But for them it’s about how do we enable those kind of processes and systems for them to learn these things and to take home and keep going with it. For example, they had somebody whose job was, I think, to shadow me and actually pick up the sort of research skillset so they could try to take that back home and do a bit more of a deeper dive in research.
Steve: It seems like there’s two sort of – I’m sure there’s multiple objectives, but it seems like one objective is go through this 20 weeks in order to launch something – in this case this sort of small web questionnaire that you created. I think it’s on the web?
Ruth: Yeah, yeah. It’s live.
Steve: So, the project has an objective and an outcome, but the objective for this whole engagement is to uplift the skills so that people can keep going afterwards?
Ruth: Yeah, yeah.
Steve: Are those two objectives in harmony when you do a program like this?
Ruth: That’s a really good question and that’s part of our learnings from this, about what is the measure of success when we’re talking about increasing kind of skillsets within the research space. They’re kind of what we call the specialty skillsets. How do we be clear about what that measuring was? I think part of my learning personally going through that was that wasn’t quite as clear back then. So, we had some tension at times where the team needed to show something to deliver and then at the same time I’m trying to uplift the research capability to show the rigueur, to show how do we work in these ways? Now our screen cycles were only a week at a time. It was very, very fast. And taking people through a very – what was like a very uncomfortable process. And it’s uncomfortable. They’re getting challenged every single day and our job is to help them through and navigate through that process. So, yeah, it creates tension. But, as I said, we have integrated as a result of learning, going through those programs to look at how do we now better serve our customers which are the other government agencies?
Steve: Where – if we were to sort of draw an org chart in the air – maybe it’s not an org chart for the government – where does DTA reside within the structure of the Australian government?
Ruth: Yeah, we have an interesting function. So, we are not a delivery arm. We touch on policy work. But we sit under the Prime Minister and Cabinet kind of function. So, what that means is we are this centralized agency. I think maybe the closest equivalent might be the 18F in the U.S. A little bit of GDS in the U.K. I’m not sure what the Canadian version would be. We have a little bit of that kind of function where it’s not tied into any particular agency because my job is to work across government agencies. But that brings its own challenges around – particularly around the research practices, how we – I’m going to come back to this at some point about how we manage our knowledge across government. A lot of research is happening across many, many agencies. So, we’ve got this uplift function, but as my role as Head of Research I’m also interested in how we manage those other functions within government as well as it relates to research, as a point of research practice?
Steve: Sorry, other functions?
Ruth: Yeah, so at the moment, because I do a lot of – I help teams do a lot of recruitment, but it’s internally within our organization or other government agencies. So, they reach out and go, “Ruth, we need to hire a researcher. We’re not quite sure what to look for. Can you help?” So, it’s about reaching out, how do we help them and craft – how do help them find the right fit of people? Are they doing a discovery piece? Or are they doing just a lot of very heavy product work that’s just iterating just a product day to day to day? And trying to work out what is it that they actually need and how does that work and what kind of researcher would they need? So, helping them find that right fit because we don’t have a pool of people. We’re a very small agency, so we just can’t pop out and just embed into teams. We did that before in the past. It doesn’t scale for us, so now we’re working how we support those functions? Other things we’re also looking at is how do we manage the knowledge that we build with government among our citizens? We have so much research and we spend a lot of money as a government looking at our citizens experience, their interactions with government, but it’s often done in those individual organizations. So, they might have lots of data internally, but what happens when we start needing to look between those touch points, between the different agencies? How does research work in those spaces in between? Because people are having a very agency focused view of their citizens. So, part of our – one of our functions is actually looking at how we provide a more holistic lens across government to show that when a citizen engages in government it’s not just a departmental time, but it’s actually a range of organizations, whether it’s government, private sector, friends, networks – whatever it is. How we show that relationship in the context of people’s lives, so it’s bigger than just one particular service or element of a service?
Steve: Are there aspects of how government is structured? I don’t know whether that’s regulatory or funding? Because every big organization has sort of siloed research and siloed data and doesn’t treat the user experience as coming across all those things. I think that’s common above a certain size, but I’m wondering are there any interesting compounding factors in the way that government works that maybe changes that – how you even try to address those problems.
Ruth: Yeah, it’s a very good point because different government agencies have different rules around data collection and different legislation around how much and what data you collect. There’s often discussion that as user researchers, the kind of data we collect may not necessarily be the same kind of data that the legislation demands that we collect a bunch of other things. So, often there’s tension of going well, we’re here to understand about our citizens. How do we collect that – the people might go the consenters only allowed it to be shared within that particular department. So, it’s like hitting up against issues around consent and do we start looking at broader consent around all of government? Or is it mostly just federal government? Or is it just between portfolios of government? We haven’t nutted out that issue yet. So, it’s a challenge that we’re working on right now because content is actually a big, big problem space for us. Like we want to do research that’s ethical and safe and it’s really challenging when you’ve got multiple agencies who are funded separately. Back in the private sector days I will come, and I will do work for one organization at a time. This feels like we just up the whole scale of the problem because now we’re taking it across multiple agencies who may not necessarily want to share, or be as comfortable sharing, what they have. For a number of reasons. Whether they feel it’s a reputational risk – because we deal a lot with risk as government. There’s always focuses about risk. And I think about what happens if people find out people said that about our service? We don’t want people to know that people say that about our service. So, you get those kinds of complicated…
Steve: Oh, so there’s a risk to the agency.
Ruth: Reputation.
Steve: If the feedback or any kind of user data that’s critical of that agency that becomes public – if you’re a civil servant you don’t want that.
Ruth: There’s some mindsets – I believe they call it reputational risks for government. So, it’s one of the challenges heading up research is how do we deal with that kind of mindset and how to shift some of that thinking as well. So, a lot of framing I’ve been using a lot more recently is about this is about risk reduction and increasing the certainty in the decisions we’re making that’s policy or your product or your features – you know whatever scale we’re talking about, research helps you with that decision making. We’re not something that you just do on the side and just throw it over. That actually becomes part of your decision-making processes and one of your many inputs. Because end of day we understand that people have a lot of pressures and they’re making decisions about why they release some things, whether it’s a policy or a government service. But how do we help them do those things a bit better. So, that framing around risk reduction or reducing certainty – it speaks a language of some people.
Steve: That also sounds like private sector product development. The consequences are different. It’s not just about finances. It’s about other – I’m gathering that you’re in a situation where there’s other consequences besides profit of the organization in the commercial sector. So, you said reputational risk is a very tangible thing that people are kind of operating around.
Ruth: Another challenge that government – we work in a four-year cycle over here in Australia. So, the cycle of government, if it’s incoming, is it a different change of government happening? The agendas may change as well and that would change the focus of your research. When you’re trying to be a little more strategic with research it can change that function, or not. Like we have that additional level of think of. So, although on the one hand we still need to understand our users, our citizens, no matter what. No matter what government is in power we still have – it’s still a part of our function in the role as a researcher to keep that understanding going and to keep developing that deep understanding. But the stuff that it aligns to in the strategic decision making does change and that’s always an interesting factor in working with government. It’s what actually made me – it’s one of the things that made me leave private sector to come here was the complexity of all these kind of nitty gritty things that look so simple on the outside, but it’s actually really complex and really messy. And I love that.
Steve: Okay, what do you love about that? That sounds like something that some people might run away from, but you’re drawn to it.
Ruth: Because it’s hard. It’s really, really hard. I’ve been a consultant for 7/8 – for many years – and I enjoyed that breadth of consulting. But what I found as a consultant, I would go – we’d go in and we’d help people do the – you get the joy of seeing something happen, but I was – I reached a point in my career where I wanted to see that really affect change on a much greater scale and affect it in a much longer kind of timeframe as well. So, I was at a point as a consultant where you do things and it affects a particular project or a program. And actually, when I started looking at how do we affect the whole scale change in government and how do we start shifting that mindset? It was hard to do it as a consultant. So, I thought the only thing I could maybe try doing is to join the public service. So, 2 ½ years ago I left the private sector and I’d come in to try to do this with a bunch of other really smart people across all levels of government, trying to look at how we do this transformation. And it’s hard and I like that because it’s – you’re in for the long haul. You’re in for this massive change. You know it’s not easy and we’re doing this kind of research that’s really – you look at the behavior in organizational and government level change and it’s just complex and it’s fun. Exhausting, but fun.
Steve: So, you’re describing in the consulting world there’s moments or points in the process where you draw satisfaction or some sense of reward from.
Ruth: Yeah.
Steve: And now you’re in this much more complex situation. It’s hard. I’m sort of inferring things take time. What rewards do you identify yourself? Whether it’s milestones or successes, like what keeps you going in a very hard endeavor that you’ve chosen?
Ruth: For me it’s about the empowering of others. One thing you said, it’s about we help people. We like to help people. That’s why we do the role that we do. And for me this is about how do we help at scale? Our government has been doing things for a very long time and there are some very, very smart people in government that often bog down in the structures of government, the bureaucracy, that sometimes doesn’t enable the kind of stuff that these smart people are working on. So, what gives me satisfaction is being able to help some of that transformation. To help others empower them to do this stuff better. And the satisfaction I get is when somebody has that kind of mindset shift and they’re taking this stuff back home. Or, they’re coming back to me later and go, “Ruth, it’s been a year since I’ve seen you, but here’s what I’ve done.” And they themselves have gone back and been this change agent for not just research, but for just human centered – good, human centered thinking. And it’s great when you see that kind of stuff happen. So, helping others to do this stuff better.
Steve: You said the long haul and that makes me think as someone who is a consultant, you’re done. You’re not there to get that feedback or to see the consequences. Or you have to kind of find your way back in to say hey what happened a year later? So, if these are longer term working relationships that you have, you’re able to hear from those people and see kind of what’s happening.
Ruth: Yeah.
Steve: Maybe we can talk about the Head of User Research role. When did that come about? How did it come about?
Ruth: So, Head of Research, I’ve been doing that for about over a year, maybe two years now. We had a previous head of – it was actually design research – Lisa Reichheldt, who you would know. When she left they actually split that role up to two. Her role was an executive level role, but they’re not splitting up the role and taking it down a level. So, it was an interesting organizational change I think – into a Head of Service Design and Head of Research. We also have Head of Content Strategy and Head of Interaction Design who are my equivalents. So, my role involves helping my agency, and across government, to really better the practice of research. I find I’m doing a lot of it – I don’t do as much research anymore nowadays. My role is really more of an enabling function. So, part of that is looking at how do we bring in the right people into the teams. When they’re here, how do we help mentor them? So, part of my role is I catch up with a lot of the researchers and check in how they’re going. I’m connecting them to other researchers as well in our communities. And also trying to look at how we put together much more – how we lift the conversation around research. So, part of my role is actually about that strategic aspect of research. How do we do it better? How do we help enable the broad decision making of government? We’re also starting to see a shift to much more policy sides rather than just product and services – up into policy land where a policy can trigger many, many services and products. So, that’s really fascinating as well. And I’m really enjoying that kind of aspect as well. The other parts of the role is really – we look after a guild. So, I don’t know if – I’ve heard other organizations have this structure too. So, our researchers who come into our organization are embedded into product teams, but we have a guild that then unites all of those researchers and those interested in research to come together once a fortnight and actually share the knowledge. So, it could be – for example, I’ll be going back and doing a sharing of what I might learn at a conference. Somebody might share about something happening in their project. A problem that they’re having. How do we solve it together? So, the guild is almost like a tribe that cuts across our organization no matter what team you’re from and it unites us as a practice. And that’s how I work through that kind of process to enable the better uplift of our practice.
And externally, my job is just to advocate for the research role. So, I often get called in to talk to other executives about what does this research mean? “Don’t we do this stuff?” “Haven’t we been doing this stuff for years?” Answering those kind of questions. Look at how it helps them in their strategic perspective. How it helps in their decision making? And also, just help their staff connect better and just do the craft better.
Steve: You said before how one of the things you’re doing is helping agencies hire into that agency as opposed to into DTA – hire researchers directly into those agencies.
Ruth: Yeah.
Steve: Are those – and then you’re talking now about sort of maintaining connections with researchers through sort of formal and informal methods. I’m just trying to understand, is that within DTA only? Is there engagement with these other agency researchers, I guess I’d call them?
Ruth: Yeah. So, the supporting through the guild is a DTA only thing that we do. Or anyone that comes into DTA on a short-term project, we connect them up into the guild, so we can support them that way. But for the broader government we actually got a Google mailing list. It’s called a service design user research mailing list. It covers design and research and we support people through that. But it’s not just us – we’re a facilitator, so our job is to connect others to each other because we’ve got amazing people out in government and if they connect – and peer – we learn from each other. That’s a really good thing. So, we’re not the be all to end all. We’re that connecting function. I think some of these we’re brokering. You know we help broker relationships as well. Otherwise, it can come across as oh we have to go through DTA, or do we have to be involved in all this stuff? No, because it’s about enabling others, empowering others to connect and keep growing together.
Steve: So, there might be a researcher in one agency that you know and another researcher in another agency that you also know, but they don’t know each other.
Ruth: Yeah.
Steve: And if there’s something going on you might say oh this is the researcher you should talk to.
Ruth: Yeah. So if somebody – say they’re trying out a particular technique for the first time and I go so and so from here has actually done it before, have a chat to them about how they do it because they would have some good learnings in the government context of what worked and what didn’t work and it will just help speed you up when you try and do this yourself.
Steve: So, in DTA itself, you listed a bunch of different functions that you have kind of peers leading. I’m sort of wondering how the team sizes vary? I don’t know if team is the right word, but sort of the number of people maybe in each of those functions?
Ruth: At the moment I think we’ve got around 13 researchers, but the number varies significantly across DTA depending on the projects that we’re supporting at the time. It also doesn’t include the projects that we support that belong to other agencies. So, that number scales. At one point I was working on project that had 18 researchers and that was just within the one project. So, that was a good question there about scaling and processes and ethics and all of that. But the number changes and what’s interesting is that the recruitment is done by the product owner and we provide a supporting function to help them get the right fit. But it’s done by the products where the person comes – they join the product team. So, they don’t belong to my team of researchers. It’s more of a virtual kind of team if that makes sense. It’s where the guild/tribe comes in. And those product teams, they usually range between the kind of 6/10 kind of sizing and there’s usually a dedicated research function in there. Not all teams can have one because as a government we often have budget constraints. So, we have people who do research who are not necessarily researchers. But we try to encourage that specialism, whether it’s a researcher or a service designer or an interaction designer because they bring a depth and they’re upskilling people on their team as they’re doing it too.
Steve: So, someone could be hired into DTA as a researcher, but the thing they’re spending their time on is a project team?
Ruth: Yes. Yeah, that’s right. Although I try to steal some time to work on our community, our practice matter things. One of the things at the moment is we’re reiterating through our processes and policies at the moment. So, through guilds – we use the guild as a bit of a working spot as well. We come together one hour a fortnight and we use it to work out what we want to tackle next and how to improve this particular process. So, I try to steal time that way from those teams to enable our better practices.
Steve: For Australian government employees, how geographically dispersed are they?
Ruth: So DTA, because we’re so small, we’re based mostly in Canberra which is the capital of Australia, and very small. I think we’re only about 400,000 people that lives in Canberra. Our other office is in Sydney which is even smaller. What makes it interesting, when you have that kind of diversity in spaces, is that other agencies are even bigger than us and they have many more people scattered around Australia. But the challenge I think that comes to research is recruiting. Finding strong talent in Canberra can be challenging as well and often people – I love Canberra, but lots of people – there’s always jokes about Canberra. Honestly, it’s too cold. It can snow in winter. It’s cold. It’s cold in Canberra and it’s hot in the summer. It’s small. We’re inland. People have the stereotypes with it being a government town. So, there are all these stereotypes around Canberra being a boring place. It’s actually not. It’s actually a beautiful place and I love it. But recruitment becomes really challenging. How do you encourage people to leave their beautiful cities of Melbourne? Leave Sydney and come spend some time. So, part of it is trying to shift the conversation around how do we work on more meaningful work instead and shifting the conversation around that. So, I think I always joked to Leisa before about do we need to have a recruitment video to show people, hey, it’s great, come work in Canberra. But it does change the way then on recruitment and how we uplift capability within existing skillsets as well.
Steve: Can we go back to something you were saying about policy? I want to understand maybe the mechanics of that more. I understand that the policy is set by parts of the government and then, as you said, that impacts the products that are being created. But you were describing maybe a different cycle, I think, or a different stroke in that cycle where the work that you’re doing can – is it correct to say that it’s impacting policy?
Ruth: Yeah.
Steve: What’s the dynamic there, I guess, is really my question?
Ruth: So, there’s been a shift – more recent shift in government. And I think you’ll see that globally – in global governments as well about how we shift this more human centered thinking and lenses over to policy lens. Traditionally in policy I do a lot of research. When you create a policy, they go out and do big scale studies. There are certain methodologies for policy research. But part of what we’re trying to help is how do we get the policy folks who are creating this policy actually out in the field, actually talking to the people who are impacted by this policy, and actually going out into the communities that will be impacted down the track by this, and actually understand that context. So, some people do do that already, but it’s not necessarily the norm. So, we’re trying to look at how do we shift some of that thinking and that’s part of my kind of focus at the moment is how we start working towards that lens and it’s really interesting. Because you know people have been doing policy creation for years. It’s not a new thing. A research lens is not a new thing, but the way we do our kind of research is actually new for them. And trying to help people, going, actually this stuff that we do is not just a digital thing. There’s a bit of a perception that oh, this kind of research is to create digital products, right? And you go, actually no, we can actually apply it across a range of things. Because it’s really about understanding people, understanding what is it like to – whether it’s government or whatever it is – what does a person’s day in the life sort of look like in this space. Especially when it’s government, we’re serving such a diverse range of people from our indigenous communities. Those are very disadvantaged. To those that might be on the other end of the scale as well. All the way to large scale businesses. It’s such an amazing, diverse set of users. So, how do we kind of work together to shift that thinking? It’s a good challenge and one that I think will take time as well.
Steve: It seems very analogous to working with those agencies – your phrase of uplifting their skills – those product people are starting in a different place than your policy people are, but it sounds like it’s the same thing. We want to help you to understand the benefit of going out to the communities and seeing what’s going on.
Ruth: Yeah. As I mentioned earlier – I remember starting government 17 years ago working on range of systems and our work involved doing usability testing. You see people using your thing. So, research in that aspect is not new, but I think the more general style in exploratory style research is new, or newer for government now because it’s been happening for the last few years. So, seeing that shift has been really interesting, to say how do we help you do more of that style of research as well. And they’re all very valuable and the stuff you were doing before, usability testing was great, and it’s got a purpose, but how do we shift that conversation to talk about are we actually solving the right problems. I think that comes under the guise of maybe a lot on the innovation labs and design thinking. At the end of the day for me it’s about the fundamentals of who are these people we’re designing for and what problems are we trying to solve, and how do we understand the needs deeply? How we get there – there will be a range in methods – how to help people work out what that combination of methods are.
Steve: When we talk about research methods that’s often about how do we get this data from these users we’re trying to understand but your emphasis seems very strongly on how do we engage the different people that work in government to learn that and do that and use it? It sounds like – the word methods to me would encompass what you’re doing there as well.
Ruth: Methods, approaches. Yeah, it is. I mean it’s one of the things – words have meaning, and words are powerful. Within government, when we talk about research, you’re talking about methods. People go what are we talking about in there? Are we talking about market message? We do a lot of that. You know is it just a design thinking thing which is also a big buzzword. Or is it this agile thing. Being clear about what’s the outcome we’re trying to achieve as government. I use the word methods, but really, it’s about that focus of we know we have an outcome, how do we best meet that and make sure we’re doing something that’s good for our users that doesn’t disadvantage them, but hopefully improves their lives as well.
Steve: It would be great to talk a little about what your trajectory was through your education and your professional work that kind of led you on the path. Talk a little bit about the consulting work before government – maybe just go back a little further and talk about where did you start to see this path that you’ve been on?
Ruth: I actually totally by accident fell into it. I started off as a business analyst designing mainframe screens. That was over 17 years ago, my first kind of proper design kind of job. I had no idea what mainframes were at that time. What is this thing? And it was just by the luck of the draw, I happened to be sitting next to a team that was the very first user centered design team in Australian government. So, I would hear them. They go out and they do these things. They come back, and they would share these stories about the people that they were meeting and the kind of feedback they had about the products they were testing. I said what are they doing? This stuff is really interesting. So, being a curious person, I stuck my head up over the fence and asked them what is it that you’re doing? Tell me about this user centered design stuff. I never covered that in my university degree. We covered all about systems gathering and requirements gathering, all these different methodologies that you do to do that – never once heard about this human centered stuff before. So, by accident I heard that, and I just asked can I join the team? I thought what’s the worst they can say? No. And then I’ll just keep doing what I’m doing and keep trying to find a way. So, I asked and I joined that team. And so, I had my grounding by learning on the job. So, starting government 17 years ago, just learning about what this user centered design stuff was. Really loved it so much and then went back to uni to try to get a masters in human factors, back in the day.
Steve: What was your first degree in?
Ruth: My first degree was information technology and systems. So, very, very IT driven. So, I come from that kind of analytic lens. But doing more of this stuff I just realized I love research. But it was a gradual thing because I didn’t know that such a job existed. And it could be a reflection of Australia at the time because 17 years ago the whole user centered design movement was happening, or starting to happen. Companies were getting set up to do this kind of space. Government agencies were thinking about this stuff and doing usability testing. That’s what my UCD was at the time. So, that was my first foray to this field. From there, and I think like many other people in Australia, we were generalists. We came from UCD to a UX generalist. We did everything, including research and prototyping, everything. But I found soon enough – I quickly realized that research was my passion. I just wanted to do more of this stuff. I think I just got really interested in what people do and say and think and why do they do certain things. And that was fascinating. So, over the next 17 years I ended up leaving government and then going out into consulting. I was head huntered to come and do this work and it was great. So, I did that and was working in the private sector for quite a while and had this broad range of experiences. And then from there, that’s when I decided okay, I’ve been doing this stuff for a while, I’m enjoying it. I’m enjoying being a principal and helping others be better at the consulting craft. But that’s when I then came back to how do I do this longer term, impactful thing around government because that’s where I live and I’m really passionate to see how they improve government. There’s just so much interesting work happening in that space. So, that’s when I made the leap back. So, I started off at DTA as one of the lead senior researchers that they had and then when Lisa left I took over and became Head of Research.
Steve: When you think about the time that you were consulting, as someone who – I’m sorry. I’m going to go back to something else. You started talking about going back to uni for – was it human factors?
Ruth: Human factors – yeah back in the day there was human factors. But we didn’t have all the amazing design degrees we have now. Back then it was – that was the closest thing I could find because I wanted to improve – I just wanted to broaden my skillset in this kind of human centered design space and the only thing we had at the time as human factors. So, I was doing that remotely. I was also one of the first few students who was doing remote uni work. They’d never had that before. It was challenging. And I got to work and learned with a bunch of people who were designing airplanes and aircrafts and hospital systems. And when I’d introduce myself it was like I work for government and I design systems, it was very jarring for me to go wow these people are doing stuff that – massive design things, so if you get it wrong it falls out of the sky or people die. And I was just designing systems. But it was such a good grounding for me because human factors really is about that processing, that cognition. It really drew upon the school of psychology and the sciences as well, from this particular course that I was doing.
Steve: Was there ever a point at which you started to identify yourself as a researcher specifically?
Ruth: That’s a good question. That’s hard because in Australia people have mostly been generalists. That’s how it just evolves. And you had to be in the consulting world to survive because if you – I would just get really jealous – I would see the kind of work you’re doing and go you can be a specialist, but in Australia we just weren’t quite there as a country. We didn’t have the maturity to say I am a researcher. So, I called myself just a UX person for a long time, but research is my passion and I always seeked projects doing that kind of work that’s research and help enable that in our clients and in our practice. I think it just evolved. I don’t know actually when. I just started calling myself that kind of thing. It’s just something that I did.
Steve: So, your first job that you came into DTA with – you had research in your title. Not to conflate the way we identify ourselves with what our job title is, but that’s a point where you went from generalist to specialist. Is that right?
Ruth: If you’re talking about titles, probably yes, because previously my title was principal consultant, UX consultant. And part of my job was – although most of my work I was in was research work – that’s just what I like, so I will do that kind of work. But that wouldn’t naturally go down well with the clients. They just weren’t ready for that in Australia. They were more familiar with UX. We need to improve our experience and however you get there, we want to help them get that outcome. So, although probably 90% of my work was research, the other part of it was actually supporting – oh, probably a little bit less – I was actually supporting the practices as well. As principal, my job was to make sure the quality of all the work coming through was also good. I was doing a lot of that kind of stuff in that kind of leadership role. It was the other half of my job before I left consulting. So, I think the detail – just use the title, to the exclusion of everything else.
Steve: One of the points you were making earlier was about the challenge of recruiting – I guess in Canberra specifically and maybe government is just a part of that. But what was – how did you get recruited to rejoin government?
Ruth: I was actually working at the former DTA, or DTO, as a consultant doing research work for them. Because it was the first time they were going through these ways of working and we were doing discovery, alpha, beta, live. And they needed people who had experience doing research, who could help lead that process and to help facilitate that process. And they were also recruiting for other specialists. So, I was one of the many specialists that was brought on because they had to scale really quickly and trying to build that capability within weeks was not going to happen. So, they had to find it externally. So, I was one of those many externals that were brought in. When I came in I saw these ways of working and going wow, I didn’t know we could do that in government. It’s a bit of a stereotype. People think about government as a very, very slow moving beast, that you can’t do a lot of things. But in reality, it’s about how do we frame it and how do we talk the language of the people who are decision makers to help bring about this change. So, for me seeing that, I was like I want to be part of this kind of stuff. It was a bit of like start up, doing all these ways. It’s like I found a home. So, seeing that was how I then made a leap. So, DTA – or DTO at that time – they kept asking can you come on board and if you like doing this stuff, if you like government, come join. So, I took a while to think about it, quite a while, and then made the leap. Because it is a big leap to leave what is a very interesting job as a consultant and you’re helping so many different clients and so many different problem spaces. You know one day you could be working for an airline, the next one for a bank and the next one for a not for profit. You know that’s really, really interesting. I like that. But there was this need and I wanted to be part of that movement as well, to transform the way we do things as government.
Steve: You’ve made this point a couple of times that even though the D in DTA is digital, the output, the outcomes don’t have to be digital. Do you have any examples where that’s kind of how it’s gone?
Ruth: Yeah. So, there was this great project a few years ago, done within DTA. It’s Medicare – Medicare is our kind of health system where if you’re a citizen you get access to major health services. It’s covered by the government. So, one of the services I was looking at was having a baby. So, in Australia you have a baby. You have to register the baby for a Medicare card. There’s a whole range of process around the registration. And as someone who has gone through it myself in the last two years, it’s actually quite a tricky timeframe. If you’re exhausted from delivering this amazing little thing, you’ve got a million things going through your mind and then you’ve got all these forms you have to fill in. And sometimes that involves, because of identity things, you have to go into offices with a newborn. It’s just hard – it’s hard. And I was particularly emphasizing now that I have a little one myself. So, this particular project is looking at how do we make that process easier. Now what was interesting was that the team had gone – and I think they were thinking, maybe we just digitize the form. Maybe that’s what we mean when we talk sort of digital transformation. That might be the easiest way we can deliver something quickly. But what they soon found out, and I had a really awesome service designer called Mira who joined the team. She worked with them and she actually suggested they bring in a lawyer, because they kept hitting up against issues like oh, well of course we’ll just digitize the form. What else will we do? But when they went out doing the research they were finding out that a lot of the stuff we were asking for as government, we really have and the hospitals have. Because when you give birth you’re giving a bunch of details over. So, they actually went out and did a bunch of research out in hospitals and what they ended up doing was actually deciding, you know what, we don’t actually need something. We don’t actually need a form. Or we don’t actually need a digital thing, like an app or a website to fill in. What happens if we even just remove that process and just assume that if we’re having a baby and we get consent, we can just issue the card. So, they were exploring this what might we kind of situation. So, it was radical, and people were going, “no you can’t do that.” But when the lawyer joined the team, there were lawyers going actually you know what, we can actually make this work. So, the trial was actually in one of the hospitals up in the Gold Coast. It was actually looking at what happens if you sequenced in from the parents at the time of the birth. They’re filling a bunch of information to the hospital about the birth of the baby, what happens if we use that and issue the Medicare card. So, it’s something that had gone from taking weeks and weeks because you have to go in and see multiple people, get the thing signed. And gone down from that to only a few days. It was amazing because you’ve now gone from filling in something to not even having a thing and stuff just appeared in your letter box. I think that’s a very great example of transformation happening where you don’t have to have a particular, physical digital thing. And it was good to see what they did with it.
Steve: Are there things outside the professional realm, like outside your work at DTA, where your skills, the things that you have a passion for and the things that you are just so great at, is there anything that you do outside – were those things manifest in other forms?
Ruth: So, Canberra is a very small place and I love community. So, things I would do are not quite – they’re not research related, but it comes under people. I’m involved a lot in organizing community events around things like TEDxCanberra. So, I’ve been involved in that for about 5 years about bringing and facilitating the amount of smart people that we have and for others to learn from that. I really enjoy that aspect and that comes through like a job as well. I really love connecting people and seeing them grow because my intent is if you can grow and be a million times better than what I can ever be, that is an awesome outcome. You know always hire people that are smarter than you. You want to encourage that growth. So, for me that stuff that manifests outside of my job has to do with things like that. Things like BarCamp back in the days. Something called GovHack, very early in the days – so we look at open government data and how do we help people flesh that out into interesting things. All those little stuff like that is what – again, we’re connecting community and people.
And something else I do that’s not even related is I love making. My hubby and I, we’re both makers. We’re like crafting things. So, I make science-themed jewelry for fun on the side, using our lasers and 3D printers. For some reason, that process of designing something that’s very, very tangible and you’re prototyping that thing and you’re wearing it and you’re seeing people use it as well. It’s very interesting. So, although it’s not necessarily a research thing, it’s a fun thing because you are trying to use a creative side of your brain in a very different way. In a much more tangible way.
Steve: So, is that the connection to the creative?
Ruth: Yeah. And again, I’m just doing research on a very small scale, again to wear my stuff and see how it works. It’s just very, very small-scale stuff. But you learn a lot about people’s behaviors, about when they say things like oh Ruth, can you make that in these colors because I think people will buy that. And what people say they want and what they actually end up doing. Like we talk about that all the time, but to see it in action with your own product is very interesting because you’ll make stuff because people ask for it. So, let’s just try it and see what happens. There’s not a need for it. You just lazily see it sit there. But just little things like that where it’s really tiny stuff that we talk about in our practice that manifests in this way. It’s interesting.
Steve: Can you describe one of your pieces?
Ruth: Yeah, so one of the things that we do is we look at data. So, my hubby actually takes the data from looking at the weather data for the cities in Australia. So, the Bureau of Meteorology here has about 100+ years of weather data. So, we take that and actually visualize it into a wearable piece. Unfortunately, I’m not wearing one today. I’m wearing a serotonin molecular today which is happiness. We take that and then actually model it into a bit of a big jewelry statement piece and then laser cut it. And that’s meant to show, well just data visualization. How do we make science interesting and communicate these things in different ways that’s not just a PowerPoint or a presentation or a survey, but actually a very tangible way of understanding data. So, I’m interested in also that crossover between when we talk about research, the data and then the data sciences folks. I just love geeking out some of the data stuff and going hey, we could find other just weird wonderful ways to display this. So, it may not have any actual use to researchers, but it’s just geeky and fun.
Steve: What do you think people who purchase your jewelry – because so much of our work is the people that make something have a model of it and other people that use it have a model, but sometimes our job is to just try to articulate that delta. So, you’re describing kind of what you’re putting into these pieces and even just the geeky pleasure. But for someone that buys one and wears something that you’ve made, what’s their interpretation or their narrative of what it is, or what it means?
Ruth: Yeah, it’s interesting. They’ll come up and they’ll go – they identify as a fellow science geek. Especially seeing that, they’re going “is that a serotonin molecule?” Like, it is. And then we have a conversation about science and about what makes them interested in science. I use it as kind of a talking point to work out how did they get into this field? Or if they’re not in the science field, how do they actually trigger – it’s a good way to practice research questions actually. But it’s just fascinating seeing that kind of shared – I think they’re having that shared platform or love of a particular field, in this case science. And that’s what connects people and they feel like oh, they’re part of this group of people who love these kind of geeky things.
Steve: There’s something, I don’t know, secret about it, that someone who gets it is going to get it when they see it and somebody else is just going to not recognize that.
Ruth: It’s funny. Not a secret club, but it is that recognition that somebody else enjoys this thing. Whether people have that in music, or whatever their particular passion is, when you connect on that kind of level it’s just so interesting. And then you start hearing their stories about how they either buy it for their daughter because their daughter had started doing this thing out in Antarctica – wow, just the stories you hear. I think that’s where it comes to my research – I just love hearing stories as well and you hear all these fascinating things that people tell you when you’re buying jewelry because it’s just a conversation starter.
Steve: Yeah, and you’re connecting on something. You’re right. It’s not secret. Secret is about being hidden. This is just about that connection, or something about the recognition, that’s a better of putting it, to get those stories. That’s really wonderful. Is there anything else that we should cover in this conversation?
Ruth: I think we had a good range of discussion.
Steve: I think so too. Well, thank you so much. It’s been really great to chat with you.
Ruth: Thank you, Steve.
Steve: Thank you for sharing so much great stuff.
Ruth: That was really fun. Thank you very much for your help, for your time.
Steve: All right. That’s the wrap on another episode! subscribe to Dollars to Donuts wherever you get podcasts. If you’re using Apple Podcasts, how’s about giving the podcast a rating (five stars?) and even a short review. This helps other people find out about the podcast. Portigal dot com slash podcast has transcripts, show notes, and all of the episodes. Follow the podcast on Twitter, and buy my books Interviewing Users and Doorbells Danger and Dead Batteries from Rosenfeld Media or Amazon. Our theme music is by Bruce Todd.
The post 21. Ruth Ellison of Digital Transformation Agency first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Ruth Ellison, Head of User Research at DTA, the Digital Transformation Agency in Australia. We discuss the challenges of user research – and digital product development – in government, embedding researchers into product teams but maintaining a guild model to connect them, and how research […]
The post 21. Ruth Ellison of Digital Transformation Agency first appeared on Portigal Consulting.
In this episode of Dollars to Donuts I speak with Ruth Ellison, Head of User Research at DTA, the Digital Transformation Agency in Australia. We discuss the challenges of user research – and digital product development – in government, embedding researchers into product teams but maintaining a guild model to connect them, and how research can impact policy.
My role is really more of an enabling function, looking at how do we bring in the right people into the teams? When they’re here, how do we help mentor them? I’m connecting them to other researchers in our communities. Also trying to look at how we lift the conversation around research. Part of my role is about that strategic aspect of research. How do we do it better? How do we help enable the broad decision making of government? – Ruth Ellison
Show Links* ~~Fundamentals of Interviewing Users (SF)~~ * Ruth on LinkedIn * Ruth on Twitter * Digital Transformation Agency * DTO becomes DTA * 18F * GDS (Government Digital Service) * Leisa Reichelt * Leisa Reichelt on Dollars to Donuts (part 1) * Leisa Reichelt on Dollars to Donuts (part 2) * Canberra * Medicare * TEDxCanberra * Science and Geek themed jewellery
Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.
TranscriptSteve Portigal: Greetings and thanks for checking out this episode of Dollars to Donuts, the podcast where I talk with people who lead user research in their organization.
Coming up in San Francisco on September 13th, I’m teaching a public workshop – Fundamentals of Interviewing Users. I’ll put the link in the show notes. I bet you know someone in the San Francisco Bay Area who would get value out of this workshop and I would appreciate you recommending it. I also work with organizations directly to help them elevate their user research practices.
Of course, supporting me and my business is the best way for you to support this podcast and help me make more episodes. If you have thoughts about the podcast, reach out to me at DONUTS AT PORTIGAL DOT COM or on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
I went to a cafe in my neighborhood. I placed my order and then swiped my card in the payment terminal. They told me “We’ll call you when your order is ready” and I went and sat down. I heard a couple of orders get called, “double cappuccino, soy milk latte” etc. After a few minutes they called me: “Steve!”
I was briefly taken aback. They never asked me for my name, how did they know that order was for me? I realized that my when I paid for my order by credit card, of course they got my name. But this seemed like a new customer service behavior. I was curious so I paid attention the next time I went to Starbucks. They asked me for my name. They do this before payment. My local Starbucks is inconsistent as to whether or not they ask for my name and whether or not they call out my order by the contents of the order or by my name.
There are many regular routines that we go through that become almost scripted, so when something goes off-script, like being called by name when I was never asked for my name, it really jumps out. Eventually the script gets rewritten and we regard the change as familiar, but those moments of change are sometimes tentative moments in an experience. This cafe could have asked me for my name not so they had my name, but so that they could signal to me that they were asking for permission to address me by name in a few minutes. I’m sure there are cases where the name on the credit card doesn’t match to how someone prefers to be addressed.
Maybe I’m just too sensitive in noticing this change, to find it an abrupt surprise. But you can just imagine the well-meaning coffee shop staff feeling excited about being able to do this, to get the customer name and call out the orders in a more personalized manner than just “double americano.” They could, but did anyone stop to think if they should?
Note that I’m not complaining about my service experience, just reflecting on it to suggest that it’s an interesting moment, when things to change. Knowing that things are going to change is an opportunity to get ahead of that change and try to understand more deeply from the people who use that service what it is that they are expecting, and where there might be mismatches between what you want to do and what that change will mean for these customers.
Well, on to the interview. It was wonderful to get to speak with Ruth Ellison and I think you’re going to really like our conversation. She is Head of User Research at the Digital Transformation Agency in Australia.
Ruth, welcome to the podcast. Thank you for being here.
Ruth Ellison: Thank you, Steve. Glad to be here. Thanks for having me.
Steve: So, why don’t we begin as I often do – as we all often do, I guess – ask you for an introduction. Tell us, what do you do?
Ruth: Hi. My name is Ruth Ellison. I’m the Head of User Research at the Digital Transformation Agency. So, at DTA – we call it DTA – we love our acronyms in government. So, DTA is a small, and relatively new, government agency set up to actually help Australian government create simple and fast clear services. So, this involves improving the skill sets of people who work in the space, the digital space, and also just helping looking at projects and how we do digital transformation across government.
Steve: So, what is digital transformation?
Ruth: Oh, that’s the million-dollar question.
Steve: Do I have to pay a million dollars to get the answer?
Ruth: It seems that we’ve got a digital transformation strategy. Um, for me it’s really about how we’re transforming the way government thinks and approach problems. How do we help our citizens interact with government in ways that are better? So, we use digital, but really for me it’s not just about digital. It’s about the people. It’s about the services and how we move to much more human centered lens to problem solving and problem definition and even just the way we deliver our services.
Steve: And then who are the – I don’t know if clients is the right word, but who do you engage with within governments?
Ruth: So, the DTA has a very interesting function. We’re a centralized government agency, very small. There’s only a few hundred of us. Our clients are actually other government agencies, mostly at the federal level. Because we have three levels of government in Australia – federal, state and local. So, we work mostly at the federal level, but part of our function is also looking – we have services cross between boundaries of government and how does that work across? So, our clients are mostly at the federal level. The other government agencies, often very, very large or very small, it doesn’t matter the size. What matters is what are they doing that involves interactions with their users, which is normally citizens or businesses and organizations.
Steve: What are different ways that those relationships get initiated in government?
Ruth: This is interesting for us. A lot of work comes through, depending on the size of the project, as opposed to the size of the problem, is actually involved in investments that’s over a certain number of dollars, but we’re also interested in projects that have a social impact as well. So, if it meets a range of our internal priorities we’re keen to get involved. So, the internet really involves a way to transform the way government interacts with citizens. We’re interested in reaching out and working with people. So, it’s not just government. We also work with government and not for profits and other private organizations to work out what’s the best way to collaborate and finding ways of working. It’s really exciting.
Steve: Can you give some examples, over the last few years, what kinds of things that DTA has worked on?
Ruth: Yeah, so part of our transformation agenda is how do we uplift the skillsets required to work in these ways? I’ve been involved in a few projects back in earlier days of DTA – it’s called Digital Transformation Office. One of them was actually around looking at how people – this might sound really boring, but it actually was fascinating, about tax obligations. So, we’re looking at the space of how do people start up small businesses and what are the kind of challenges and barriers that they face when interacting with government. So, we went to the full discovery process, going through discovery. We do alpha, beta and live. So, discovery was really about what is this problem space looking like for starting a business. As a government we have a lot of assumptions around how people interact with us in our role within people’s lives, but it’s really – discovery, quickly discover that there’s a lot of other factors and lots of other interactions that happen that can be quite surprising. Based on that the team actually – we actually narrowed it down and actually looked at this particularly interesting space, like the maker movement and the people who are making a lot of jewelry, or they’re crafting beers, or they’re doing very niche kind of things where it’s kind of a hobby at the moment. They start shifting over into potentially a small business. That space is fraught with a lot of questions and uncertainty from our citizens because they’re not sure if, “what happens if I don’t say anything to the tax office?” Do I get a tax debt down the track? You don’t want to end up with a $20,000 debt. It’s very scary. How do we actually help solve that particular problem? So, through this kind of discovery and alpha process we narrow it down and actually helped, I think, three agencies, work on this particular problem and how we define what people’s obligations are to the government, and how we can make that a little bit easier?
Steve: What gets output from that process and what gets made or implemented by an agency as a result of that kind of program?
Ruth: I think people hear the word digital and they tend to assume is it a digital product? In this case it was, but some other projects were not. In this particular case we end up with like a little smart, online, questionnaire toolset, but we didn’t know that was what we were going to have. The team was quite deliberate in starting out very open to look at what is the actual problems and what’s the context of these lives that we’re trying to understand because we can’t just assume that a digital tool is going to solve that particular theme. And it’s a bit of a mind shift for government because often it’s quicker just to go down, and just – we’re going to have an app or a new website to fix this. And part of this way of working is going actually, it’s not actually true. Are there other ways we can solve this? And how do we make people’s lives easier. So, we did end up the digital thing and end with this and it actually crosses over multiple agencies which for those that don’t work in government, it’s actually a very challenging space. Agencies tend to work in silos, just for the nature of the way we fund projects and the way we work, and the way skillsets are built, that it’s very sell into one agency at a time. So, to have something that worked across multiple departments and agencies was actually a really amazing achievement and was well done to the teams for being part of this process and actually being willing to change as we went along and learn lots of new stuff about our users?
Steve: You’re talking about the teams in these agencies, that they’re open to that?
Ruth: Yes. So, I think in the earlier days of DTO or DTA people treated this as a bit of an experiment. So, what we did was we formed multidisciplinary teams. So, we didn’t have user research going out doing research and throwing it over the fence. We’re trying to shift that mindset because traditionally a lot of research happening to government is very much on the valuative side. We do a lot of usability testing. We do a report so you kind of give it back to the teams and they do stuff with it and you hope they implement some of it. This was about trying to change the way we approach research, but approach the whole problem landscape. So, the team was formed from, I think, 3 or 4 different agencies, including us. I was actually one of the researchers embedded into that particular team at that time. So, this was before my Head of Research job. And then we actually worked with the teams, with developers, business subject matter experts, taxation experts, and my favorite, a lawyer, as well on how we actually tackle this. And the reason we start looking at those kind of skillsets was because we find if we’re just doing a very IT focused problem and we don’t then look at the problem from a human lens, it comes down to a very technical solution. So, by bringing people like the business folks in and the lawyers, we start having this kind of transformation that occurs that’s very different. So, the lawyer, I was working with her and before she joined the team she was saying we have to send stuff up to her team and they’ll approve the wording, and everything go out to the users. And it was slowing down the processes of how we do design and research. So, the team asked her to join our team and she became embedded. She went from this process of “research stuff, that’s not really what I do. I’m here to be your taxation expert and your lawyer, or the legal expert” to coming on all the research and just changing her mindset on how we actually approach the way we word things. So, going from “we can’t say that because it’s not how government says things,” to “that’s not going to work for the people we saw,” was a massive mindset shift and it was fascinating.
Steve: What do you think – how did you create the conditions for her to have that kind of personal/professional transformation?
Ruth: So, that’s an interesting question. It really comes to the whole team. We were very lucky to have a team of people from different agencies who were willing to try these new ways of working. A lot of them had done our job before, but not necessarily worked where you have all these multiple skillsets in one team, embedded for a 20-week process of trying to go through a full discovery all the way up to releasing something. So, part of that was we have social contracts in place as a team and how do we create a safe environment – psychologically safe – where we can raise things and challenge each other and it’s okay to do that? It was an active work every day. You know the whole team is working on these kind of ways of working to say it’s okay to come along. It’s okay to not agree with things, but come along and let’s just try having an open mind when we’re going out to do this research. And let’s just observe and we’ll come back and as a team we do our analysis and our synthesis together. And that really helped her and the other team members transformation of their mindset because everyone was involved in the process. Not just me as a researcher bringing somebody else out with me. And I think that was a significant mindset change.
Steve: So, this group of people, 20 weeks – like how big was the team?
Ruth: The team was – I think we had about 8-10. The size of the team changes and even now in my role as Head of Research we help teams to understand that the team does change. The nature of the team changes over the course of these phases. So, that was – we kept it small for that particular team because we had to move fairly quickly. We had to deliver something. But we also had to show the teams how we could uplift the kind of capability in both research and the other kind of specialist skillsets. There was agile skillsets. There was things around development. But my job was to lead the research transformation.
Steve: When you say uplift, does this mean that people that participate in this 20 weeks come back to their agency afterwards with – they’ve gone through some professional development?
Ruth: Yeah. So, the goal is for them to go back to their home agencies or departments and actually continue the momentum. So, we learn a lot about that process. That’s a whole = how do you build momentum and continue momentum when you’ve taken the team out of their context in their daily working lives? So, they were embedded into our agency for that 20 weeks where while we create the space for them to do that. We had the walls. We had the equipment that we could take them out. It was very different when they went home. So, we learned a lot about that process and we iterate throughout our services as a result. But for them it’s about how do we enable those kind of processes and systems for them to learn these things and to take home and keep going with it. For example, they had somebody whose job was, I think, to shadow me and actually pick up the sort of research skillset so they could try to take that back home and do a bit more of a deeper dive in research.
Steve: It seems like there’s two sort of – I’m sure there’s multiple objectives, but it seems like one objective is go through this 20 weeks in order to launch something – in this case this sort of small web questionnaire that you created. I think it’s on the web?
Ruth: Yeah, yeah. It’s live.
Steve: So, the project has an objective and an outcome, but the objective for this whole engagement is to uplift the skills so that people can keep going afterwards?
Ruth: Yeah, yeah.
Steve: Are those two objectives in harmony when you do a program like this?
Ruth: That’s a really good question and that’s part of our learnings from this, about what is the measure of success when we’re talking about increasing kind of skillsets within the research space. They’re kind of what we call the specialty skillsets. How do we be clear about what that measuring was? I think part of my learning personally going through that was that wasn’t quite as clear back then. So, we had some tension at times where the team needed to show something to deliver and then at the same time I’m trying to uplift the research capability to show the rigueur, to show how do we work in these ways? Now our screen cycles were only a week at a time. It was very, very fast. And taking people through a very – what was like a very uncomfortable process. And it’s uncomfortable. They’re getting challenged every single day and our job is to help them through and navigate through that process. So, yeah, it creates tension. But, as I said, we have integrated as a result of learning, going through those programs to look at how do we now better serve our customers which are the other government agencies?
Steve: Where – if we were to sort of draw an org chart in the air – maybe it’s not an org chart for the government – where does DTA reside within the structure of the Australian government?
Ruth: Yeah, we have an interesting function. So, we are not a delivery arm. We touch on policy work. But we sit under the Prime Minister and Cabinet kind of function. So, what that means is we are this centralized agency. I think maybe the closest equivalent might be the 18F in the U.S. A little bit of GDS in the U.K. I’m not sure what the Canadian version would be. We have a little bit of that kind of function where it’s not tied into any particular agency because my job is to work across government agencies. But that brings its own challenges around – particularly around the research practices, how we – I’m going to come back to this at some point about how we manage our knowledge across government. A lot of research is happening across many, many agencies. So, we’ve got this uplift function, but as my role as Head of Research I’m also interested in how we manage those other functions within government as well as it relates to research, as a point of research practice?
Steve: Sorry, other functions?
Ruth: Yeah, so at the moment, because I do a lot of – I help teams do a lot of recruitment, but it’s internally within our organization or other government agencies. So, they reach out and go, “Ruth, we need to hire a researcher. We’re not quite sure what to look for. Can you help?” So, it’s about reaching out, how do we help them and craft – how do help them find the right fit of people? Are they doing a discovery piece? Or are they doing just a lot of very heavy product work that’s just iterating just a product day to day to day? And trying to work out what is it that they actually need and how does that work and what kind of researcher would they need? So, helping them find that right fit because we don’t have a pool of people. We’re a very small agency, so we just can’t pop out and just embed into teams. We did that before in the past. It doesn’t scale for us, so now we’re working how we support those functions? Other things we’re also looking at is how do we manage the knowledge that we build with government among our citizens? We have so much research and we spend a lot of money as a government looking at our citizens experience, their interactions with government, but it’s often done in those individual organizations. So, they might have lots of data internally, but what happens when we start needing to look between those touch points, between the different agencies? How does research work in those spaces in between? Because people are having a very agency focused view of their citizens. So, part of our – one of our functions is actually looking at how we provide a more holistic lens across government to show that when a citizen engages in government it’s not just a departmental time, but it’s actually a range of organizations, whether it’s government, private sector, friends, networks – whatever it is. How we show that relationship in the context of people’s lives, so it’s bigger than just one particular service or element of a service?
Steve: Are there aspects of how government is structured? I don’t know whether that’s regulatory or funding? Because every big organization has sort of siloed research and siloed data and doesn’t treat the user experience as coming across all those things. I think that’s common above a certain size, but I’m wondering are there any interesting compounding factors in the way that government works that maybe changes that – how you even try to address those problems.
Ruth: Yeah, it’s a very good point because different government agencies have different rules around data collection and different legislation around how much and what data you collect. There’s often discussion that as user researchers, the kind of data we collect may not necessarily be the same kind of data that the legislation demands that we collect a bunch of other things. So, often there’s tension of going well, we’re here to understand about our citizens. How do we collect that – the people might go the consenters only allowed it to be shared within that particular department. So, it’s like hitting up against issues around consent and do we start looking at broader consent around all of government? Or is it mostly just federal government? Or is it just between portfolios of government? We haven’t nutted out that issue yet. So, it’s a challenge that we’re working on right now because content is actually a big, big problem space for us. Like we want to do research that’s ethical and safe and it’s really challenging when you’ve got multiple agencies who are funded separately. Back in the private sector days I will come, and I will do work for one organization at a time. This feels like we just up the whole scale of the problem because now we’re taking it across multiple agencies who may not necessarily want to share, or be as comfortable sharing, what they have. For a number of reasons. Whether they feel it’s a reputational risk – because we deal a lot with risk as government. There’s always focuses about risk. And I think about what happens if people find out people said that about our service? We don’t want people to know that people say that about our service. So, you get those kinds of complicated…
Steve: Oh, so there’s a risk to the agency.
Ruth: Reputation.
Steve: If the feedback or any kind of user data that’s critical of that agency that becomes public – if you’re a civil servant you don’t want that.
Ruth: There’s some mindsets – I believe they call it reputational risks for government. So, it’s one of the challenges heading up research is how do we deal with that kind of mindset and how to shift some of that thinking as well. So, a lot of framing I’ve been using a lot more recently is about this is about risk reduction and increasing the certainty in the decisions we’re making that’s policy or your product or your features – you know whatever scale we’re talking about, research helps you with that decision making. We’re not something that you just do on the side and just throw it over. That actually becomes part of your decision-making processes and one of your many inputs. Because end of day we understand that people have a lot of pressures and they’re making decisions about why they release some things, whether it’s a policy or a government service. But how do we help them do those things a bit better. So, that framing around risk reduction or reducing certainty – it speaks a language of some people.
Steve: That also sounds like private sector product development. The consequences are different. It’s not just about finances. It’s about other – I’m gathering that you’re in a situation where there’s other consequences besides profit of the organization in the commercial sector. So, you said reputational risk is a very tangible thing that people are kind of operating around.
Ruth: Another challenge that government – we work in a four-year cycle over here in Australia. So, the cycle of government, if it’s incoming, is it a different change of government happening? The agendas may change as well and that would change the focus of your research. When you’re trying to be a little more strategic with research it can change that function, or not. Like we have that additional level of think of. So, although on the one hand we still need to understand our users, our citizens, no matter what. No matter what government is in power we still have – it’s still a part of our function in the role as a researcher to keep that understanding going and to keep developing that deep understanding. But the stuff that it aligns to in the strategic decision making does change and that’s always an interesting factor in working with government. It’s what actually made me – it’s one of the things that made me leave private sector to come here was the complexity of all these kind of nitty gritty things that look so simple on the outside, but it’s actually really complex and really messy. And I love that.
Steve: Okay, what do you love about that? That sounds like something that some people might run away from, but you’re drawn to it.
Ruth: Because it’s hard. It’s really, really hard. I’ve been a consultant for 7/8 – for many years – and I enjoyed that breadth of consulting. But what I found as a consultant, I would go – we’d go in and we’d help people do the – you get the joy of seeing something happen, but I was – I reached a point in my career where I wanted to see that really affect change on a much greater scale and affect it in a much longer kind of timeframe as well. So, I was at a point as a consultant where you do things and it affects a particular project or a program. And actually, when I started looking at how do we affect the whole scale change in government and how do we start shifting that mindset? It was hard to do it as a consultant. So, I thought the only thing I could maybe try doing is to join the public service. So, 2 ½ years ago I left the private sector and I’d come in to try to do this with a bunch of other really smart people across all levels of government, trying to look at how we do this transformation. And it’s hard and I like that because it’s – you’re in for the long haul. You’re in for this massive change. You know it’s not easy and we’re doing this kind of research that’s really – you look at the behavior in organizational and government level change and it’s just complex and it’s fun. Exhausting, but fun.
Steve: So, you’re describing in the consulting world there’s moments or points in the process where you draw satisfaction or some sense of reward from.
Ruth: Yeah.
Steve: And now you’re in this much more complex situation. It’s hard. I’m sort of inferring things take time. What rewards do you identify yourself? Whether it’s milestones or successes, like what keeps you going in a very hard endeavor that you’ve chosen?
Ruth: For me it’s about the empowering of others. One thing you said, it’s about we help people. We like to help people. That’s why we do the role that we do. And for me this is about how do we help at scale? Our government has been doing things for a very long time and there are some very, very smart people in government that often bog down in the structures of government, the bureaucracy, that sometimes doesn’t enable the kind of stuff that these smart people are working on. So, what gives me satisfaction is being able to help some of that transformation. To help others empower them to do this stuff better. And the satisfaction I get is when somebody has that kind of mindset shift and they’re taking this stuff back home. Or, they’re coming back to me later and go, “Ruth, it’s been a year since I’ve seen you, but here’s what I’ve done.” And they themselves have gone back and been this change agent for not just research, but for just human centered – good, human centered thinking. And it’s great when you see that kind of stuff happen. So, helping others to do this stuff better.
Steve: You said the long haul and that makes me think as someone who is a consultant, you’re done. You’re not there to get that feedback or to see the consequences. Or you have to kind of find your way back in to say hey what happened a year later? So, if these are longer term working relationships that you have, you’re able to hear from those people and see kind of what’s happening.
Ruth: Yeah.
Steve: Maybe we can talk about the Head of User Research role. When did that come about? How did it come about?
Ruth: So, Head of Research, I’ve been doing that for about over a year, maybe two years now. We had a previous head of – it was actually design research – Lisa Reichheldt, who you would know. When she left they actually split that role up to two. Her role was an executive level role, but they’re not splitting up the role and taking it down a level. So, it was an interesting organizational change I think – into a Head of Service Design and Head of Research. We also have Head of Content Strategy and Head of Interaction Design who are my equivalents. So, my role involves helping my agency, and across government, to really better the practice of research. I find I’m doing a lot of it – I don’t do as much research anymore nowadays. My role is really more of an enabling function. So, part of that is looking at how do we bring in the right people into the teams. When they’re here, how do we help mentor them? So, part of my role is I catch up with a lot of the researchers and check in how they’re going. I’m connecting them to other researchers as well in our communities. And also trying to look at how we put together much more – how we lift the conversation around research. So, part of my role is actually about that strategic aspect of research. How do we do it better? How do we help enable the broad decision making of government? We’re also starting to see a shift to much more policy sides rather than just product and services – up into policy land where a policy can trigger many, many services and products. So, that’s really fascinating as well. And I’m really enjoying that kind of aspect as well. The other parts of the role is really – we look after a guild. So, I don’t know if – I’ve heard other organizations have this structure too. So, our researchers who come into our organization are embedded into product teams, but we have a guild that then unites all of those researchers and those interested in research to come together once a fortnight and actually share the knowledge. So, it could be – for example, I’ll be going back and doing a sharing of what I might learn at a conference. Somebody might share about something happening in their project. A problem that they’re having. How do we solve it together? So, the guild is almost like a tribe that cuts across our organization no matter what team you’re from and it unites us as a practice. And that’s how I work through that kind of process to enable the better uplift of our practice.
And externally, my job is just to advocate for the research role. So, I often get called in to talk to other executives about what does this research mean? “Don’t we do this stuff?” “Haven’t we been doing this stuff for years?” Answering those kind of questions. Look at how it helps them in their strategic perspective. How it helps in their decision making? And also, just help their staff connect better and just do the craft better.
Steve: You said before how one of the things you’re doing is helping agencies hire into that agency as opposed to into DTA – hire researchers directly into those agencies.
Ruth: Yeah.
Steve: Are those – and then you’re talking now about sort of maintaining connections with researchers through sort of formal and informal methods. I’m just trying to understand, is that within DTA only? Is there engagement with these other agency researchers, I guess I’d call them?
Ruth: Yeah. So, the supporting through the guild is a DTA only thing that we do. Or anyone that comes into DTA on a short-term project, we connect them up into the guild, so we can support them that way. But for the broader government we actually got a Google mailing list. It’s called a service design user research mailing list. It covers design and research and we support people through that. But it’s not just us – we’re a facilitator, so our job is to connect others to each other because we’ve got amazing people out in government and if they connect – and peer – we learn from each other. That’s a really good thing. So, we’re not the be all to end all. We’re that connecting function. I think some of these we’re brokering. You know we help broker relationships as well. Otherwise, it can come across as oh we have to go through DTA, or do we have to be involved in all this stuff? No, because it’s about enabling others, empowering others to connect and keep growing together.
Steve: So, there might be a researcher in one agency that you know and another researcher in another agency that you also know, but they don’t know each other.
Ruth: Yeah.
Steve: And if there’s something going on you might say oh this is the researcher you should talk to.
Ruth: Yeah. So if somebody – say they’re trying out a particular technique for the first time and I go so and so from here has actually done it before, have a chat to them about how they do it because they would have some good learnings in the government context of what worked and what didn’t work and it will just help speed you up when you try and do this yourself.
Steve: So, in DTA itself, you listed a bunch of different functions that you have kind of peers leading. I’m sort of wondering how the team sizes vary? I don’t know if team is the right word, but sort of the number of people maybe in each of those functions?
Ruth: At the moment I think we’ve got around 13 researchers, but the number varies significantly across DTA depending on the projects that we’re supporting at the time. It also doesn’t include the projects that we support that belong to other agencies. So, that number scales. At one point I was working on project that had 18 researchers and that was just within the one project. So, that was a good question there about scaling and processes and ethics and all of that. But the number changes and what’s interesting is that the recruitment is done by the product owner and we provide a supporting function to help them get the right fit. But it’s done by the products where the person comes – they join the product team. So, they don’t belong to my team of researchers. It’s more of a virtual kind of team if that makes sense. It’s where the guild/tribe comes in. And those product teams, they usually range between the kind of 6/10 kind of sizing and there’s usually a dedicated research function in there. Not all teams can have one because as a government we often have budget constraints. So, we have people who do research who are not necessarily researchers. But we try to encourage that specialism, whether it’s a researcher or a service designer or an interaction designer because they bring a depth and they’re upskilling people on their team as they’re doing it too.
Steve: So, someone could be hired into DTA as a researcher, but the thing they’re spending their time on is a project team?
Ruth: Yes. Yeah, that’s right. Although I try to steal some time to work on our community, our practice matter things. One of the things at the moment is we’re reiterating through our processes and policies at the moment. So, through guilds – we use the guild as a bit of a working spot as well. We come together one hour a fortnight and we use it to work out what we want to tackle next and how to improve this particular process. So, I try to steal time that way from those teams to enable our better practices.
Steve: For Australian government employees, how geographically dispersed are they?
Ruth: So DTA, because we’re so small, we’re based mostly in Canberra which is the capital of Australia, and very small. I think we’re only about 400,000 people that lives in Canberra. Our other office is in Sydney which is even smaller. What makes it interesting, when you have that kind of diversity in spaces, is that other agencies are even bigger than us and they have many more people scattered around Australia. But the challenge I think that comes to research is recruiting. Finding strong talent in Canberra can be challenging as well and often people – I love Canberra, but lots of people – there’s always jokes about Canberra. Honestly, it’s too cold. It can snow in winter. It’s cold. It’s cold in Canberra and it’s hot in the summer. It’s small. We’re inland. People have the stereotypes with it being a government town. So, there are all these stereotypes around Canberra being a boring place. It’s actually not. It’s actually a beautiful place and I love it. But recruitment becomes really challenging. How do you encourage people to leave their beautiful cities of Melbourne? Leave Sydney and come spend some time. So, part of it is trying to shift the conversation around how do we work on more meaningful work instead and shifting the conversation around that. So, I think I always joked to Leisa before about do we need to have a recruitment video to show people, hey, it’s great, come work in Canberra. But it does change the way then on recruitment and how we uplift capability within existing skillsets as well.
Steve: Can we go back to something you were saying about policy? I want to understand maybe the mechanics of that more. I understand that the policy is set by parts of the government and then, as you said, that impacts the products that are being created. But you were describing maybe a different cycle, I think, or a different stroke in that cycle where the work that you’re doing can – is it correct to say that it’s impacting policy?
Ruth: Yeah.
Steve: What’s the dynamic there, I guess, is really my question?
Ruth: So, there’s been a shift – more recent shift in government. And I think you’ll see that globally – in global governments as well about how we shift this more human centered thinking and lenses over to policy lens. Traditionally in policy I do a lot of research. When you create a policy, they go out and do big scale studies. There are certain methodologies for policy research. But part of what we’re trying to help is how do we get the policy folks who are creating this policy actually out in the field, actually talking to the people who are impacted by this policy, and actually going out into the communities that will be impacted down the track by this, and actually understand that context. So, some people do do that already, but it’s not necessarily the norm. So, we’re trying to look at how do we shift some of that thinking and that’s part of my kind of focus at the moment is how we start working towards that lens and it’s really interesting. Because you know people have been doing policy creation for years. It’s not a new thing. A research lens is not a new thing, but the way we do our kind of research is actually new for them. And trying to help people, going, actually this stuff that we do is not just a digital thing. There’s a bit of a perception that oh, this kind of research is to create digital products, right? And you go, actually no, we can actually apply it across a range of things. Because it’s really about understanding people, understanding what is it like to – whether it’s government or whatever it is – what does a person’s day in the life sort of look like in this space. Especially when it’s government, we’re serving such a diverse range of people from our indigenous communities. Those are very disadvantaged. To those that might be on the other end of the scale as well. All the way to large scale businesses. It’s such an amazing, diverse set of users. So, how do we kind of work together to shift that thinking? It’s a good challenge and one that I think will take time as well.
Steve: It seems very analogous to working with those agencies – your phrase of uplifting their skills – those product people are starting in a different place than your policy people are, but it sounds like it’s the same thing. We want to help you to understand the benefit of going out to the communities and seeing what’s going on.
Ruth: Yeah. As I mentioned earlier – I remember starting government 17 years ago working on range of systems and our work involved doing usability testing. You see people using your thing. So, research in that aspect is not new, but I think the more general style in exploratory style research is new, or newer for government now because it’s been happening for the last few years. So, seeing that shift has been really interesting, to say how do we help you do more of that style of research as well. And they’re all very valuable and the stuff you were doing before, usability testing was great, and it’s got a purpose, but how do we shift that conversation to talk about are we actually solving the right problems. I think that comes under the guise of maybe a lot on the innovation labs and design thinking. At the end of the day for me it’s about the fundamentals of who are these people we’re designing for and what problems are we trying to solve, and how do we understand the needs deeply? How we get there – there will be a range in methods – how to help people work out what that combination of methods are.
Steve: When we talk about research methods that’s often about how do we get this data from these users we’re trying to understand but your emphasis seems very strongly on how do we engage the different people that work in government to learn that and do that and use it? It sounds like – the word methods to me would encompass what you’re doing there as well.
Ruth: Methods, approaches. Yeah, it is. I mean it’s one of the things – words have meaning, and words are powerful. Within government, when we talk about research, you’re talking about methods. People go what are we talking about in there? Are we talking about market message? We do a lot of that. You know is it just a design thinking thing which is also a big buzzword. Or is it this agile thing. Being clear about what’s the outcome we’re trying to achieve as government. I use the word methods, but really, it’s about that focus of we know we have an outcome, how do we best meet that and make sure we’re doing something that’s good for our users that doesn’t disadvantage them, but hopefully improves their lives as well.
Steve: It would be great to talk a little about what your trajectory was through your education and your professional work that kind of led you on the path. Talk a little bit about the consulting work before government – maybe just go back a little further and talk about where did you start to see this path that you’ve been on?
Ruth: I actually totally by accident fell into it. I started off as a business analyst designing mainframe screens. That was over 17 years ago, my first kind of proper design kind of job. I had no idea what mainframes were at that time. What is this thing? And it was just by the luck of the draw, I happened to be sitting next to a team that was the very first user centered design team in Australian government. So, I would hear them. They go out and they do these things. They come back, and they would share these stories about the people that they were meeting and the kind of feedback they had about the products they were testing. I said what are they doing? This stuff is really interesting. So, being a curious person, I stuck my head up over the fence and asked them what is it that you’re doing? Tell me about this user centered design stuff. I never covered that in my university degree. We covered all about systems gathering and requirements gathering, all these different methodologies that you do to do that – never once heard about this human centered stuff before. So, by accident I heard that, and I just asked can I join the team? I thought what’s the worst they can say? No. And then I’ll just keep doing what I’m doing and keep trying to find a way. So, I asked and I joined that team. And so, I had my grounding by learning on the job. So, starting government 17 years ago, just learning about what this user centered design stuff was. Really loved it so much and then went back to uni to try to get a masters in human factors, back in the day.
Steve: What was your first degree in?
Ruth: My first degree was information technology and systems. So, very, very IT driven. So, I come from that kind of analytic lens. But doing more of this stuff I just realized I love research. But it was a gradual thing because I didn’t know that such a job existed. And it could be a reflection of Australia at the time because 17 years ago the whole user centered design movement was happening, or starting to happen. Companies were getting set up to do this kind of space. Government agencies were thinking about this stuff and doing usability testing. That’s what my UCD was at the time. So, that was my first foray to this field. From there, and I think like many other people in Australia, we were generalists. We came from UCD to a UX generalist. We did everything, including research and prototyping, everything. But I found soon enough – I quickly realized that research was my passion. I just wanted to do more of this stuff. I think I just got really interested in what people do and say and think and why do they do certain things. And that was fascinating. So, over the next 17 years I ended up leaving government and then going out into consulting. I was head huntered to come and do this work and it was great. So, I did that and was working in the private sector for quite a while and had this broad range of experiences. And then from there, that’s when I decided okay, I’ve been doing this stuff for a while, I’m enjoying it. I’m enjoying being a principal and helping others be better at the consulting craft. But that’s when I then came back to how do I do this longer term, impactful thing around government because that’s where I live and I’m really passionate to see how they improve government. There’s just so much interesting work happening in that space. So, that’s when I made the leap back. So, I started off at DTA as one of the lead senior researchers that they had and then when Lisa left I took over and became Head of Research.
Steve: When you think about the time that you were consulting, as someone who – I’m sorry. I’m going to go back to something else. You started talking about going back to uni for – was it human factors?
Ruth: Human factors – yeah back in the day there was human factors. But we didn’t have all the amazing design degrees we have now. Back then it was – that was the closest thing I could find because I wanted to improve – I just wanted to broaden my skillset in this kind of human centered design space and the only thing we had at the time as human factors. So, I was doing that remotely. I was also one of the first few students who was doing remote uni work. They’d never had that before. It was challenging. And I got to work and learned with a bunch of people who were designing airplanes and aircrafts and hospital systems. And when I’d introduce myself it was like I work for government and I design systems, it was very jarring for me to go wow these people are doing stuff that – massive design things, so if you get it wrong it falls out of the sky or people die. And I was just designing systems. But it was such a good grounding for me because human factors really is about that processing, that cognition. It really drew upon the school of psychology and the sciences as well, from this particular course that I was doing.
Steve: Was there ever a point at which you started to identify yourself as a researcher specifically?
Ruth: That’s a good question. That’s hard because in Australia people have mostly been generalists. That’s how it just evolves. And you had to be in the consulting world to survive because if you – I would just get really jealous – I would see the kind of work you’re doing and go you can be a specialist, but in Australia we just weren’t quite there as a country. We didn’t have the maturity to say I am a researcher. So, I called myself just a UX person for a long time, but research is my passion and I always seeked projects doing that kind of work that’s research and help enable that in our clients and in our practice. I think it just evolved. I don’t know actually when. I just started calling myself that kind of thing. It’s just something that I did.
Steve: So, your first job that you came into DTA with – you had research in your title. Not to conflate the way we identify ourselves with what our job title is, but that’s a point where you went from generalist to specialist. Is that right?
Ruth: If you’re talking about titles, probably yes, because previously my title was principal consultant, UX consultant. And part of my job was – although most of my work I was in was research work – that’s just what I like, so I will do that kind of work. But that wouldn’t naturally go down well with the clients. They just weren’t ready for that in Australia. They were more familiar with UX. We need to improve our experience and however you get there, we want to help them get that outcome. So, although probably 90% of my work was research, the other part of it was actually supporting – oh, probably a little bit less – I was actually supporting the practices as well. As principal, my job was to make sure the quality of all the work coming through was also good. I was doing a lot of that kind of stuff in that kind of leadership role. It was the other half of my job before I left consulting. So, I think the detail – just use the title, to the exclusion of everything else.
Steve: One of the points you were making earlier was about the challenge of recruiting – I guess in Canberra specifically and maybe government is just a part of that. But what was – how did you get recruited to rejoin government?
Ruth: I was actually working at the former DTA, or DTO, as a consultant doing research work for them. Because it was the first time they were going through these ways of working and we were doing discovery, alpha, beta, live. And they needed people who had experience doing research, who could help lead that process and to help facilitate that process. And they were also recruiting for other specialists. So, I was one of the many specialists that was brought on because they had to scale really quickly and trying to build that capability within weeks was not going to happen. So, they had to find it externally. So, I was one of those many externals that were brought in. When I came in I saw these ways of working and going wow, I didn’t know we could do that in government. It’s a bit of a stereotype. People think about government as a very, very slow moving beast, that you can’t do a lot of things. But in reality, it’s about how do we frame it and how do we talk the language of the people who are decision makers to help bring about this change. So, for me seeing that, I was like I want to be part of this kind of stuff. It was a bit of like start up, doing all these ways. It’s like I found a home. So, seeing that was how I then made a leap. So, DTA – or DTO at that time – they kept asking can you come on board and if you like doing this stuff, if you like government, come join. So, I took a while to think about it, quite a while, and then made the leap. Because it is a big leap to leave what is a very interesting job as a consultant and you’re helping so many different clients and so many different problem spaces. You know one day you could be working for an airline, the next one for a bank and the next one for a not for profit. You know that’s really, really interesting. I like that. But there was this need and I wanted to be part of that movement as well, to transform the way we do things as government.
Steve: You’ve made this point a couple of times that even though the D in DTA is digital, the output, the outcomes don’t have to be digital. Do you have any examples where that’s kind of how it’s gone?
Ruth: Yeah. So, there was this great project a few years ago, done within DTA. It’s Medicare – Medicare is our kind of health system where if you’re a citizen you get access to major health services. It’s covered by the government. So, one of the services I was looking at was having a baby. So, in Australia you have a baby. You have to register the baby for a Medicare card. There’s a whole range of process around the registration. And as someone who has gone through it myself in the last two years, it’s actually quite a tricky timeframe. If you’re exhausted from delivering this amazing little thing, you’ve got a million things going through your mind and then you’ve got all these forms you have to fill in. And sometimes that involves, because of identity things, you have to go into offices with a newborn. It’s just hard – it’s hard. And I was particularly emphasizing now that I have a little one myself. So, this particular project is looking at how do we make that process easier. Now what was interesting was that the team had gone – and I think they were thinking, maybe we just digitize the form. Maybe that’s what we mean when we talk sort of digital transformation. That might be the easiest way we can deliver something quickly. But what they soon found out, and I had a really awesome service designer called Mira who joined the team. She worked with them and she actually suggested they bring in a lawyer, because they kept hitting up against issues like oh, well of course we’ll just digitize the form. What else will we do? But when they went out doing the research they were finding out that a lot of the stuff we were asking for as government, we really have and the hospitals have. Because when you give birth you’re giving a bunch of details over. So, they actually went out and did a bunch of research out in hospitals and what they ended up doing was actually deciding, you know what, we don’t actually need something. We don’t actually need a form. Or we don’t actually need a digital thing, like an app or a website to fill in. What happens if we even just remove that process and just assume that if we’re having a baby and we get consent, we can just issue the card. So, they were exploring this what might we kind of situation. So, it was radical, and people were going, “no you can’t do that.” But when the lawyer joined the team, there were lawyers going actually you know what, we can actually make this work. So, the trial was actually in one of the hospitals up in the Gold Coast. It was actually looking at what happens if you sequenced in from the parents at the time of the birth. They’re filling a bunch of information to the hospital about the birth of the baby, what happens if we use that and issue the Medicare card. So, it’s something that had gone from taking weeks and weeks because you have to go in and see multiple people, get the thing signed. And gone down from that to only a few days. It was amazing because you’ve now gone from filling in something to not even having a thing and stuff just appeared in your letter box. I think that’s a very great example of transformation happening where you don’t have to have a particular, physical digital thing. And it was good to see what they did with it.
Steve: Are there things outside the professional realm, like outside your work at DTA, where your skills, the things that you have a passion for and the things that you are just so great at, is there anything that you do outside – were those things manifest in other forms?
Ruth: So, Canberra is a very small place and I love community. So, things I would do are not quite – they’re not research related, but it comes under people. I’m involved a lot in organizing community events around things like TEDxCanberra. So, I’ve been involved in that for about 5 years about bringing and facilitating the amount of smart people that we have and for others to learn from that. I really enjoy that aspect and that comes through like a job as well. I really love connecting people and seeing them grow because my intent is if you can grow and be a million times better than what I can ever be, that is an awesome outcome. You know always hire people that are smarter than you. You want to encourage that growth. So, for me that stuff that manifests outside of my job has to do with things like that. Things like BarCamp back in the days. Something called GovHack, very early in the days – so we look at open government data and how do we help people flesh that out into interesting things. All those little stuff like that is what – again, we’re connecting community and people.
And something else I do that’s not even related is I love making. My hubby and I, we’re both makers. We’re like crafting things. So, I make science-themed jewelry for fun on the side, using our lasers and 3D printers. For some reason, that process of designing something that’s very, very tangible and you’re prototyping that thing and you’re wearing it and you’re seeing people use it as well. It’s very interesting. So, although it’s not necessarily a research thing, it’s a fun thing because you are trying to use a creative side of your brain in a very different way. In a much more tangible way.
Steve: So, is that the connection to the creative?
Ruth: Yeah. And again, I’m just doing research on a very small scale, again to wear my stuff and see how it works. It’s just very, very small-scale stuff. But you learn a lot about people’s behaviors, about when they say things like oh Ruth, can you make that in these colors because I think people will buy that. And what people say they want and what they actually end up doing. Like we talk about that all the time, but to see it in action with your own product is very interesting because you’ll make stuff because people ask for it. So, let’s just try it and see what happens. There’s not a need for it. You just lazily see it sit there. But just little things like that where it’s really tiny stuff that we talk about in our practice that manifests in this way. It’s interesting.
Steve: Can you describe one of your pieces?
Ruth: Yeah, so one of the things that we do is we look at data. So, my hubby actually takes the data from looking at the weather data for the cities in Australia. So, the Bureau of Meteorology here has about 100+ years of weather data. So, we take that and actually visualize it into a wearable piece. Unfortunately, I’m not wearing one today. I’m wearing a serotonin molecular today which is happiness. We take that and then actually model it into a bit of a big jewelry statement piece and then laser cut it. And that’s meant to show, well just data visualization. How do we make science interesting and communicate these things in different ways that’s not just a PowerPoint or a presentation or a survey, but actually a very tangible way of understanding data. So, I’m interested in also that crossover between when we talk about research, the data and then the data sciences folks. I just love geeking out some of the data stuff and going hey, we could find other just weird wonderful ways to display this. So, it may not have any actual use to researchers, but it’s just geeky and fun.
Steve: What do you think people who purchase your jewelry – because so much of our work is the people that make something have a model of it and other people that use it have a model, but sometimes our job is to just try to articulate that delta. So, you’re describing kind of what you’re putting into these pieces and even just the geeky pleasure. But for someone that buys one and wears something that you’ve made, what’s their interpretation or their narrative of what it is, or what it means?
Ruth: Yeah, it’s interesting. They’ll come up and they’ll go – they identify as a fellow science geek. Especially seeing that, they’re going “is that a serotonin molecule?” Like, it is. And then we have a conversation about science and about what makes them interested in science. I use it as kind of a talking point to work out how did they get into this field? Or if they’re not in the science field, how do they actually trigger – it’s a good way to practice research questions actually. But it’s just fascinating seeing that kind of shared – I think they’re having that shared platform or love of a particular field, in this case science. And that’s what connects people and they feel like oh, they’re part of this group of people who love these kind of geeky things.
Steve: There’s something, I don’t know, secret about it, that someone who gets it is going to get it when they see it and somebody else is just going to not recognize that.
Ruth: It’s funny. Not a secret club, but it is that recognition that somebody else enjoys this thing. Whether people have that in music, or whatever their particular passion is, when you connect on that kind of level it’s just so interesting. And then you start hearing their stories about how they either buy it for their daughter because their daughter had started doing this thing out in Antarctica – wow, just the stories you hear. I think that’s where it comes to my research – I just love hearing stories as well and you hear all these fascinating things that people tell you when you’re buying jewelry because it’s just a conversation starter.
Steve: Yeah, and you’re connecting on something. You’re right. It’s not secret. Secret is about being hidden. This is just about that connection, or something about the recognition, that’s a better of putting it, to get those stories. That’s really wonderful. Is there anything else that we should cover in this conversation?
Ruth: I think we had a good range of discussion.
Steve: I think so too. Well, thank you so much. It’s been really great to chat with you.
Ruth: Thank you, Steve.
Steve: Thank you for sharing so much great stuff.
Ruth: That was really fun. Thank you very much for your help, for your time.
Steve: All right. That’s the wrap on another episode! subscribe to Dollars to Donuts wherever you get podcasts. If you’re using Apple Podcasts, how’s about giving the podcast a rating (five stars?) and even a short review. This helps other people find out about the podcast. Portigal dot com slash podcast has transcripts, show notes, and all of the episodes. Follow the podcast on Twitter, and buy my books Interviewing Users and Doorbells Danger and Dead Batteries from Rosenfeld Media or Amazon. Our theme music is by Bruce Todd.
The post 21. Ruth Ellison of Digital Transformation Agency first appeared on Portigal Consulting.
This episode of Dollars to Donuts is part 2 of my conversation with Leisa Reichelt of Atlassian. If you haven’t listened to part 1 yet, you can find it here. We talk about corporate versus government work, scaling research, and changing organizational DNA.
I love research, I love the way that we learn things and what that means, but the thing that really drives me is seeing an organization almost like a design problem and thinking about like what do we – what levers can we pull? What do we choose to do? How do we position ourselves so that we cannot just do fun research, but we can actually really have this knowledge and this insight and this practice fundamentally change how this organization operates? – Leisa Reichelt
Show Links* ~~Fundamentals of Interviewing Users (SF)~~ * Part 1 with Leisa * Leisa on LinkedIn * Leisa on Twitter * Atlassian * Francis Maude * Martha Lane Fox * Tom Loosemore * GDS (Government Digital Service) * A Country Practice * DTA (Digital Transformation Agency) * Flow Interactive * NPS (Net Promoter Score) * Kate Towsey
Follow Dollars to Donuts on Twitter and help other people discover the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Hey, and here we are with another episode of Dollars to Donuts, the podcast where I talk to the people who are leading user research in their organization. This is part two of my interview with Leisa Reichelt. If you’re just joining the podcast with this episode, I’d encourage you to go back to the previous one for the first part of our conversation.
As a reminder, my public workshop Fundamentals of Interviewing Users is happening September 13th in San Francisco. There’ll be a link for more information in the show notes. It’d be great if you recommended to this a friend or colleague. I also teach classes directly to in-house teams so reach out to learn more. Beyond teaching, in my consulting practice I also lead user research studies, so let’s talk if that’s a way that I can help your team.
Of course, supporting my own consulting work is the best way to support Dollars to Donuts. Share your feedback about the podcast by email at DONUTS AT PORTIGAL DOT COM or on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
You know, as user researchers, we love our sticky notes. A few years ago, in an article about how IBM was being transformed by design, a key achievement in this transformation was the ability for staff to order post-it notes. In some ways, that was the saddest thing that I ever heard, that IBM was so broken that ordering a quotidian office supply item was verboten, and that enabling this was seen as a victory worthy of mention. But it also was very real and acknowledged how much of uphill battle these kinds of corporate transformation efforts really are.
So this carrier of innovative meaning both in its legendary origins and in its rapaciously consuming audience, this product of the 3M company clearly struggles itself with innovation. We’ve had the weird dispensers, the odd sizes and shapes, the reverse fan fold which may be tied to the dispenser but I’ve mostly just found them showing up in the most aggravating moments in a session. I think many years ago they came out with Super Sticky notes which seems like a complete contradiction of the value proposition but I think it’s just an acknowledgement that the original formulation fell off more than we’d like. I don’t think of them as Super Sticky, just the proper amount of sticky for most uses. Anyway, the latest thing I came across really made me scratch my head. It’s a pack of bright orange Post-Its, probably part of a series of exciting new notes with the aspirational branding A WORLD OF COLOR. This particular pack had an even more aspiration and even less relevant tagline “Rio De Janeiro” Collection. I can imagine interior paint, fabric even car finishes being marketed this way but it’s so strange to see on a package of sticky notes. The post-it, for the researcher, the designer is a backdrop, a carrier for something else. It already is aspirational, because of what we’re putting on it. Making it a “collection,” associating with a far-off fabulous city, is just ridiculous.
Okay, here’s part two of my interview with Leisa Rechelt, the Head of Research and Insights at Atlassian in Sydney Australia. It was quite an in-depth interview that’s been broken up into two parts. This is part two, you can check out the previous episode for part 1. Let’s get to it!
I’d love to hear about some of the other kinds of organizations you’ve worked in and what those have been like.
Leisa Reichelt: Sure. Well, for probably 5 or 6 years before I joined Atlassian I worked in government. First of all, in the UK and then I moved back to Australia and worked in the Australian government for about months. Something like that, I think. And that was super different, super different to Atlassian. So, it was a much more kind of familiar ground for me in that it was organizations that you’d go into going we should really go and like involve people in our design process and they would go why would you possibly want to do that? So, that’s a whole different problem set than what we’re dealing with, I think, in some of the tech companies. But hugely rewarded as well. So, yeah, really very different.
Steve: Both those governments seem like their commitment – I guess they’re just different cases, but their respective commitments to sort of design – I don’t know, digital services seems to be the term that gets thrown around for that. But it seems like there’s been significant commitments in both those cases. I mean you’re coming into environments where someone has said we want to do this, we want to change that default. And – like in the UK what was sort of the – how did that get initiated?
Leisa: Okay. So, in the UK we had a MP, Francis Maude, late in his career. I don’t exactly know how it came to him that we could probably be doing better with our digital services than we are. I don’t fully have that back story, but it did come to his attention that we could do better. And he recruited a lovely woman by the name of Martha Lane Fox – Dame Martha Lane Fox now – to basically help him come up with an approach to how we should solve the problem of the UK digital government services not being what they need to be. Martha worked with a bunch of very smart people. In particular, a guy called Tom Loosemore, to come up with some recommendations. And it was off the back of those recommendations that the government digital service was put together. Tom and a bunch of other people that he’d worked with in various places around London and the UK came together and started working on trying to transform how government thought about approaching digital services. And they had – their design principles, I think, were really the best way of setting out what their beliefs were. And fortunately for me, and number one of those principals, was around putting user needs and not government needs first. And so yeah, culturally there was a cohort in there who were well supported within the political system and were able to really kind of make great shifts and changes and progress on that front as a result.
Steve: It seems to me, just from like watching Twitter or LinkedIn, or just who I keep coming across, that there are research people, titled researchers in every kind of nook and cranny of government services – digital and otherwise, I think in the UK. It seems like from the time that you got involved it’s built into something – it seems like it’s sort of changed the way that government is delivering services. I don’t know if that’s an accurate assessment.
Leisa: Yeah. So, it’s when I started at GDS I kind of came off the back of like – there was a lot of talk about being user centered, but I couldn’t really see exactly where the users were in the process. So, I was publicly a little bit critical of them at one point kind of saying well you know I see that you’re thinking about users a lot and you’re looking at the data that they leave behind a lot, but are you actually – actually involving getting a good understanding of them in the process of designing and transforming these services? And so, I was given the opportunity to come in and put my money where my mouth was, such as it was. And at that point there was the odd kind of researcher here and there. There were like 3 or 4, I think, at GDS at the time, and they were stretched across about 25 different projects. I remember sitting down at their team meetings and the team meetings were basically sitting in front of a spreadsheet, looking at all of the projects that they were supposed to be covering, dividing their days into quarter days and working out how on earth they were going to try to help to support these teams, not all of which were in London. So, a lot of them kind of theoretically required either quite a bit of traveling or just dealing with on the telephone. And that was it. So, they were there, but they were really not well set up to be effective. And I was not really – you know I’d worked with government as a consultant in the past and was pretty skeptical about whether or not that would be a brilliant place for me to work long term. And so, when it came time for me to say what I thought we should do I wasn’t really that worried if I lost my job. So, I was able to say what I thought we should do which is I thought we should have one researcher per team, which at that point in time was like just an outrageous thing to be saying. It was absurd. And we didn’t quite get one researcher per team, but we did get quite a big chunk of researchers and so – and the other thing that I did that I kind of look back on and think that was a really important thing to do was I put one researcher on as many teams as I had researchers and I just left the rest, pretty much unsupported. And the thinking behind that was if I could just give a bunch of people an opportunity to show what good looks like then we could create demand for that good. And if we stretched everybody across multiple things we’d never be able to create that showcase, that exemplar that let people see what they could have. And I think that kind of made all the difference. In the early stages again, speaking about getting proactive rather than reactive. One of the big challenges that we had was like how do we get people even to plan for this? Like how do we get researchers, and budget for what researchers need to do, in the projects ahead of time because, you know, people would come to us saying, “we’re starting discovery next week, I’ve heard you can help us with that.” And by then, like you know, the budgets had been approved like 3 or 4 months before. You had to like kick off an engineer to get a researcher which is always a super popular thing to do – not. And then there were things that we do where we just sort of made rules about you have to have one researcher per team, at least 3 days a week. And it was 3 days because then you couldn’t split them across two projects equally. You had to do research at least every sprint, which was 2 weeks for us. And you had to involve at least this many people. That basically gave us the formula for working out how much research would cost in a project which let us get like that number into the project budget really early on. And these are like really totally boring things that have got really nothing to do with research at all, but were hugely enabling. By the time I was leaving GSA, instead of having the problem of going you should have a researcher in there, you should have a researcher in there, I would have people coming to me, initiating projects, going I’m starting a project, I’m going to need at least 3 or 4 researchers and you go no, no, no, that’s too many, just start with one and kind of go from there. Yeah, it was something that really – I think because people were able to see the difference that it made – and researchers were really facilitators for the rest of the team. So, their job, yes was to run the research and set it up and facilitate it and do the analysis and that kind of thing, but they created these opportunities for the entire team to be able to see how what they were working on was solving problems. Or, how what they were working on worked really well, or didn’t work so well. I think that was like the most powerful thing that they did was really to be very open and inclusive in the way that the whole team was invited and expected to come along and see for themselves, first hand, what was happening. Our research labs that we had onsite we’re always full. Like our observation room, we just had to keep making it bigger and bigger all the time because so many people would come in and watch these sessions. And I think it was that – a) it’s super helpful, but b) I think it’s really – it’s great feedback that the team gets. It provides – it shows the meaningfulness of the work that they’re doing and I think that’s a really desirable thing for teams to be able to see what I’m doing makes a difference. It has an impact. It matters. And I like to think that that contributes to team health as well.
Steve: What’s the difference in the culture of government vs. software technology companies where that kind of response, that kind of commitment and engagement can happen?
Leisa: Well, I was going to say governments don’t have growth teams. They do have behavioral scientists though – behavioral economists – which are kind of a similarish type thing, but they tend to still experiment on letters more than they do on digital things. But I think that it’s a few things. I think that people in government feel – it’s such a generalization. I don’t even know if I believe it 100%. Certainly, in places that I’ve worked people in government have felt their responsibility to the people who are the service receivers and the idea of experimenting on them is something that you would approach much more carefully then what we see, I think, in a lot of other sort of software/technical organizations. I think that’s one part of it. I think the other thing is very few people that I met in government thought that they could do my job. They didn’t know that my job was a thing and they didn’t think that they could do it probably as well as I could. Whereas I find in tech companies there’s a lot more of that. Like everyone’s got some kind of research background. Everyone feels a little more confident and capable at doing this stuff themselves. And I know that that’s not universally true. I know there are a lot of companies where everyone is terrified of going out and talking to customers. But I do feel as though in tech companies – some tech companies – there is a lot more kind of over-confidence about your ability to go out and do this, especially this – and I don’t want to sound like I’m a complete downer on the lean sort of thing because done well it’s great, but I think that has really encouraged people to think that there’s not too much too this, that anyone can go and have a chat with a customer and that’s what we do. I think there are probably a few other things that are really different.
Steve: That’s a good comparison. Can we go back in time a little bit further maybe. I’d love to hear about how you found research as a thing? How is it that you ended up in this field?
Leisa: Well, I was one of those people who growing up didn’t know what I wanted to be. I briefly wanted to be a vet, but that was mostly because of a popular television show that was on in Australia at the time.
Steve: Which one?
Leisa: It was called A Country Practice and it had a character called Vicky who was the vet and I thought she was the bees’ knees. But then kind of beyond that – that was like when I was in primary school and then beyond that I honestly really don’t know what I want to do when I grow up. After high school I went and did a university degree in communications and I could have gone and done a bunch of other different degrees. I kind of applied for like this variety of all different kinds of things in different cities. Like it was a toss-up between doing music in Melbourne or communications in Sydney. In the end I chose communications in Sydney because they had 9 hours face to face time and it was pass/fail and I thought that sounded pretty cruisey enough, after being a bit of a study nerd in high school. I thought maybe I’d earned a break. It was not my best thought out decision and I didn’t really know what I was doing there. And then I was working at the same time. I really didn’t know what I wanted to do at all. I met the Internet at uni for the first time and then in one of my kind of weird jobs that I was doing, I was working for a legal services company and the guy who had started that company, one of his kind of big market advantages was this software that he had had designed basically, which ran all of the kind of logistics of his company. And the guy – you know as software does sometimes, it just stops working or it needs changing. And the guy who was in charge of the software was a supplier and when he couldn’t be bothered coming on site to fix stuff he would ask me to do it. So, I got to understand a little bit about kind of how software happened. And then we started thinking about doing internet stuff and it morphed into a really early E-commece offering and I was asked to be the project manager of that. And I thought to myself, this internet this – this is it, right? This is why I didn’t know what I wanted to do when I grew up because the Internet didn’t exist when I was growing up. So, I thought that was really exciting and then I went and worked in some digital agencies. I had a job called a producer and a producer basically did everything except for writing most of the code and the visual design. So, you did the project management, the account management, the interaction design, the information architecture, all of that sort of stuff. A lot of the sort of project strategy stuff as well. This was in the early days when agencies were still kind of really small. And then over time I just kind of got rid of the stuff that was less interesting to me to focus in on the stuff that was very interesting to me. At the time it was information architecture and the research that goes behind that. And I kept doing both of those things for quite a while. At one point I was working on this huge project in Australia and trying to get budget to do some research. And it was a tiny, tiny budget that I wanted and I had to fight so hard for it. And it was a project that was being run by an ad agency and I’m pretty sure they spent on lunch every week what I was asking for, for research for the entire project. I got very frustrated. And that was basically what kind of drove me to go and move to London. I could see that they had agencies there where they had lots of like like me working in the same company, which just like sounded extraordinary and I wanted to go and see what that was like. So, I moved to London and went and worked for a place called Flow Interactive, which at the time was filled with what I thought were the smartest and most interesting people I’d ever met in my life and I learned loads from that experience. Then I started getting to the point where I realized that in a lot of large organizations, if you stayed in one place long enough the same brief would come around over and over again. And that you could have a great time doing research and nobody would do a blind thing with it. And that stopped being fun when I realized that it was fun to do the research and then nobody would do anything with it. It was good the first few times. The third time I’m like, hmm.
And then I was – my boss at the time apologetically said to me, “I’m really sorry Leisa, but I’ve got this friend and he’s got this start-up, would you mind – can you do me a favor? I know it’s not very glamorous to work on start-ups, but would you mind working on this start-up?” And so, I did and I remember sitting in the usability lab doing usability testing, as you did, and in the observation room was the entire team, which was amazing that the team actually turned up. Not only that, but they were changing the prototype as I was doing the testing. So, like compared to the other companies that did nothing with anything ever, like they were doing it as I was doing the research which was surprising, but you know, but good. And that was it. From then, I just sort of decided I need to go work with people who are actually going to use the research and that led me into working with tech start-ups for a long time. Yeah
Steve: You worked for yourself as a consultant?
Leisa: Yeah, I had my own business for a long time and worked a lot with tech start-ups in London and then also with some larger companies as well. It was like some big publishing companies and universities and places like that too. It was actually a combination of all of those things over time led me to believe that actually probably the biggest, most important challenge is how you change the DNA of the organization that you’re working in. That you can do the best research project in the world, but if the company that you’re working for, the organization that you’re working for doesn’t have the motivation to actually do something with it, then it’s kind of pointless. I remember looking back on like of work and just going what have I got to show for this? It felt like very little. It was not long after that actually that the opportunity to go join GDS emerged and I thought to myself, right, well this – if I can’t have an impact at this organization I can’t have an impact anywhere. Because, like you said before, it was so full of people who were really smart and really orientated to do the right thing. And super meaningful work as well. And I think that’s where I still find myself today where I love research, I love the way that we learn things and what that means, but the thing that really drives me is seeing an organization almost like a design problem and thinking about like what do we – what levers can we pull? What do we choose to do? How do we position ourselves so that we cannot just do fun research, but we can actually really have this knowledge and this insight and this practice fundamentally change how this organization operates?
Steve: That’s a great arc that you just articulated for yourself. It’s really fabulous. So, let’s go back to you were describing sort of the make-up of the team at Atlassian and you talked about two pieces which are unusual but may be worth digging into a little bit. One was research ops, which is sort of an emergent practice. I’d love to hear you explain a little more about what that looks like for you all. And that market research is part of the team as well. Let’s talk about both of those.
Leisa: Sure. So, our ops component is relatively new. I was fortunate enough to convince Kate Towsey to come and meet me in Sydney and start sort of really building that capability for us. At the moment it really comprises kind of three main bits, I think. One is the recruitment thing, so how do we create a really good infrastructure for doing what’s for us mostly kind of B2B research recruitment, which is pretty tricky, in a way that serves the needs or our organization. And that’s – you know that turns out to be interesting in a lot of ways in terms of set out an infrastructure for that. But also seeing that as potentially an opportunity to try to help shape that kind of who’s doing what kind of research when thing. It gives us visibility into what the current activity is? What kind of stuff is happening, which is good. We don’t really have a good infrastructure for taking every bit of demand and shaping it because that would be a huge bit of work. And also, going back to the time stuff. So, recruitment is one bit of it.
We have another part which is really looking at the technical infrastructure. There’s actually, these days – there’s a big technology component to setting up research well in organizations. Whether it’s how the recruiting happens? Where it’s where the data goes? We run kind of the large-scale surveys out of our organization as well. There’s a surprising amount of tech involved in doing that reasonably well also. So, we have somebody whose job it is really just to completely look at the research tech side of things. And then we also have what we call an engagement and impact team which is a new experiment for us. And this is really looking at how can we try to build a really strong muscle around making sure that the work that we’re doing in the team that’s not embedded is consumed and understood and acted on by the rest of the organization. So, this might be some of the customer happiness surveys that we’re doing. Or it could be their top tasks work or some other sort of strategic work that we’re doing.
How do we make sure that when everybody’s got their nose to the grindstone, focused on the thing that they’re interested in right now, they have this kind of curiosity and interest and understanding for how this other research that’s happening that they haven’t commissioned could be useful to them? So, yeah. So, that’s interesting. And so, the ops thing for me is really thinking about like how can we help make sure that the researchers are really focused on research and not all of the kind of logistical stuff around it. And then secondly around thinking about scale. How can we try and make sure that the work that we are doing in our team scales out to the rest of the organization and that the things that we do at scale are of good quality?
Steve: So, you described in your own arc how you discovered that the change of the DNA, changing minds, was maybe the most compelling part of the research for you. But in building up that third piece, the engagement – and, I’m sorry, there was…
Leisa: Impact.
Steve: …engagement and impact part of ops – you’re sort of asking researchers to not be involved in that part of it as much.
Leisa: So, in the way that we are setting up our team, and it’s a work in progress right now, we have chosen – I have chosen to centralize a bunch of the work that we are doing. Mostly in order to create the opportunity to do the work that I think we need to do that wouldn’t come naturally out of demand within the product teams. So, to do some of this work that steps back from the features and gets to this really kind of grounding understanding of what the needs are. So, we chose to do that because we would never be asked to do that by the teams, but I felt that we really needed to do it. Increasingly now we also have researchers who are working embedded in the teams which is much more like what we did at GDS and DTA and other places. And so, the engagement impact teams are really looking at supporting the work of the centralized teams, but because the people that are embedded tend not to need it so much. They have really close collaboration with their product teams and they’re doing – they’re either meeting the demand or really kind of closely shaping the demand in the teams for the research work that’s happening. But I think what we want to do over time is really have this kind of good, creative tension between the crosscutting strategic research, all that sort of foundational work, and the stuff that’s happening in the teams as well. When you have a large – Atlassian is really not that big. We’re about three and half thousand people. So, tiny compared to some organizations, but quite fragmented. Like lots of products, lots of product teams. When you have that kind of complexity in your organization you have to work really hard to make sure that all the people who should be seeing the thing see the thing and that they understand why it’s important. Because a lot of
people don’t make the connection. Like for us when we do the research we go ah, it’s completely obvious why that person over there should find this really interesting. But they are so focused on something else, they often see the work and then go why should I care about that? So, you often have to really do that sort of bridging exercise as well. So, that’s where engagement and impact are really focused on, I think, is looking at really supporting the crosscutting work, the work that’s not done for any one particular team, but actually is super relevant to them if we can get them to attend to it.
Steve: I think ops in general brings up lots of interesting questions about where do you delineate what a researcher does? I appreciate your point that it depends on the context of the organization and what they’re sort of – the optimal remit for them is to lead to this larger kinds of change. So, if I’m a researcher I want my research to be consumed, to be relevant. I don’t think you’re doing this, but were I to be completely isolated from the people consuming it then I don’t get that feedback loop. You’ve talked about sort of that feedback loop of the researcher is an important thing. That being said, if there is a team that can support me, or that I can support to help drive impact, to help drive that interaction, then that enhances the researcher’s ability to kind of reach people. You’re hearing me thinking out loud about where is an ops person an enhancement to what the researcher can do versus – ‘cuz a lot of ops, like the way I hear people talk about dev ops is like it’s stuff we don’t want the developers to be doing. I don’t know. I think there’s things that research ops does that, in my judgment, I want researchers to be doing. And maybe it’s about building up infrastructure to help them do that better, not take it away from them.
Leisa: Exactly. So, the idea is definitely not that there’s like a handover point that the researcher goes here’s the insights and then engagement and impact take those and spread them out to the world. That’s definitely not what it is. I see engagement and impact people as being people who know how to get things seen by the right people and know how to bring people with interest to come and pay attention. And so, a lot of it really is about making those connections. But the other thing is like I find in my organization I’m one of very few people who has got visibility across everything, across all of the research work that we’re doing and lots of things that are happening in the organizations. And so, I can draw connections that other people can’t draw. But I’m just one person and I spend most of my time in meetings. So, I’m kind of hoping – and it’s very early days for us with this whole engagement/impact thing. So, we’re shaping it on a weekly basis right now. But my hope is that because these people are tasked with having really good visibility across what’s happening in the organization, and they’re helping to facilitate these connections between the researches and the organization, they’re going to get a similar view to what I have where they can start also to make these connections. Because I feel as though that’s one of the opportunities that we have in a research team, looking out across the entire organization, is to be able to see opportunities for connection, to reduce duplication, to sort of build sort of sense making from seeing things from all these different perspectives. To connect people to things that they didn’t even know existed that would be really useful to them. And so, I’m kind of hoping that that’s something that over time this team will be able to build as well is that ability to see where the opportunities and connections are across the entire organization that really nobody else, except for me right now, can see.
Steve: I see a parallel with the design education effort where there’s all these things anyone who has worked in research for any amount of time and says, “oh yeah, we have a lot of responsibilities. In addition to doing the research we have to be teaching people and we have to be trying to advocate for our research and planning.” It seems – I don’t know I’m inspired the more you explain this because you’re basically saying yeah, if that’s a responsibility, maybe we need a team. Maybe it has to be somebody’s job. As opposed to sort of bearing it. And when we make it all part of one person’s job we don’t necessarily even acknowledge that hey, education is part of my job. Hey evangelism is part of my job. Hey driving impact. By creating a team to me it makes me more mindful. Like oh yeah, that’s a thing that you could spend fulltime on. And it doesn’t mean, to your point, you’re not taking it away. You’re creating specialties for all the different facets of – we talk about research, but it’s so many different things.
Leisa: Absolutely.
Steve: And you’re naming and staffing specialties that I haven’t seen named and staffed before.
Leisa: Yeah. It’s really interesting to try to get people to take these jobs because they’re like what’s the career path for this? And I’m like I don’t know. I have no idea what the career path is for this. Let’s work it out together. So, it requires really special people to be willing to take these roles and to kind of – to believe in what the opportunity is and what the purpose is. But we are taking a lean approach to shaping the team, but like I said, I always think about this as a design problem. I’m always thinking like what’s the need. And like we talked before about the huge amount of specialism that’s involved in being a great educator. So, the idea that you can just do that in your spare time, in between projects, is just not realistic, I don’t think, if you want to do a good job at it. And it’s the same with the engagement piece, right. As soon as – like you’re always under pressure to get a project wrapped up and there’s always something that you should be starting next. And so, it can be really difficult for the researchers to go, right, now I’m going to do the engagement and impact part of my job and I’m going to spend 3 weeks just making sure that everyone knows. I’m going to go and do all of these town halls and I’m going to go and run these workshops and I’m going to write these blogs – I’m going to do all of these things. Like that’s – it takes a lot of work, especially if you’re exhausted from all you’ve just done – all of your analysis work. You want to crawl under a rock somewhere and rest for ages and now you have to go out and do all of this like roadshowing of what you’ve learned. They still – the researchers still have to do most of that work, but having engagement and impact there I think can make it so that it’s much more streamlined. We don’t have to sit down and go alright, who do we have to go and talk to? Like when are they having their meetings? How can we get into their team meeting and just present there? Part of what this team is doing is just really having all of that knowledge and going right, okay, so what you want to do after you’ve done a project is this, this, this, this, this, this. And it just takes a lot of that repetitive effort out and it means that the whole process of engaging and being impactful becomes more streamlined.
Steve: I talked to someone a little while ago and they described hiring a librarian, like a reference librarian. We had the conversation about the data repository and they’d come to the realization that it’s not about the infrastructure, it’s about having someone own it, advocate for it. It’s the same thing. Here’s a problem that we have, that we always talk about, what would happen if we created a role for that and could take the pressure off the other people that have other things to do.
Leisa: I have a research librarian on my list of roles to hire yet. I was just waiting to get the headcount for it. But yeah, I completely, completely agree with that point of view.
Steve: So, now market research. That also is part of your group.
Leisa: Yeah, yeah. Market research came about because I inherited NPS and that’s a mixed blessing, obviously. But I wanted to kind of embrace it as an opportunity because it was something that particularly executive people in the organization cared about a lot. It was a good way – it was a good sort of vehicle for getting access at the highest levels and getting people’s attention at the highest levels. But obviously we were pretty – I was pretty unhappy with our methodology which was really surveying way too many people in product when they were trying to get their jobs done. And also, the methodology behind how we did the analysis of that. So, we spent a bunch of time thinking about what we wanted to do that was different to that and when we sat down and thought about the kinds of people that we needed to run a better program of work it became really clear to me that, certainly in Sydney, we don’t have quantitative user researchers. I don’t know, in the valley you can hire them fairly – well they exist. People acknowledge them as a thing. It’s not a thing in Sydney. And so, the closest thing that I could get to that was to look at market research and particular kinds of market researchers who would have that ability to actually really design good surveys and to have really strong skills in thinking about how to analyze them and share that information back as well. So, yeah, so that’s – we’ve gone kind of large on that now and that team is now pretty much staffed with people who have got market research background. And it is really that ability to bring in the quantitative analysis that has made a big difference. And it opens up all kinds of opportunities for us. So, we now are able to work super closely, for example, with our brand and marketing part of the organization because we have skillsets that they value in our team now. We can start to pick up research that they would either have built a separate team for or would outsource. And I think having those opportunities to have, you know, what might have been a voice of the customer team, what might have been a marketing research team, all in the one place is super exciting for us and we would not have been able to do that without bringing some marketing researchers into our team. So, that’s been something that I probably – like if I had not inherited the NPS probably wouldn’t have naturally thought to do. But it’s one of those kind of happy accidents that I’m really, really pleased has happened and I would definitely do again in the future.
Steve: The phrase mixed methods research seems to be sort of a – like a hot term right now which to me is about the collaboration between these different types of research tools or research professionals or research teams. I mean the term seems newish. I’m not sure that the practice is that new. I don’t know, is that – so, with these different capabilities that you have is that part of how you’re able to work?
Leisa: Yeah, absolutely. And I think also it gives us a really strong conduit into some of the other more quantitative parts of the organization as well. So, working much more closely with product analytics is part of what we’ve kind of naturally had to and being able to do as a result of the work that our quant researchers are doing. So, yeah it does, it really – I mean this is something that we’ve talked about a lot as well is trying to look at these mixed methodologies and looking at all the different data sources that we have available to us. And that includes people like our support organization and our kind of field and sales operations as well. But yeah, so I think – I think that having – yeah, having that particular capability in the team has been really critical to helping bring the other sources closer to ours. And we’re starting to get into a much better place at being able to sort of cross-reference each other’s wok and demonstrate that being able to do that is a good thing and kind of go back to the reliability question again, right, that that’s something that you should look for when you’re looking to the reliability of the work that you’re doing. Like can I actually see this in other data sources in this organization? Well, yeah, I can. Okay, great. Then I feel like I can rely on them more.
Steve: If we were to have an episode of Dollars to Donuts in in 2025, let’s imagine that there’s still podcasts and there’s still Atlassian, and you’re still in the role, like what would we be talking about? What would you be highlighting?
Leisa: I don’t know. Hopefully a whole, new different set of problems than what I have right now. I have a picture that I drew. I call it my FY20++ next gen research plan, vision type thing and it has a whole bunch of different kind of sections, components to what a research and insights team might look like, that we would build over the next couple of years. And I – so, that’s – I have kind of a sense of where I want to get to with it but can’t – I’m not yet at the point of being able to anticipate what the problems would be then. Here I am assuming that we’d be talking about the problems and not all of the great accomplishments that we’ve had since then.
Steve: But that’s – I mean that’s the way I think a research leader – your response isn’t unique. I think finding problems that are big ones and tackling them is why – of course that’s our response because that’s the kind of person that ends up in the role that you’re in.
Leisa: The thing that I would be most proud of, the thing that I’m really looking for as we go forward is across the organization for us, whether it’s in product teams, whether it’s at executive level, whether it’s in like the sales or the support, wherever in the organization it is, I would really love for us to have developed this natural tendency to talk about problems and opportunities from the users and the customers point of view, and not from our own product or feature point of view. That for me feels like the big transformation that could make a huge difference. And so, a lot of the things that we’re working on now is how do we build that understanding? How do we build the connections between that understanding and the things that we’re working on – the things we’re thinking about working on – what are the big things and the tiny things that we can do to start to change the way that we think about things? And so, I remember when I was at GDS one of the tiny things that we did was we just said it’s not user testing, it’s usability testing or user research. And tons of people will listen to this and go, “it really doesn’t matter.” But what it did do is by getting people to do that little correction in their head every time they accidentally said the wrong thing, or accidentally almost said the wrong thing, and they’d go we don’t test users, we test ourselves. We test our work, we test our ideas, we test our designs. That, in a government context, was a really super important thing. And so, it’s that kind of – it’s almost that linguistic correction of like making sure that when we talk about things we don’t – like I often say to people that I’m working with at the moment, like if you can’t describe the problem without saying an Atlassian word then you don’t understand the problem. And so that’s what I’m looking for. I’m looking for a future where when we talk about things we don’t talk about it by the product name or the feature name or some other kind of internal, buzz wordy thing what we have, but we use language that clearly comes from an understanding of our users and our customers and the needs that they have and the problems that they are trying to solve. That’s, that’s – that’s my kind of mission, I think, more than anything else right now because I feel like that will have the biggest cultural impact.
Steve: That’s fabulous. Anything else we want to talk about?
Leisa: No, I think we’ve done it to death, Steve.
Steve: Alright, well thank you very much Leisa. It was really interesting to talk to you. I really appreciate all your time.
Leisa: Thank you very much. It’s been fun.
Steve: Thanks for listening to Dollars to Donuts. Follow the podcast on Twitter, and subscribe to the podcast at portigal dot com slash podcast, or Apple Podcasts, Stitcher or Spotify, and the rest. The website has transcripts, show notes, and all the episodes. Get another copy of my books Interviewing Users and Doorbells Danger and Dead Batteries from Rosenfeld Media dot com or Amazon. Thank you Bruce Todd for the great Dollars to Donuts theme.
The post 20. Leisa Reichelt of Atlassian (Part 2) first appeared on Portigal Consulting.
This episode of Dollars to Donuts is part 2 of my conversation with Leisa Reichelt of Atlassian. If you haven’t listened to part 1 yet, you can find it here. We talk about corporate versus government work, scaling research, and changing organizational DNA. I love research, I love the way that we learn things and […]
The post 20. Leisa Reichelt of Atlassian (Part 2) first appeared on Portigal Consulting.
This episode of Dollars to Donuts is part 2 of my conversation with Leisa Reichelt of Atlassian. If you haven’t listened to part 1 yet, you can find it here. We talk about corporate versus government work, scaling research, and changing organizational DNA.
I love research, I love the way that we learn things and what that means, but the thing that really drives me is seeing an organization almost like a design problem and thinking about like what do we – what levers can we pull? What do we choose to do? How do we position ourselves so that we cannot just do fun research, but we can actually really have this knowledge and this insight and this practice fundamentally change how this organization operates? – Leisa Reichelt
Show Links* ~~Fundamentals of Interviewing Users (SF)~~ * Part 1 with Leisa * Leisa on LinkedIn * Leisa on Twitter * Atlassian * Francis Maude * Martha Lane Fox * Tom Loosemore * GDS (Government Digital Service) * A Country Practice * DTA (Digital Transformation Agency) * Flow Interactive * NPS (Net Promoter Score) * Kate Towsey
Follow Dollars to Donuts on Twitter and help other people discover the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Hey, and here we are with another episode of Dollars to Donuts, the podcast where I talk to the people who are leading user research in their organization. This is part two of my interview with Leisa Reichelt. If you’re just joining the podcast with this episode, I’d encourage you to go back to the previous one for the first part of our conversation.
As a reminder, my public workshop Fundamentals of Interviewing Users is happening September 13th in San Francisco. There’ll be a link for more information in the show notes. It’d be great if you recommended to this a friend or colleague. I also teach classes directly to in-house teams so reach out to learn more. Beyond teaching, in my consulting practice I also lead user research studies, so let’s talk if that’s a way that I can help your team.
Of course, supporting my own consulting work is the best way to support Dollars to Donuts. Share your feedback about the podcast by email at DONUTS AT PORTIGAL DOT COM or on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
You know, as user researchers, we love our sticky notes. A few years ago, in an article about how IBM was being transformed by design, a key achievement in this transformation was the ability for staff to order post-it notes. In some ways, that was the saddest thing that I ever heard, that IBM was so broken that ordering a quotidian office supply item was verboten, and that enabling this was seen as a victory worthy of mention. But it also was very real and acknowledged how much of uphill battle these kinds of corporate transformation efforts really are.
So this carrier of innovative meaning both in its legendary origins and in its rapaciously consuming audience, this product of the 3M company clearly struggles itself with innovation. We’ve had the weird dispensers, the odd sizes and shapes, the reverse fan fold which may be tied to the dispenser but I’ve mostly just found them showing up in the most aggravating moments in a session. I think many years ago they came out with Super Sticky notes which seems like a complete contradiction of the value proposition but I think it’s just an acknowledgement that the original formulation fell off more than we’d like. I don’t think of them as Super Sticky, just the proper amount of sticky for most uses. Anyway, the latest thing I came across really made me scratch my head. It’s a pack of bright orange Post-Its, probably part of a series of exciting new notes with the aspirational branding A WORLD OF COLOR. This particular pack had an even more aspiration and even less relevant tagline “Rio De Janeiro” Collection. I can imagine interior paint, fabric even car finishes being marketed this way but it’s so strange to see on a package of sticky notes. The post-it, for the researcher, the designer is a backdrop, a carrier for something else. It already is aspirational, because of what we’re putting on it. Making it a “collection,” associating with a far-off fabulous city, is just ridiculous.
Okay, here’s part two of my interview with Leisa Rechelt, the Head of Research and Insights at Atlassian in Sydney Australia. It was quite an in-depth interview that’s been broken up into two parts. This is part two, you can check out the previous episode for part 1. Let’s get to it!
I’d love to hear about some of the other kinds of organizations you’ve worked in and what those have been like.
Leisa Reichelt: Sure. Well, for probably 5 or 6 years before I joined Atlassian I worked in government. First of all, in the UK and then I moved back to Australia and worked in the Australian government for about months. Something like that, I think. And that was super different, super different to Atlassian. So, it was a much more kind of familiar ground for me in that it was organizations that you’d go into going we should really go and like involve people in our design process and they would go why would you possibly want to do that? So, that’s a whole different problem set than what we’re dealing with, I think, in some of the tech companies. But hugely rewarded as well. So, yeah, really very different.
Steve: Both those governments seem like their commitment – I guess they’re just different cases, but their respective commitments to sort of design – I don’t know, digital services seems to be the term that gets thrown around for that. But it seems like there’s been significant commitments in both those cases. I mean you’re coming into environments where someone has said we want to do this, we want to change that default. And – like in the UK what was sort of the – how did that get initiated?
Leisa: Okay. So, in the UK we had a MP, Francis Maude, late in his career. I don’t exactly know how it came to him that we could probably be doing better with our digital services than we are. I don’t fully have that back story, but it did come to his attention that we could do better. And he recruited a lovely woman by the name of Martha Lane Fox – Dame Martha Lane Fox now – to basically help him come up with an approach to how we should solve the problem of the UK digital government services not being what they need to be. Martha worked with a bunch of very smart people. In particular, a guy called Tom Loosemore, to come up with some recommendations. And it was off the back of those recommendations that the government digital service was put together. Tom and a bunch of other people that he’d worked with in various places around London and the UK came together and started working on trying to transform how government thought about approaching digital services. And they had – their design principles, I think, were really the best way of setting out what their beliefs were. And fortunately for me, and number one of those principals, was around putting user needs and not government needs first. And so yeah, culturally there was a cohort in there who were well supported within the political system and were able to really kind of make great shifts and changes and progress on that front as a result.
Steve: It seems to me, just from like watching Twitter or LinkedIn, or just who I keep coming across, that there are research people, titled researchers in every kind of nook and cranny of government services – digital and otherwise, I think in the UK. It seems like from the time that you got involved it’s built into something – it seems like it’s sort of changed the way that government is delivering services. I don’t know if that’s an accurate assessment.
Leisa: Yeah. So, it’s when I started at GDS I kind of came off the back of like – there was a lot of talk about being user centered, but I couldn’t really see exactly where the users were in the process. So, I was publicly a little bit critical of them at one point kind of saying well you know I see that you’re thinking about users a lot and you’re looking at the data that they leave behind a lot, but are you actually – actually involving getting a good understanding of them in the process of designing and transforming these services? And so, I was given the opportunity to come in and put my money where my mouth was, such as it was. And at that point there was the odd kind of researcher here and there. There were like 3 or 4, I think, at GDS at the time, and they were stretched across about 25 different projects. I remember sitting down at their team meetings and the team meetings were basically sitting in front of a spreadsheet, looking at all of the projects that they were supposed to be covering, dividing their days into quarter days and working out how on earth they were going to try to help to support these teams, not all of which were in London. So, a lot of them kind of theoretically required either quite a bit of traveling or just dealing with on the telephone. And that was it. So, they were there, but they were really not well set up to be effective. And I was not really – you know I’d worked with government as a consultant in the past and was pretty skeptical about whether or not that would be a brilliant place for me to work long term. And so, when it came time for me to say what I thought we should do I wasn’t really that worried if I lost my job. So, I was able to say what I thought we should do which is I thought we should have one researcher per team, which at that point in time was like just an outrageous thing to be saying. It was absurd. And we didn’t quite get one researcher per team, but we did get quite a big chunk of researchers and so – and the other thing that I did that I kind of look back on and think that was a really important thing to do was I put one researcher on as many teams as I had researchers and I just left the rest, pretty much unsupported. And the thinking behind that was if I could just give a bunch of people an opportunity to show what good looks like then we could create demand for that good. And if we stretched everybody across multiple things we’d never be able to create that showcase, that exemplar that let people see what they could have. And I think that kind of made all the difference. In the early stages again, speaking about getting proactive rather than reactive. One of the big challenges that we had was like how do we get people even to plan for this? Like how do we get researchers, and budget for what researchers need to do, in the projects ahead of time because, you know, people would come to us saying, “we’re starting discovery next week, I’ve heard you can help us with that.” And by then, like you know, the budgets had been approved like 3 or 4 months before. You had to like kick off an engineer to get a researcher which is always a super popular thing to do – not. And then there were things that we do where we just sort of made rules about you have to have one researcher per team, at least 3 days a week. And it was 3 days because then you couldn’t split them across two projects equally. You had to do research at least every sprint, which was 2 weeks for us. And you had to involve at least this many people. That basically gave us the formula for working out how much research would cost in a project which let us get like that number into the project budget really early on. And these are like really totally boring things that have got really nothing to do with research at all, but were hugely enabling. By the time I was leaving GSA, instead of having the problem of going you should have a researcher in there, you should have a researcher in there, I would have people coming to me, initiating projects, going I’m starting a project, I’m going to need at least 3 or 4 researchers and you go no, no, no, that’s too many, just start with one and kind of go from there. Yeah, it was something that really – I think because people were able to see the difference that it made – and researchers were really facilitators for the rest of the team. So, their job, yes was to run the research and set it up and facilitate it and do the analysis and that kind of thing, but they created these opportunities for the entire team to be able to see how what they were working on was solving problems. Or, how what they were working on worked really well, or didn’t work so well. I think that was like the most powerful thing that they did was really to be very open and inclusive in the way that the whole team was invited and expected to come along and see for themselves, first hand, what was happening. Our research labs that we had onsite we’re always full. Like our observation room, we just had to keep making it bigger and bigger all the time because so many people would come in and watch these sessions. And I think it was that – a) it’s super helpful, but b) I think it’s really – it’s great feedback that the team gets. It provides – it shows the meaningfulness of the work that they’re doing and I think that’s a really desirable thing for teams to be able to see what I’m doing makes a difference. It has an impact. It matters. And I like to think that that contributes to team health as well.
Steve: What’s the difference in the culture of government vs. software technology companies where that kind of response, that kind of commitment and engagement can happen?
Leisa: Well, I was going to say governments don’t have growth teams. They do have behavioral scientists though – behavioral economists – which are kind of a similarish type thing, but they tend to still experiment on letters more than they do on digital things. But I think that it’s a few things. I think that people in government feel – it’s such a generalization. I don’t even know if I believe it 100%. Certainly, in places that I’ve worked people in government have felt their responsibility to the people who are the service receivers and the idea of experimenting on them is something that you would approach much more carefully then what we see, I think, in a lot of other sort of software/technical organizations. I think that’s one part of it. I think the other thing is very few people that I met in government thought that they could do my job. They didn’t know that my job was a thing and they didn’t think that they could do it probably as well as I could. Whereas I find in tech companies there’s a lot more of that. Like everyone’s got some kind of research background. Everyone feels a little more confident and capable at doing this stuff themselves. And I know that that’s not universally true. I know there are a lot of companies where everyone is terrified of going out and talking to customers. But I do feel as though in tech companies – some tech companies – there is a lot more kind of over-confidence about your ability to go out and do this, especially this – and I don’t want to sound like I’m a complete downer on the lean sort of thing because done well it’s great, but I think that has really encouraged people to think that there’s not too much too this, that anyone can go and have a chat with a customer and that’s what we do. I think there are probably a few other things that are really different.
Steve: That’s a good comparison. Can we go back in time a little bit further maybe. I’d love to hear about how you found research as a thing? How is it that you ended up in this field?
Leisa: Well, I was one of those people who growing up didn’t know what I wanted to be. I briefly wanted to be a vet, but that was mostly because of a popular television show that was on in Australia at the time.
Steve: Which one?
Leisa: It was called A Country Practice and it had a character called Vicky who was the vet and I thought she was the bees’ knees. But then kind of beyond that – that was like when I was in primary school and then beyond that I honestly really don’t know what I want to do when I grow up. After high school I went and did a university degree in communications and I could have gone and done a bunch of other different degrees. I kind of applied for like this variety of all different kinds of things in different cities. Like it was a toss-up between doing music in Melbourne or communications in Sydney. In the end I chose communications in Sydney because they had 9 hours face to face time and it was pass/fail and I thought that sounded pretty cruisey enough, after being a bit of a study nerd in high school. I thought maybe I’d earned a break. It was not my best thought out decision and I didn’t really know what I was doing there. And then I was working at the same time. I really didn’t know what I wanted to do at all. I met the Internet at uni for the first time and then in one of my kind of weird jobs that I was doing, I was working for a legal services company and the guy who had started that company, one of his kind of big market advantages was this software that he had had designed basically, which ran all of the kind of logistics of his company. And the guy – you know as software does sometimes, it just stops working or it needs changing. And the guy who was in charge of the software was a supplier and when he couldn’t be bothered coming on site to fix stuff he would ask me to do it. So, I got to understand a little bit about kind of how software happened. And then we started thinking about doing internet stuff and it morphed into a really early E-commece offering and I was asked to be the project manager of that. And I thought to myself, this internet this – this is it, right? This is why I didn’t know what I wanted to do when I grew up because the Internet didn’t exist when I was growing up. So, I thought that was really exciting and then I went and worked in some digital agencies. I had a job called a producer and a producer basically did everything except for writing most of the code and the visual design. So, you did the project management, the account management, the interaction design, the information architecture, all of that sort of stuff. A lot of the sort of project strategy stuff as well. This was in the early days when agencies were still kind of really small. And then over time I just kind of got rid of the stuff that was less interesting to me to focus in on the stuff that was very interesting to me. At the time it was information architecture and the research that goes behind that. And I kept doing both of those things for quite a while. At one point I was working on this huge project in Australia and trying to get budget to do some research. And it was a tiny, tiny budget that I wanted and I had to fight so hard for it. And it was a project that was being run by an ad agency and I’m pretty sure they spent on lunch every week what I was asking for, for research for the entire project. I got very frustrated. And that was basically what kind of drove me to go and move to London. I could see that they had agencies there where they had lots of like like me working in the same company, which just like sounded extraordinary and I wanted to go and see what that was like. So, I moved to London and went and worked for a place called Flow Interactive, which at the time was filled with what I thought were the smartest and most interesting people I’d ever met in my life and I learned loads from that experience. Then I started getting to the point where I realized that in a lot of large organizations, if you stayed in one place long enough the same brief would come around over and over again. And that you could have a great time doing research and nobody would do a blind thing with it. And that stopped being fun when I realized that it was fun to do the research and then nobody would do anything with it. It was good the first few times. The third time I’m like, hmm.
And then I was – my boss at the time apologetically said to me, “I’m really sorry Leisa, but I’ve got this friend and he’s got this start-up, would you mind – can you do me a favor? I know it’s not very glamorous to work on start-ups, but would you mind working on this start-up?” And so, I did and I remember sitting in the usability lab doing usability testing, as you did, and in the observation room was the entire team, which was amazing that the team actually turned up. Not only that, but they were changing the prototype as I was doing the testing. So, like compared to the other companies that did nothing with anything ever, like they were doing it as I was doing the research which was surprising, but you know, but good. And that was it. From then, I just sort of decided I need to go work with people who are actually going to use the research and that led me into working with tech start-ups for a long time. Yeah
Steve: You worked for yourself as a consultant?
Leisa: Yeah, I had my own business for a long time and worked a lot with tech start-ups in London and then also with some larger companies as well. It was like some big publishing companies and universities and places like that too. It was actually a combination of all of those things over time led me to believe that actually probably the biggest, most important challenge is how you change the DNA of the organization that you’re working in. That you can do the best research project in the world, but if the company that you’re working for, the organization that you’re working for doesn’t have the motivation to actually do something with it, then it’s kind of pointless. I remember looking back on like of work and just going what have I got to show for this? It felt like very little. It was not long after that actually that the opportunity to go join GDS emerged and I thought to myself, right, well this – if I can’t have an impact at this organization I can’t have an impact anywhere. Because, like you said before, it was so full of people who were really smart and really orientated to do the right thing. And super meaningful work as well. And I think that’s where I still find myself today where I love research, I love the way that we learn things and what that means, but the thing that really drives me is seeing an organization almost like a design problem and thinking about like what do we – what levers can we pull? What do we choose to do? How do we position ourselves so that we cannot just do fun research, but we can actually really have this knowledge and this insight and this practice fundamentally change how this organization operates?
Steve: That’s a great arc that you just articulated for yourself. It’s really fabulous. So, let’s go back to you were describing sort of the make-up of the team at Atlassian and you talked about two pieces which are unusual but may be worth digging into a little bit. One was research ops, which is sort of an emergent practice. I’d love to hear you explain a little more about what that looks like for you all. And that market research is part of the team as well. Let’s talk about both of those.
Leisa: Sure. So, our ops component is relatively new. I was fortunate enough to convince Kate Towsey to come and meet me in Sydney and start sort of really building that capability for us. At the moment it really comprises kind of three main bits, I think. One is the recruitment thing, so how do we create a really good infrastructure for doing what’s for us mostly kind of B2B research recruitment, which is pretty tricky, in a way that serves the needs or our organization. And that’s – you know that turns out to be interesting in a lot of ways in terms of set out an infrastructure for that. But also seeing that as potentially an opportunity to try to help shape that kind of who’s doing what kind of research when thing. It gives us visibility into what the current activity is? What kind of stuff is happening, which is good. We don’t really have a good infrastructure for taking every bit of demand and shaping it because that would be a huge bit of work. And also, going back to the time stuff. So, recruitment is one bit of it.
We have another part which is really looking at the technical infrastructure. There’s actually, these days – there’s a big technology component to setting up research well in organizations. Whether it’s how the recruiting happens? Where it’s where the data goes? We run kind of the large-scale surveys out of our organization as well. There’s a surprising amount of tech involved in doing that reasonably well also. So, we have somebody whose job it is really just to completely look at the research tech side of things. And then we also have what we call an engagement and impact team which is a new experiment for us. And this is really looking at how can we try to build a really strong muscle around making sure that the work that we’re doing in the team that’s not embedded is consumed and understood and acted on by the rest of the organization. So, this might be some of the customer happiness surveys that we’re doing. Or it could be their top tasks work or some other sort of strategic work that we’re doing.
How do we make sure that when everybody’s got their nose to the grindstone, focused on the thing that they’re interested in right now, they have this kind of curiosity and interest and understanding for how this other research that’s happening that they haven’t commissioned could be useful to them? So, yeah. So, that’s interesting. And so, the ops thing for me is really thinking about like how can we help make sure that the researchers are really focused on research and not all of the kind of logistical stuff around it. And then secondly around thinking about scale. How can we try and make sure that the work that we are doing in our team scales out to the rest of the organization and that the things that we do at scale are of good quality?
Steve: So, you described in your own arc how you discovered that the change of the DNA, changing minds, was maybe the most compelling part of the research for you. But in building up that third piece, the engagement – and, I’m sorry, there was…
Leisa: Impact.
Steve: …engagement and impact part of ops – you’re sort of asking researchers to not be involved in that part of it as much.
Leisa: So, in the way that we are setting up our team, and it’s a work in progress right now, we have chosen – I have chosen to centralize a bunch of the work that we are doing. Mostly in order to create the opportunity to do the work that I think we need to do that wouldn’t come naturally out of demand within the product teams. So, to do some of this work that steps back from the features and gets to this really kind of grounding understanding of what the needs are. So, we chose to do that because we would never be asked to do that by the teams, but I felt that we really needed to do it. Increasingly now we also have researchers who are working embedded in the teams which is much more like what we did at GDS and DTA and other places. And so, the engagement impact teams are really looking at supporting the work of the centralized teams, but because the people that are embedded tend not to need it so much. They have really close collaboration with their product teams and they’re doing – they’re either meeting the demand or really kind of closely shaping the demand in the teams for the research work that’s happening. But I think what we want to do over time is really have this kind of good, creative tension between the crosscutting strategic research, all that sort of foundational work, and the stuff that’s happening in the teams as well. When you have a large – Atlassian is really not that big. We’re about three and half thousand people. So, tiny compared to some organizations, but quite fragmented. Like lots of products, lots of product teams. When you have that kind of complexity in your organization you have to work really hard to make sure that all the people who should be seeing the thing see the thing and that they understand why it’s important. Because a lot of
people don’t make the connection. Like for us when we do the research we go ah, it’s completely obvious why that person over there should find this really interesting. But they are so focused on something else, they often see the work and then go why should I care about that? So, you often have to really do that sort of bridging exercise as well. So, that’s where engagement and impact are really focused on, I think, is looking at really supporting the crosscutting work, the work that’s not done for any one particular team, but actually is super relevant to them if we can get them to attend to it.
Steve: I think ops in general brings up lots of interesting questions about where do you delineate what a researcher does? I appreciate your point that it depends on the context of the organization and what they’re sort of – the optimal remit for them is to lead to this larger kinds of change. So, if I’m a researcher I want my research to be consumed, to be relevant. I don’t think you’re doing this, but were I to be completely isolated from the people consuming it then I don’t get that feedback loop. You’ve talked about sort of that feedback loop of the researcher is an important thing. That being said, if there is a team that can support me, or that I can support to help drive impact, to help drive that interaction, then that enhances the researcher’s ability to kind of reach people. You’re hearing me thinking out loud about where is an ops person an enhancement to what the researcher can do versus – ‘cuz a lot of ops, like the way I hear people talk about dev ops is like it’s stuff we don’t want the developers to be doing. I don’t know. I think there’s things that research ops does that, in my judgment, I want researchers to be doing. And maybe it’s about building up infrastructure to help them do that better, not take it away from them.
Leisa: Exactly. So, the idea is definitely not that there’s like a handover point that the researcher goes here’s the insights and then engagement and impact take those and spread them out to the world. That’s definitely not what it is. I see engagement and impact people as being people who know how to get things seen by the right people and know how to bring people with interest to come and pay attention. And so, a lot of it really is about making those connections. But the other thing is like I find in my organization I’m one of very few people who has got visibility across everything, across all of the research work that we’re doing and lots of things that are happening in the organizations. And so, I can draw connections that other people can’t draw. But I’m just one person and I spend most of my time in meetings. So, I’m kind of hoping – and it’s very early days for us with this whole engagement/impact thing. So, we’re shaping it on a weekly basis right now. But my hope is that because these people are tasked with having really good visibility across what’s happening in the organization, and they’re helping to facilitate these connections between the researches and the organization, they’re going to get a similar view to what I have where they can start also to make these connections. Because I feel as though that’s one of the opportunities that we have in a research team, looking out across the entire organization, is to be able to see opportunities for connection, to reduce duplication, to sort of build sort of sense making from seeing things from all these different perspectives. To connect people to things that they didn’t even know existed that would be really useful to them. And so, I’m kind of hoping that that’s something that over time this team will be able to build as well is that ability to see where the opportunities and connections are across the entire organization that really nobody else, except for me right now, can see.
Steve: I see a parallel with the design education effort where there’s all these things anyone who has worked in research for any amount of time and says, “oh yeah, we have a lot of responsibilities. In addition to doing the research we have to be teaching people and we have to be trying to advocate for our research and planning.” It seems – I don’t know I’m inspired the more you explain this because you’re basically saying yeah, if that’s a responsibility, maybe we need a team. Maybe it has to be somebody’s job. As opposed to sort of bearing it. And when we make it all part of one person’s job we don’t necessarily even acknowledge that hey, education is part of my job. Hey evangelism is part of my job. Hey driving impact. By creating a team to me it makes me more mindful. Like oh yeah, that’s a thing that you could spend fulltime on. And it doesn’t mean, to your point, you’re not taking it away. You’re creating specialties for all the different facets of – we talk about research, but it’s so many different things.
Leisa: Absolutely.
Steve: And you’re naming and staffing specialties that I haven’t seen named and staffed before.
Leisa: Yeah. It’s really interesting to try to get people to take these jobs because they’re like what’s the career path for this? And I’m like I don’t know. I have no idea what the career path is for this. Let’s work it out together. So, it requires really special people to be willing to take these roles and to kind of – to believe in what the opportunity is and what the purpose is. But we are taking a lean approach to shaping the team, but like I said, I always think about this as a design problem. I’m always thinking like what’s the need. And like we talked before about the huge amount of specialism that’s involved in being a great educator. So, the idea that you can just do that in your spare time, in between projects, is just not realistic, I don’t think, if you want to do a good job at it. And it’s the same with the engagement piece, right. As soon as – like you’re always under pressure to get a project wrapped up and there’s always something that you should be starting next. And so, it can be really difficult for the researchers to go, right, now I’m going to do the engagement and impact part of my job and I’m going to spend 3 weeks just making sure that everyone knows. I’m going to go and do all of these town halls and I’m going to go and run these workshops and I’m going to write these blogs – I’m going to do all of these things. Like that’s – it takes a lot of work, especially if you’re exhausted from all you’ve just done – all of your analysis work. You want to crawl under a rock somewhere and rest for ages and now you have to go out and do all of this like roadshowing of what you’ve learned. They still – the researchers still have to do most of that work, but having engagement and impact there I think can make it so that it’s much more streamlined. We don’t have to sit down and go alright, who do we have to go and talk to? Like when are they having their meetings? How can we get into their team meeting and just present there? Part of what this team is doing is just really having all of that knowledge and going right, okay, so what you want to do after you’ve done a project is this, this, this, this, this, this. And it just takes a lot of that repetitive effort out and it means that the whole process of engaging and being impactful becomes more streamlined.
Steve: I talked to someone a little while ago and they described hiring a librarian, like a reference librarian. We had the conversation about the data repository and they’d come to the realization that it’s not about the infrastructure, it’s about having someone own it, advocate for it. It’s the same thing. Here’s a problem that we have, that we always talk about, what would happen if we created a role for that and could take the pressure off the other people that have other things to do.
Leisa: I have a research librarian on my list of roles to hire yet. I was just waiting to get the headcount for it. But yeah, I completely, completely agree with that point of view.
Steve: So, now market research. That also is part of your group.
Leisa: Yeah, yeah. Market research came about because I inherited NPS and that’s a mixed blessing, obviously. But I wanted to kind of embrace it as an opportunity because it was something that particularly executive people in the organization cared about a lot. It was a good way – it was a good sort of vehicle for getting access at the highest levels and getting people’s attention at the highest levels. But obviously we were pretty – I was pretty unhappy with our methodology which was really surveying way too many people in product when they were trying to get their jobs done. And also, the methodology behind how we did the analysis of that. So, we spent a bunch of time thinking about what we wanted to do that was different to that and when we sat down and thought about the kinds of people that we needed to run a better program of work it became really clear to me that, certainly in Sydney, we don’t have quantitative user researchers. I don’t know, in the valley you can hire them fairly – well they exist. People acknowledge them as a thing. It’s not a thing in Sydney. And so, the closest thing that I could get to that was to look at market research and particular kinds of market researchers who would have that ability to actually really design good surveys and to have really strong skills in thinking about how to analyze them and share that information back as well. So, yeah, so that’s – we’ve gone kind of large on that now and that team is now pretty much staffed with people who have got market research background. And it is really that ability to bring in the quantitative analysis that has made a big difference. And it opens up all kinds of opportunities for us. So, we now are able to work super closely, for example, with our brand and marketing part of the organization because we have skillsets that they value in our team now. We can start to pick up research that they would either have built a separate team for or would outsource. And I think having those opportunities to have, you know, what might have been a voice of the customer team, what might have been a marketing research team, all in the one place is super exciting for us and we would not have been able to do that without bringing some marketing researchers into our team. So, that’s been something that I probably – like if I had not inherited the NPS probably wouldn’t have naturally thought to do. But it’s one of those kind of happy accidents that I’m really, really pleased has happened and I would definitely do again in the future.
Steve: The phrase mixed methods research seems to be sort of a – like a hot term right now which to me is about the collaboration between these different types of research tools or research professionals or research teams. I mean the term seems newish. I’m not sure that the practice is that new. I don’t know, is that – so, with these different capabilities that you have is that part of how you’re able to work?
Leisa: Yeah, absolutely. And I think also it gives us a really strong conduit into some of the other more quantitative parts of the organization as well. So, working much more closely with product analytics is part of what we’ve kind of naturally had to and being able to do as a result of the work that our quant researchers are doing. So, yeah it does, it really – I mean this is something that we’ve talked about a lot as well is trying to look at these mixed methodologies and looking at all the different data sources that we have available to us. And that includes people like our support organization and our kind of field and sales operations as well. But yeah, so I think – I think that having – yeah, having that particular capability in the team has been really critical to helping bring the other sources closer to ours. And we’re starting to get into a much better place at being able to sort of cross-reference each other’s wok and demonstrate that being able to do that is a good thing and kind of go back to the reliability question again, right, that that’s something that you should look for when you’re looking to the reliability of the work that you’re doing. Like can I actually see this in other data sources in this organization? Well, yeah, I can. Okay, great. Then I feel like I can rely on them more.
Steve: If we were to have an episode of Dollars to Donuts in in 2025, let’s imagine that there’s still podcasts and there’s still Atlassian, and you’re still in the role, like what would we be talking about? What would you be highlighting?
Leisa: I don’t know. Hopefully a whole, new different set of problems than what I have right now. I have a picture that I drew. I call it my FY20++ next gen research plan, vision type thing and it has a whole bunch of different kind of sections, components to what a research and insights team might look like, that we would build over the next couple of years. And I – so, that’s – I have kind of a sense of where I want to get to with it but can’t – I’m not yet at the point of being able to anticipate what the problems would be then. Here I am assuming that we’d be talking about the problems and not all of the great accomplishments that we’ve had since then.
Steve: But that’s – I mean that’s the way I think a research leader – your response isn’t unique. I think finding problems that are big ones and tackling them is why – of course that’s our response because that’s the kind of person that ends up in the role that you’re in.
Leisa: The thing that I would be most proud of, the thing that I’m really looking for as we go forward is across the organization for us, whether it’s in product teams, whether it’s at executive level, whether it’s in like the sales or the support, wherever in the organization it is, I would really love for us to have developed this natural tendency to talk about problems and opportunities from the users and the customers point of view, and not from our own product or feature point of view. That for me feels like the big transformation that could make a huge difference. And so, a lot of the things that we’re working on now is how do we build that understanding? How do we build the connections between that understanding and the things that we’re working on – the things we’re thinking about working on – what are the big things and the tiny things that we can do to start to change the way that we think about things? And so, I remember when I was at GDS one of the tiny things that we did was we just said it’s not user testing, it’s usability testing or user research. And tons of people will listen to this and go, “it really doesn’t matter.” But what it did do is by getting people to do that little correction in their head every time they accidentally said the wrong thing, or accidentally almost said the wrong thing, and they’d go we don’t test users, we test ourselves. We test our work, we test our ideas, we test our designs. That, in a government context, was a really super important thing. And so, it’s that kind of – it’s almost that linguistic correction of like making sure that when we talk about things we don’t – like I often say to people that I’m working with at the moment, like if you can’t describe the problem without saying an Atlassian word then you don’t understand the problem. And so that’s what I’m looking for. I’m looking for a future where when we talk about things we don’t talk about it by the product name or the feature name or some other kind of internal, buzz wordy thing what we have, but we use language that clearly comes from an understanding of our users and our customers and the needs that they have and the problems that they are trying to solve. That’s, that’s – that’s my kind of mission, I think, more than anything else right now because I feel like that will have the biggest cultural impact.
Steve: That’s fabulous. Anything else we want to talk about?
Leisa: No, I think we’ve done it to death, Steve.
Steve: Alright, well thank you very much Leisa. It was really interesting to talk to you. I really appreciate all your time.
Leisa: Thank you very much. It’s been fun.
Steve: Thanks for listening to Dollars to Donuts. Follow the podcast on Twitter, and subscribe to the podcast at portigal dot com slash podcast, or Apple Podcasts, Stitcher or Spotify, and the rest. The website has transcripts, show notes, and all the episodes. Get another copy of my books Interviewing Users and Doorbells Danger and Dead Batteries from Rosenfeld Media dot com or Amazon. Thank you Bruce Todd for the great Dollars to Donuts theme.
The post 20. Leisa Reichelt of Atlassian (Part 2) first appeared on Portigal Consulting.
This episode of Dollars to Donuts features part 1 of my two-part conversation with Leisa Reichelt of Atlassian. We talk about educating the organization to do user research better, the limitations of horizontal products, and the tension between “good” and “bad” research. If you’re working on a product that has got some more foundational issues […]
The post 19. Leisa Reichelt of Atlassian (Part 1) first appeared on Portigal Consulting.
This episode of Dollars to Donuts features part 1 of my two-part conversation with Leisa Reichelt of Atlassian. We talk about educating the organization to do user research better, the limitations of horizontal products, and the tension between “good” and “bad” research.
If you’re working on a product that has got some more foundational issues that need to be addressed, but the vast majority of the work is happening at that very detailed feature level, how are you going to ever going to stop kind of circling the drain? You get stuck in this kind of local maxima. How are you ever going to take that big substantial step to really move your product forward if it’s nobody’s job, nobody’s priority, to do that? – Leisa Reichelt
Show Links* ~~Fundamentals of Interviewing Users (SF)~~ * The Art of Noticing, by Rob Walker * The Art of Noticing newsletter * Objectified * Did you see that? Tapping into your super-noticing power * Leisa on LinkedIn * Leisa on Twitter * Atlassian * Jira * Confluence * Trello * Quantifying Qualitative Research – Mind the Product * Gerry McGovern’s Top Tasks * What is Jobs to be Done (JTBD)? * Build Measure Learn
Follow Dollars to Donuts on Twitter and help other people discover the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Howdy, and here we are with another episode of Dollars to Donuts, the podcast where I talk to the people who are leading user research in their organization.
I just taught a public workshop in New York City about user research, organized by Rosenfeld Media. But don’t despair – this workshop, Fundamentals of Interviewing Users is also happening September 13th in San Francisco. I’ll put the link in the show notes. Send your team! Recommend this workshop to your friends!
If you aren’t in San Francisco, or you can’t make it September 13th, you can hire me to come into your organization and lead a training workshop. Recently I’ve taught classes for companies in New York City, and coming up will be San Diego, as well as the Midwest, and Texas. I’d love to talk with you about coming into your company and teaching people about research.
As always, a reminder that supporting me in my business is a way to support this podcast and ensure that I can keep making new episodes. If you have feedback about the podcast, I’d love to hear from you at DONUTS AT PORTIGAL DOT COM or on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
I was pretty excited this week to receive a new book in the mail. It’s called The Art of Noticing, by Rob Walker, whose name you may recognize from his books, or New York Times columns, or his appearance in Gary Hustwit’s documentary “Objectified.” I’ve only just started the book but I am eager to read it, which is not something I say that often about a book of non-fiction. The book is structured around 131 different exercises to practice noticing. Each page has really great pull-quotes and the exercises seem to come from a bunch of interesting sources like people who focus on creativity or art or storytelling. Rob also publishes a great newsletter with lots of additional tips and examples around noticing, and I’ve even sent him a few references that he’s included. I’ll put all this info in the notes for this episode. This topic takes me back to a workshop I ran a few years ago at one of the first user research conferences I ever attended, called About, With and For. The workshop is about noticing and II wonder if it’s time to revisit that workshop, and I can look to Rob’s book as a new resource.
Well, let’s get to the episode. I had a fascinating and in-depth conversation with Leisa Reichelt. She is the Head of Research and Insights at Atlassian in Sydney Australia. Our interview went on for a long time and I’m going to break it up into two parts. So, let’s get to part one here. Thank you very much for being here.
Leisa Reichelt: Thank you for inviting me.
Steve: Let’s start with maybe some background introduction. Who are you? What do you do? Maybe a little bit of how we got here – by we, I mean you.
Leisa: I am the Head of Research and Insights at Atlassian. Probably best known for creating software such as Jira and Confluence. Basically, tools that people use to make software. And then we also have Trello in our stable as well. So, there are a bunch of tools that are used by people who don’t make software as well. A whole bunch of stuff.
Steve: It seems like Jira and Confluence, if you’re any kind of software developer, those are just words you’re using, and terms from within those tools. It’s just part of the vocabulary.
Leisa: Yeah.
Steve: But if you’re outside, you maybe have never heard those words before?
Leisa: Exactly. Atlassian is quite a famous company in Australia because it’s kind of big and successful. But I think if you did a poll across Australia to find out who knew what Atlassian actually does, the brand awareness is high. The knowledge of what the company does is pretty low, unless you’re a software developer or a sort of project manager of software teams in which case you probably have heard of or used or have an opinion about Jira and probably Confluence as well.
Steve: And then Trello is used by people that aren’t necessarily software makers.
Leisa: Correct. A bunch of people do use it for making software as well, but it’s also used for people running – like in businesses, running non-technical projects and then a huge number of people just use it kind of personally – planning holidays, or weddings. I plan my kids weekends of Trello sometimes and I know I’m not alone. So, yeah it’s a real – it’s a very what we call a horizontal product.
Steve: A horizontal product can go into a lot of different industries.
Leisa: Exactly, exactly. I’m very skeptical about that term, by the way.
Steve: Horizontal? Yeah.
Leisa: Or the fact that it won’t necessarily be a good thing, but that’s another topic probably.
Steve: So, I can’t follow-up on that?
Leisa: Well, yeah, you can. Well, the problem with horizontal products, I think, is that they only do a certain amount for everybody and then people reach a point where they really want to be able to do more. And if your product is too horizontal then they will graduate to other products. And that gives you interesting business model challenges, I think. So, you have to be continually kind of seeking new people who only want to use your product up to a certain point in order to maintain your marketplace really.
Steve: When I think about my own small business and just any research I’ve done in small and just under medium sized businesses where everything is in Excel, sort of historically, where Excel is the – there may be a product, Cloud-based or otherwise, to do a thing. That someone has built kind of a custom Excel tool to do it. So, is Excel a horizontal product that way?
Leisa: I think so, yeah. In fact, I was talking to someone about this yesterday. I think that for a lot of people the first protocol for everything is a spreadsheet. They try to do everything that they can possibly do in a spreadsheet. And then there are some people who are like the, “ooh, new shiny tool. Let’s always try to find an application for a new shiny tool.” I think actually the vast majority of people take the first tool that they knew that had some kind of flexibility in it. So, if you can’t do it in Word or Excel – people will compromise a lot to be able to do things in tools that they have great familiarity with.
Steve: Yeah. But from the maker point of view you’re saying that a risk in the horizontalness, the lack of specificity, creates kind of a marketplace for the maker of the tool?
Leisa: Can do. I think for it to be successful you just have to be at such a huge scale to be able to always meet the needs of enough people. But I think things like Excel and Word and Trello, for example, they’ll always do some things for some people. Like just ‘cuz you moved to a more kind of sophisticated tool doesn’t mean that you completely abandon the old tool. You probably still use it for a bunch of things.
Steve: So, your title is Head of Research and Insights?
Leisa: Correct.
Steve: So, what’s the difference between research and insights?
Leisa: Yeah, good question. I didn’t make up my own title. I kind of inherited it. If I remember correctly, the way that it came about was that when I came into my role it was a new combination of people in the team in that we were bringing together the people who had been doing design research in the organization and the voice of the customer team who were effectively running the MPS. And I think because we were putting the two of them together it sounded weird to say research and voice of the customer. So, they went with research and insights instead. And, honestly, I haven’t spent any time really thinking about whether that was a good title or not. I’ve got other problems on my mind that are probably more pressing, so I’ve just kind of left it. If you want to get into it, I think research is the act of going out and gathering the data and pulling it altogether to make sense and insights. So, hopefully the sense that you get from it and we do try to do both of those things in my team, so I think it’s a reasonably accurate description.
Steve: How long have you been in this role?
Leisa: It’s about 18 months now.
Steve: If you look back on 18 months what are the notable things? For the experience of people inside the organization what has changed ins?
Leisa: Quite a few things. The shape and make up of the team has changed quite a lot. We’re bigger and differently composed to what we were previously. When I first came in it was literally made up of researchers who were working in products. Prior to me coming in they were reporting to design managers and we, for various reasons, pulled them out of products pretty quickly once I started. And then we had the other part of the business who were running MPS that was being gathered in product and we don’t do that anymore either. So, that team is doing quite different things now. So, we’ve changed a lot of things. We’ve introduced a couple of big programs of work as well. One of them being a big piece of work around top tasks. So, taking Gerry McGovern’s approach to top tasks. Trying to build that kind of foundational knowledge in the organization. So, that’s kind of new. There have always been those people doing Jobs To Be Done type stuff, but very very close to the product level. So, we’ve tried to pull back a little bit to really build a bigger understanding of what are the larger problems that we’re trying to solve for people and how might we see the opportunities to address those more effectively?
Steve: So, trying to create shared knowledge around what those top tasks are – I’m guessing for different products, different user scenarios.
Leisa: One of the things that we really tried to do is to get away from product top tasks and get more into really understanding what problems-based is the product, or combinations of products, trying to address. So, we don’t do top tasks for Jira. We do top tasks for agile software teams. And then through that we can then sort of ladder down to what that means for Jira or Confluence or Bitbucket or Trello, or whichever individual or combination of products we have. But it means that we sort of pull away a little bit from that feature focus. I think it can be very seductive and also very limiting.
Steve: What’s the size of the team now?
Leisa: I have a general rule in life of never count how many researchers you have in the organization because it’s always too many, according to whoever is asking you – not you, but like senior people. I think we’re probably around the mid-20s now.
Steve: And around the world, or around different locations?
Leisa: Mostly we’re in Sydney and California. So, we’re across three different offices there and we have a couple of people working remotely as well. So, I have somebody remote in Europe and another person in California who’s working out of a remote office too.
Steve: Can we sort of talk about the make up of the team evolving – what kinds of – without sort of enumerating, what are sort of the background, skillsets? Any way that you want to segment researchers, what kinds of – what’s the mix that you’re putting together?
Leisa: So, I think at the highest level we’ve got people who do design research, predominantly qualitative and they do a mixture of discovery work and evaluative work. We’ve got a team of what we call quantitative researchers and those are generally people who have got a marketing research type background and so they bring in a lot of those data and statistical skills that the other crew don’t necessarily have quite so much of. And then we have a research ops team as well who are doing some different things. And then we have a handful of research educators.
Steve: And what are the research educators doing?
Leisa: Educating people about how to do research.
Steve: These are hard questions and hard answers!
Leisa: Well, if you dig into it too much it gets complicated pretty quickly. So, I think the reality of research at Atlassian is the people in the research team probably do – best case – 20% of the research that happens in the organization and that’s probably an optimistic estimate as well. A huge amount of research is done by product managers and by designers and by other people in the organization, most of whom haven’t had any kind of training and so take a very intuitive approach to how they might do research. So, the research education team are really trying to help shape the way that this work is done so that it can be more effective.
Steve: I’m thinking about a talk I saw you give at the Mind the Product Conference where you began the talk – I think you kind of bookended the talk with a question that you didn’t answer – I don’t think you answered it definitively which is, is bad research better than no research, or words close to that. It was a great provocation to raise the question and then sort of talk about what the different aspects of that – how we might answer that? What the tradeoffs are? When you talk about this design education effort I can’t help but think that that’s connected to that question. If 80% of the research is done by people acting intuitively then yeah, how do you level up the quality of that?
Leisa: Absolutely.
Steve: Which implies that – well, I don’t know if it answers – does that answer the question? I’m not trying to gotcha here, but if you are trying to level up the quality that suggests that at some point better research is – I don’t know, there’s some equation here about bad vs. better vs. no. I’m not sure what the math of it is.
Leisa: So, this has been the real thing that’s occupied my mind a lot in the last couple of years really. And I’ve seen different organizations have really different kind of appetites for participating in research in different ways. I think at the time that I did that talk, which was probably, what?
Steve: Almost a year ago.
Leisa: Yeah. I think I was still trying to come to terms with exactly where I’ve felt – what I’ve felt about all of this, because as somebody who – well, as somebody in my role, everybody in an organization is – a lot of people in the organization are going to be watching you to see you trying to be overly precious and to become a gatekeeper or a bottleneck or all of these kinds of things. So, I’ve always felt like I had to be very careful about what you enabled and what you stopped because everyone has work that they need to get done, right. And the fact that people want to involve their customers and their users in the design process is something that I want to be seen to be supporting. Like I don’t want to be – I don’t want to stop that and I certainly don’t want to message that we shouldn’t have a closeness to our users and customers when we’re doing the research.
But, I’ve also seen a lot of practices that are done with the best of intentions that just get us to some crazy outcomes. And that really worries me. It worries me on two levels. It worries me in terms of the fact that we do that and we don’t move our products forward in the way that we could and it worries me because I think it reflects really poorly on research as a profession. I think most of us have seen situations where people have said well I did the research and nothing got better, so I’m not going to do it anymore. Clearly it’s a waste of time, right. And almost always what’s happened there is that the way that people have executed it has not been in a way that has helped them to see the things that they need to see or make the decisions that they need to make. So, it’s this really hard like to walk to try to understand how to enable, but how to enable in a way that is actually enabling in a positive way and is not actually facilitating really problematic outcomes. So, that’s, yeah – that’s my conundrum is balancing that. One the one hand I feel really uncomfortable saying no research is better than bad research. But on the other hand, I’ve seen plenty of evidence that makes me feel like actually maybe it’s true.
Steve: So, that’s kind of a time horizon there that the bad research may lead to the wrong decisions that impact the product and then sort of harm the prospects of it. I’m just reflecting back. That harm the prospects of a research to kind of go forward. Right, every researcher has heard that, “well we already knew that” response which to me is part of what you’re talking about. It’s that when research doesn’t – isn’t conducted in a way and sort of isn’t facilitated so that people have those learning moments where they – I think you said something about sort of helping them see the thing that’s going to help them make the right decision. And that’s – right, that’s not a – that’s different than what methodology do you use, or are you asking leading questions? Maybe leading questions are part of it because you confirm what you knew and so you can’t see something new because you’re not kind of surfacing it.
Leisa: Loads of it comes around framing, right. Loads of it comes around where are you in the organization? What are you focused on right now? What’s your remit? What’s your scope? What are you allowed to or interested in asking questions about? In a lot of cases this high volume of research comes from very downstream, very feature focused areas, right. So, if you’re working on a product that has got some more foundational issues that need to be addressed, but the vast majority of the work is happening at that very detailed feature level, how are you going to ever going to stop kind of circling the drain. You get stuck in this kind of local maxima. How are you ever going to take that big substantial step to really move your product forward if it’s nobody’s job, nobody’s priority, to do that. So, a lot of this is kind of structural. That so many of our people who are conducting this research are so close to the machine of delivery, and shipping and shipping and shipping as quickly as possible, that they don’t have the opportunity to think – to look sideways and see what’s happening on either side of their feature. Even when there are teams that are working on really similar things. They’ve run so heads down and feature driven. So, doing the least possible that you can to validate and then ship to learn, which is a whole other area of bad practice, I think, in many cases. It’s really limiting and you can see organizations that spend a huge amount of time and effort doing this research, but it’s at such a micro level that they don’t see the problems that their customers are dying to tell them about and they never ask about the problems that the customers are dying to tell them about because the customers just answer the questions that they asked and that’s kind of what bothers me. And so, it’s not about – in a lot of cases, some of it is about practice. Like I think it’s amazing how few people can just resist the temptation to say hey I’ve got this idea for a feature, how much would you like it? They can get 7 out of who said they loved my feature. Like that feels so definitive and so reliable and that’s very desirable and hard to resist. So, yes that happens.
But the bigger thing for me I think is where research is situated and the fact that you don’t have both that kind of big picture view as well as that micro view. We just have all of these fragments of microness and a huge amount of effort expended to do it. And I don’t feel like it’s helping us take those big steps forward.
Steve: But Leisa, all you need is a data repository so that you can surface those micro findings for the rest of the organization, right?
Leisa: You’re a troll Steve.
Steve: I am trolling you.
Leisa: You’re a troll, but again, I mean in some – kind of yes, but again, like all of those micro things don’t necessarily collectively give you the full view. And a lot of those micro things, because of the way they’re being asked, actually give you information that’s useless. If all of your questioning is around validating a particular feature that you’ve already decided that you think you want to do and you go in asking about that, you never really ask for the why? Like why would you use – like who is actually using this? Why would they use this? What actual real problem are they solving with this, right? So, the problem becomes the feature and then all of that research then becomes very disposable because the feature shifts and then nobody uses it as much as everybody thought they would, or they’re not as satisfied with it as what they thought they would be. So, we just keep iterating and iterating and iterating on these tiny things. We move buttons around. We change buttons. We like add more stuff in – surely that will fix it. But it’s because we haven’t taken that step back to actually understand the why? If you’re talking about those big user need level questions, the big whys, then you know what, you can put those things in a repository and you can build more and more detail into those because those tend to be long lasting.
And then at the other end, just the basic interaction design type stuff that we learn through research. A lot of those things don’t change much either. But it’s that bit in the middle, that feature level stuff, is the most disposable and often the least useful and I feel like that’s where we spend a huge amount of our time.
Steve: Do you have any theories as to why in large software enterprise technology companies that there is a focus on leaning heavily on that kind of feature validation research? What causes that?
Leisa: I think that there’s probably at least two things that I can think about that contribute to that. One is around what’s driving people? What gets people their promotions? What makes people look good in organizations? Shipping stuff – shipping stuff gets you good feedback when you’re going for your promotion. They want a list of the stuff that you’ve done, the stuff that you’ve shipped. In a lot of organizations just the fact that you’ve shipped it is enough. Nobody is actually following up to see whether or not it actually made a substantive difference in customers’ lives or not. So, I think that drive to ship is incredibly important. And our org structures as well, like the way that we divide teams up now, especially in organizations where you’ve got this kind of microservice platform kind of environment. You can have teams who’ve got huge dependencies on each other and they really have to try to componentize their work quite a lot. So, you have all of these kind of little micro teams who their customer’s experience is all of their work combined, but they all have different bosses with different KPIs or different OKRs or whatever the case may be. And I think that’s a problem. And then the other thing is this like build, measure, build – what is it? I’ve forgotten how you say it.
Steve: I’m the wrong person.
Leisa: Build/measure/learn.
Steve: Yeah, okay.
Leisa: The lean thing, right, which is this kind of idea that you can just build it and ship it and then learn. And that’s – that’s – that means that like if it had been learn/build/measure/learn we would be in a different situation, right, because we would be doing some discovery up front and then we would have a lot of stuff that we already knew before we had to ship it out to the customers. But it’s not. It’s build – you have an idea, build/measure/learn. And then people are often not particularly fussy about thinking about the learn bit. So, we ship it, put some analytics on it and we’ll put a feedback collector on it and then we’ll learn.
What are you going to learn? Whatever. What does success look like? I don’t know. It’s very – it’s kind of lazy and it makes – we treat our customers like lab rats when we do that. And I feel like they can tell. There’s lots of stuff that gets shipped that shouldn’t get shipped, that we know shouldn’t get shipped. I’m not talking about Atlassian in specific. I’m talking about in general. We all see it in our day to day digital lives that people ship stuff that we really should know better, but this build/measure/learn thing means that unless you can see it in the product analytics, unless you can see thousands of customers howling with anger and distress, it doesn’t count.
Steve: It reminds me of a client I worked with a few years ago where we had some pretty deep understandings of certain points of the interactions that were happening and what the value proposition was and where there was meaning. Just a lot of sort of – some pretty rich stuff. And the client was great because they kept introducing us to different product teams to help apply what we had learned to really, really specific decisions that were going to be made. But what we started to see – this pattern was they were interested in setting up experiments, not improving the product. And in fact, this is a product that has an annual use cycle. So, the time horizon for actually making changes was really far and some of the stuff was not rocket science. Like it was clear for this decision – A versus B versus C – like here’s the research said very clearly like this is what people care about, or what’s going to have impact, or what’s going to make them feel secure in these decisions. They were like great, we can conduct an experiment. We have three weeks to set up the experiment and then we’ll get some data. And I was – I just hadn’t really encountered that mindset before. I don’t know if they were really build/measure/learn literally, if that was their philosophy, but I didn’t know how to sort of help – I wasn’t able to move them to the way that I wanted it done. I’d really just encountered it for the first time and it seemed like there was a missed opportunity there. Like we knew how to improve the product and I wasn’t against – like they are conducting experiments and measuring and learnings – awesome. That’s research. But acting on what you’ve learned seems like why you’re doing the research in the first place.
Leisa: It feels as though we have this great toolset that we could be using, right? We’ve got going out and doing your kind of ethnography, contextual type stuff. And then right at the other end we’ve got the product analytics and running experiments and a whole bunch of stuff in between. And it really feels to me as though organizations tend to really just get excited about one thing and go hard on that one thing. And growth – doing the experiments is a big thing right now. I see loads of situations where we look at data and we go, oh look the graph is going in this direction. Well, the graph is not going in that direction. Why? And I’ll kind of – like there’s so much guessing behind all of that, right? And if it doesn’t go quite right, well then let’s have another guess, let’s have another guess, let’s have another guess. And this – like you say, there’s so much stuff that we probably already know if we could connect two parts of the organization together to take together more effectively. Or, there are methods that we could use to find out the why pretty quickly without having to just put another experiment, another experiment onto our customers, our users. But the knowledge of this toolset and the ability to choose the right tool and apply it, and to apply these approaches in combination seems to be where the challenge is.
Steve: What’s the role of design in this? In terms of here’s the thing that we know and here’s the thing we want to make. We haven’t talked about designers. Ideally for me, I sort of hope designers are the instruments of making that translation. Without design then you can sort of test implementations, but you can’t synthesize and make anything new necessarily.
Leisa: Well, I mean yeah. Theoretically design has a really important role to play here because I think design hopefully understands the users in a design process better than anybody else. Understands the opportunities for iteration and levels of fidelity for exploring problems in a way that nobody else on the team does. And lots of designers do know that. But the time pressure is enormous, especially in these kind of larger tech companies where a lot of times designers – designers are also concerned about being bottlenecks. They have to feed the engineering machine. And it can be really difficult for them to have the conversations to talk about all of the work that we should be doing before we ship something. So, I feel as though they have a lot of pressure on them, a lot of time pressure on them. They’re being pressured to really contract the amount of effort that they put in before we ship something. And there is – yeah, there’s this huge desire amongst product teams and their bosses to just get something shipped, get something live and then we’ll learn that I think design really struggles with. And I don’t know – it can be really difficult in those kinds of environments to be the person who stands up and says we need to slow down and we need to do less and do it better. So, I have empathy for designers in their inability to shift the system – these problems – necessarily because of this big pressure that’s being put on them just to keep the engineers coding.
Steve: Right. If you say you want more time to work on something that’s – what are the risks to you for doing that?
Leisa: Exactly, exactly. On a personal level everyone wants to look good in front of their boss. Everyone would like a pay raise and a promotion and the easiest way to get those things is to do what you’re told and ship the stuff. Keep everyone busy, produce the shiny stuff, let it get coded up, let it go live, learn, carry on. That’s thinking short term. That’s how to have a happy life. Is that how you’re going to fundamentally improve the product or service that you’re working on? Probably not. But it takes a lot of bravery to try to change that, to try to stop this crazy train that’s out of control. Throw yourself in front of the bus. All that kind of stuff. Like you said, it’s hard, especially when there are loads of people all around you who are very happy to keep doing it the way that it’s being done right now. So, yeah. And I think that’s research’s role, that’s design’s role. I would like that to be PM’s role as well. And most engineers are – a lot of engineers are very, very, very interested in making sure that the work that they’re doing is actually solving real problems and delivering real value as well.
Steve: So, as you point to the individuals there’s a lot of shared objectives. But if you point the system, which is – it’s the system of – it’s the rewards system and the incentive system, but there’s also some – I think there’s just sort of what day to day looks like. Sort of the operations of the system of producing technology products.
Leisa: I think there’s also – there’s something about like what do people like to see? What gets people excited, right? And graphs pointing upwards in the right direction is really exciting. Like certain outcomes are really exciting. Ambiguous outcomes, really not exciting at all. Having things that go fast, very exciting. Things that go slow, not exciting. So, I think there are all of these things that this collection of humans, that form an organization have a strong bias towards, that we get really excited about, that don’t necessary help us in the long run. You know you see lots of people who get really excited about the graph. Very few people dig in behind the graph to the data. Where did this data come from? How much can I actually believe it? Like people are so willing to just accept experiment findings without actually digging in behind it. And a lot of the time, when you do dig in behind it, you go huh, that doesn’t look very reliable at all. So, there’s something about we love to tell ourselves that we’re data informed or data driven, but a huge number of the people who are excited about that don’t have very much data literacy to be able to check in and make sure that the stuff they’re getting excited about is actually reliable.
Steve: So, how could someone improve their data literacy?
Leisa: I think that there’s a lot of work that we need to do to make sure that we actually understand how experiments should be structured. And understand more about – being more curious about where is this data coming from and what are the different ways that we could get tricked by this, right? And there are tons of like books and papers and all kinds of things on the subject. But actually, when you go looking for and you’re coming at it with a critical mind, rather than with a mind that gets excited by graphs, you know a lot of it is pretty logical. When you think about like – like surveys, right? Who did this survey actually – who actually answered these questions? Instead of just going hey, there’s an answer that supports my argument, I’m just going to grab that. To dig behind it and go like are these – are the people who are being surveyed actually the same people that we’re talking about when we’re making this decision. Is there something about the nature of those group of people that is going to bias their response? It’s remarkable to me how few people actually kind of check the source to make sure that it’s worthy of relying on. Everyone – a lot of people are really keen to just grab whatever data they can get, but that supports their argument. I think this is another one of those kind of human things, those human inclinations that we have that lead us towards kind of bad behaviors.
Steve: I can imagine that with this research education effort that you’re doing that people that participate in that are going to learn how to make better choices in the research that they then run, some practical skills, some planning, some framing, as you talked about. But it seems like that literacy is a likely side effect of that, or maybe it’s not the side effect. Maybe it’s the effect. How to be better consumers of research. Once you understand how the sausage is made a little bit then you understand, oh yeah, that’s a biased question, or that’s bad sampling, and there’s a choice to make in all of these and we have to question those choices to understand the research that’s presented to us. I hadn’t really thought of training people to do research as a way to help then in their consumption or critical thinking around research.
Leisa: Something else that we’ve really come to realize recently as well is that because of all of the other pressures that the people who are making these decisions are having to deal with as well, we think we’ll do much better if we train the entire team together instead of just training the people who are tasked with doing the research. Because something that we’ve observed is that you can train people and they can go this is great – I really see what I was doing before was creating these kind of not great outcomes and what I need to do to do better. And then you’ll see them, not that long later, doing exactly the opposite to what we thought we had agreed we were going to do going forward. And we’re like well what’s happening? These are like smart, good people who have been given the information and are still doing crazy stuff. Like what’s going on with that? And you realize they go back into their context where everyone is just trying to drive to get stuff done faster, faster, faster, faster, and you have to plan to do good research. The easiest research to do really, really quickly is the crappy research. If you want to do good research you do have to do some planning. So, you need to get into a proactive mindset for it rather than a reactive mindset for it. And you need the entire team to be able to do that. So, one of the things that we’re looking to do moving forward is not just to train the designers and the PMs, but actually to train a ton of people all around them in the teams to help them understand that the way that you ask the questions and who you ask them of and where – all the different things that could impact the reliability of your research – requires planning and thinking about in advance. So, we hope that means that the whole team will understand the importance of taking the time to do it and it won’t just be like one or two people in a large team fighting to do the right thing and being pressured by everybody else. So, I think it is – the education aspect, I suspect, is super important and it goes way beyond just helping people who are doing research to do better research. It goes to helping the whole organization to understand the impact and risk that goes with doing a lot of research activity in the wrong way or the wrong place.
Steve: Just fascinating to me and I think a big challenge for all of us that lots of researchers are involved in education of one form or another – workshops – a big program like you’ve been building. But most of us are not trained educators. We don’t have a pedagogical, theoretical background about – it’s about communicating information. It’s about influence and changing minds. And it just seems like a lot of researchers, from an individual researcher on a product team to someone like you that’s looking at the whole organization, we’re sort of experimenting and trying to build tools and processes that help people learn. And learn is not just imparting the information, but reframe and empower, like these big words where – I wish I had – I wish you had a PhD in education. I wish I had that. Or that I had been a college professor for a number of years, or whatever it would take – however I would get that level of insight. Personally, I have none of that. So, to hear us – you know we all talk about these kinds of things. I think research gives you some skill and prototyping and iterating and measuring in order to make the kinds of changes in the implementation that your making. I don’t know about you. I feel like I am amateurish, I guess, in terms of my educational theory.
Leisa: Absolutely. And I think talking to the team who are doing this work, like it’s really, really, really hard – really hard work to come up with the best way to share this knowledge with people in your organizational context, in a way that is always time constrained. At Atlassian we have a long history of doing kind of internal training. We have this thing called bootcamps. We’ve got almost like a voluntary system where people come in and they run these bootcamps and you can go and learn about the finer details of advanced Jira administration or you can come in and learn about how to do a customer interview. But the timeframe around that was like – like a two-hour bootcamp was a long bootcamp. Most bootcamps are like an hour. And so, when we started thinking about this we were like we’re going to need at least a day, maybe two days. And everyone was like nobody will turn up. But yeah – fortunately we have – we do day long sessions and people have been turning up. So, that’s great. But yeah, it’s a huge effort to try to come up with something that works well. And every time we do these courses the trainers and educators go away and think about what worked and how can we do it better. So, we iterate every single time. So, yeah, it’s a huge amount of effort. I think in larger organizations too, there are other people in the organizations who are also tasked with learning type stuff. So, we have a couple of different teams at Atlassian who are kind of involved in helping educate within the organization. So, we’re building a lot of relationships with different parts of the org than we have before in order to try to get some support and even just infrastructure. Like there’s just a whole lot of like logistics behind setting this up if you want to do it in any kind of scale. It’s great to have that support internally and it’s really good to start to build these relationships across the organization in different ways. But yeah, I think that we certainly underestimated the challenge of just designing what education looks like and how to make it run well. It’s a massive, massive effort.
Steve: It’s not the material – you guys probably have at hand a pretty good set of here’s how to go do this. If you brought an intern onto your team you could probably get them up to speed with material that you have, but you’re trying to change a culture. And I think the advantage that I have, the context that I have as an external educator is that people are opting in and that I have to by the assumption that they want to be there and I don’t have access to what the outcomes are. I might through someone that’s my gatekeeper, but it’s kind of one them. I have the responsibility for the training, but not the responsibility for the outcomes, which is what you all are kind of working with. So, I envy you and I don’t envy you.
Leisa: Well, I think I – I like it because, you know, going back to the build/measure/learn thing, right. Again, we did a learn before we did our build and measure because we’re researchers and that’s what we do, but it is – it’s super interesting to see the behaviors of people who have come through the training and see whether they shift or not. That gives us – we get feedback from people after the course who tell us whether they thought it was useful or not useful and what they liked and didn’t like, but then we actually get to observe because through our ops team, they come through us to do their recruiting and get their incentives. So, we can keep a little bit more of an eye on what activity is coming out and see if there is any sort of shifting in that as well. And it’s not just courses as well. It’s thinking about like what’s the overall ecosystem? What are other things that we can be doing where we are sort of reaching out to help support people on this journey as well. Before we did educating, we had advisors who kind of made themselves available to go and sort of support teams who recognized that they might have a need for some help with getting their research right. So, that was kind of our first attempt. But we had to pivot into educating because of the time factor. We would go in and give advice and everybody would go it’s great advice. We’d totally love to do that, but I have to get this done by next Thursday. So, I’m going to ignore your advice for now and carry on with what I was going to do anyway. And that was pretty frustrating. So, we felt like we have to invest in trying to get ahead of the curve a little bit – try to get ahead. Try to not necessarily influence the stuff that has to happen by next Thursday, but try to encourage teams to start being proactive and planning to do research instead of doing this really reactive work instead. Or as well as the reactive work perhaps. I don’t know.
Steve: Have the right mix.
Leisa: Yeah.
Steve: I wonder if – and this is probably not going to turn out to be true, but I wonder about being proactive and planning versus time. The time pressure to me is about oh we only have so many hours to spend on this, or that calendar wise we need to be there next Thursday. But being proactive says, well if we started thinking about it three weeks ago we’d be ready for it Thursday to do it “the right way.” I’m wondering, can we tease apart the pressures. One is like no proactivity, the sort of very short-term kind of thinking, is different than hours required. Is that true?
Leisa: I think so. Because I think that even want to do the short-term stuff we still spend like quite a lot of time and effort on it. And the planning in advance, like the more proactive work doesn’t necessarily, I don’t think, entail more actual work. It just might be that you put in your recruitment request a couple of weeks beforehand so that we can try to make sure that the people that you meet are the right kinds of people, instead of if you have to have it done in 3 or 4 days time then your ability to be careful and selective in terms of who you recruit to participate in the research is very much limited. So, we see – when we have those kind of time constraints you see everybody just going to unmoderated usability testing and their panel and that introduces a whole lot of problems in terms of what you’re able to learn and how reliable that might be. Yeah. I was thinking for a second about you know when – theoretically when you do your unmoderated usability testing you should still be watching the videos, right. So, that should take as much time as watching a handful of carefully recruited people being – doing these sessions in a facilitated way. But the reality is, I think, that most people don’t watch the videos, which speaks to quality.
Steve: Here we are back again. It seems like there’s a, for the profession overall, maybe one way to start sort of framing the way around this time pressure thing is to decouple proactiveness versus sort of hours burned. That it’s going to be the similar number of hours burned, but if you start earlier the quality goes up. I had never really thought about those two things as being separate.
Leisa: Yeah. And I don’t think people do. I think when people think about – and this is – I mean I sort of said the word quality. I’m trying not to say quality anymore. I’m trying to talk about meaningfulness more now, I think, because whenever you talk about quality to pushback that you get is well it doesn’t need to be perfect, so it doesn’t need to be academic. I just need enough data to be able to make a decision and understand that. But then I see the data upon which they’re making the decision and that makes me worry, right? And I think that’s – we want to get out of this discussion of like quality levels, more into sort of reliability levels, like how reliable does it need to be? Because surely there has to be a bar of reliability that you have to meet before you feel like you’ve got that information that you need to make a decision. But I see loads of people making decisions off the back of really dreadful and misleading data and that’s what worries me. And they feel confident with that data – going back to the data literacy problem. Like they really haven’t dug into why they might be given really misleading answers as a result of the who and the how and the why, all those decisions that are made around how to do the research, most of which have been driven by time constraints.
Steve: Okay, that’s the end of part one of my interview with Leisa. What a cliffhanger! There’s more to come from Leisa in the next episode of Dollars to Donuts. Meanwhile, subscribe to the podcast at portigal dot com slash podcast, or go to your favorite podcasting tool like Apple Podcasts, Stitcher or Spotify, among others. The website has transcripts, show notes, and the complete set of episodes. Follow the podcast on Twitter, and buy my books Interviewing Users and Doorbells Danger and Dead Batteries from Rosenfeld Media or Amazon. Thanks to Bruce Todd for the Dollars to Donuts theme music.
The post 19. Leisa Reichelt of Atlassian (Part 1) first appeared on Portigal Consulting.
This episode of Dollars to Donuts features part 1 of my two-part conversation with Leisa Reichelt of Atlassian. We talk about educating the organization to do user research better, the limitations of horizontal products, and the tension between “good” and “bad” research.
If you’re working on a product that has got some more foundational issues that need to be addressed, but the vast majority of the work is happening at that very detailed feature level, how are you going to ever going to stop kind of circling the drain? You get stuck in this kind of local maxima. How are you ever going to take that big substantial step to really move your product forward if it’s nobody’s job, nobody’s priority, to do that? – Leisa Reichelt
Show Links* ~~Fundamentals of Interviewing Users (SF)~~ * The Art of Noticing, by Rob Walker * The Art of Noticing newsletter * Objectified * Did you see that? Tapping into your super-noticing power * Leisa on LinkedIn * Leisa on Twitter * Atlassian * Jira * Confluence * Trello * Quantifying Qualitative Research – Mind the Product * Gerry McGovern’s Top Tasks * What is Jobs to be Done (JTBD)? * Build Measure Learn
Follow Dollars to Donuts on Twitter and help other people discover the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Howdy, and here we are with another episode of Dollars to Donuts, the podcast where I talk to the people who are leading user research in their organization.
I just taught a public workshop in New York City about user research, organized by Rosenfeld Media. But don’t despair – this workshop, Fundamentals of Interviewing Users is also happening September 13th in San Francisco. I’ll put the link in the show notes. Send your team! Recommend this workshop to your friends!
If you aren’t in San Francisco, or you can’t make it September 13th, you can hire me to come into your organization and lead a training workshop. Recently I’ve taught classes for companies in New York City, and coming up will be San Diego, as well as the Midwest, and Texas. I’d love to talk with you about coming into your company and teaching people about research.
As always, a reminder that supporting me in my business is a way to support this podcast and ensure that I can keep making new episodes. If you have feedback about the podcast, I’d love to hear from you at DONUTS AT PORTIGAL DOT COM or on Twitter at Dollars To Donuts, that’s d o l l R s T O D o n u t s.
I was pretty excited this week to receive a new book in the mail. It’s called The Art of Noticing, by Rob Walker, whose name you may recognize from his books, or New York Times columns, or his appearance in Gary Hustwit’s documentary “Objectified.” I’ve only just started the book but I am eager to read it, which is not something I say that often about a book of non-fiction. The book is structured around 131 different exercises to practice noticing. Each page has really great pull-quotes and the exercises seem to come from a bunch of interesting sources like people who focus on creativity or art or storytelling. Rob also publishes a great newsletter with lots of additional tips and examples around noticing, and I’ve even sent him a few references that he’s included. I’ll put all this info in the notes for this episode. This topic takes me back to a workshop I ran a few years ago at one of the first user research conferences I ever attended, called About, With and For. The workshop is about noticing and II wonder if it’s time to revisit that workshop, and I can look to Rob’s book as a new resource.
Well, let’s get to the episode. I had a fascinating and in-depth conversation with Leisa Reichelt. She is the Head of Research and Insights at Atlassian in Sydney Australia. Our interview went on for a long time and I’m going to break it up into two parts. So, let’s get to part one here. Thank you very much for being here.
Leisa Reichelt: Thank you for inviting me.
Steve: Let’s start with maybe some background introduction. Who are you? What do you do? Maybe a little bit of how we got here – by we, I mean you.
Leisa: I am the Head of Research and Insights at Atlassian. Probably best known for creating software such as Jira and Confluence. Basically, tools that people use to make software. And then we also have Trello in our stable as well. So, there are a bunch of tools that are used by people who don’t make software as well. A whole bunch of stuff.
Steve: It seems like Jira and Confluence, if you’re any kind of software developer, those are just words you’re using, and terms from within those tools. It’s just part of the vocabulary.
Leisa: Yeah.
Steve: But if you’re outside, you maybe have never heard those words before?
Leisa: Exactly. Atlassian is quite a famous company in Australia because it’s kind of big and successful. But I think if you did a poll across Australia to find out who knew what Atlassian actually does, the brand awareness is high. The knowledge of what the company does is pretty low, unless you’re a software developer or a sort of project manager of software teams in which case you probably have heard of or used or have an opinion about Jira and probably Confluence as well.
Steve: And then Trello is used by people that aren’t necessarily software makers.
Leisa: Correct. A bunch of people do use it for making software as well, but it’s also used for people running – like in businesses, running non-technical projects and then a huge number of people just use it kind of personally – planning holidays, or weddings. I plan my kids weekends of Trello sometimes and I know I’m not alone. So, yeah it’s a real – it’s a very what we call a horizontal product.
Steve: A horizontal product can go into a lot of different industries.
Leisa: Exactly, exactly. I’m very skeptical about that term, by the way.
Steve: Horizontal? Yeah.
Leisa: Or the fact that it won’t necessarily be a good thing, but that’s another topic probably.
Steve: So, I can’t follow-up on that?
Leisa: Well, yeah, you can. Well, the problem with horizontal products, I think, is that they only do a certain amount for everybody and then people reach a point where they really want to be able to do more. And if your product is too horizontal then they will graduate to other products. And that gives you interesting business model challenges, I think. So, you have to be continually kind of seeking new people who only want to use your product up to a certain point in order to maintain your marketplace really.
Steve: When I think about my own small business and just any research I’ve done in small and just under medium sized businesses where everything is in Excel, sort of historically, where Excel is the – there may be a product, Cloud-based or otherwise, to do a thing. That someone has built kind of a custom Excel tool to do it. So, is Excel a horizontal product that way?
Leisa: I think so, yeah. In fact, I was talking to someone about this yesterday. I think that for a lot of people the first protocol for everything is a spreadsheet. They try to do everything that they can possibly do in a spreadsheet. And then there are some people who are like the, “ooh, new shiny tool. Let’s always try to find an application for a new shiny tool.” I think actually the vast majority of people take the first tool that they knew that had some kind of flexibility in it. So, if you can’t do it in Word or Excel – people will compromise a lot to be able to do things in tools that they have great familiarity with.
Steve: Yeah. But from the maker point of view you’re saying that a risk in the horizontalness, the lack of specificity, creates kind of a marketplace for the maker of the tool?
Leisa: Can do. I think for it to be successful you just have to be at such a huge scale to be able to always meet the needs of enough people. But I think things like Excel and Word and Trello, for example, they’ll always do some things for some people. Like just ‘cuz you moved to a more kind of sophisticated tool doesn’t mean that you completely abandon the old tool. You probably still use it for a bunch of things.
Steve: So, your title is Head of Research and Insights?
Leisa: Correct.
Steve: So, what’s the difference between research and insights?
Leisa: Yeah, good question. I didn’t make up my own title. I kind of inherited it. If I remember correctly, the way that it came about was that when I came into my role it was a new combination of people in the team in that we were bringing together the people who had been doing design research in the organization and the voice of the customer team who were effectively running the MPS. And I think because we were putting the two of them together it sounded weird to say research and voice of the customer. So, they went with research and insights instead. And, honestly, I haven’t spent any time really thinking about whether that was a good title or not. I’ve got other problems on my mind that are probably more pressing, so I’ve just kind of left it. If you want to get into it, I think research is the act of going out and gathering the data and pulling it altogether to make sense and insights. So, hopefully the sense that you get from it and we do try to do both of those things in my team, so I think it’s a reasonably accurate description.
Steve: How long have you been in this role?
Leisa: It’s about 18 months now.
Steve: If you look back on 18 months what are the notable things? For the experience of people inside the organization what has changed ins?
Leisa: Quite a few things. The shape and make up of the team has changed quite a lot. We’re bigger and differently composed to what we were previously. When I first came in it was literally made up of researchers who were working in products. Prior to me coming in they were reporting to design managers and we, for various reasons, pulled them out of products pretty quickly once I started. And then we had the other part of the business who were running MPS that was being gathered in product and we don’t do that anymore either. So, that team is doing quite different things now. So, we’ve changed a lot of things. We’ve introduced a couple of big programs of work as well. One of them being a big piece of work around top tasks. So, taking Gerry McGovern’s approach to top tasks. Trying to build that kind of foundational knowledge in the organization. So, that’s kind of new. There have always been those people doing Jobs To Be Done type stuff, but very very close to the product level. So, we’ve tried to pull back a little bit to really build a bigger understanding of what are the larger problems that we’re trying to solve for people and how might we see the opportunities to address those more effectively?
Steve: So, trying to create shared knowledge around what those top tasks are – I’m guessing for different products, different user scenarios.
Leisa: One of the things that we really tried to do is to get away from product top tasks and get more into really understanding what problems-based is the product, or combinations of products, trying to address. So, we don’t do top tasks for Jira. We do top tasks for agile software teams. And then through that we can then sort of ladder down to what that means for Jira or Confluence or Bitbucket or Trello, or whichever individual or combination of products we have. But it means that we sort of pull away a little bit from that feature focus. I think it can be very seductive and also very limiting.
Steve: What’s the size of the team now?
Leisa: I have a general rule in life of never count how many researchers you have in the organization because it’s always too many, according to whoever is asking you – not you, but like senior people. I think we’re probably around the mid-20s now.
Steve: And around the world, or around different locations?
Leisa: Mostly we’re in Sydney and California. So, we’re across three different offices there and we have a couple of people working remotely as well. So, I have somebody remote in Europe and another person in California who’s working out of a remote office too.
Steve: Can we sort of talk about the make up of the team evolving – what kinds of – without sort of enumerating, what are sort of the background, skillsets? Any way that you want to segment researchers, what kinds of – what’s the mix that you’re putting together?
Leisa: So, I think at the highest level we’ve got people who do design research, predominantly qualitative and they do a mixture of discovery work and evaluative work. We’ve got a team of what we call quantitative researchers and those are generally people who have got a marketing research type background and so they bring in a lot of those data and statistical skills that the other crew don’t necessarily have quite so much of. And then we have a research ops team as well who are doing some different things. And then we have a handful of research educators.
Steve: And what are the research educators doing?
Leisa: Educating people about how to do research.
Steve: These are hard questions and hard answers!
Leisa: Well, if you dig into it too much it gets complicated pretty quickly. So, I think the reality of research at Atlassian is the people in the research team probably do – best case – 20% of the research that happens in the organization and that’s probably an optimistic estimate as well. A huge amount of research is done by product managers and by designers and by other people in the organization, most of whom haven’t had any kind of training and so take a very intuitive approach to how they might do research. So, the research education team are really trying to help shape the way that this work is done so that it can be more effective.
Steve: I’m thinking about a talk I saw you give at the Mind the Product Conference where you began the talk – I think you kind of bookended the talk with a question that you didn’t answer – I don’t think you answered it definitively which is, is bad research better than no research, or words close to that. It was a great provocation to raise the question and then sort of talk about what the different aspects of that – how we might answer that? What the tradeoffs are? When you talk about this design education effort I can’t help but think that that’s connected to that question. If 80% of the research is done by people acting intuitively then yeah, how do you level up the quality of that?
Leisa: Absolutely.
Steve: Which implies that – well, I don’t know if it answers – does that answer the question? I’m not trying to gotcha here, but if you are trying to level up the quality that suggests that at some point better research is – I don’t know, there’s some equation here about bad vs. better vs. no. I’m not sure what the math of it is.
Leisa: So, this has been the real thing that’s occupied my mind a lot in the last couple of years really. And I’ve seen different organizations have really different kind of appetites for participating in research in different ways. I think at the time that I did that talk, which was probably, what?
Steve: Almost a year ago.
Leisa: Yeah. I think I was still trying to come to terms with exactly where I’ve felt – what I’ve felt about all of this, because as somebody who – well, as somebody in my role, everybody in an organization is – a lot of people in the organization are going to be watching you to see you trying to be overly precious and to become a gatekeeper or a bottleneck or all of these kinds of things. So, I’ve always felt like I had to be very careful about what you enabled and what you stopped because everyone has work that they need to get done, right. And the fact that people want to involve their customers and their users in the design process is something that I want to be seen to be supporting. Like I don’t want to be – I don’t want to stop that and I certainly don’t want to message that we shouldn’t have a closeness to our users and customers when we’re doing the research.
But, I’ve also seen a lot of practices that are done with the best of intentions that just get us to some crazy outcomes. And that really worries me. It worries me on two levels. It worries me in terms of the fact that we do that and we don’t move our products forward in the way that we could and it worries me because I think it reflects really poorly on research as a profession. I think most of us have seen situations where people have said well I did the research and nothing got better, so I’m not going to do it anymore. Clearly it’s a waste of time, right. And almost always what’s happened there is that the way that people have executed it has not been in a way that has helped them to see the things that they need to see or make the decisions that they need to make. So, it’s this really hard like to walk to try to understand how to enable, but how to enable in a way that is actually enabling in a positive way and is not actually facilitating really problematic outcomes. So, that’s, yeah – that’s my conundrum is balancing that. One the one hand I feel really uncomfortable saying no research is better than bad research. But on the other hand, I’ve seen plenty of evidence that makes me feel like actually maybe it’s true.
Steve: So, that’s kind of a time horizon there that the bad research may lead to the wrong decisions that impact the product and then sort of harm the prospects of it. I’m just reflecting back. That harm the prospects of a research to kind of go forward. Right, every researcher has heard that, “well we already knew that” response which to me is part of what you’re talking about. It’s that when research doesn’t – isn’t conducted in a way and sort of isn’t facilitated so that people have those learning moments where they – I think you said something about sort of helping them see the thing that’s going to help them make the right decision. And that’s – right, that’s not a – that’s different than what methodology do you use, or are you asking leading questions? Maybe leading questions are part of it because you confirm what you knew and so you can’t see something new because you’re not kind of surfacing it.
Leisa: Loads of it comes around framing, right. Loads of it comes around where are you in the organization? What are you focused on right now? What’s your remit? What’s your scope? What are you allowed to or interested in asking questions about? In a lot of cases this high volume of research comes from very downstream, very feature focused areas, right. So, if you’re working on a product that has got some more foundational issues that need to be addressed, but the vast majority of the work is happening at that very detailed feature level, how are you going to ever going to stop kind of circling the drain. You get stuck in this kind of local maxima. How are you ever going to take that big substantial step to really move your product forward if it’s nobody’s job, nobody’s priority, to do that. So, a lot of this is kind of structural. That so many of our people who are conducting this research are so close to the machine of delivery, and shipping and shipping and shipping as quickly as possible, that they don’t have the opportunity to think – to look sideways and see what’s happening on either side of their feature. Even when there are teams that are working on really similar things. They’ve run so heads down and feature driven. So, doing the least possible that you can to validate and then ship to learn, which is a whole other area of bad practice, I think, in many cases. It’s really limiting and you can see organizations that spend a huge amount of time and effort doing this research, but it’s at such a micro level that they don’t see the problems that their customers are dying to tell them about and they never ask about the problems that the customers are dying to tell them about because the customers just answer the questions that they asked and that’s kind of what bothers me. And so, it’s not about – in a lot of cases, some of it is about practice. Like I think it’s amazing how few people can just resist the temptation to say hey I’ve got this idea for a feature, how much would you like it? They can get 7 out of who said they loved my feature. Like that feels so definitive and so reliable and that’s very desirable and hard to resist. So, yes that happens.
But the bigger thing for me I think is where research is situated and the fact that you don’t have both that kind of big picture view as well as that micro view. We just have all of these fragments of microness and a huge amount of effort expended to do it. And I don’t feel like it’s helping us take those big steps forward.
Steve: But Leisa, all you need is a data repository so that you can surface those micro findings for the rest of the organization, right?
Leisa: You’re a troll Steve.
Steve: I am trolling you.
Leisa: You’re a troll, but again, I mean in some – kind of yes, but again, like all of those micro things don’t necessarily collectively give you the full view. And a lot of those micro things, because of the way they’re being asked, actually give you information that’s useless. If all of your questioning is around validating a particular feature that you’ve already decided that you think you want to do and you go in asking about that, you never really ask for the why? Like why would you use – like who is actually using this? Why would they use this? What actual real problem are they solving with this, right? So, the problem becomes the feature and then all of that research then becomes very disposable because the feature shifts and then nobody uses it as much as everybody thought they would, or they’re not as satisfied with it as what they thought they would be. So, we just keep iterating and iterating and iterating on these tiny things. We move buttons around. We change buttons. We like add more stuff in – surely that will fix it. But it’s because we haven’t taken that step back to actually understand the why? If you’re talking about those big user need level questions, the big whys, then you know what, you can put those things in a repository and you can build more and more detail into those because those tend to be long lasting.
And then at the other end, just the basic interaction design type stuff that we learn through research. A lot of those things don’t change much either. But it’s that bit in the middle, that feature level stuff, is the most disposable and often the least useful and I feel like that’s where we spend a huge amount of our time.
Steve: Do you have any theories as to why in large software enterprise technology companies that there is a focus on leaning heavily on that kind of feature validation research? What causes that?
Leisa: I think that there’s probably at least two things that I can think about that contribute to that. One is around what’s driving people? What gets people their promotions? What makes people look good in organizations? Shipping stuff – shipping stuff gets you good feedback when you’re going for your promotion. They want a list of the stuff that you’ve done, the stuff that you’ve shipped. In a lot of organizations just the fact that you’ve shipped it is enough. Nobody is actually following up to see whether or not it actually made a substantive difference in customers’ lives or not. So, I think that drive to ship is incredibly important. And our org structures as well, like the way that we divide teams up now, especially in organizations where you’ve got this kind of microservice platform kind of environment. You can have teams who’ve got huge dependencies on each other and they really have to try to componentize their work quite a lot. So, you have all of these kind of little micro teams who their customer’s experience is all of their work combined, but they all have different bosses with different KPIs or different OKRs or whatever the case may be. And I think that’s a problem. And then the other thing is this like build, measure, build – what is it? I’ve forgotten how you say it.
Steve: I’m the wrong person.
Leisa: Build/measure/learn.
Steve: Yeah, okay.
Leisa: The lean thing, right, which is this kind of idea that you can just build it and ship it and then learn. And that’s – that’s – that means that like if it had been learn/build/measure/learn we would be in a different situation, right, because we would be doing some discovery up front and then we would have a lot of stuff that we already knew before we had to ship it out to the customers. But it’s not. It’s build – you have an idea, build/measure/learn. And then people are often not particularly fussy about thinking about the learn bit. So, we ship it, put some analytics on it and we’ll put a feedback collector on it and then we’ll learn.
What are you going to learn? Whatever. What does success look like? I don’t know. It’s very – it’s kind of lazy and it makes – we treat our customers like lab rats when we do that. And I feel like they can tell. There’s lots of stuff that gets shipped that shouldn’t get shipped, that we know shouldn’t get shipped. I’m not talking about Atlassian in specific. I’m talking about in general. We all see it in our day to day digital lives that people ship stuff that we really should know better, but this build/measure/learn thing means that unless you can see it in the product analytics, unless you can see thousands of customers howling with anger and distress, it doesn’t count.
Steve: It reminds me of a client I worked with a few years ago where we had some pretty deep understandings of certain points of the interactions that were happening and what the value proposition was and where there was meaning. Just a lot of sort of – some pretty rich stuff. And the client was great because they kept introducing us to different product teams to help apply what we had learned to really, really specific decisions that were going to be made. But what we started to see – this pattern was they were interested in setting up experiments, not improving the product. And in fact, this is a product that has an annual use cycle. So, the time horizon for actually making changes was really far and some of the stuff was not rocket science. Like it was clear for this decision – A versus B versus C – like here’s the research said very clearly like this is what people care about, or what’s going to have impact, or what’s going to make them feel secure in these decisions. They were like great, we can conduct an experiment. We have three weeks to set up the experiment and then we’ll get some data. And I was – I just hadn’t really encountered that mindset before. I don’t know if they were really build/measure/learn literally, if that was their philosophy, but I didn’t know how to sort of help – I wasn’t able to move them to the way that I wanted it done. I’d really just encountered it for the first time and it seemed like there was a missed opportunity there. Like we knew how to improve the product and I wasn’t against – like they are conducting experiments and measuring and learnings – awesome. That’s research. But acting on what you’ve learned seems like why you’re doing the research in the first place.
Leisa: It feels as though we have this great toolset that we could be using, right? We’ve got going out and doing your kind of ethnography, contextual type stuff. And then right at the other end we’ve got the product analytics and running experiments and a whole bunch of stuff in between. And it really feels to me as though organizations tend to really just get excited about one thing and go hard on that one thing. And growth – doing the experiments is a big thing right now. I see loads of situations where we look at data and we go, oh look the graph is going in this direction. Well, the graph is not going in that direction. Why? And I’ll kind of – like there’s so much guessing behind all of that, right? And if it doesn’t go quite right, well then let’s have another guess, let’s have another guess, let’s have another guess. And this – like you say, there’s so much stuff that we probably already know if we could connect two parts of the organization together to take together more effectively. Or, there are methods that we could use to find out the why pretty quickly without having to just put another experiment, another experiment onto our customers, our users. But the knowledge of this toolset and the ability to choose the right tool and apply it, and to apply these approaches in combination seems to be where the challenge is.
Steve: What’s the role of design in this? In terms of here’s the thing that we know and here’s the thing we want to make. We haven’t talked about designers. Ideally for me, I sort of hope designers are the instruments of making that translation. Without design then you can sort of test implementations, but you can’t synthesize and make anything new necessarily.
Leisa: Well, I mean yeah. Theoretically design has a really important role to play here because I think design hopefully understands the users in a design process better than anybody else. Understands the opportunities for iteration and levels of fidelity for exploring problems in a way that nobody else on the team does. And lots of designers do know that. But the time pressure is enormous, especially in these kind of larger tech companies where a lot of times designers – designers are also concerned about being bottlenecks. They have to feed the engineering machine. And it can be really difficult for them to have the conversations to talk about all of the work that we should be doing before we ship something. So, I feel as though they have a lot of pressure on them, a lot of time pressure on them. They’re being pressured to really contract the amount of effort that they put in before we ship something. And there is – yeah, there’s this huge desire amongst product teams and their bosses to just get something shipped, get something live and then we’ll learn that I think design really struggles with. And I don’t know – it can be really difficult in those kinds of environments to be the person who stands up and says we need to slow down and we need to do less and do it better. So, I have empathy for designers in their inability to shift the system – these problems – necessarily because of this big pressure that’s being put on them just to keep the engineers coding.
Steve: Right. If you say you want more time to work on something that’s – what are the risks to you for doing that?
Leisa: Exactly, exactly. On a personal level everyone wants to look good in front of their boss. Everyone would like a pay raise and a promotion and the easiest way to get those things is to do what you’re told and ship the stuff. Keep everyone busy, produce the shiny stuff, let it get coded up, let it go live, learn, carry on. That’s thinking short term. That’s how to have a happy life. Is that how you’re going to fundamentally improve the product or service that you’re working on? Probably not. But it takes a lot of bravery to try to change that, to try to stop this crazy train that’s out of control. Throw yourself in front of the bus. All that kind of stuff. Like you said, it’s hard, especially when there are loads of people all around you who are very happy to keep doing it the way that it’s being done right now. So, yeah. And I think that’s research’s role, that’s design’s role. I would like that to be PM’s role as well. And most engineers are – a lot of engineers are very, very, very interested in making sure that the work that they’re doing is actually solving real problems and delivering real value as well.
Steve: So, as you point to the individuals there’s a lot of shared objectives. But if you point the system, which is – it’s the system of – it’s the rewards system and the incentive system, but there’s also some – I think there’s just sort of what day to day looks like. Sort of the operations of the system of producing technology products.
Leisa: I think there’s also – there’s something about like what do people like to see? What gets people excited, right? And graphs pointing upwards in the right direction is really exciting. Like certain outcomes are really exciting. Ambiguous outcomes, really not exciting at all. Having things that go fast, very exciting. Things that go slow, not exciting. So, I think there are all of these things that this collection of humans, that form an organization have a strong bias towards, that we get really excited about, that don’t necessary help us in the long run. You know you see lots of people who get really excited about the graph. Very few people dig in behind the graph to the data. Where did this data come from? How much can I actually believe it? Like people are so willing to just accept experiment findings without actually digging in behind it. And a lot of the time, when you do dig in behind it, you go huh, that doesn’t look very reliable at all. So, there’s something about we love to tell ourselves that we’re data informed or data driven, but a huge number of the people who are excited about that don’t have very much data literacy to be able to check in and make sure that the stuff they’re getting excited about is actually reliable.
Steve: So, how could someone improve their data literacy?
Leisa: I think that there’s a lot of work that we need to do to make sure that we actually understand how experiments should be structured. And understand more about – being more curious about where is this data coming from and what are the different ways that we could get tricked by this, right? And there are tons of like books and papers and all kinds of things on the subject. But actually, when you go looking for and you’re coming at it with a critical mind, rather than with a mind that gets excited by graphs, you know a lot of it is pretty logical. When you think about like – like surveys, right? Who did this survey actually – who actually answered these questions? Instead of just going hey, there’s an answer that supports my argument, I’m just going to grab that. To dig behind it and go like are these – are the people who are being surveyed actually the same people that we’re talking about when we’re making this decision. Is there something about the nature of those group of people that is going to bias their response? It’s remarkable to me how few people actually kind of check the source to make sure that it’s worthy of relying on. Everyone – a lot of people are really keen to just grab whatever data they can get, but that supports their argument. I think this is another one of those kind of human things, those human inclinations that we have that lead us towards kind of bad behaviors.
Steve: I can imagine that with this research education effort that you’re doing that people that participate in that are going to learn how to make better choices in the research that they then run, some practical skills, some planning, some framing, as you talked about. But it seems like that literacy is a likely side effect of that, or maybe it’s not the side effect. Maybe it’s the effect. How to be better consumers of research. Once you understand how the sausage is made a little bit then you understand, oh yeah, that’s a biased question, or that’s bad sampling, and there’s a choice to make in all of these and we have to question those choices to understand the research that’s presented to us. I hadn’t really thought of training people to do research as a way to help then in their consumption or critical thinking around research.
Leisa: Something else that we’ve really come to realize recently as well is that because of all of the other pressures that the people who are making these decisions are having to deal with as well, we think we’ll do much better if we train the entire team together instead of just training the people who are tasked with doing the research. Because something that we’ve observed is that you can train people and they can go this is great – I really see what I was doing before was creating these kind of not great outcomes and what I need to do to do better. And then you’ll see them, not that long later, doing exactly the opposite to what we thought we had agreed we were going to do going forward. And we’re like well what’s happening? These are like smart, good people who have been given the information and are still doing crazy stuff. Like what’s going on with that? And you realize they go back into their context where everyone is just trying to drive to get stuff done faster, faster, faster, faster, and you have to plan to do good research. The easiest research to do really, really quickly is the crappy research. If you want to do good research you do have to do some planning. So, you need to get into a proactive mindset for it rather than a reactive mindset for it. And you need the entire team to be able to do that. So, one of the things that we’re looking to do moving forward is not just to train the designers and the PMs, but actually to train a ton of people all around them in the teams to help them understand that the way that you ask the questions and who you ask them of and where – all the different things that could impact the reliability of your research – requires planning and thinking about in advance. So, we hope that means that the whole team will understand the importance of taking the time to do it and it won’t just be like one or two people in a large team fighting to do the right thing and being pressured by everybody else. So, I think it is – the education aspect, I suspect, is super important and it goes way beyond just helping people who are doing research to do better research. It goes to helping the whole organization to understand the impact and risk that goes with doing a lot of research activity in the wrong way or the wrong place.
Steve: Just fascinating to me and I think a big challenge for all of us that lots of researchers are involved in education of one form or another – workshops – a big program like you’ve been building. But most of us are not trained educators. We don’t have a pedagogical, theoretical background about – it’s about communicating information. It’s about influence and changing minds. And it just seems like a lot of researchers, from an individual researcher on a product team to someone like you that’s looking at the whole organization, we’re sort of experimenting and trying to build tools and processes that help people learn. And learn is not just imparting the information, but reframe and empower, like these big words where – I wish I had – I wish you had a PhD in education. I wish I had that. Or that I had been a college professor for a number of years, or whatever it would take – however I would get that level of insight. Personally, I have none of that. So, to hear us – you know we all talk about these kinds of things. I think research gives you some skill and prototyping and iterating and measuring in order to make the kinds of changes in the implementation that your making. I don’t know about you. I feel like I am amateurish, I guess, in terms of my educational theory.
Leisa: Absolutely. And I think talking to the team who are doing this work, like it’s really, really, really hard – really hard work to come up with the best way to share this knowledge with people in your organizational context, in a way that is always time constrained. At Atlassian we have a long history of doing kind of internal training. We have this thing called bootcamps. We’ve got almost like a voluntary system where people come in and they run these bootcamps and you can go and learn about the finer details of advanced Jira administration or you can come in and learn about how to do a customer interview. But the timeframe around that was like – like a two-hour bootcamp was a long bootcamp. Most bootcamps are like an hour. And so, when we started thinking about this we were like we’re going to need at least a day, maybe two days. And everyone was like nobody will turn up. But yeah – fortunately we have – we do day long sessions and people have been turning up. So, that’s great. But yeah, it’s a huge effort to try to come up with something that works well. And every time we do these courses the trainers and educators go away and think about what worked and how can we do it better. So, we iterate every single time. So, yeah, it’s a huge amount of effort. I think in larger organizations too, there are other people in the organizations who are also tasked with learning type stuff. So, we have a couple of different teams at Atlassian who are kind of involved in helping educate within the organization. So, we’re building a lot of relationships with different parts of the org than we have before in order to try to get some support and even just infrastructure. Like there’s just a whole lot of like logistics behind setting this up if you want to do it in any kind of scale. It’s great to have that support internally and it’s really good to start to build these relationships across the organization in different ways. But yeah, I think that we certainly underestimated the challenge of just designing what education looks like and how to make it run well. It’s a massive, massive effort.
Steve: It’s not the material – you guys probably have at hand a pretty good set of here’s how to go do this. If you brought an intern onto your team you could probably get them up to speed with material that you have, but you’re trying to change a culture. And I think the advantage that I have, the context that I have as an external educator is that people are opting in and that I have to by the assumption that they want to be there and I don’t have access to what the outcomes are. I might through someone that’s my gatekeeper, but it’s kind of one them. I have the responsibility for the training, but not the responsibility for the outcomes, which is what you all are kind of working with. So, I envy you and I don’t envy you.
Leisa: Well, I think I – I like it because, you know, going back to the build/measure/learn thing, right. Again, we did a learn before we did our build and measure because we’re researchers and that’s what we do, but it is – it’s super interesting to see the behaviors of people who have come through the training and see whether they shift or not. That gives us – we get feedback from people after the course who tell us whether they thought it was useful or not useful and what they liked and didn’t like, but then we actually get to observe because through our ops team, they come through us to do their recruiting and get their incentives. So, we can keep a little bit more of an eye on what activity is coming out and see if there is any sort of shifting in that as well. And it’s not just courses as well. It’s thinking about like what’s the overall ecosystem? What are other things that we can be doing where we are sort of reaching out to help support people on this journey as well. Before we did educating, we had advisors who kind of made themselves available to go and sort of support teams who recognized that they might have a need for some help with getting their research right. So, that was kind of our first attempt. But we had to pivot into educating because of the time factor. We would go in and give advice and everybody would go it’s great advice. We’d totally love to do that, but I have to get this done by next Thursday. So, I’m going to ignore your advice for now and carry on with what I was going to do anyway. And that was pretty frustrating. So, we felt like we have to invest in trying to get ahead of the curve a little bit – try to get ahead. Try to not necessarily influence the stuff that has to happen by next Thursday, but try to encourage teams to start being proactive and planning to do research instead of doing this really reactive work instead. Or as well as the reactive work perhaps. I don’t know.
Steve: Have the right mix.
Leisa: Yeah.
Steve: I wonder if – and this is probably not going to turn out to be true, but I wonder about being proactive and planning versus time. The time pressure to me is about oh we only have so many hours to spend on this, or that calendar wise we need to be there next Thursday. But being proactive says, well if we started thinking about it three weeks ago we’d be ready for it Thursday to do it “the right way.” I’m wondering, can we tease apart the pressures. One is like no proactivity, the sort of very short-term kind of thinking, is different than hours required. Is that true?
Leisa: I think so. Because I think that even want to do the short-term stuff we still spend like quite a lot of time and effort on it. And the planning in advance, like the more proactive work doesn’t necessarily, I don’t think, entail more actual work. It just might be that you put in your recruitment request a couple of weeks beforehand so that we can try to make sure that the people that you meet are the right kinds of people, instead of if you have to have it done in 3 or 4 days time then your ability to be careful and selective in terms of who you recruit to participate in the research is very much limited. So, we see – when we have those kind of time constraints you see everybody just going to unmoderated usability testing and their panel and that introduces a whole lot of problems in terms of what you’re able to learn and how reliable that might be. Yeah. I was thinking for a second about you know when – theoretically when you do your unmoderated usability testing you should still be watching the videos, right. So, that should take as much time as watching a handful of carefully recruited people being – doing these sessions in a facilitated way. But the reality is, I think, that most people don’t watch the videos, which speaks to quality.
Steve: Here we are back again. It seems like there’s a, for the profession overall, maybe one way to start sort of framing the way around this time pressure thing is to decouple proactiveness versus sort of hours burned. That it’s going to be the similar number of hours burned, but if you start earlier the quality goes up. I had never really thought about those two things as being separate.
Leisa: Yeah. And I don’t think people do. I think when people think about – and this is – I mean I sort of said the word quality. I’m trying not to say quality anymore. I’m trying to talk about meaningfulness more now, I think, because whenever you talk about quality to pushback that you get is well it doesn’t need to be perfect, so it doesn’t need to be academic. I just need enough data to be able to make a decision and understand that. But then I see the data upon which they’re making the decision and that makes me worry, right? And I think that’s – we want to get out of this discussion of like quality levels, more into sort of reliability levels, like how reliable does it need to be? Because surely there has to be a bar of reliability that you have to meet before you feel like you’ve got that information that you need to make a decision. But I see loads of people making decisions off the back of really dreadful and misleading data and that’s what worries me. And they feel confident with that data – going back to the data literacy problem. Like they really haven’t dug into why they might be given really misleading answers as a result of the who and the how and the why, all those decisions that are made around how to do the research, most of which have been driven by time constraints.
Steve: Okay, that’s the end of part one of my interview with Leisa. What a cliffhanger! There’s more to come from Leisa in the next episode of Dollars to Donuts. Meanwhile, subscribe to the podcast at portigal dot com slash podcast, or go to your favorite podcasting tool like Apple Podcasts, Stitcher or Spotify, among others. The website has transcripts, show notes, and the complete set of episodes. Follow the podcast on Twitter, and buy my books Interviewing Users and Doorbells Danger and Dead Batteries from Rosenfeld Media or Amazon. Thanks to Bruce Todd for the Dollars to Donuts theme music.
The post 19. Leisa Reichelt of Atlassian (Part 1) first appeared on Portigal Consulting.
This episode of Dollars to Donuts features Kathleen Asjes of Schibsted Media. In our conversation, we talk about what happens when research need exceeds resources, the importance of keeping the knowledge inside the organization, and the benefit of diversity in a research team.
It’s not so much about which university do I go to and which master program do I follow, or which kind of courses do I need to pay a lot of money for. It’s more about reading up on it and trying to get started and trying to get it into your every day work. Or if you’re a designer you can be doing it. If you’re a product manager you can be doing it. But also, if you’re working in HR, you can start trying it out, experimenting. You just need to do it a lot to get good at it. And you can come from any kind of background to become a user researcher really. You just need to get started. – Kathleen Asjes
Show Links* Original Stroopwafels * Van Stapele Koekmakerij * Kathleen on LinkedIn * Schibsted Media * Svenska Dagbladet * Aftonbladet * Just Enough Research by Erika Hall * Interviewing Users
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Welcome to this episode of Dollars to Donuts, the podcast where I talk to people who lead user research in their organization.
I’m going to be teaching two public workshops about user research, coordinated by Rosenfeld Media. The workshop is called Fundamentals of Interviewing Users and will be held May 20th in New York City and September 13th in San Francisco. You can find more details about the workshop and register at http://rosenfeldmedia.com/public-workshops. I’ll put the links in the show notes. You can send people from your team, or you can help me out by recommending the workshop to someone else.
I also run user research training workshops for in-house teams as well as work collaboratively with teams to run user research projects and bring new insights into the organizational culture and decision-making processes.
I enjoy making this podcast, and I appreciate your support in hiring me – and in recommending me – that helps make it possible.
I was in Amsterdam recently and in preparation for the trip I did a little bit of research about interesting dessert options that the city had to offer. It had been ten years since my last visit and I already knew I wanted to head back to Original Stroopwafels at the Albert Cuyp Market. But what else was there? Looking online I found multiple enthusiastic posts about a chocolate cookie bakery. The reviews were breathless hyperbole — the best cookie you’ll ever eat, and so on. So on our first day, jetlag notwithstanding, we walked over to Van Stapele Koekmakerij. We found it on a tiny street, with a small line of people across the street, and a cookie bouncer, maybe more of a butler, keeping people in place, and letting people into the store, in small groups. The store itself was also quite tiny, and it turns out they only sell one product – just this chocolate cookie, filled with white chocolate. The main option we had is simply how many of this cookie we wanted to buy (with a secondary option around what you put the cookies in – a regular bag or box, or a gift tin).
The store itself was beautiful, with a classic logo on the window, and lots of wood and gilt and glass, just what I pictured an old Dutch bakery would look like, the kind of place that has a time-tested recipe. We bought our cookies and then stood on the street corner to meet a local friend. Meanwhile a passerby – an expat Brit – heard us speaking English and stopped to ask us about the cookies. Are they really that good to justify the lines of people that she’s always seeing in this neighborhood? We thought they were pretty good, and we only waited in line a few minutes. We didn’t tell her this but it’s hard for any cookie to live up to the Best Cookie Ever hype that we found online, but it was a good cookie! Anyway, this woman went on to explain that this store was pretty new, like in the last couple of years. I was very surprised, as the appearance of the space, and the overall narrative, led me to assume it was someplace that had been doing this one thing – selling only one kind of cookie – for decades if not longer. The cookie was still good, but this took away a small bit of the magic away, this idea that that we had finally connected with a long-standing Best of Amsterdam experience.
It got me thinking about authenticity, that slippery topic. This bakery created a seemingly authentic experience, implying a time-worn history – even though it’s very new and only tourists shop there. The fact that the truth belied my assumptions suggests that it was somehow inauthentic, but I would still put them on the authentic side of the equation. T hat the place looked and felt (and tasted) authentic, even though technically, it isn’t?
I’m reminded of one of the most fun aspects of user research, trying to tease apart these underlying meanings, when even the words we use are inadequate to fully capture or communicate the nuance. Imagine trying to interview someone about their experience with this bakery and how you would go about unpacking their interpretation of the value proposition. I should note I am available for this work and I may be willing to accept payment in the form of desserts.
Moving on from philosophical cookie musings, let’s get to my interview with Kathleen. Kathleen Asjes leads the user research team at Schibsted Media in Sweden. Alright Kathleen, thanks for being on the podcast.
Kathleen Asjes: Thank you for having me. Great to be here.
Steve: Let’s start with an introduction. Who are you? Where do you work? What do you do there?
Kathleen: So, I work at Schibsted Media in Sweden, but I’m originally Dutch. I moved to Sweden around 8 years ago and before that I also moved around quite a bit. So, been here and there. And I lead the user research team at Schibsted. We are 9 people altogether right now.
Steve: Nine?
Kathleen: Nine researchers. And yeah. I manage the team. I make sure that we advance our research practice. That the research maturity of the organization kind of advances. For those who don’t know Schibsted Media, which is basically the whole world, we are a media company based in Scandinavia and it consists of – it’s a conglomerate of all kinds of online newspapers – on and offline newspapers – media houses basically. And they are the biggest ones in Norway and Sweden. Eight altogether, but also some like regional, coastal newspapers. So, it’s a mix of both national and regional newspapers as well as other niche products that we have.
Steve: That are not newspapers?
Kathleen: Yeah. So, in total we work with around 34 product teams. Some of our teams – there’s many, many different kinds of product teams. Some of them are related to these newspapers, but others are like Scandinavian Weight Watchers and price comparison sites and so on.
Steve: That are all part of this media conglomerate?
Kathleen: Yeah, yeah. And Schibsted consists of both media and marketplaces. So, it started out as a Scandinavian company, I think 200 years ago almost. It’s old, old, well-established, but it also has marketplaces all over the world. So, in France, in Spain, in South America even.
Steve: What’s a marketplace in this context?
Kathleen: Oh, it’s online selling and reselling of used goods.
Steve: And are these all different brands? There’s not a global, international marketplace brand?
Kathleen: No, no. This is all kind of local brands. Local marketplaces that are like national marketplaces in these different countries, but they are all hosted by Schibsted.
Steve: Okay. Now what’s – this is a broad question, I guess. What’s the mix of things that were acquired – I guess how does this media conglomerate come to have such a more diverse portfolio than you’re describing?
Kathleen: I have no idea, to be honest. I think it just, as a media house it’s in their DNA to experiment and to be first and to be fast. Actually, kind of like agile. So, when it comes to kind of new products or new opportunities they act fast and they experiment and they, yeah, widen their horizons and do it in a very sustainable way. Like focus on growth and kind of maturing products.
Steve: Have you – during your time there, have you seen them move into a new – not product category, but sort of geographic region? Like a part of the world where they didn’t have any business before?
Kathleen: Yeah. So, they do this all the time, especially in the marketplaces, but now also with kind of personal finance and borrowing money services and so on. So, yeah, they’re expanding and looking at new regions all the time.
Steve: So, in those examples, that’s businesses that they’re already in and then how can we do that in another territory?
Kathleen: Yeah. How can we take this to a neighboring country? Or how can we expand this in a different regions? We see opportunities somewhere else, what can we do?
Steve: I’m starting to see a little why there’s so many different product teams that you’re working with.
Kathleen: Yeah, but this is only – like, I’m only on the media side. I’m sorry, I should be really clear about that. All the marketplaces stuff, I’m not in that group. My colleagues are.
Steve: Do you have peers who do research in that part of the business as well?
Kathleen: Yeah. So, they’re researchers. My team is based in Oslo and Stockholm, but there’s also researchers based in London, Paris, Barcelona, South America. So, altogether, I think, in Schibsted wide there are around 25-30 researchers.
Steve: So, you’re involved in the media side.
Kathleen: Um-hmm
Steve: I’m just wondering sort of how many clusters are there? Media is one.
Kathleen: And marketplace is the other. They’re like the big two classes.
Steve: So, you’re 9 and they’re – I’m doing math now in my head – so that’s about 16 people probably in the marketplace.
Kathleen: It’s a community of researchers where you make an effort to see each other every year which is good.
Steve: So even though the things you’re working on are quite different – different industries, different businesses – there’s still a sense of trying to keep connection?
Kathleen: Yes.
Steve: What are you trying to accomplish by bringing these different researchers together?
Kathleen: I think since user research is not a new kind of field, but is ever evolving and it’s really important for us to make sure that we evolve as user researchers. And you do this in dialogue. Like this is why we attend conferences, and this is why you explore new methods and new things and you want to share these things of course. So, I think that’s why it’s important that we meet and talk about these things face to face because from that kind of dialog you get a very different experience than just seeing a presentation or talking to each other online. So, to have a community, to have your peers, to have others that give feedback on work and they make you reflect on what you’re doing and what’s going well and what maybe isn’t going that well, I think it’s really crucial to kind of advance your practice and to become more mature in your research in the organization you’re in.
Steve: How are those conversations different when you’re in person than through all of our collaboration platforms that we work in?
Kathleen: I think, as always, it’s not always just in meeting and having an agenda and talking about certain things. It’s also the discussions that you have during breakfast if you’re staying at the same hotel. Or after work while you’re having a drink. As well as also, well we hosted during our last meetup – we met in Budapest where we also did a visit. We visited another company and talked with their user research team and talked a bit about okay, what are you struggling with? What are we struggling with? And then you start asking each other questions and you get these really brilliant discussions going. So, one of the questions there was someone asked like okay, when it comes to really, um, keeping track of all the research, all the learnings that you have to know in an organization, how do you do that? Like, what is a good database or platform to store your insights? And we were like darn, we have the same issue. We were looking for a solution as well. And basically, everyone in industry is looking for a good solution to do this over time. And if anyone knows, please let me know. Reach out to me. I’m happy to hear.
Steve: Let’s maybe switch topics back and go back to setting more context. How long have you been in this role?
Kathleen: I came to Schibsted around four years ago, I think, and I came in as a consultant first on a very kind of different – one of those niche products. I worked with lifestyle which was like online fashion and interior decoration – really strange, but fun to work with. And then after doing that for almost a year my manager said okay we would like to have you board, would you consider becoming a permanent employee here? And I said, ah, only if we can make sure that this is like pure, like a research role. Before, as a consultant, I would kind of be – often be hired as a UX designer. Like the industry in Scandinavia was not that ready for user research as a profession, or expertise. Often when I would say, like 6/7 years ago – it was like oh the user research people – “like what?” I said, “oh you know finding out about user needs and really tailoring the product to meet their needs.” And then like, “okay, but that’s what you do at the end, right? So, what else?” We need someone at the beginning. You really had to explain well actually it’s at the beginning where you start and where you need to really understand what you’re doing and who you’re doing it for. So, often we’ll be hired as a UX designer and have to wing the design part most of the time, but do all the fundamental research kind of early on. So, I said okay. I was the one of the first user researchers in Schibsted. I said, okay, you need to make this a role and it needs to have – I see that all the teams need to be doing more user research, so I want to coach them and train them and make sure that we become more mature. So, that’s how it kind of started and I’ve been changing my role about every year. So, from research lead, working by myself, then I got like a small team to work with, with two other researchers. And then last year we finally kind of combined all the different researchers that were in media, but were working with their own separate kind of product, into one kind of – one team that works more independent of the different product teams and works more like okay, where are we needed? Where can we add most value? Where can we add most impact?
Steve: Was there a change in title for you when that happened?
Kathleen: Yeah. So, that’s when I became head of research and got the 8 or 9 people team up and running.
Steve: I’m curious about – I don’t know if it’s a fair question for you, but at what point do you think organizations go from – like there’s some evolution here of we’re doing research, or we have people that have the title of researcher, to now we have someone who is responsible who is a leader. That’s, I think, sort of the evolution that you described. But what is that point? I don’t know, is there a point at which that becomes clear that we’re ready or it’s time for leadership in research as opposed to sort of just the function being staffed?
Kathleen: Yeah. I think it depends a lot from organization to organization and I think it depends a lot on how – you know who is there? Like in the organizational leadership you need to have those people who want to promote it and see the value and really want to kind of make sure that research becomes leading in so many ways. And one way to get there is to really show your impact again and again and again. And do a lot of that lobbying and showcasing and communicating. Another thing can be that one of the senior VPs or someone who comes in has that experience from another company and goes like, “okay, I see where we’re struggling. What we need is professional user research. Let’s bring those people in or promote people into those roles and structure it more.” So, it can be bottom up, or top down. It can go both ways. I think in our case it was more bottom up.
Steve: Right. You hear stories about people were brought in with the intention of building up a function capability.
Kathleen: Yup. Build a team. Build a function. Change the organization. I never had that mandate to begin with. I created it.
Steve: You didn’t have the mandate, but it sounds like you had that trajectory. I mean in the conversations you had about taking a fulltime job versus being a consultant, it sounds like – I’m inferring from some of what you said that you’re looking towards more robust establishment of research that you would want to champion and drive.
Kathleen: Definitely, definitely. I think it’s so important because user research, for us as a function, as a role in the organization, we’re there to make sure that everyone in the product teams makes better informed decisions. We do a lot of gut feeling and looking at competitors and just coming up with stuff. We need to make those decisions more robust and we need better insights. So, that’s what we’re doing and for that to really get to a professional level and a good level you need to have it embedded in the organization and have it at decision making levels. Yeah. So, it’s always been – like I see where it’s needed, and I try to make sure it gets there.
Steve: So, part of the evolution that you were describing was there were individual researchers working on different product teams and then you reorganized so that you’re all working together and then putting research – you’re looking for where it’s needed and then putting research where it’s needed.
Kathleen: Yes
Steve: And I find this, and maybe you do as well, this is like a – this is a hot topic. Are researchers embedded, or are researchers centralized, are the terms I hear used most often.
Kathleen: Yeah. So, we’re in a matrix organization and then you’re a bit of both. So, every product has UX designers, product managers, developers. And then you have all the different – yeah, brands, across the other side of the matrix. And then yeah, we feed research into each of these kind of product teams, wherever it’s needed. So, you’re half embedded and half not. And ideally, of course, it would be wonderful if I had 34 researchers. Then yeah, everyone could be embedded. But I don’t see anyone granting me that kind of capacity anytime soon.
Steve: At least 34 right. Maybe there’s work for more than one researcher on that team.
Kathleen: Definitely. Let’s go for 40 or 50. Yeah, I would like to, you know, pose that question to the CEO or CTO, or whomever is making the decisions these days. And I’m not going to get at all, no. And I’m not sure if we do a better job at that scale because now what you see is when you work with multiple product teams is that they have similar challenges, similar issues, similar audience. So, you have more of that dialogue going on between the product teams and you’re facilitating that as a researcher because you have an overview of all the different issues and the similar issues that they’re trying to deal with. So, I think in that sense, yeah, we’re usually understaffed, as always, and there’s way more requests than there are people on my team, for sure. At the same time, it makes you make really tough decisions on okay, where can we add most value? Where do we add most? But also, you’re spread thin, so you need to make sure that teams do their own research, but you support them doing it. And actually, I think they do – I think the outcomes are used more and better if teams do it themselves. With your help, obviously. But yeah.
Steve: Would you have a theory as to why?
Kathleen: Well it’s all about that kind of firsthand experience. You know if you talk with your audience or users directly you’ve shaken their hands. You’ve seen what they struggle with. You know what’s happening to them and you will use that in your everyday decision making for the product. And also, you know, you can do a study, for example working on news consumption patterns. Like how often during the day do you access online news and which sources do you look at, for example? But during those conversations maybe really interesting things are coming up about subscription models. Or when you are willing to pay for news, or not. And if the product owner, or the designer, has those conversations themselves with users, then in the next project that comes up where they’re actively looking at subscription models or changing those, they will have that knowledge in their back pocket and take that up again. Whereas, you know, if I write a report, or share information about news consumption, that information about subscriptions will not be in there. Like that’s in my mind. So, if I’m the expert who knows everything and has all the conversations and all the interviews and gets all the raw data and starts analyzing that, and I’m the only one doing it, then I am the expert of everything. But I can’t be there for their everyday decision making, or their backlog grooming, or they’re deciding on the next step for their product strategy. So, the more they do themselves the more all those insights will feed into their everyday work.
Steve: So, can I just try reflecting back? So, a team that does its own research, one has the benefit of direct experience and two, has the things that seem extraneous to your initial question, these sort of weak signals, these other bits and pieces that you hear, that that rests with them, where they’re more likely to reuse it in the future when it comes up.
Kathleen: Yeah, definitely, definitely. And then you do have – on the flip side there’s the challenge of synthesis, data analysis. You know that’s quite – can be quite advanced and of course not everyone has the capacity to do so, both in time as well as mentally. So, it’s a fine balance between grooming and training them to do their own things – we’re still keeping an eye on it and helping them in the places and spaces where they struggle.
Steve: So, something like synthesis, which is advanced, what do you do to help a team that has not been…
Kathleen: Yeah. So, I would sit in on those sessions with them. So, for example, I myself support two product teams now and we’ve been doing this for around a year and a half, so they’re pretty good at writing their own kind of discussion guides, doing their own interviews and so on. But I noticed like, okay, when it comes to the analysis part they’re still – you know the findings are still kind of superficial. So, I would just try and sit in on those sessions. So, we’re listening to one or two of the interviews and sit in on the session, look through the raw data with them. And we start digging. And then you know, when people jump to conclusions too quickly – so, like “okay, where did you get that from?” It’s like, “oh the fear of missing out. It’s here and there.” “Well, actually no one every said that. It’s a term we use a lot when it comes to filter bubbles and so on. I don’t think I’ve heard any users say that in interviews.” And then make them dig to find the evidence and they go, “oh, no, I jumped too quick there.” Like “okay, let’s step back then and look again at what we did find.” So, yeah, but it’s different for every team really, what they need and where they’re at and some are at that basic level of okay what do I ask and how do I ask it? And others are a bit more advanced.
Steve: So, what’s your decision-making process in allocating your own resources?
Kathleen: Yeah.
Steve: Across these different teams where maybe a researcher in your group is going to do the research, or is going to support this other team doing the research themselves? You know, how do you plan all that?
Kathleen: Well, it’s really hard and actually lately there have been – I wouldn’t say conflicts, but there are – you know different teams are fighting with each other for who has access to which resource, which of course is a terrible situation to be in. But we do it in a way where – we’re 9 people now and we make sure that each researcher has multiple product teams that they focus on and they’re the first point of contact for. So, all the requests and so in come in for them and they follow a product team and the kind of research roadmap on what they’re doing. And then when a new request comes in it comes through me and then I’ll have to just, yeah, interact with the team and ask, okay, who has space? Do we think this is more important than something else? Do we need to prioritize this? And then we prioritize based on several things. So, first it is like where do we have most impact? Like where do we think we can generate the most important insights that are really needed for a team, but also where are we at ourselves in our professional, personal development and what kind of opportunities come with this research assignment or challenge that will help this or that person in our team grow personally?
Steve: Can you give a generic example of what that would be like?
Kathleen: Yeah. So, for example, we had an assignment coming up and we saw like okay, well looking at it we saw that okay this kind of challenge requires a diary study. And then there was someone on the team who really wanted to do a diary study and never done it before and wanted to come to try out that as a method. So, we said okay, well this is a great opportunity for you to work this out and experiment with this. So, now we put that person onto that assignment. So, it’s a big mix of different things really. Like where can we have most impact, but where can we also learn new things and try and challenge ourselves and grow, both as a team and as well as research professionals.
Steve: If you don’t have resources for the request that comes in, I’m sure – there’s a couple of things that could happen. That research could just not happen? Or it could be done by the team? I don’t know what – maybe I should put it as a question. If you can’t handle something, what happens to it?
Kathleen: I don’t think it’s done. It doesn’t get done. Or, we hire external agencies to do it, which I think often is the worst thing to do. And this is why I’m happy that we kind of came together as a research team because I think around 2 years ago it was like every individual researcher would be stuck doing a lot of tactical validation or kind of research and be really busy with that kind of everyday work, really trying to make sure that product teams look more at these kinds of insights and validate whatever they push out. Um, and at the same time we were thinking like okay we need this really big, strategic research done on the whole – you know what is the ecosystem in Scandinavia for both news consumption as well as online marketplaces and selling and reselling of used goods. And then we hired this agency to do this for us and spent loads of money on it. And then so all the really challenging, really big research questions are done by an external agency that deliver a report, but then all the super useful insights that are in extra information kind of – you know they stick there. They don’t really reach us. Or at least not into the organization. And we can’t do it ourselves because we’re too busy. That’s, of course, a terrible place to be in.
Steve: It sounds analogous to when a product team does research versus when the research team does it for them.
Kathleen: Yes. Yeah. So, it’s even one step further removed. It’s not even in the organization. So, some consultants know a lot about media consumption and the whole ecosystem in Scandinavia. Good for them. They got paid to do that as well.
Steve: So, as someone who works as a consultant, I’m definitely sensitive to – when you hear that your way of working is the worst possible option, that definitely makes my ears perk up.
Kathleen: I hope I didn’t offend you.
Steve: No, I’m not offended. It seems to me, in my experiences, that the mitigation for that is to have it be a collaboration, which is a different way. I think you were even talking about this in the way that you’re supporting your internal clients versus kind of doing it for them. I think for consultants, this is my opinion and my experience, like it’s a go take the research – to go take the tasks, execute it and report back on it. As a consultant, that’s not very fun. It’s not very rewarding. I don’t think it’s very effective. But I’m interested in sort of finding collaborative models. Again, just kind of like sometimes with your teams you’re coaching them, but they’re handling it and sometimes you’re in there doing the research with them. I think these kind of hybrid – they’re more blurry kinds of collaborations. Because I don’t want to know everything about some topic and then…
Kathleen: No, no.
Steve: That doesn’t suit my objectives. That doesn’t suit my clients’ objectives.
Kathleen: No, exactly. But that’s why you as a consultant need to have a really thorough discussion about that at the start, right? Like how can we make sure that these findings and insights kind of land in the organization and are usable by everyone. Rather than it just being the deliverable that then kind of goes from one’s inbox to another inbox. And then four weeks later it’s forgotten. Or in some kind of drive somewhere.
Steve: Right. Your point earlier about trying to help them have some experience where something sticks with them. Like that’s definitely the model of my own business and I think as researchers we’re capable of facilitating that experience in a way that – I mean this is all what you were talking about, setting up these teams to be successful. So, how do you help them have those experiences? It sounds like that’s what you’ve been working on a lot.
Kathleen: Yeah, definitely.
Steve: I think we agree more than we disagree.
Kathleen: We do. We do.
Steve: Let’s see if we can loop back to something you mentioned before. You were talking about one of the advantages of this kind of matrix model is that researchers have experience across different products and different product teams and they’re able to bring something that they’ve learned into a different context.
Kathleen: Yeah, exactly. And also, I think our – we try to make sure that our product teams do kind of continuous research, or rolling research, whatever you want to call it, so that they bring in their end users every month – at least every four weeks or so. And then, you know, you have a batch of 5/6 people coming in every month for all these different product teams. So, it adds up to a lot of people in the end. And over times it adds up to talking to very many people across time. So, we try to make sure that they have some kind of fundamental questions that they ask everyone. So, across these different product teams that we all ask about media consumption habits. Like just to really understand, okay, you are reading Svenska Dagbladet, the Swedish national newspaper, but which other sources do you have for your information? And then it might turn out that well actually I’m also daily visiting Aftonbladet, which is the other big national newspaper that we also own. And how do these two compliment each other? To ask about which other products are you using? How does this complement each other? Are there competitors there? So, these are the kind of questions every product team asks and that we can also kind of look in across, or over time. Do we see things changing? Or do we see like across the board to kind of see overlap, or learn things that other product teams should know about. While talking with Svenska Dagbladet readers, what do we learn about? Aftonbladet at the same time.
Steve: Just yesterday someone was asking me about their organization and describing – they put it as very siloed research and they were feeling frustrated as a researcher – I think they were a designer who’s doing a lot of research – that they were seeing things that they knew could apply elsewhere and they didn’t know what to do with them. And you’re kind of describing sort of a proactive structure for that. That there’s some things that are going to come up and you’re going to intentionally ask in a lot of different research.
Kathleen: Yes.
Steve: And then, I guess this would be the challenge for this other group – then you have to put the time in to say well what does that tell us?
Kathleen: Yeah, exactly. And then not just keep it on the shelf, but actually use it. And I think there we still have a way to go, to be honest. Like we try to make sure these things are in place, but to actively spend time digging through it and making sure it’s useful. Like I think we can take extra steps there. Yeah.
Steve: At one level it’s there if you need it, to go look at it, but to be proactive you would have to go and say well what are we learning?
Kathleen: Yeah. What did we learn this quarter? What did we learn this half year? We compile all that data.
Steve: One thing I’ve heard people talk about is – you know you’re describing managing these different requests that are coming in and prioritizing them based on these different criteria. I’ve heard people talk about one thing they can do with those requests is say, “we already have that information.” Like recycling data, or recycling insights, as opposed to redoing the research.
Kathleen: Yeah, exactly. So, often it is often about seeing like okay, you want to have information about willingness to pay for online use and so on. We know from your counterparts in Norway that they’ve done this in this study, have a look at that and see if you can reuse some of those insights, or if you think they apply to you, or if you can build on that. So, there’s a lot of that kind of cross-functional learning. But then still it becomes – I’ve been there for four years now, so I’ve seen and heard quite many things, but then still the way – you know what is the one central database or place where you will know that that information is there, even if people have left. Or what if I leave? Like where does all that knowledge go?
Steve: That’s a theme I think we keep coming back to. Sort of where does knowledge go?
Kathleen: Yeah.
Steve: Right, the external consultant? Or you delivering the report? The kind of matrix way of working for the team and then where does that live? It seems like this is a really significant challenge. As you said, the folks that you met with in Budapest, that was also an issue for them.
Kathleen: Yes. I think it’s an issue a lot of people struggle with. Like across everything. And then, yeah, do you need an organizational Wiki or database or, I don’t know? Do you have the answer? Have you heard the solution?
Steve: I had a really interesting conversation with someone about this and what she said to me was yes you can have the infrastructure, whatever database tool you want, but they hired basically like a reference librarian role. In her experience it would fail unless you staffed it. This is a job and it takes time and it takes effort. So, that you have somebody there…
Kathleen: And is it worth it? Because what is the shelf life of each insight or thing over time? You know things are changing rapidly. Sometimes I wonder a bit about that. Like things that happened four or five years ago and it’s still relevant for us now. Is that insight still valid? Or have things changed?
Steve: That’s interesting for a 200 year old company. Some things clearly are changing very rapidly around us, but what are the fundamental things that – I don’t know. I don’t know how you even kind of assess that.
Kathleen: Yeah, but if you think about reading news online and something that for many of us is an everyday habit now, but 5 or 10 years ago it wasn’t.
Steve: But the role of – the role of news in being a citizen, I mean has that changed?
Kathleen: No, no, no. And that’s also what’s so interesting about working in this field. When I graduated I wanted to be a journalist actually because I thought it was really exciting and really such an important thing. Free and independent journalism is such an important thing in society. Now, all these years later, I get to work with journalism. So, I’m almost there. Just in a different kind of capacity. Yeah. So, that’s quite good.
Steve: Right. Is there a sense among the team that you are – I mean do you all identify as working in the news industry?
Kathleen: Yeah. I think that mission that Schibsted has about really providing free and independent journalism is really important for most of us. People really align along that mission.
Steve: I think there’s something about the work of research – as someone who has worked as a consultant, you get into just a wide variety of industries and topic areas and I feel like among researchers that’s a bonus.
Kathleen: Yeah.
Steve: Sometimes I’ve had this conversation with people when they approach me about some research problem they have and exploring if there’s a possibility to collaborate? They’re describing something they’re doing, and they say, “is this interesting to you?” I’ll have to kind of laugh because…
Kathleen: It’s all interesting, isn’t it?
Steve: It’s all interesting, right. And so – but you were describing maybe the opposite. And then obviously there’s thousands of problems, even within the space that you’re in, but I guess it’s breadth versus depth that you’re in an industry and you’re sort of focused on the commitment to that mission. Whereas if you’re sort of a journeyman researcher like myself you move across very different kinds of things. And that in itself is rewarding, but I don’t have the mission thing that you have.
Kathleen: But I’ve been there as well. I’ve also worked as a consultant and I’ve worked with offshore engineers that have high voltage cables across the North Sea and so on. And who would have thought when I came out to Sweden that I would be working on this factory site with a yellow helmet on my head and looking at what these engineers are coming up with and what they’re doing and so on. But that’s an amazing thing, isn’t it? You get to really explore things you didn’t know about were happening and what they’re like and get to ask all the dumb questions to really get a good understanding. But after doing that for four or five years, I also at some point felt like yeah, often you are asked to be there at the end where things have failed, or things are not going well. How interesting would it be to be there along the way? Like I said really helping teams be better informed and make better decisions. For that you need to stick around for a while. That’s also super interesting right now. Yeah.
Steve: So, as you’re talking a little bit about some of the career – elements of your career path, can you maybe step back even further and talk about what you pursued in education, when you were thinking about journalism and then where you ended up in the work world and sort of what led you to where you are now?
Kathleen: Yeah, good one. I studied – well first, I managed to study psychology for about 3 or 4 months when I came out of high school because I was so interested in people. I thought oh I need to – that will help me understand the world and people better. And I got super frustrated with it in more time and I thought the course in statistics was more interesting than the introduction to psychology. So, I noticed very quickly like okay, if statistics is more interesting than actually the introduction to the thing that I’m supposed to study for the next 4 or 5 years, then I’m in the wrong place. So, then I switched to political science. So, I have a masters degree in political science which has really given me a really good fundamental basis in research methods and skills, both quantitative, as well as the qualitative side of things, which I think is really great, now in hindsight. Because, you know, after studying for several years you start hating statistics after a while as well. But I’m happy I did it because it’s given me good insights into mixed methods and how you need to combine both qualitative and quantitative data to really, yeah, validate your own findings, as well as make your findings kind of – and insights stronger and have more impact, which is something that we really need to do as user researchers. Because we’re still being questioned about okay, why did you only talk to five people? I had that the other day, like why are you only – and I was like oh, these questions, are they still coming up? Interesting. So, I studied political science and after that I worked in research, like just plain academic research as well as in policy research for a municipality, but I noticed that I was more interested in technology and what it does with people and how people use it and how it enables people. So, after a while, also moving around from Holland to Australia and coming back again, managed to find a job in the Library for the Blind where we were producing materials for visually impaired people. And there we spent – I was a product manager for a while and I noticed that we were building things and making things – and I was in the research and development department – and we were spending lots of money and lots of our time on things that were actually not really helping visually impaired people at all because we were stuck kind of thinking in old ways, or thinking that we know what visually impaired people need, whereas we didn’t know anything, really. We were just doing it out of goodwill, thinking we were helping. But actually, things change. So, that’s where I started to really focus on user research and looking at how do we develop these products, how do we develop an online library for visually impaired people and what do they need in their everyday lives to access things online? What devices do they use? So that was when I really started to focus on user research.
Steve: Were you thinking with that label, user research, at that point?
Kathleen: No, not at all. In the U.S. I think user research is more established than what’s in Europe and it wasn’t until, I think, six or seven years ago, I was at a conference in the U.S. where actually someone, I asked them, “so what do you do?” And she said, “oh I’m a user researcher at Google.” I’m like, user research, that’s what I’m doing. We talked about what her job was and what she was doing, and I was like that’s exactly what I’m doing. So, now I had a title for it and actually, you know, a good understanding of what it was and what you could call it. But then no one else in the industry got that yet, in Europe. So, it’s a bit behind in some ways.
Steve: So, what happened after the working on Library for the Blind?
Kathleen: So, after that, I managed a team for a while to really become more user centered. I did it for 3 years almost and then we moved to Sweden, from Holland to Sweden and then had to reinvent myself and what I was and what I was doing. And then I started consulting. So, I started consulting and working with UX (user research) in different industries). So, I’ve done everything, like I said, from offshore engineers to 3D kitchen modeling comparison. I’ve done a very interesting study on heating systems, heating pumps and how to explore interfaces for that. It was a really fun and weird assignment. And ended up doing an assignment for Schibsted and stuck around because they’re interesting to work with.
Steve: You know your whole journey is – I love that you were doing research and didn’t know that it was called that. That you had sort of found your way to it without the benefit of that label or that identity or some of the professional community stuff. It just makes me think about – I think your example is common. If you look at sort of trajectories over time – and I think you’re right, geographically it’s different – U.S. versus Europe versus other parts of the world – but there was a time where anyone who was in user research was there through an indirect – I don’t even want to say non-traditional because there was no alternative – but everyone was there through a mostly somewhat kind of indirect path. And as time moves on and the research industry grows there are people who identified that early on, chose an educational path that would sort of prepare them for that.
Kathleen: You can study it nowadays.
Steve: You can study it, right. Whereas we remember when people would say, what is that? That’s not a thing. I don’t know, so what are the – as time moves on and we have a mix of people that have studied it versus people who have – I don’t know. This is a little pretentious, but been called to it because we identified it – I guess you can study it and be called to it. But someone like me that – I was not trained. I found my way like you did. What does it mean for the practice, kind of collectively, as we have this mix – as that mix changes, I’m going to guess more people are being training in it now. The mix includes more people who studied than it did in the past. That’s a hypothesis.
Kathleen: Yeah. I think it does, but at the same time I don’t think it should. I think the richness of our practice, research practice, actually comes from people with different backgrounds and different kind of previous experience. So, that’s also what I look for in my team. Like everyone comes from a different background and when I recruit for my team then I also actively seek people that come from something else. Different nationality, different professional background, different academic background, different place in life from juniors – really junior – I just hired someone straight out of, psychology she studies – to more senior and have been around for a while and have all that expertise from just industry or academically. And I think that’s how you as a team, and as a person, can grow most, through that mix. And that’s how we can do our best work. I would not want to have a team of nine trained user researchers.
Steve: Why?
Kathleen: Best quality comes from diversity, quite often. That’s been proven time and time again. From the more different angles you can look at a problem the better your answer or solution will be. And I think that’s the same for a user research team. You need, you know, a qualitative research expert. You need quant research experts. You need people who are more experienced in design so that they can come with really good recommendations on how to solve problems. You need people who have you know – yeah, different – my team as well. There are some anthropologists, psychologist, someone who comes from product design, like physical product design, or from digital product design, human computer interaction, UX – you know we’re a bit of everything and we actively want to be – the more diversity, the better. The more we can learn from each other then the better we will get at what we do.
Steve: Do you think there’s any connection between – you know you mentioned before that people are still asking the sample size question. Is there any question between that, as kind of how well research is understood or not understood, and – your diversity of thought, your diversity of educational background is not – I don’t think that’s an unusual way of articulating it. And I’m with you that that’s a strength, but I wonder – you know a practice that is made up as kind of a conglomeration of different types of thoughts and educational background, are we creating challenges for ourselves in being understood by others because we have that? I mean diversity – I agree, diversity is a good thing, but there’s sort of almost like a ragtag aspect of it. Like oh, we’ve got a psychologist and a product designer and someone from political science. It could be misconstrued as sort of a lack of intentionality or standardization.
Kathleen: The more well established it is through education and in like product development processes, the more standardization there is the easier it will be to get recognition for what you’re doing. At the same time, if you really deliver high quality insights and really can also show the impact of these are the insights, this is what we’re learning, this is what the product team is also changing based on that, this is how it’s changing our revenue, this is how it’s changing our product, this is how it’s really improving the user experience, then yeah, it’s in that proof and in showcasing and also labeling it, naming it. Like we got here through this and this insight and this is how we managed to change things around. And people can also get there. But that level of understanding and acceptance of we talked to six people and we know more now than we know from just staring at our bellybuttons and looking in-house to ourselves or our competitors.
Steve: I think what I’m hearing you saying is that the thing that helps advance the field is the work. The work being good, the work being relevant and kind of advocating for the work. And my thing about what kind of backgrounds do people have and so on, that’s probably not as relevant as the thing that we’re here to do, that people care about. That’s what they care about.
Kathleen: At the same time, of course, it is tremendously helpful that large companies have really professional research teams and that they actively do this and that there is education and training in this and that level of, yeah, making it more professional. That helps us, of course, as well, because then I can also say, “you know what, at The Guardian, they’re doing it this way. We could be doing something similar.” Or, they have learned this and this and we should be looking at that as well. Yeah, it goes both ways.
Steve: So, you’ve talked about just the diverse team where people have exposure to just different modes of thought, different backgrounds. And then you gave that example where you might choose how to prioritize a program, among many that you’re being asked to do, so that someone on the team – you mentioned the diary study example. And I guess, so just more broadly how do you think about the team, where they are at now and developing them. Kind of helping create the conditions that can support professional development within your team.
Kathleen: Yeah, so I think part of being – what’s so good about being part of a team is that you have people with different kinds of expertise that you can learn from and that will help you kind of grow as a research professional. And I think it’s really important. There’s a high demand for user researchers. So, they’re being poached like left, right and center if you get the opportunity. LinkedIn requests galore if you have user research in your title. Some recruiter will at some point go like, “hey, should we be talking with you?” So, to counter that you need to make sure that people are challenged in their everyday work and that they feel that they’re growing and that they’re developing, and they get to do really challenging things. And in our team we’re looking at that from kind of – we have a sparring system where every month you team up with a different person in the team and every week you spend at least an hour sparring on what am I doing, how’s it going, what are my challenges right now, can you give me feedback on this script, or this method or this approach, set up, you name it. It’s up to them themselves. But just to make sure that you get active input from not just within the team setting, but also one on one, to help you kind of get different inspiration, different feedback, all the time. And as well, besides the sparring, making sure that people get to do really challenging assignments and that we put them on kind of the research that’s not just, you know, whoever, which product team yells the hardest gets to put a researcher and get to say what they have to do. But to make sure that we focus on the team so where we can have high impact and really mean something for the organization as well as the product team as well as for the researcher themselves. So, I think there we’re seeing, compared to a year ago when everyone was in their own product, in their own silo, doing their own thing, we’re seeing lots of benefit from that now. Being a central team, of course, can be challenging because you’re getting more requests on what you can do, but at the same time you get to really learn from each other and learn from your peers, and yeah, develop yourself, your own practice, quite a bit. So, I think that’s really what’s super inspiring about being a research team, or having that. And I can recommend to everyone who is like a research team of one – because there’s many of us around – to try and find your peers and try and find a research community around you. It can be a different organization, or it can be even online. It doesn’t matter. Try and find people that you can spar with and you can talk with and learn that way.
Steve: I have to admit, I haven’t heard the word sparring used in that context. I don’t know if we have a regional difference here, or my lack of exposure to certain kinds of environments. I don’t know.
Kathleen: Maybe it’s a Scandinavian thing. I don’t know.
Steve: I think of sparring as like practicing in – like in boxing.
Kathleen: Yeah.
Steve: So, when you said the word it brought to mind something adversarial. But as you unpack it I’m like oh, this is a trusted partner that you can – I think a sparring partner in sports, in boxing, is someone that – it’s like a friend that you can hit, right.
Kathleen: Yeah. Someone that challenges you.
Steve: Yes. And you have to have trust.
Kathleen: Exactly
Steve: It’s a safe place to go outside your comfort zone.
Kathleen: Yeah. And that trust is something that has to grow over time. Because I remember when we came together as a team at first there really was some doubts and questions about so what is this going to add for me? Like why should I be part of this? So, it was hard to work to make sure that people really, you know, yeah, got an understanding for what is your expertise? What is my expertise? What are our different backgrounds and how do we complement each other and how do you add to my work and how can I add to your work? So, it took a couple of months. And yeah. It’s a lot about meeting each other face to face and making an effort to see each other every month, even though we’re in two – well, Oslo and Stockholm is not far from each other. It’s an hours flight. You can see each other. So, we do that every month and it is a mix of both professional, as well as eating out.
Steve: Right. We’re looping back to what you were saying before about just the overall research community at the company.
Kathleen: Yup.
Steve: That even within the team that’s how you sort of turned a bunch of individuals into a team was through spending time together.
Kathleen: Yeah, exactly. And then, you know, if you’ve seen someone, if you’ve met them, it is easier to reach out online and post a question and get someone to then maybe follow up and talk with you or have like, yeah, that kind of communication with.
Steve: You’re describing something that’s at least somewhat parallel to the work itself. That being in the same space with people and talking to them. You talked before about meals. Maybe you’re not literally breaking bread with somebody, but you are sharing space with them. There’s things that happen, that I think can only happen in that environment.
Kathleen: Yeah.
Steve: That’s the research, but that’s also team building and establishing a shared culture and a shared set of beliefs. I think you described going from like what’s in it for me to what’s in it for us. I’m paraphrasing you, but that kind of shift.
Kathleen: Yeah, but I think you’re right there. Yeah. And it is – yeah, it’s both. You need to – by supporting your research community you get lots back as well. So, it goes both ways.
Steve: So just looking into the future, the researcher’s future looking question, whether this is about your own personal, professional trajectory or about the organization you are in that you’re building, or the practice in general of research, you know some number of years in the future, what kind of things should we or you or your team be heading towards?
Kathleen: I think when it comes to my team and the organization I really think we’re maturing quite well in terms of how well research is established and what we’re doing and so on. But I think the next hurdle for us, or for me, professionally what I want to see is that – in the moment I’m talking a lot about how can we make sure that product teams understand their audience better and understand their user better and have that firsthand experience. And now I’m working towards the next phase which is okay, how can we get the organization to understand their audience better. So, now okay Schibsted Media, there’s a product and tech organization of around 300 people or so, and I want them to do and understand user research and we need to spend a lot of time training them and coaching them and mentoring them. Great. But there are 2000 people, employees, working in Schibsted Media, and this includes like the journalists and it includes like salespeople. Ad space, or the work with subscription. And it includes customer service. And it includes HR even. So, how can we get them to start doing user research and get them to have that kind of first kind of access. Because otherwise it’s still a silo. It’s still the product and tech silo that gets the empathy and gets the understanding. Well, what about the people that create the content on the platform? What about the people that sell it? Or sell parts of the platform? For an end user that’s all the same experience. It all goes hand in hand. So, all those people need to be involved in it. So, that’s where I see our next challenge.
We have a really cool pilot project coming up in April where in one of our media houses in Bergen, which is in Norway, we’re going to do an experiment with this kind of a pilot where in one week we will get around almost half of the employees to do all kinds of user interviews and be part of workshops and so on and to have kind of an intensive week exposure to their audience and their users and see how that impacts the organization and the product. We’ll do research on that pilot as well. Like look at it before, during and after. How do the employees of that institute, the local newspaper there, how do they think it impacts their work, their understanding? What are the short term effects? Long term effects? And how can we build on that pilot to see what we can do with it? That’s next. So, it’s not even that far in the future. It’s quite soon. Then we have to kind of take that to the rest of the organization and all the other product teams. It will be interesting to see how it goes.
Steve: It’s a reimagining of research, as you say. Moving from product – research is a thing that helps product versus research is a thing that serves the organization and the culture. That to me feels really big and very future looking and so exciting.
Kathleen: Yeah. It will be a lot of fun. Let’s hope it goes well. I’m not sure yet how we’re going to synthesize data. It still needs a lot of work before April, but we’re gearing up for it and it will be, I think – yeah.
Steve: Is there anything else I should have asked you about?
Kathleen: I was thinking about before, that people that want to shift their careers to user research always ask – what should I be reading? What should I be doing? Where do I even start? And there I think – I always recommend them like two or three books I think can be worth mentioning. I think Erika Hall’s Just Enough Research is great. But I also really always buy and share your book, Interviewing Users, with people.
Steve: Thank you. This is not a paid endorsement.
Kathleen: No, no. Not at all. But I think this is like the fundamental basics of just getting started and it’s a lot about that. It’s not so much about which university do I go to and which master program do I follow, or which kind of courses do I need to pay a lot of money for. It’s more about reading up on it and trying to get started and trying to get it into your every day work. Or if you’re a designer you can be doing it. If you’re a product manager you can be doing it. But also, if you’re working in HR, you can start trying it out, experimenting. You just need to do it a lot to get good at it. And you can come from any kind of background to become a user researcher really. You just need to get started.
Steve: What if when you were doing political science research, what if Erika’s book had been around then, or my book had been around then? Like would that have – I mean I realize you didn’t know that that was a thing that you were going to go learn from. I’m just thinking of you at the point. You’re kind of advising a hypothetical person, but just going back in time, what did you have to sort of teach you how to do the research that you were doing?
Kathleen: Yeah. Well, I think in social sciences that’s all you do. You learn research methodology and you learn to master that. But I think I would have loved to see both these books, but also online there are so many great resources – articles and so on – that show you how it’s practically applied in the industry. And it’s so different from academic research and it’s so tangible. You can see direct usage impact, change and so on. I think if I would have seen and known about that I would have switched earlier on. Then I would not have gone into academic research in the beginning at all. I would not have toyed around with should I do a PhD, or not? No.
Steve: No.
Kathleen: No way. So, yeah I think it would have been really inspiring. And I think for all these people who are in academia it is sometimes hard to – you know, what can I do with my skills? What can I do with my knowledge? But in the industry there is so much. There’s such a big need for people with these kind of skills and such a big need for that kind of expertise. I highly recommend.
Steve: I keep meeting more people who are hiring PhD and similar level academics to come into industry. And more people in academia that are excited, and a little lost, but excited about making the movement into industry.
Kathleen: Yeah.
Steve: And kind of applying – I don’t know – back in my early days that seemed like those were two camps that philosophically didn’t trust each other.
Kathleen: Yup.
Steve: And I don’t encounter that attitude anymore.
Kathleen: No.
Steve: There’s just people from both worlds are looking to embrace each other and so on. So, that makes me hopeful for just the field growing in that way.
Kathleen: But you know what’s so funny? ‘Cuz like I feel like I’ve come so far from any kind of academic style research and so on, but it was a couple of months ago that one of the designers – you know I was talking with him about how you standardize and how you can get like frequent research going and talking about the research objective and he just said, “you know what, you’re just so academic.” Just getting the basics, for some people that still feels too academic and you think like oh my, I need to do more – think more about how I share these things or what I say. Is the fundamental basics, you need to have a goal too, or an objective for any kind of study? But maybe we should hold something different because it just sounds too farfetched, or too kind of highbrow for someone.
Steve: I’m impressed that you took that as a growth opportunity or a challenge for yourself.
Kathleen: Not at first moment. At first I was just like, what, you’re kidding me? And then I thought about it a bit more and like okay, if I’m getting this kind of feedback then maybe I need to change and do my own kind of internal research. If I want them to do their own research then I need to approach them at their own level and using the words and kind of things that they can relate to. Otherwise it’s like how am I going to convince them otherwise?
Steve: Well great. Do you have anything else we should cover?
Kathleen: Don’t think so. Thank you so much.
Steve: Well thanks. This has been a great conversation. I’ve learned a lot and lots of ideas. It was great to have the time to chat with you.
Kathleen: Alright.
Steve: Thank you.
Kathleen: Thanks.
Steve: Thanks for being here for this episode of Dollars to Donuts. Follow the podcast on Twitter, and subscribe at portigal.com/podcast, or iTunes, or Spotify, or Stitcher, or your favorite podcast distribution platform. Also at portigal.com/podcast you can find the transcript and links for this and all episodes. Buy my books from Rosenfeld Media or Amazon. Thanks to Bruce Todd for the Dollars to Donuts theme.
The post 18. Kathleen Asjes of Schibsted Media first appeared on Portigal Consulting.
This episode of Dollars to Donuts features Kathleen Asjes of Schibsted Media. In our conversation, we talk about what happens when research need exceeds resources, the importance of keeping the knowledge inside the organization, and the benefit of diversity in a research team.
It’s not so much about which university do I go to and which master program do I follow, or which kind of courses do I need to pay a lot of money for. It’s more about reading up on it and trying to get started and trying to get it into your every day work. Or if you’re a designer you can be doing it. If you’re a product manager you can be doing it. But also, if you’re working in HR, you can start trying it out, experimenting. You just need to do it a lot to get good at it. And you can come from any kind of background to become a user researcher really. You just need to get started. – Kathleen Asjes
Show Links* Original Stroopwafels * Van Stapele Koekmakerij * Kathleen on LinkedIn * Schibsted Media * Svenska Dagbladet * Aftonbladet * Just Enough Research by Erika Hall * Interviewing Users
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Welcome to this episode of Dollars to Donuts, the podcast where I talk to people who lead user research in their organization.
I’m going to be teaching two public workshops about user research, coordinated by Rosenfeld Media. The workshop is called Fundamentals of Interviewing Users and will be held May 20th in New York City and September 13th in San Francisco. You can find more details about the workshop and register at http://rosenfeldmedia.com/public-workshops. I’ll put the links in the show notes. You can send people from your team, or you can help me out by recommending the workshop to someone else.
I also run user research training workshops for in-house teams as well as work collaboratively with teams to run user research projects and bring new insights into the organizational culture and decision-making processes.
I enjoy making this podcast, and I appreciate your support in hiring me – and in recommending me – that helps make it possible.
I was in Amsterdam recently and in preparation for the trip I did a little bit of research about interesting dessert options that the city had to offer. It had been ten years since my last visit and I already knew I wanted to head back to Original Stroopwafels at the Albert Cuyp Market. But what else was there? Looking online I found multiple enthusiastic posts about a chocolate cookie bakery. The reviews were breathless hyperbole — the best cookie you’ll ever eat, and so on. So on our first day, jetlag notwithstanding, we walked over to Van Stapele Koekmakerij. We found it on a tiny street, with a small line of people across the street, and a cookie bouncer, maybe more of a butler, keeping people in place, and letting people into the store, in small groups. The store itself was also quite tiny, and it turns out they only sell one product – just this chocolate cookie, filled with white chocolate. The main option we had is simply how many of this cookie we wanted to buy (with a secondary option around what you put the cookies in – a regular bag or box, or a gift tin).
The store itself was beautiful, with a classic logo on the window, and lots of wood and gilt and glass, just what I pictured an old Dutch bakery would look like, the kind of place that has a time-tested recipe. We bought our cookies and then stood on the street corner to meet a local friend. Meanwhile a passerby – an expat Brit – heard us speaking English and stopped to ask us about the cookies. Are they really that good to justify the lines of people that she’s always seeing in this neighborhood? We thought they were pretty good, and we only waited in line a few minutes. We didn’t tell her this but it’s hard for any cookie to live up to the Best Cookie Ever hype that we found online, but it was a good cookie! Anyway, this woman went on to explain that this store was pretty new, like in the last couple of years. I was very surprised, as the appearance of the space, and the overall narrative, led me to assume it was someplace that had been doing this one thing – selling only one kind of cookie – for decades if not longer. The cookie was still good, but this took away a small bit of the magic away, this idea that that we had finally connected with a long-standing Best of Amsterdam experience.
It got me thinking about authenticity, that slippery topic. This bakery created a seemingly authentic experience, implying a time-worn history – even though it’s very new and only tourists shop there. The fact that the truth belied my assumptions suggests that it was somehow inauthentic, but I would still put them on the authentic side of the equation. T hat the place looked and felt (and tasted) authentic, even though technically, it isn’t?
I’m reminded of one of the most fun aspects of user research, trying to tease apart these underlying meanings, when even the words we use are inadequate to fully capture or communicate the nuance. Imagine trying to interview someone about their experience with this bakery and how you would go about unpacking their interpretation of the value proposition. I should note I am available for this work and I may be willing to accept payment in the form of desserts.
Moving on from philosophical cookie musings, let’s get to my interview with Kathleen. Kathleen Asjes leads the user research team at Schibsted Media in Sweden. Alright Kathleen, thanks for being on the podcast.
Kathleen Asjes: Thank you for having me. Great to be here.
Steve: Let’s start with an introduction. Who are you? Where do you work? What do you do there?
Kathleen: So, I work at Schibsted Media in Sweden, but I’m originally Dutch. I moved to Sweden around 8 years ago and before that I also moved around quite a bit. So, been here and there. And I lead the user research team at Schibsted. We are 9 people altogether right now.
Steve: Nine?
Kathleen: Nine researchers. And yeah. I manage the team. I make sure that we advance our research practice. That the research maturity of the organization kind of advances. For those who don’t know Schibsted Media, which is basically the whole world, we are a media company based in Scandinavia and it consists of – it’s a conglomerate of all kinds of online newspapers – on and offline newspapers – media houses basically. And they are the biggest ones in Norway and Sweden. Eight altogether, but also some like regional, coastal newspapers. So, it’s a mix of both national and regional newspapers as well as other niche products that we have.
Steve: That are not newspapers?
Kathleen: Yeah. So, in total we work with around 34 product teams. Some of our teams – there’s many, many different kinds of product teams. Some of them are related to these newspapers, but others are like Scandinavian Weight Watchers and price comparison sites and so on.
Steve: That are all part of this media conglomerate?
Kathleen: Yeah, yeah. And Schibsted consists of both media and marketplaces. So, it started out as a Scandinavian company, I think 200 years ago almost. It’s old, old, well-established, but it also has marketplaces all over the world. So, in France, in Spain, in South America even.
Steve: What’s a marketplace in this context?
Kathleen: Oh, it’s online selling and reselling of used goods.
Steve: And are these all different brands? There’s not a global, international marketplace brand?
Kathleen: No, no. This is all kind of local brands. Local marketplaces that are like national marketplaces in these different countries, but they are all hosted by Schibsted.
Steve: Okay. Now what’s – this is a broad question, I guess. What’s the mix of things that were acquired – I guess how does this media conglomerate come to have such a more diverse portfolio than you’re describing?
Kathleen: I have no idea, to be honest. I think it just, as a media house it’s in their DNA to experiment and to be first and to be fast. Actually, kind of like agile. So, when it comes to kind of new products or new opportunities they act fast and they experiment and they, yeah, widen their horizons and do it in a very sustainable way. Like focus on growth and kind of maturing products.
Steve: Have you – during your time there, have you seen them move into a new – not product category, but sort of geographic region? Like a part of the world where they didn’t have any business before?
Kathleen: Yeah. So, they do this all the time, especially in the marketplaces, but now also with kind of personal finance and borrowing money services and so on. So, yeah, they’re expanding and looking at new regions all the time.
Steve: So, in those examples, that’s businesses that they’re already in and then how can we do that in another territory?
Kathleen: Yeah. How can we take this to a neighboring country? Or how can we expand this in a different regions? We see opportunities somewhere else, what can we do?
Steve: I’m starting to see a little why there’s so many different product teams that you’re working with.
Kathleen: Yeah, but this is only – like, I’m only on the media side. I’m sorry, I should be really clear about that. All the marketplaces stuff, I’m not in that group. My colleagues are.
Steve: Do you have peers who do research in that part of the business as well?
Kathleen: Yeah. So, they’re researchers. My team is based in Oslo and Stockholm, but there’s also researchers based in London, Paris, Barcelona, South America. So, altogether, I think, in Schibsted wide there are around 25-30 researchers.
Steve: So, you’re involved in the media side.
Kathleen: Um-hmm
Steve: I’m just wondering sort of how many clusters are there? Media is one.
Kathleen: And marketplace is the other. They’re like the big two classes.
Steve: So, you’re 9 and they’re – I’m doing math now in my head – so that’s about 16 people probably in the marketplace.
Kathleen: It’s a community of researchers where you make an effort to see each other every year which is good.
Steve: So even though the things you’re working on are quite different – different industries, different businesses – there’s still a sense of trying to keep connection?
Kathleen: Yes.
Steve: What are you trying to accomplish by bringing these different researchers together?
Kathleen: I think since user research is not a new kind of field, but is ever evolving and it’s really important for us to make sure that we evolve as user researchers. And you do this in dialogue. Like this is why we attend conferences, and this is why you explore new methods and new things and you want to share these things of course. So, I think that’s why it’s important that we meet and talk about these things face to face because from that kind of dialog you get a very different experience than just seeing a presentation or talking to each other online. So, to have a community, to have your peers, to have others that give feedback on work and they make you reflect on what you’re doing and what’s going well and what maybe isn’t going that well, I think it’s really crucial to kind of advance your practice and to become more mature in your research in the organization you’re in.
Steve: How are those conversations different when you’re in person than through all of our collaboration platforms that we work in?
Kathleen: I think, as always, it’s not always just in meeting and having an agenda and talking about certain things. It’s also the discussions that you have during breakfast if you’re staying at the same hotel. Or after work while you’re having a drink. As well as also, well we hosted during our last meetup – we met in Budapest where we also did a visit. We visited another company and talked with their user research team and talked a bit about okay, what are you struggling with? What are we struggling with? And then you start asking each other questions and you get these really brilliant discussions going. So, one of the questions there was someone asked like okay, when it comes to really, um, keeping track of all the research, all the learnings that you have to know in an organization, how do you do that? Like, what is a good database or platform to store your insights? And we were like darn, we have the same issue. We were looking for a solution as well. And basically, everyone in industry is looking for a good solution to do this over time. And if anyone knows, please let me know. Reach out to me. I’m happy to hear.
Steve: Let’s maybe switch topics back and go back to setting more context. How long have you been in this role?
Kathleen: I came to Schibsted around four years ago, I think, and I came in as a consultant first on a very kind of different – one of those niche products. I worked with lifestyle which was like online fashion and interior decoration – really strange, but fun to work with. And then after doing that for almost a year my manager said okay we would like to have you board, would you consider becoming a permanent employee here? And I said, ah, only if we can make sure that this is like pure, like a research role. Before, as a consultant, I would kind of be – often be hired as a UX designer. Like the industry in Scandinavia was not that ready for user research as a profession, or expertise. Often when I would say, like 6/7 years ago – it was like oh the user research people – “like what?” I said, “oh you know finding out about user needs and really tailoring the product to meet their needs.” And then like, “okay, but that’s what you do at the end, right? So, what else?” We need someone at the beginning. You really had to explain well actually it’s at the beginning where you start and where you need to really understand what you’re doing and who you’re doing it for. So, often we’ll be hired as a UX designer and have to wing the design part most of the time, but do all the fundamental research kind of early on. So, I said okay. I was the one of the first user researchers in Schibsted. I said, okay, you need to make this a role and it needs to have – I see that all the teams need to be doing more user research, so I want to coach them and train them and make sure that we become more mature. So, that’s how it kind of started and I’ve been changing my role about every year. So, from research lead, working by myself, then I got like a small team to work with, with two other researchers. And then last year we finally kind of combined all the different researchers that were in media, but were working with their own separate kind of product, into one kind of – one team that works more independent of the different product teams and works more like okay, where are we needed? Where can we add most value? Where can we add most impact?
Steve: Was there a change in title for you when that happened?
Kathleen: Yeah. So, that’s when I became head of research and got the 8 or 9 people team up and running.
Steve: I’m curious about – I don’t know if it’s a fair question for you, but at what point do you think organizations go from – like there’s some evolution here of we’re doing research, or we have people that have the title of researcher, to now we have someone who is responsible who is a leader. That’s, I think, sort of the evolution that you described. But what is that point? I don’t know, is there a point at which that becomes clear that we’re ready or it’s time for leadership in research as opposed to sort of just the function being staffed?
Kathleen: Yeah. I think it depends a lot from organization to organization and I think it depends a lot on how – you know who is there? Like in the organizational leadership you need to have those people who want to promote it and see the value and really want to kind of make sure that research becomes leading in so many ways. And one way to get there is to really show your impact again and again and again. And do a lot of that lobbying and showcasing and communicating. Another thing can be that one of the senior VPs or someone who comes in has that experience from another company and goes like, “okay, I see where we’re struggling. What we need is professional user research. Let’s bring those people in or promote people into those roles and structure it more.” So, it can be bottom up, or top down. It can go both ways. I think in our case it was more bottom up.
Steve: Right. You hear stories about people were brought in with the intention of building up a function capability.
Kathleen: Yup. Build a team. Build a function. Change the organization. I never had that mandate to begin with. I created it.
Steve: You didn’t have the mandate, but it sounds like you had that trajectory. I mean in the conversations you had about taking a fulltime job versus being a consultant, it sounds like – I’m inferring from some of what you said that you’re looking towards more robust establishment of research that you would want to champion and drive.
Kathleen: Definitely, definitely. I think it’s so important because user research, for us as a function, as a role in the organization, we’re there to make sure that everyone in the product teams makes better informed decisions. We do a lot of gut feeling and looking at competitors and just coming up with stuff. We need to make those decisions more robust and we need better insights. So, that’s what we’re doing and for that to really get to a professional level and a good level you need to have it embedded in the organization and have it at decision making levels. Yeah. So, it’s always been – like I see where it’s needed, and I try to make sure it gets there.
Steve: So, part of the evolution that you were describing was there were individual researchers working on different product teams and then you reorganized so that you’re all working together and then putting research – you’re looking for where it’s needed and then putting research where it’s needed.
Kathleen: Yes
Steve: And I find this, and maybe you do as well, this is like a – this is a hot topic. Are researchers embedded, or are researchers centralized, are the terms I hear used most often.
Kathleen: Yeah. So, we’re in a matrix organization and then you’re a bit of both. So, every product has UX designers, product managers, developers. And then you have all the different – yeah, brands, across the other side of the matrix. And then yeah, we feed research into each of these kind of product teams, wherever it’s needed. So, you’re half embedded and half not. And ideally, of course, it would be wonderful if I had 34 researchers. Then yeah, everyone could be embedded. But I don’t see anyone granting me that kind of capacity anytime soon.
Steve: At least 34 right. Maybe there’s work for more than one researcher on that team.
Kathleen: Definitely. Let’s go for 40 or 50. Yeah, I would like to, you know, pose that question to the CEO or CTO, or whomever is making the decisions these days. And I’m not going to get at all, no. And I’m not sure if we do a better job at that scale because now what you see is when you work with multiple product teams is that they have similar challenges, similar issues, similar audience. So, you have more of that dialogue going on between the product teams and you’re facilitating that as a researcher because you have an overview of all the different issues and the similar issues that they’re trying to deal with. So, I think in that sense, yeah, we’re usually understaffed, as always, and there’s way more requests than there are people on my team, for sure. At the same time, it makes you make really tough decisions on okay, where can we add most value? Where do we add most? But also, you’re spread thin, so you need to make sure that teams do their own research, but you support them doing it. And actually, I think they do – I think the outcomes are used more and better if teams do it themselves. With your help, obviously. But yeah.
Steve: Would you have a theory as to why?
Kathleen: Well it’s all about that kind of firsthand experience. You know if you talk with your audience or users directly you’ve shaken their hands. You’ve seen what they struggle with. You know what’s happening to them and you will use that in your everyday decision making for the product. And also, you know, you can do a study, for example working on news consumption patterns. Like how often during the day do you access online news and which sources do you look at, for example? But during those conversations maybe really interesting things are coming up about subscription models. Or when you are willing to pay for news, or not. And if the product owner, or the designer, has those conversations themselves with users, then in the next project that comes up where they’re actively looking at subscription models or changing those, they will have that knowledge in their back pocket and take that up again. Whereas, you know, if I write a report, or share information about news consumption, that information about subscriptions will not be in there. Like that’s in my mind. So, if I’m the expert who knows everything and has all the conversations and all the interviews and gets all the raw data and starts analyzing that, and I’m the only one doing it, then I am the expert of everything. But I can’t be there for their everyday decision making, or their backlog grooming, or they’re deciding on the next step for their product strategy. So, the more they do themselves the more all those insights will feed into their everyday work.
Steve: So, can I just try reflecting back? So, a team that does its own research, one has the benefit of direct experience and two, has the things that seem extraneous to your initial question, these sort of weak signals, these other bits and pieces that you hear, that that rests with them, where they’re more likely to reuse it in the future when it comes up.
Kathleen: Yeah, definitely, definitely. And then you do have – on the flip side there’s the challenge of synthesis, data analysis. You know that’s quite – can be quite advanced and of course not everyone has the capacity to do so, both in time as well as mentally. So, it’s a fine balance between grooming and training them to do their own things – we’re still keeping an eye on it and helping them in the places and spaces where they struggle.
Steve: So, something like synthesis, which is advanced, what do you do to help a team that has not been…
Kathleen: Yeah. So, I would sit in on those sessions with them. So, for example, I myself support two product teams now and we’ve been doing this for around a year and a half, so they’re pretty good at writing their own kind of discussion guides, doing their own interviews and so on. But I noticed like, okay, when it comes to the analysis part they’re still – you know the findings are still kind of superficial. So, I would just try and sit in on those sessions. So, we’re listening to one or two of the interviews and sit in on the session, look through the raw data with them. And we start digging. And then you know, when people jump to conclusions too quickly – so, like “okay, where did you get that from?” It’s like, “oh the fear of missing out. It’s here and there.” “Well, actually no one every said that. It’s a term we use a lot when it comes to filter bubbles and so on. I don’t think I’ve heard any users say that in interviews.” And then make them dig to find the evidence and they go, “oh, no, I jumped too quick there.” Like “okay, let’s step back then and look again at what we did find.” So, yeah, but it’s different for every team really, what they need and where they’re at and some are at that basic level of okay what do I ask and how do I ask it? And others are a bit more advanced.
Steve: So, what’s your decision-making process in allocating your own resources?
Kathleen: Yeah.
Steve: Across these different teams where maybe a researcher in your group is going to do the research, or is going to support this other team doing the research themselves? You know, how do you plan all that?
Kathleen: Well, it’s really hard and actually lately there have been – I wouldn’t say conflicts, but there are – you know different teams are fighting with each other for who has access to which resource, which of course is a terrible situation to be in. But we do it in a way where – we’re 9 people now and we make sure that each researcher has multiple product teams that they focus on and they’re the first point of contact for. So, all the requests and so in come in for them and they follow a product team and the kind of research roadmap on what they’re doing. And then when a new request comes in it comes through me and then I’ll have to just, yeah, interact with the team and ask, okay, who has space? Do we think this is more important than something else? Do we need to prioritize this? And then we prioritize based on several things. So, first it is like where do we have most impact? Like where do we think we can generate the most important insights that are really needed for a team, but also where are we at ourselves in our professional, personal development and what kind of opportunities come with this research assignment or challenge that will help this or that person in our team grow personally?
Steve: Can you give a generic example of what that would be like?
Kathleen: Yeah. So, for example, we had an assignment coming up and we saw like okay, well looking at it we saw that okay this kind of challenge requires a diary study. And then there was someone on the team who really wanted to do a diary study and never done it before and wanted to come to try out that as a method. So, we said okay, well this is a great opportunity for you to work this out and experiment with this. So, now we put that person onto that assignment. So, it’s a big mix of different things really. Like where can we have most impact, but where can we also learn new things and try and challenge ourselves and grow, both as a team and as well as research professionals.
Steve: If you don’t have resources for the request that comes in, I’m sure – there’s a couple of things that could happen. That research could just not happen? Or it could be done by the team? I don’t know what – maybe I should put it as a question. If you can’t handle something, what happens to it?
Kathleen: I don’t think it’s done. It doesn’t get done. Or, we hire external agencies to do it, which I think often is the worst thing to do. And this is why I’m happy that we kind of came together as a research team because I think around 2 years ago it was like every individual researcher would be stuck doing a lot of tactical validation or kind of research and be really busy with that kind of everyday work, really trying to make sure that product teams look more at these kinds of insights and validate whatever they push out. Um, and at the same time we were thinking like okay we need this really big, strategic research done on the whole – you know what is the ecosystem in Scandinavia for both news consumption as well as online marketplaces and selling and reselling of used goods. And then we hired this agency to do this for us and spent loads of money on it. And then so all the really challenging, really big research questions are done by an external agency that deliver a report, but then all the super useful insights that are in extra information kind of – you know they stick there. They don’t really reach us. Or at least not into the organization. And we can’t do it ourselves because we’re too busy. That’s, of course, a terrible place to be in.
Steve: It sounds analogous to when a product team does research versus when the research team does it for them.
Kathleen: Yes. Yeah. So, it’s even one step further removed. It’s not even in the organization. So, some consultants know a lot about media consumption and the whole ecosystem in Scandinavia. Good for them. They got paid to do that as well.
Steve: So, as someone who works as a consultant, I’m definitely sensitive to – when you hear that your way of working is the worst possible option, that definitely makes my ears perk up.
Kathleen: I hope I didn’t offend you.
Steve: No, I’m not offended. It seems to me, in my experiences, that the mitigation for that is to have it be a collaboration, which is a different way. I think you were even talking about this in the way that you’re supporting your internal clients versus kind of doing it for them. I think for consultants, this is my opinion and my experience, like it’s a go take the research – to go take the tasks, execute it and report back on it. As a consultant, that’s not very fun. It’s not very rewarding. I don’t think it’s very effective. But I’m interested in sort of finding collaborative models. Again, just kind of like sometimes with your teams you’re coaching them, but they’re handling it and sometimes you’re in there doing the research with them. I think these kind of hybrid – they’re more blurry kinds of collaborations. Because I don’t want to know everything about some topic and then…
Kathleen: No, no.
Steve: That doesn’t suit my objectives. That doesn’t suit my clients’ objectives.
Kathleen: No, exactly. But that’s why you as a consultant need to have a really thorough discussion about that at the start, right? Like how can we make sure that these findings and insights kind of land in the organization and are usable by everyone. Rather than it just being the deliverable that then kind of goes from one’s inbox to another inbox. And then four weeks later it’s forgotten. Or in some kind of drive somewhere.
Steve: Right. Your point earlier about trying to help them have some experience where something sticks with them. Like that’s definitely the model of my own business and I think as researchers we’re capable of facilitating that experience in a way that – I mean this is all what you were talking about, setting up these teams to be successful. So, how do you help them have those experiences? It sounds like that’s what you’ve been working on a lot.
Kathleen: Yeah, definitely.
Steve: I think we agree more than we disagree.
Kathleen: We do. We do.
Steve: Let’s see if we can loop back to something you mentioned before. You were talking about one of the advantages of this kind of matrix model is that researchers have experience across different products and different product teams and they’re able to bring something that they’ve learned into a different context.
Kathleen: Yeah, exactly. And also, I think our – we try to make sure that our product teams do kind of continuous research, or rolling research, whatever you want to call it, so that they bring in their end users every month – at least every four weeks or so. And then, you know, you have a batch of 5/6 people coming in every month for all these different product teams. So, it adds up to a lot of people in the end. And over times it adds up to talking to very many people across time. So, we try to make sure that they have some kind of fundamental questions that they ask everyone. So, across these different product teams that we all ask about media consumption habits. Like just to really understand, okay, you are reading Svenska Dagbladet, the Swedish national newspaper, but which other sources do you have for your information? And then it might turn out that well actually I’m also daily visiting Aftonbladet, which is the other big national newspaper that we also own. And how do these two compliment each other? To ask about which other products are you using? How does this complement each other? Are there competitors there? So, these are the kind of questions every product team asks and that we can also kind of look in across, or over time. Do we see things changing? Or do we see like across the board to kind of see overlap, or learn things that other product teams should know about. While talking with Svenska Dagbladet readers, what do we learn about? Aftonbladet at the same time.
Steve: Just yesterday someone was asking me about their organization and describing – they put it as very siloed research and they were feeling frustrated as a researcher – I think they were a designer who’s doing a lot of research – that they were seeing things that they knew could apply elsewhere and they didn’t know what to do with them. And you’re kind of describing sort of a proactive structure for that. That there’s some things that are going to come up and you’re going to intentionally ask in a lot of different research.
Kathleen: Yes.
Steve: And then, I guess this would be the challenge for this other group – then you have to put the time in to say well what does that tell us?
Kathleen: Yeah, exactly. And then not just keep it on the shelf, but actually use it. And I think there we still have a way to go, to be honest. Like we try to make sure these things are in place, but to actively spend time digging through it and making sure it’s useful. Like I think we can take extra steps there. Yeah.
Steve: At one level it’s there if you need it, to go look at it, but to be proactive you would have to go and say well what are we learning?
Kathleen: Yeah. What did we learn this quarter? What did we learn this half year? We compile all that data.
Steve: One thing I’ve heard people talk about is – you know you’re describing managing these different requests that are coming in and prioritizing them based on these different criteria. I’ve heard people talk about one thing they can do with those requests is say, “we already have that information.” Like recycling data, or recycling insights, as opposed to redoing the research.
Kathleen: Yeah, exactly. So, often it is often about seeing like okay, you want to have information about willingness to pay for online use and so on. We know from your counterparts in Norway that they’ve done this in this study, have a look at that and see if you can reuse some of those insights, or if you think they apply to you, or if you can build on that. So, there’s a lot of that kind of cross-functional learning. But then still it becomes – I’ve been there for four years now, so I’ve seen and heard quite many things, but then still the way – you know what is the one central database or place where you will know that that information is there, even if people have left. Or what if I leave? Like where does all that knowledge go?
Steve: That’s a theme I think we keep coming back to. Sort of where does knowledge go?
Kathleen: Yeah.
Steve: Right, the external consultant? Or you delivering the report? The kind of matrix way of working for the team and then where does that live? It seems like this is a really significant challenge. As you said, the folks that you met with in Budapest, that was also an issue for them.
Kathleen: Yes. I think it’s an issue a lot of people struggle with. Like across everything. And then, yeah, do you need an organizational Wiki or database or, I don’t know? Do you have the answer? Have you heard the solution?
Steve: I had a really interesting conversation with someone about this and what she said to me was yes you can have the infrastructure, whatever database tool you want, but they hired basically like a reference librarian role. In her experience it would fail unless you staffed it. This is a job and it takes time and it takes effort. So, that you have somebody there…
Kathleen: And is it worth it? Because what is the shelf life of each insight or thing over time? You know things are changing rapidly. Sometimes I wonder a bit about that. Like things that happened four or five years ago and it’s still relevant for us now. Is that insight still valid? Or have things changed?
Steve: That’s interesting for a 200 year old company. Some things clearly are changing very rapidly around us, but what are the fundamental things that – I don’t know. I don’t know how you even kind of assess that.
Kathleen: Yeah, but if you think about reading news online and something that for many of us is an everyday habit now, but 5 or 10 years ago it wasn’t.
Steve: But the role of – the role of news in being a citizen, I mean has that changed?
Kathleen: No, no, no. And that’s also what’s so interesting about working in this field. When I graduated I wanted to be a journalist actually because I thought it was really exciting and really such an important thing. Free and independent journalism is such an important thing in society. Now, all these years later, I get to work with journalism. So, I’m almost there. Just in a different kind of capacity. Yeah. So, that’s quite good.
Steve: Right. Is there a sense among the team that you are – I mean do you all identify as working in the news industry?
Kathleen: Yeah. I think that mission that Schibsted has about really providing free and independent journalism is really important for most of us. People really align along that mission.
Steve: I think there’s something about the work of research – as someone who has worked as a consultant, you get into just a wide variety of industries and topic areas and I feel like among researchers that’s a bonus.
Kathleen: Yeah.
Steve: Sometimes I’ve had this conversation with people when they approach me about some research problem they have and exploring if there’s a possibility to collaborate? They’re describing something they’re doing, and they say, “is this interesting to you?” I’ll have to kind of laugh because…
Kathleen: It’s all interesting, isn’t it?
Steve: It’s all interesting, right. And so – but you were describing maybe the opposite. And then obviously there’s thousands of problems, even within the space that you’re in, but I guess it’s breadth versus depth that you’re in an industry and you’re sort of focused on the commitment to that mission. Whereas if you’re sort of a journeyman researcher like myself you move across very different kinds of things. And that in itself is rewarding, but I don’t have the mission thing that you have.
Kathleen: But I’ve been there as well. I’ve also worked as a consultant and I’ve worked with offshore engineers that have high voltage cables across the North Sea and so on. And who would have thought when I came out to Sweden that I would be working on this factory site with a yellow helmet on my head and looking at what these engineers are coming up with and what they’re doing and so on. But that’s an amazing thing, isn’t it? You get to really explore things you didn’t know about were happening and what they’re like and get to ask all the dumb questions to really get a good understanding. But after doing that for four or five years, I also at some point felt like yeah, often you are asked to be there at the end where things have failed, or things are not going well. How interesting would it be to be there along the way? Like I said really helping teams be better informed and make better decisions. For that you need to stick around for a while. That’s also super interesting right now. Yeah.
Steve: So, as you’re talking a little bit about some of the career – elements of your career path, can you maybe step back even further and talk about what you pursued in education, when you were thinking about journalism and then where you ended up in the work world and sort of what led you to where you are now?
Kathleen: Yeah, good one. I studied – well first, I managed to study psychology for about 3 or 4 months when I came out of high school because I was so interested in people. I thought oh I need to – that will help me understand the world and people better. And I got super frustrated with it in more time and I thought the course in statistics was more interesting than the introduction to psychology. So, I noticed very quickly like okay, if statistics is more interesting than actually the introduction to the thing that I’m supposed to study for the next 4 or 5 years, then I’m in the wrong place. So, then I switched to political science. So, I have a masters degree in political science which has really given me a really good fundamental basis in research methods and skills, both quantitative, as well as the qualitative side of things, which I think is really great, now in hindsight. Because, you know, after studying for several years you start hating statistics after a while as well. But I’m happy I did it because it’s given me good insights into mixed methods and how you need to combine both qualitative and quantitative data to really, yeah, validate your own findings, as well as make your findings kind of – and insights stronger and have more impact, which is something that we really need to do as user researchers. Because we’re still being questioned about okay, why did you only talk to five people? I had that the other day, like why are you only – and I was like oh, these questions, are they still coming up? Interesting. So, I studied political science and after that I worked in research, like just plain academic research as well as in policy research for a municipality, but I noticed that I was more interested in technology and what it does with people and how people use it and how it enables people. So, after a while, also moving around from Holland to Australia and coming back again, managed to find a job in the Library for the Blind where we were producing materials for visually impaired people. And there we spent – I was a product manager for a while and I noticed that we were building things and making things – and I was in the research and development department – and we were spending lots of money and lots of our time on things that were actually not really helping visually impaired people at all because we were stuck kind of thinking in old ways, or thinking that we know what visually impaired people need, whereas we didn’t know anything, really. We were just doing it out of goodwill, thinking we were helping. But actually, things change. So, that’s where I started to really focus on user research and looking at how do we develop these products, how do we develop an online library for visually impaired people and what do they need in their everyday lives to access things online? What devices do they use? So that was when I really started to focus on user research.
Steve: Were you thinking with that label, user research, at that point?
Kathleen: No, not at all. In the U.S. I think user research is more established than what’s in Europe and it wasn’t until, I think, six or seven years ago, I was at a conference in the U.S. where actually someone, I asked them, “so what do you do?” And she said, “oh I’m a user researcher at Google.” I’m like, user research, that’s what I’m doing. We talked about what her job was and what she was doing, and I was like that’s exactly what I’m doing. So, now I had a title for it and actually, you know, a good understanding of what it was and what you could call it. But then no one else in the industry got that yet, in Europe. So, it’s a bit behind in some ways.
Steve: So, what happened after the working on Library for the Blind?
Kathleen: So, after that, I managed a team for a while to really become more user centered. I did it for 3 years almost and then we moved to Sweden, from Holland to Sweden and then had to reinvent myself and what I was and what I was doing. And then I started consulting. So, I started consulting and working with UX (user research) in different industries). So, I’ve done everything, like I said, from offshore engineers to 3D kitchen modeling comparison. I’ve done a very interesting study on heating systems, heating pumps and how to explore interfaces for that. It was a really fun and weird assignment. And ended up doing an assignment for Schibsted and stuck around because they’re interesting to work with.
Steve: You know your whole journey is – I love that you were doing research and didn’t know that it was called that. That you had sort of found your way to it without the benefit of that label or that identity or some of the professional community stuff. It just makes me think about – I think your example is common. If you look at sort of trajectories over time – and I think you’re right, geographically it’s different – U.S. versus Europe versus other parts of the world – but there was a time where anyone who was in user research was there through an indirect – I don’t even want to say non-traditional because there was no alternative – but everyone was there through a mostly somewhat kind of indirect path. And as time moves on and the research industry grows there are people who identified that early on, chose an educational path that would sort of prepare them for that.
Kathleen: You can study it nowadays.
Steve: You can study it, right. Whereas we remember when people would say, what is that? That’s not a thing. I don’t know, so what are the – as time moves on and we have a mix of people that have studied it versus people who have – I don’t know. This is a little pretentious, but been called to it because we identified it – I guess you can study it and be called to it. But someone like me that – I was not trained. I found my way like you did. What does it mean for the practice, kind of collectively, as we have this mix – as that mix changes, I’m going to guess more people are being training in it now. The mix includes more people who studied than it did in the past. That’s a hypothesis.
Kathleen: Yeah. I think it does, but at the same time I don’t think it should. I think the richness of our practice, research practice, actually comes from people with different backgrounds and different kind of previous experience. So, that’s also what I look for in my team. Like everyone comes from a different background and when I recruit for my team then I also actively seek people that come from something else. Different nationality, different professional background, different academic background, different place in life from juniors – really junior – I just hired someone straight out of, psychology she studies – to more senior and have been around for a while and have all that expertise from just industry or academically. And I think that’s how you as a team, and as a person, can grow most, through that mix. And that’s how we can do our best work. I would not want to have a team of nine trained user researchers.
Steve: Why?
Kathleen: Best quality comes from diversity, quite often. That’s been proven time and time again. From the more different angles you can look at a problem the better your answer or solution will be. And I think that’s the same for a user research team. You need, you know, a qualitative research expert. You need quant research experts. You need people who are more experienced in design so that they can come with really good recommendations on how to solve problems. You need people who have you know – yeah, different – my team as well. There are some anthropologists, psychologist, someone who comes from product design, like physical product design, or from digital product design, human computer interaction, UX – you know we’re a bit of everything and we actively want to be – the more diversity, the better. The more we can learn from each other then the better we will get at what we do.
Steve: Do you think there’s any connection between – you know you mentioned before that people are still asking the sample size question. Is there any question between that, as kind of how well research is understood or not understood, and – your diversity of thought, your diversity of educational background is not – I don’t think that’s an unusual way of articulating it. And I’m with you that that’s a strength, but I wonder – you know a practice that is made up as kind of a conglomeration of different types of thoughts and educational background, are we creating challenges for ourselves in being understood by others because we have that? I mean diversity – I agree, diversity is a good thing, but there’s sort of almost like a ragtag aspect of it. Like oh, we’ve got a psychologist and a product designer and someone from political science. It could be misconstrued as sort of a lack of intentionality or standardization.
Kathleen: The more well established it is through education and in like product development processes, the more standardization there is the easier it will be to get recognition for what you’re doing. At the same time, if you really deliver high quality insights and really can also show the impact of these are the insights, this is what we’re learning, this is what the product team is also changing based on that, this is how it’s changing our revenue, this is how it’s changing our product, this is how it’s really improving the user experience, then yeah, it’s in that proof and in showcasing and also labeling it, naming it. Like we got here through this and this insight and this is how we managed to change things around. And people can also get there. But that level of understanding and acceptance of we talked to six people and we know more now than we know from just staring at our bellybuttons and looking in-house to ourselves or our competitors.
Steve: I think what I’m hearing you saying is that the thing that helps advance the field is the work. The work being good, the work being relevant and kind of advocating for the work. And my thing about what kind of backgrounds do people have and so on, that’s probably not as relevant as the thing that we’re here to do, that people care about. That’s what they care about.
Kathleen: At the same time, of course, it is tremendously helpful that large companies have really professional research teams and that they actively do this and that there is education and training in this and that level of, yeah, making it more professional. That helps us, of course, as well, because then I can also say, “you know what, at The Guardian, they’re doing it this way. We could be doing something similar.” Or, they have learned this and this and we should be looking at that as well. Yeah, it goes both ways.
Steve: So, you’ve talked about just the diverse team where people have exposure to just different modes of thought, different backgrounds. And then you gave that example where you might choose how to prioritize a program, among many that you’re being asked to do, so that someone on the team – you mentioned the diary study example. And I guess, so just more broadly how do you think about the team, where they are at now and developing them. Kind of helping create the conditions that can support professional development within your team.
Kathleen: Yeah, so I think part of being – what’s so good about being part of a team is that you have people with different kinds of expertise that you can learn from and that will help you kind of grow as a research professional. And I think it’s really important. There’s a high demand for user researchers. So, they’re being poached like left, right and center if you get the opportunity. LinkedIn requests galore if you have user research in your title. Some recruiter will at some point go like, “hey, should we be talking with you?” So, to counter that you need to make sure that people are challenged in their everyday work and that they feel that they’re growing and that they’re developing, and they get to do really challenging things. And in our team we’re looking at that from kind of – we have a sparring system where every month you team up with a different person in the team and every week you spend at least an hour sparring on what am I doing, how’s it going, what are my challenges right now, can you give me feedback on this script, or this method or this approach, set up, you name it. It’s up to them themselves. But just to make sure that you get active input from not just within the team setting, but also one on one, to help you kind of get different inspiration, different feedback, all the time. And as well, besides the sparring, making sure that people get to do really challenging assignments and that we put them on kind of the research that’s not just, you know, whoever, which product team yells the hardest gets to put a researcher and get to say what they have to do. But to make sure that we focus on the team so where we can have high impact and really mean something for the organization as well as the product team as well as for the researcher themselves. So, I think there we’re seeing, compared to a year ago when everyone was in their own product, in their own silo, doing their own thing, we’re seeing lots of benefit from that now. Being a central team, of course, can be challenging because you’re getting more requests on what you can do, but at the same time you get to really learn from each other and learn from your peers, and yeah, develop yourself, your own practice, quite a bit. So, I think that’s really what’s super inspiring about being a research team, or having that. And I can recommend to everyone who is like a research team of one – because there’s many of us around – to try and find your peers and try and find a research community around you. It can be a different organization, or it can be even online. It doesn’t matter. Try and find people that you can spar with and you can talk with and learn that way.
Steve: I have to admit, I haven’t heard the word sparring used in that context. I don’t know if we have a regional difference here, or my lack of exposure to certain kinds of environments. I don’t know.
Kathleen: Maybe it’s a Scandinavian thing. I don’t know.
Steve: I think of sparring as like practicing in – like in boxing.
Kathleen: Yeah.
Steve: So, when you said the word it brought to mind something adversarial. But as you unpack it I’m like oh, this is a trusted partner that you can – I think a sparring partner in sports, in boxing, is someone that – it’s like a friend that you can hit, right.
Kathleen: Yeah. Someone that challenges you.
Steve: Yes. And you have to have trust.
Kathleen: Exactly
Steve: It’s a safe place to go outside your comfort zone.
Kathleen: Yeah. And that trust is something that has to grow over time. Because I remember when we came together as a team at first there really was some doubts and questions about so what is this going to add for me? Like why should I be part of this? So, it was hard to work to make sure that people really, you know, yeah, got an understanding for what is your expertise? What is my expertise? What are our different backgrounds and how do we complement each other and how do you add to my work and how can I add to your work? So, it took a couple of months. And yeah. It’s a lot about meeting each other face to face and making an effort to see each other every month, even though we’re in two – well, Oslo and Stockholm is not far from each other. It’s an hours flight. You can see each other. So, we do that every month and it is a mix of both professional, as well as eating out.
Steve: Right. We’re looping back to what you were saying before about just the overall research community at the company.
Kathleen: Yup.
Steve: That even within the team that’s how you sort of turned a bunch of individuals into a team was through spending time together.
Kathleen: Yeah, exactly. And then, you know, if you’ve seen someone, if you’ve met them, it is easier to reach out online and post a question and get someone to then maybe follow up and talk with you or have like, yeah, that kind of communication with.
Steve: You’re describing something that’s at least somewhat parallel to the work itself. That being in the same space with people and talking to them. You talked before about meals. Maybe you’re not literally breaking bread with somebody, but you are sharing space with them. There’s things that happen, that I think can only happen in that environment.
Kathleen: Yeah.
Steve: That’s the research, but that’s also team building and establishing a shared culture and a shared set of beliefs. I think you described going from like what’s in it for me to what’s in it for us. I’m paraphrasing you, but that kind of shift.
Kathleen: Yeah, but I think you’re right there. Yeah. And it is – yeah, it’s both. You need to – by supporting your research community you get lots back as well. So, it goes both ways.
Steve: So just looking into the future, the researcher’s future looking question, whether this is about your own personal, professional trajectory or about the organization you are in that you’re building, or the practice in general of research, you know some number of years in the future, what kind of things should we or you or your team be heading towards?
Kathleen: I think when it comes to my team and the organization I really think we’re maturing quite well in terms of how well research is established and what we’re doing and so on. But I think the next hurdle for us, or for me, professionally what I want to see is that – in the moment I’m talking a lot about how can we make sure that product teams understand their audience better and understand their user better and have that firsthand experience. And now I’m working towards the next phase which is okay, how can we get the organization to understand their audience better. So, now okay Schibsted Media, there’s a product and tech organization of around 300 people or so, and I want them to do and understand user research and we need to spend a lot of time training them and coaching them and mentoring them. Great. But there are 2000 people, employees, working in Schibsted Media, and this includes like the journalists and it includes like salespeople. Ad space, or the work with subscription. And it includes customer service. And it includes HR even. So, how can we get them to start doing user research and get them to have that kind of first kind of access. Because otherwise it’s still a silo. It’s still the product and tech silo that gets the empathy and gets the understanding. Well, what about the people that create the content on the platform? What about the people that sell it? Or sell parts of the platform? For an end user that’s all the same experience. It all goes hand in hand. So, all those people need to be involved in it. So, that’s where I see our next challenge.
We have a really cool pilot project coming up in April where in one of our media houses in Bergen, which is in Norway, we’re going to do an experiment with this kind of a pilot where in one week we will get around almost half of the employees to do all kinds of user interviews and be part of workshops and so on and to have kind of an intensive week exposure to their audience and their users and see how that impacts the organization and the product. We’ll do research on that pilot as well. Like look at it before, during and after. How do the employees of that institute, the local newspaper there, how do they think it impacts their work, their understanding? What are the short term effects? Long term effects? And how can we build on that pilot to see what we can do with it? That’s next. So, it’s not even that far in the future. It’s quite soon. Then we have to kind of take that to the rest of the organization and all the other product teams. It will be interesting to see how it goes.
Steve: It’s a reimagining of research, as you say. Moving from product – research is a thing that helps product versus research is a thing that serves the organization and the culture. That to me feels really big and very future looking and so exciting.
Kathleen: Yeah. It will be a lot of fun. Let’s hope it goes well. I’m not sure yet how we’re going to synthesize data. It still needs a lot of work before April, but we’re gearing up for it and it will be, I think – yeah.
Steve: Is there anything else I should have asked you about?
Kathleen: I was thinking about before, that people that want to shift their careers to user research always ask – what should I be reading? What should I be doing? Where do I even start? And there I think – I always recommend them like two or three books I think can be worth mentioning. I think Erika Hall’s Just Enough Research is great. But I also really always buy and share your book, Interviewing Users, with people.
Steve: Thank you. This is not a paid endorsement.
Kathleen: No, no. Not at all. But I think this is like the fundamental basics of just getting started and it’s a lot about that. It’s not so much about which university do I go to and which master program do I follow, or which kind of courses do I need to pay a lot of money for. It’s more about reading up on it and trying to get started and trying to get it into your every day work. Or if you’re a designer you can be doing it. If you’re a product manager you can be doing it. But also, if you’re working in HR, you can start trying it out, experimenting. You just need to do it a lot to get good at it. And you can come from any kind of background to become a user researcher really. You just need to get started.
Steve: What if when you were doing political science research, what if Erika’s book had been around then, or my book had been around then? Like would that have – I mean I realize you didn’t know that that was a thing that you were going to go learn from. I’m just thinking of you at the point. You’re kind of advising a hypothetical person, but just going back in time, what did you have to sort of teach you how to do the research that you were doing?
Kathleen: Yeah. Well, I think in social sciences that’s all you do. You learn research methodology and you learn to master that. But I think I would have loved to see both these books, but also online there are so many great resources – articles and so on – that show you how it’s practically applied in the industry. And it’s so different from academic research and it’s so tangible. You can see direct usage impact, change and so on. I think if I would have seen and known about that I would have switched earlier on. Then I would not have gone into academic research in the beginning at all. I would not have toyed around with should I do a PhD, or not? No.
Steve: No.
Kathleen: No way. So, yeah I think it would have been really inspiring. And I think for all these people who are in academia it is sometimes hard to – you know, what can I do with my skills? What can I do with my knowledge? But in the industry there is so much. There’s such a big need for people with these kind of skills and such a big need for that kind of expertise. I highly recommend.
Steve: I keep meeting more people who are hiring PhD and similar level academics to come into industry. And more people in academia that are excited, and a little lost, but excited about making the movement into industry.
Kathleen: Yeah.
Steve: And kind of applying – I don’t know – back in my early days that seemed like those were two camps that philosophically didn’t trust each other.
Kathleen: Yup.
Steve: And I don’t encounter that attitude anymore.
Kathleen: No.
Steve: There’s just people from both worlds are looking to embrace each other and so on. So, that makes me hopeful for just the field growing in that way.
Kathleen: But you know what’s so funny? ‘Cuz like I feel like I’ve come so far from any kind of academic style research and so on, but it was a couple of months ago that one of the designers – you know I was talking with him about how you standardize and how you can get like frequent research going and talking about the research objective and he just said, “you know what, you’re just so academic.” Just getting the basics, for some people that still feels too academic and you think like oh my, I need to do more – think more about how I share these things or what I say. Is the fundamental basics, you need to have a goal too, or an objective for any kind of study? But maybe we should hold something different because it just sounds too farfetched, or too kind of highbrow for someone.
Steve: I’m impressed that you took that as a growth opportunity or a challenge for yourself.
Kathleen: Not at first moment. At first I was just like, what, you’re kidding me? And then I thought about it a bit more and like okay, if I’m getting this kind of feedback then maybe I need to change and do my own kind of internal research. If I want them to do their own research then I need to approach them at their own level and using the words and kind of things that they can relate to. Otherwise it’s like how am I going to convince them otherwise?
Steve: Well great. Do you have anything else we should cover?
Kathleen: Don’t think so. Thank you so much.
Steve: Well thanks. This has been a great conversation. I’ve learned a lot and lots of ideas. It was great to have the time to chat with you.
Kathleen: Alright.
Steve: Thank you.
Kathleen: Thanks.
Steve: Thanks for being here for this episode of Dollars to Donuts. Follow the podcast on Twitter, and subscribe at portigal.com/podcast, or iTunes, or Spotify, or Stitcher, or your favorite podcast distribution platform. Also at portigal.com/podcast you can find the transcript and links for this and all episodes. Buy my books from Rosenfeld Media or Amazon. Thanks to Bruce Todd for the Dollars to Donuts theme.
The post 18. Kathleen Asjes of Schibsted Media first appeared on Portigal Consulting.
This episode of Dollars to Donuts features Kathleen Asjes of Schibsted Media. In our conversation, we talk about what happens when research need exceeds resources, the importance of keeping the knowledge inside the organization, and the benefit of diversity in a research team. It’s not so much about which university do I go to and […]
The post 18. Kathleen Asjes of Schibsted Media first appeared on Portigal Consulting.
In this episode of Dollars to Donuts, I talk with Tomer Sharon, the Head of User Research and Metrics at Goldman Sachs. We talk about how to assess potential hires for user research positions, infrastructure for capturing and searching a body of data, and developing a practice inside a willing, yet large, organization. Some parts […]
The post 17. Tomer Sharon of Goldman Sachs first appeared on Portigal Consulting.
In this episode of Dollars to Donuts, I talk with Tomer Sharon, the Head of User Research and Metrics at Goldman Sachs. We talk about how to assess potential hires for user research positions, infrastructure for capturing and searching a body of data, and developing a practice inside a willing, yet large, organization.
Some parts of kind of pure research are creative. Probably the biggest one is translating a set of questions that a team has into okay, what are we going to do to get answers? If it was that easy to come up with an answer to that, then anybody could do that well. That’s not the case. A lot of people are having a lot of trouble with that part. So, I think that’s a creative part. You’re not going to see a beautiful painting coming out of that, but it is creative. – Tomer Sharon
Show Links* Tomer on LinkedIn * Tomer on Twitter * Goldman Sachs * WeWork * It’s OUR Research on Twitter * It’s OUR Research on Amazon * Validating Product Ideas Through Lean User Research * Goldman Sachs Private Wealth Management * Marcus by Goldman Sachs * UserZoom * UserTesting * OKRs * ResearchOps * Democratizing UX (and Polaris) * Masters In Human Factors at Bentley * Adam Neumann, WeWork CEO * Key Experience Indicators: How to decide what to measure? (Medium) * Google’s HEART Framework for Measuring UX * Face of Finance NYC 2019 * User Research London 2019
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Greetings, humans! Thanks for listening to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization.
Over the past while I’ve been putting together a household emergency kit. It’s primarily shopping exercise, and I’ve ordered a hand crank and solar powered radio, a replacement for matches, latex gloves, bandages, and air filter masks (which we made use of during a period of dangerously poor air quality recently). The last step was getting some food that will last – cans of soup and stew, crackers, single-serve breakfast cereals. There’s something satisfying about acquiring a bunch of stuff and storing it away, somewhat organized. And that led to a stray thought that I noticed – “Oh, I can’t wait to use all this great stuff!” And then I realized how crazy that sounded. I don’t want to use it! I don’t want there to be some emergency that is bad enough that I’m drinking the emergency water stored in the garage and eating canned stew, also stored in the garage. I mean, yes, we’ll eat or donate the food before it expires and replace it, but it’s a whole set of preparations that I hope I’ll not use, which leaves me with the hope for no shopping gratification, kind of a confusing way to feel.
But it did remind of the recent workshop I led with researchers in Sydney, Australia. We looked at a lot of the user research war stories I’ve been collecting, like those published in Doorbells, Danger, and Dead Batteries, and we pulled out lessons and best practices. There was – as there always is – a lot of discussion about safety and preparation. It seemed to me that people who worked in organizations with established safety cultures already had a strong baseline of safety procedures for user research and fieldwork, like texting a contact before and after a session, not going out alone, and so on. Their work cultures were strong on processes, especially for safety, so thinking about this for research was obvious. But not everyone works in that kind of environment, and plenty of researchers work for themselves, without that corporate structure to support them in creating best practices for safety. Anyway it led to a lot of discussion beyond just safety about running through possibilities ahead of time so that when any situation comes up it’s not a surprise and there’s at least a starting point already established about how to respond. I think this is a great idea, but I think we have to acknowledge the limitations – you can’t plan for every possible situation, there are always going to be things that come up that you probably haven’t ever considered. I think that some planning for the unexpected will help you to adapt in the moment to surprises, but that’s different than the false comfort of assuming you have every contingency planned for.
I hope I never have to make use of our large cache of sterile latex gloves, but maybe just having acquired them I’m in a slightly better situation for some other unexpected situation?
You can help me continue to produce this podcast for you. I run my own small business, and you can hire me! I lead in-depth user research that helps drive product and strategy decisions, that helps drive internal change. I provide guidance to teams while they are learning about their customers, and I deliver training that teaches people how to be better at user research and analysis. You can buy either of my two books – the classic Interviewing Users and Doorbells, Danger, and Dead Batteries, a book of stories from other researchers about the kinds of things that happen when they go out into the field. You can also review this podcast on iTunes and review the books on Amazon. With your support, I can keep doing this podcast for you.
All right! Time for my interview with Tomer. Tomer Sharon is the Head of User Research and Metrics at Goldman Sachs. He’s worked at Google and WeWork, and written two books – It’s Our Research, and Validating Product Ideas Through Lean User Research.
Thanks for being on the podcast. It’s great to have you here – virtually here in audio space that we’re all sharing. Why don’t you start off – just do a little introduction. Who are you? Where do you work? What are you doing?
Tomer Sharon: Okay. Thank you for having me, first. My name is Tomer Sharon. I am currently Head of User Research and Metrics at Goldman Sachs. I do have a second day job. I’m also heading a design group for a product called PWM (Private Wealth Management). Yeah, this is where I’m at in the past – well, almost a year.
Steve: Alright. So, what is Goldman Sachs?
Tomer: Goldman Sachs is, I would say, an investment bank. Probably one of the more important banks in the world. Big corporate. Definitely not one you would associated with design and research, at least not that type of research. But they are changing and they’re celebrating 150 years this year and they’re moving towards what’s called outside digital transformation and that includes learning more from their audiences and investing a lot more in design.
Steve: What’s the relationship between the existence of your role and this larger shift that’s going on?
Tomer: I think there’s a strong relationship. They have been realizing that they can’t just be living in their own box and they have to open up and try and understand audiences that they’re engaged with already and new audiences. I’ll give an example. Goldman has a commercial bank that’s called Marcus. It’s been around for a couple of years, but still these are consumers that Goldman is now trying to attract. So, it’s definitely not the kind of typical audience that they’re used to. So, they understand that they need to open up, learn from them, and design for them and with them, and that is a shift that has been happening in the past few years. And my role – I didn’t replace anyone. I’m the first one. – is a part of that shift.
Steve: Is there any sort of one point or one incident, or key moment, I guess, which sort of marks that transition in a company like Goldman to say yeah, we’ve been doing it this way, we need to do it this way.
Tomer: I think it happened – I don’t know if there was one event, but I think it happened in the past maybe two years. The user experience team there was very small and then suddenly they decided that it’s time to invest more in that. And from zero it went to several dozens, many dozens, within a year. And the more people do their work, show their work, share their work, and their work is very successful, the more teams and leaders talk about that and then it’s like a cycle that feeds itself and then it grows and grows. So, that’s been happening a lot in the past few years.
Steve: And do you have a perspective on – this awareness that you’re describing at Goldman seems to align with what I’ve heard and observed in financial services in general. That it’s an industry that maybe wasn’t seen as paying attention to consumers, users, design, and over the past number of years.
Tomer: Yeah. I will admit, this is my first financial services job. So, I’m not really familiar with that world other than Goldman. But, so I hear. I don’t really know from first experience.
Steve: But I’m inferring – correct me if I’m wrong – I’m inferring that that also is not a conversation that you’re having inside Goldman?
Tomer: No. It’s very, I would say, kind of practical and tactical. We’re not talking about the concept of having me and people like me there. We’re just focusing on doing the work that we know how to do, we’ve been doing for many years, and just bringing that insight and understanding of that world to an organization that wasn’t aware of that previously.
Steve: That world is the process and tools of learning about people?
Tomer: Um, yeah. I would say process, tools, people. They’re used to hiring different people. I saw that in Google at the time. I saw that at WeWork at the time where even formally you don’t have in the – I don’t know HR systems – you don’t have names for roles for what we do, for who we are. I can’t remember the names, but we were engineers or UI engineers, or things like that. Until you get recognized, and I experienced that at Google, and then you do have a job family for design and for research and so on. It’s a process.
Steve: So, at Goldman you’re trying to then like – that’s part of the…
Tomer: Well, I’m not – we’re too, especially in research, we’re too few people to start having a job family in the HR systems, but we’ll get there. What I’m trying to do is first a lot of evangelism. A lot of conversations with people to plant seeds in their minds that they might need somebody like that. That they might engage in a project, a one-off project, and learn more about it. And with several groups and divisions it works already. So, we have people there and they start doing their work.
Steve: It’s interesting that you’re sort of highlighting evangelism vs. like conducting research.
Tomer: Right. Right. Sometimes it’s integrated. Sometimes I’m asking people to – some kind of a leap of faith. And then we actually, we do the research and then the work talks for itself. I don’t need to evangelize anymore. They get it. They understand. They want more of it and from there the doors are open. I prefer it this way. Even in the past, I prefer to kind of do the work show then kind of wave my hands and talk about it. I found it always be more meaningful to people and more persuading.
Steve: I’ve heard this. I don’t know if this is what’s playing out for you in these situations, but that you help someone take a leap of faith. You do this work. The results speak for themselves for them. Do those results – or how can you help those results spread elsewhere in the organization?
Tomer: Um, I use that example when I talk with others and I invite people to connect. I mean in the same company you can hear from them. You can meet them. I sometimes facilitate those meetings. And then they hear from others. I know it will take time, that it’s not something that happens over a day or a week or a month. Sometimes a year. But these conversations happen, whether I’m aware of them, or not. Whether I facilitate them or not. I’m confident that the more work we do the more people we’ll have, and we’ll be more impactful.
Steve: How does that work across – I don’t know if they’re called business units. Goldman as you said speaks to different kinds of users, different kinds of customers, with its products and services.
Tomer: So, what I do – so, as soon as we have researchers joining the group, we assign them to a – let’s call it a business unit, or a product team. And then they’re theirs 100% of the time. I only support them with infrastructure, career growth and things like that. And then they do the work with the team. I do a lot of legwork kind of before that happens to make sure that they have a team that wants them, that needs them, and that knows what to expect. So, it’s not the first time that they hear about that research thing with the appearance of the person. So, that’s how I intend to kind of continue doing rather than hire to – like work in an agency model. I have a pool of people that kind of come and go kind of based on projects.
Steve: Okay, so, just to reiterate, you talk about help someone to take a leap of faith, and I’m maybe changing your words a little bit. And a next step from that is to hire someone and put them in that team.
Tomer: Yes.
Steve: And then they do the work and then the work, as you said, the work speaks for itself, but it’s the work that you’ve staffed someone into that team?
Tomer: Yeah. I’ll just add to that. In many cases it’s not really a leap of faith because there are people who are – I call myself an outsider – who didn’t grow up in Goldman, that know about this field, know about research, know about people like us. So, they’re very open to having them. So, that’s much easier.
Steve: So, you said you’re not looking to do an agency model.
Tomer: No.
Steve: From your face, I think there’s a point of view.
Tomer: Yes, definitely.
Steve: What’s the point of view behind that?
Tomer: The point of view is that I want people to feel they belong; the researchers feel they belong to a team. And I feel that if I just send them off to short term projects they can’t grow with the team. They can’t have any history with the product. They’re not familiar. They’re really like consultants that come and go. I want them to feel a part of the team, understand all the history, sit with them. I don’t look for them to sit next to me. I want them to sit with their teams. And then going to be a part of that team and then be more impactful. I strongly believe that this is the way to go.
Steve: Yeah.
Tomer: And I’ve seen that happen before with my own eyes. Not WeWork, because we didn’t do that at WeWork, but definitely at Google where we experienced, at some point, a decentralization. So, there was one UX group and they were decentralized into the different business units. Not much has changed because people were assigned to teams already, but it just felt like the right thing to do, to have people sit with their teams, work with the same team, same product, for long periods of time.
Steve: So what implications, if any, does that have for what kind of skillset someone that you’re hiring needs to have?
Tomer: Yeah. That’s a good one. So, I’m looking for more experienced people. More senior people. There will be a time where we will hire more junior people and that’s not the time when we’re just starting. And I had the same thing at WeWork when we were building a group from scratch. You need experienced people. So, I’m looking for people who are eager and passionate about building something from scratch, taking a team that maybe doesn’t know anything about user research and try and kind of build that relationship and build those results. We talked about that earlier. With that team and grow with them, grow the practice with them. And maybe if it’s extremely successful grow a team there at some point.
Steve: There’s always a thing with questions and answers, at least for me. I ask a question, I have an idea what answer I’m going to get. And it’s interesting when the answer is different. I know it doesn’t always cleave this way, but there’s sort of the soft skills and the hard skills thing. I imagined I was going to get a hard skills answer because I was thinking about some of the pragmatic constraints of this, but your answer emphasized more, I think, softer skills. Growth, advocacy, leadership from a research point of view.
Tomer: I think it kind of almost goes without saying that once you’re – I don’t know, maybe I’m wrong. But once you’re – I always like to – I tell that to my people when we hire. We look for resumes that scream researcher. So, if your resume screams researcher, then that’s covered. I’m more interested in the soft skills.
Steve: Yeah. Can I push on that just a little bit more?
Tomer: Sure.
Steve: And I think – my bias here, or the perspective I come from is the consultant. So, I come in and out. So, I know what the limitations of that are and so just from where I sit, I see the challenges of what you’re describing because – I mean there’s a life cycle of when a team is developing – there’s probably a smarter way to say that. But something is being built and it goes through stages over weeks/months/years and so the research needs change and so the way the researcher has to respond to that, or lead that, or support that changes. So, that’s kind of the – I’m just revealing what I was probing on here, what I thought the answer was going to be. When you say screams researcher in a resume does that sort of imply that the ability to – are you inferring from that resume the ability to support that team and all these different areas of its evolution?
Tomer: Yeah. A resume that screams researcher is a resume that you see all the methods, you see the flexibility in kind of not sticking to one method or one approach. Doing that in multiple cultures, multiple companies. Have a trajectory of growth. So, you’re doing kind of from less meaningful things to more meaningful things, and so on. So, that to me screams researcher. Not, you know, you look at the job titles and you ask yourself why am I even reading this resume. And then you read a cover letter that says something along the lines of “I’ve always wanted to be.” Maybe at some point, but not now.
Steve: Yes. There is an archetype of a person I think that identifies strongly with researchers and sort of what they’re about, but their experience doesn’t match that.
Tomer: And I’m smiling because there’s always an exception. I remember one at WeWork where – and I was talking – speaking the same way – only senior people. I’m not looking for more junior people. And then I hired two junior people because there’s always an exception. Because you see somebody who makes me ignore everything I just said and hire them.
Steve: What’s an example of something that overrides that?
Tomer: I don’t even know how to explain it. It’s just – sometimes it’s the spark in the eyes that you see immediately with people and you see that they are going to be like a sponge. That’s a good metaphor. And you just know it’s going to work.
Steve: And you said passion early on.
Tomer: Yeah, yeah, yeah.
Steve: And I think evidence of passion is past performance. But that’s not the only evidence of passion if you’re seeing that with people that you’re meeting.
Tomer: Right, right. Although in most cases I’m very reluctant to hire based on passion if you don’t have experience. Because you can have a lot of passion and motivation, but you know nothing. It doesn’t mean you’re going to be bad, but it means a lot of people are going to need to support. So, I think kind of my approach is that I’ll try and do zero to very little of that because we’re just starting. We’re just a few people. We don’t really have the time that we need to support our teams. We don’t have the time to support another person. Not now. When we grow, yeah, but that’s definitely – you know, if we were 10/20 senior people on the team I would say we have to have more junior people because we can’t have a team of only senior people. But right now, we’re not even close to that.
Steve: Can you say the size that you are now?
Tomer: Yeah, sure. We are four. In a couple of weeks, we’ll be five. Worldwide, both here in New York and in London.
Steve: And is there like a roadmap for number of people to get to that 10 or 20?
Tomer: Yeah. I don’t want to mention numbers, but I think we’re on a path of growth.
Steve: Okay. So, you talk about sort of the time commitment required to support people at different levels. I want to go back and clarify one thing. It was almost an aside. You said that you don’t want to do the agency model. You want people with these teams, but you’re trying to provide infrastructure. And then you used a phrase like career growth. So, given the size you’re at now, acknowledging that there’s bandwidth challenges here, so what does infrastructure look like? What does career growth look like now?
Tomer: Um, infrastructure. So, that’s easy. So, there’s a lot of – when I look around me there’s a lot of motivation to do research, but there are needs in terms of – and gaps in terms of tools, knowledge, guidance, process. So, research was happening before there were researchers, but it was kind of very kind of based on sporadic motivation from different people who are really trying to do the best they could do. So, we need to support them with services that are out there, industry standard services. So, we need to sign agreements and get those licenses on board, different vendors.
Steve: What’s an example?
Tomer: UserTesting, UserZoom. These would be two, but there are more. And, um – let’s see what else. We are kind of creating process for, okay what happens if you want to do research, but you don’t have a researcher and you probably won’t have a researcher in the next few years? But you still – you acknowledge the fact that you need research now. So, um – so we have some – we have established some kind of a way to ask for that and then sign up for office hours or something like that, and then get some help from us. We will advise you and help you kind of get going without the researcher. What else? We started working with OKRs. So, if that’s – to me that’s a part of kind of infrastructure. Or the cool kids now call it research ops. So, research ops.
Steve: So, explain what OKRs are and explain what research ops is.
Tomer: OKRs is Objectives and Key Results. It’s a goal setting approach. There are many. This is just the one that I’m used to, familiar with. And that’s just a way to set goals and see how you’re doing. So, we do that as a research group as well. Research ops – I would take the easy route and say this is everything that helps research happen without the actual research.
Steve: Are the OKRs an example of research ops? That’s what you were saying?
Tomer: So, the tools that I mentioned – the OKRs, hiring, knowledge management.
Steve: All of the infrastructure?
Tomer: Yes. If somebody wants to do – I don’t know, whatever – a usability test, and they don’t know how. So, you want them to have the tool. We want them to have a knowledge base that they can access and see okay, what do I need to do to run the usability test? And we want them to have guidance and support from a person, from a researcher who knows what they’re doing. And I would say all of that is research ops. For researchers, a big part of research ops is participant recruitment. So, finding people to learn from. That’s a big part. And something that I’m truly passionate about is kind of insight repositories. Building some kind of a repository that we can pull from later on. I would add to that any infrastructure involving measuring the user experience. So, building that. I would also include that under research ops.
Steve: So, you said you’re passionate about insight repository?
Tomer: Yeah.
Steve: Do I have to ask an actual question?
Tomer: What do you want to know?
Steve: What are you trying to get to? At Goldman, what are you working towards with that?
Tomer: It’s the same as everywhere. I, for many years, I realized I was kind of bothered by how wasteful research is. That even I felt that I’m “learning the same things over and over again.” I know that other people did research, in the same company, did research that I’m about to do and even if I get their insights I’m going to do that again. And I know it’s really, really messy and hard to retain all the knowledge that you gather from research. The second I had an opportunity to do something about it, I did. That was at WeWork. And we built a system that we called Polaris for that, to solve these problems. We identified the – it’s going to sound funny, but we identified that the main problem, the main root cause for these problems are – or is, the research report, that was what I called back then the atomic unit of a research insight and we changed that unit into – that’s why the metaphor breaks – a smaller atom, that we called a nugget, a research nugget. And that’s what we stored in this repository. So, a nugget was a combination of an observation, evidence and tags.
Steve: What was the last one?
Tomer: Tags. And then due to these tags you can then search through this database and find answers to questions that you didn’t design studies around. So, it happened many times at WeWork where people came to us and said, “what do we know about…?” or, “can we do a study about blah blah?” And we said let’s try Polaris first and realized that we have all the answers without even needing to do more research. This will only happen – you have to change your ways a little bit, not just work with a system like that – this will only happen if you do kind of continuous research – continuous and open-ended research. Back then at WeWork we did exit interviews. So, every WeWork member, customer that decided to leave, we pinged them and talked with them, interviewed them, and asked them kind of very open-ended questions such as why are you leaving? What worked well at WeWork? What didn’t work so well? If you had 15 minutes with the CEO, what would you tell him? Things like that. And then that allowed us to have answers to many, many questions because these research participants, these exiting members decided what they wanted to talk about. If they wanted to talk about – I don’t know, whatever – the price that was too high for them, or the coffee that was too great for them to leave to another place, or whatever it is that they chose, went into the system and then we could pull it out later on and then see okay we heard – and combining with – combining that with a user experience measurement system would lead you to a situation where you can say okay, I saw that satisfaction with coffee in our WeWork buildings in the Netherlands has gone down in the past month. Here is a play list of three Dutch members bitch about coffee from the past month. And then you have the what happened – the numbers. You have the why it happened from these videos. And if you’re going to “serve that” to the person that buys or decides how to brew coffee in the Netherlands then that’s half way through to the solution. So, we imagine something like that at Goldman as well.
Steve: So, it only works, you said, if you have kind of ongoing open-ended research?
Tomer: Yeah.
Steve: Why is that?
Tomer: Because if you always – so, the other type of research I had to give it a name. So, I would call it dedicated research. Dedicated research is research that you do, and you know what research questions you have beforehand, and you answer those questions. And then you can create nuggets and it’s all good. But then you’ll only have answers to those questions. When you do open-ended you have answers to questions you never imagined that you might have, or may have in the future.
Steve: What if my research question is what are the highs and lows of a WeWork member?
Tomer: So, if you do that once you’ll get a snapshot of that point in time. But if you do that continuously, all the time – and at WeWork we, at some point, interviewed -all the UX team members did that on a regular frequency. Then you have thousands and thousands of data points.
Steve: Okay. So, there’s sort of a scale here – scope or scale. I don’t even know which one it is, but there’s an amount of data that covers a breadth of topics and that is also refreshed.
Tomer: Yes.
Steve: Okay. So, what’s – I would hate if anyone asked me this question, but I’m going to ask you. What’s an insight? Because you were kind of saying a nugget is this sort of stuff, but you brought up insight. So, what’s an insight.
Tomer: I actually have a definition for that. I’m thinking about that these days. An insight to me is a deep understanding of the situation. So, you – I’m trying to think of an example. I’ll go to WeWork again. So, imagine a researcher that walks in a WeWork building, looking around, and then sees WeWork has shared open spaces, but also private offices. Okay. So, let’s say they notice in private offices that a lot of them have printers that they’ve brought in and that looks odd because WeWork offers printing. So, we have printers and we offer that as a part of your WeWork membership and if you’re going to go over you pay for more. It’s a nice stream of revenue for WeWork and when they counted it – let’s say the count it and saw that half of the private offices brought in their own printers. They go to a second building and a third building and they count, and they see the same. Half of the members that have private offices brought in their own printer. So, let’s say they stop here, go back to the office and add an insight, nugget, whatever we call it, to the system saying half of WeWork members in those buildings brought in their own printer. That to me is not a deep understanding of the situation. It’s very interesting. It may be indicative of something that’s going on that we’re not aware of, but that’s not enough. We have to understand why? So, I would encourage that researcher to knock on the doors and ask why? And then we may hear things like, “oh, you know you have a 15-page manual on how to install printers and I’m not going to waste my time on that.” Or, “you have to log in each time you go to a printer and that’s taking more time.” And so on and so forth. “We do steal your paper, so we enjoy that.” That’s what I mean by deeper understanding, so we can better understand the situation, know to answer why that is happening, have some evidence and then I would say that’s an insight. That’s a deep understanding of the situation. And to me the system that we built is going to enforce that, so you cannot submit nuggets or insights without that why part? Just facts are not enough. We don’t need facts. We need facts plus why they are what they are.
Steve: Right. There’s a behavior and what’s the reason for that behavior. I struggle when I hear people talk about insights because sometimes they talk about why a single person is doing something, as opposed to sort of why users of a system are experiencing something. Like, in your scenario, you knock on somebody’s door and say what’s up with your own printer? “It’s too hard to install.” And you come back and say – like there’s a difference between we understand why this person was doing it and then sort of the generalized conclusion. I don’t know, I’m putting my own language into your framework. It’s probably not working, but…
Tomer: That’s alright. What we would do with Polaris is gather those individual insights. So, each one would be a nugget. And then if you have 100 of these, the only difference would be the video, the person in front of the camera explaining why they brought in a printer, or whatever it is. And then you can create a playlist and show it to IT, or whoever decided that we’re going to go with this system for printing, and have them decide what they’re going to do about it.
Steve: Okay, so let me push on that a little bit. Because when you talk to say 5 people or something, about this behavior, bringing in your own printers, people are going to give related, but sort of seemingly individualized explanations. And there’s an act of interpretation, analysis and synthesis – words that we often use – to sort of say well let’s look at all of those. Like what’s the overarching reason? Talk about deep understanding. It’s not that we loose track of the fact that the installation manual is messy. There are not enough plugs. There are a lot of sort of reasons, but the larger issue is something like complexity, or not seen as adding to my business or something. There’s a higher order thing. To me, that’s what the insight is. I think you were talking about doing that.
Tomer: Yeah.
Steve: The words are slippery here when we’re talking about insights and nuggets and sort of explanations.
Tomer: Nugget is more the kind of the technical way we called it. But, I agree. And the way it happened in Polaris is through those playlists. So, we encourage people, both in – you know people who belong to the UX team, and ones that are not, to collect these nuggets into playlists and then prove a point, or do this analysis, get to an insight and share it. So, it depends on what you find and what you collect there, but let’s say you search for printers and then you get 73 results. You sift through them and you see that 15 are not really related to what you’re trying to communicate here. The rest is too many, so you’ll pick maybe 7 that the videos are really good, like “good participants” that eloquently explain the point and then you can add your analysis in writing and describe that higher insight, or deeper insight.
Steve: Right. So, Polaris kind of sets you up to make that interpretation.
Tomer: Yeah, yeah.
Steve: It gives you some structure or some way to quickly…
Tomer: Yes, yes.
Steve: Does Polaris capture that thing it facilitates you to make?
Tomer: What do you mean?
Steve: So, create this playlist. You watch the playlist and you come up with sort of new articulation. The biggest issue around printers for us is that we are doing this, and they are thinking that. Right? That’s a new piece of knowledge that’s created by reviewing what Polaris gives you.
Tomer: Yeah. Polaris was not smart. It’s just a tool. It would do whatever – it allows you to do whatever you want to do with it. So, it allows you to add text to do that. So, if that’s a yes then it allows you to do that. Polaris, in its kind of essence, is very simple. Just a tool that facilitates the kind of storage of those nuggets and creation of those playlists.
Steve: Yeah. So, that really helps me understand sort of what you’re aiming at when you talk about insight repository. It’s to me something you can re-query to come up with new conclusions.
Tomer: Yes. Exactly.
Steve: As opposed to sort of here’s all the things that we’ve concluded.
Tomer: Yes. Exactly. Exactly.
Steve: Okay. Wow. Here’s the GIF of the mind blowing up a little bit.
Tomer: You want more?
Steve: If you have more, yes.
Tomer: Now it’s just ideas and that’s not something I’m kind of working towards in Goldman, but I’m thinking – so imagine every company that should have a Polaris has it. That’s also a waste because then every company has its own repository and then I’m sure there’s a lot of overlap. So, maybe in the future there should be a kind of open…
Steve: Panopticon.
Tomer: And open Polaris, or whatever we call it, that anybody can contribute to and anybody can pull from. I’m not doing anything about that.
Steve: A friend of mine – so, this is like a third hand quote that I’m sure I’m misquoting, but a friend of mine told me this and he was quoting the head of knowledge management at NASA. This guy says, “the best knowledge management tool is lunch.”
Tomer: I can see where – I can see why that was said. Yet, I wholeheartedly reject the idea.
Steve: Yeah
Tomer: I mean, that’s not scalable. That’s not what if I went to the bathroom when that happened? What if I started working in NASA the day after that important insight was shared? I can understand the kind of anecdotal part of it, how it’s useful. But to me there has to be something – I don’t know what to call it – more solid.
Steve: Yeah. Analogously, you started off our conversation by describing the lunching that you’re doing at Goldman, connecting people, meeting people. That’s different I think than transmitting nuggets, but you are using sort of lunch, and I mean very vaguely lunch, time with people, talking to them as a way to – I don’t know, as a way to do what? What’s the difference between sort of the NASA lunch thing and what you’re doing with socializing and connecting people?
Tomer: I think what I’m trying to do is kind of socialize a discipline. And if I understand it correctly, the NASA lunch is to socialize an insight maybe. So, yeah, I don’t think we’re there yet in terms of socializing insights.
Steve: And who knows what the context of that quote, which has been quoted, which I’m requoting. It may be exactly coherent with what you’re talking about.
Tomer: Maybe
Steve: I want to loop back to something that you talked about as part of this infrastructure. You said there’s groups that don’t have a researcher and may never have a researcher. So, what are sort of tools, knowledgebase, that can help them do things? I feel like that’s – there’s a thing in our field about who does research? And I’m not even sure what the label for that is?
Tomer: Job security.
Steve: Yeah. Is that what it is?
Tomer: Should we let them do research?
Steve: Right.
Tomer: Yeah. It makes me laugh. Of course, we should let them. As if we’re the authority. But, yeah, of course. I mean why would anyone not be allowed to do research? Because they didn’t go to school? I don’t think so. If somebody wants to do it, to me that’s huge. So, we should give them everything we can to let them do it. Are they going to be doing a bad job? Maybe. But to me bad research is better than no research. It’s a first step and we are good about the tools, the socializing what we do, socializing best practices, things will get better. Yeah, there will probably be crap added in the first few times. Maybe the first 20 times. But if they really want to and they have this passion, then why kill it by saying that it’s our job, or something like that. So, yes, I’m all for “letting them” do research. Definitely.
Steve: I mean I think you highlighted exactly – so there’s – I think job security is a fear, but I think bad research is also a fear, as you said.
Tomer: Yeah. I’m okay with that.
Steve: And you said bad research is better than no research.
Tomer: To, me, yeah. 100%. Yes
Steve: I like how definitive you are.
Tomer: I’m…
Steve: Because that’s a hot topic, I think. I’ve heard people go back and forth on it.
Tomer: I know. I heard that too. I’m definitely on that side.
Steve: I also will say you’re describing ways to limit or mitigate bad research.
Tomer: Help make it better.
Steve: Yeah.
Tomer: Yeah, yeah. I mean first they need to know about it. What happens – there are so many people who develop, let’s say software, at Goldman. They’re not even aware of our existence. It’s not that they think about it and say, “uh, no, I’m not going to do it.” They don’t even know that this is happening, that we are there. So, I think there’s a long way to go. We need to kind of be more popular and be more known and then provide all the tools, help, guidance, knowledge that we can. Knowing that we can’t support everybody. It’s not going to happen. It happened even at Google that had, at the time 100s, and today probably a lot more researchers. There are teams that build stuff. Why not allow them to do research to help them?
Steve: You made the comment that research was happening before there were researchers.
Tomer: Yeah.
Steve: In the history of the world that’s also true, right.
Tomer: Yeah. True. We can’t stop that or control that.
Steve: So, maybe just a slight shift. We’re sort of talking about who’s allowed to do what, or what are researchers and what do we do? You mentioned early on that you also take on an additional role. Do you want to give some context to that? And what does that mean for you?
Tomer: Yeah, I also lead a design group for a product, Private Wealth Management. It allows very rich people to manage their money. It’s a part of a service that Goldman offers. And the digital aspect of it is not the primary aspect of it. It’s just kind of a supporting role. It’s mostly based on a relationship between an advisor, a Goldman Sachs advisor, and a client. And there’s the usual suspect of website apps and so on. I’m now leading a group of people that design that. And I mean design with the expanded way we define it. So, it’s not just designers. It’s also researchers and data people and a writer and prototyping and so on.
Steve: Are there differences between managing researchers – we talked before about people coming in with resumes that scream researchers – vs. all the different kinds of functions you’re working with on that team?
Tomer: Honestly, no. I don’t think so. I would be the first to admit it. I’m personally not the best designer. Not in the world definitely and not that I could be. But, I think there are things that are similar no matter what kind of group you’re leading. It’s good that you know something about what the group is doing, but I think it’s mostly about empowering the right people, giving them what they need, releasing them of things that are just stupid, that they don’t need to do, and they don’t need to be involved in, and focusing them on what they are passionate about. This doesn’t have any direct relationship with design or research or whatever it is.
Steve: Yeah. There’s an email list that I think you and I are both on that’s about design and user research. And there was a thread, or maybe people were having a conference call. I can’t even remember how it manifested, but the topic was researchers managing designers, which seems like it’s a newer thing. If you look historically, like research was sort of the accessory or adjacency to design, so design teams kind of managed researchers. But as research has grown there’s other people in the situation like yours where their label for themselves would lean more towards researcher, but they’re managing designers. So, it’s interesting that you sort of don’t see a difference because I feel like the thrust of this group needing to talk was hey, there’s something different here and so how are we going to deal with it?
Tomer: I know I have kind of my internal bias toward research. So, I’m probably more kind of attentive to mostly when that is not happening. Maybe than a person who would be a designer that manages designers. But I’m just guessing. I don’t know. I know that I’m definitely – I care about research and I notice and say something when it doesn’t happen. I don’t know – does it have to be a designer – designers need to know if that’s their thing.
Steve: I think I might switch gears entirely here.
Tomer: Go for it.
Steve: I’d love to just go way back, like as far back as you want to go and maybe give the story of things you did in your life to kind of get here, whether those are work or school or other things?
Tomer: That got me here?
Steve: Yeah. What’s your sort of background, or your narrative arc, if you will?
Tomer: So, I’ll tell you something from a long time ago that probably really was the tipping point, that I wasn’t even aware of that that was the tipping point at the time. I mean it’s not a secret I’m originally from Israel and I relocated, it was 12 years ago. And while in Israel I served in the Army. I signed for a career in the Army. So, the whole shebang. I was going to be a career officer for the long term. But then when I was 24, long story short, I was a paraglider. And I took a course and I got injured badly and I was out of the Army for a year. I was at home recovering. And that was a bad thing. Bad injury, but that opened my eyes. And during that year I came back to the Army and said I want to cancel the whole thing; I don’t want to stay; I want out. I will do my next job because we planned for one more job, but that’s it. I’m out. And that’s what happened. I think without that accident I would probably – I’d probably be retired by now, but I would be a career officer and not what I am today. And kind of looking back, I’m happy that that’s what happened. I would say that’s the biggest thing that affected what I’m doing today.
Steve: So, opening your eyes was realizing that you didn’t want to go on the path that you were on. Was there any hints for you of what path you did want to pursue?
Tomer: I knew it was creative. I was in a wheelchair for four months and then crutches and then learn how to walk again. And I made my way, once I was able to kind of get up, to a local artist that gave kind of very open-ended lessons in his basement, like a couple of blocks from my house at the time. So, I started painting and tried all kinds of ways to paint. And I know – I didn’t know to say that I will be an artist and honestly, I wasn’t really good at it. But I knew it would be something creative. I didn’t know exactly what.
Steve: If you look at your work today, does it match that?
Tomer: Um, not 100% overlap, but I feel some of it is, yeah.
Steve: I mean I have wrestled, mostly privately, with just the idea is research a creative field? Or are we creative?
Tomer: Some of it is.
Steve: I found myself in a collaboration with people that I think more traditionally fit that job description and I kind of had my hair blown back, just on sort of the speed and breadth of making stuff. It was definitely intimidating.
Tomer: So, I would call myself a researcher, but still I was heavily involved in shaping Polaris. That’s a product. It’s not research work. I’m now involved in creating a system that – I also lead a small team of engineers that build a system to measure the user experience. So, that is definitely more creative than maybe research. But some parts of kind of pure research are creative. To me probably the biggest one is translating one or a set of questions that a team has into okay, what are we going to do to get answers. That’s not always – if it was that easy to come up with an answer to that, then anybody could do that well. That’s not the case. A lot of people are having a lot of trouble with that part. So, I think that’s a creative part. You’re not going to see a beautiful painting coming out of that, but it is creative.
Steve: Right. I think, for me, creating the new story out of a bunch of experiences or nuggets, or whatever you’re pulling from…
Tomer: Realizing, getting to an insight. Yeah.
Steve: So, what did you do after the art class? What do you end up doing?
Tomer: I applied – I applied to what we call today, probably a visual communications program. Got accepted to one of the best ones, if not the best one in Israel at the time, and last minute decided that it’s not for me. And then I learned – I studied copywriting. So, I’m a certified copywriter, in Hebrew though. And then I took my first job – or, I worked. I worked for 3 years as – I didn’t really know what I was going to do, so I did something that I knew how to do and that was I worked in a very small consultancy for military oriented industries. So, I did that for – I was a project manager. I did that for I think 3 years. That was the time where I kind of learned all these things and really set my mind that anything army related is not for me. And then my first real job in that direction was I was – funny how names were at that time. I was an internet copywriter, which we would probably call today a content strategist, for a website that I would compare it to Monster or Indeed, or something like that today. And there I got exposed to – probably at the time it was Jakob Nielsen and people in that area. I started reading more. I did some, again what we would call today, product management work there. And then I asked them to switch to what we would call today a researcher. They said, “no.” And I was like, okay, and I looked for a company who would take me. And there was one company that took me as – again, it wasn’t researcher. It was called a usability something. And that’s it. That’s how it started. They were very brave, I should admit, because I didn’t know much.
Steve: But they took you as a researcher?
Tomer: Yeah.
Steve: So, that was sort of your first time with the title.
Tomer: Yeah. You want an even funnier story. The only person who actually knew what it was was the CEO. He was the one who interviewed me. And then he hired me and then a month later he decided to leave. So, the only person who really knew what I was doing left about a month in. But that went well. That’s how it started.
Steve: What was the point at which you came to the U.S.?
Tomer: Um, so, I had a couple more jobs and I realized very quickly that, at the time at least, there weren’t enough, or even at all, opportunities to grow into managing a group or a team of people who do that. I was always the only person in the company that did that. And I realized, or I got to the conclusion that the only place for me to work in a company that had a lot of people of my tribe would be here in the states. And I also realized the army there is not like the army here. I didn’t have any academic degree. I also realized that I needed the right degree because these companies that I was thinking about would not even read my resume. My resume didn’t scream researcher. So, I went to school. I continued working and went to school. I completed my Bachelor’s and then applied to Bentley University here and then moved. And as I was studying there I – during school there, in Massachusetts, I contacted Google and things rolled from there.
Steve: We talked about WeWork a little bit. Can you – like what was the – what was your role at Google and maybe what was your role at WeWork?
Tomer: Google, I was a user researcher, a senior user researcher. First in advertising. They’ve changed all the names by now, but at the time it was DoubleClick for Publishers. That was the product I was involved in. Ironically, I was the only researcher there in that group of hundreds of people. And after, I think, 2 ½ years I switched transferred to Search and in Search, I was first Voice Search and then we called it, at the time, Core Search – the bag of 10 blue links and how they developed from that point to what you see today as all the visual aspects of the result and so on. So, that started back then, vertical by vertical – TV, movies, music. I was doing a lot of research into search results for sports. I know a lot more than I should about all kinds of sports, cricket and so on. Yeah.
Steve: And then what was the role you took at WeWork?
Tomer: WeWork, I was head of user experience. So, started a group from scratch. The goal there, and this is kind of following several conversations with the CEO and co-founder that hired me. At the time – again, I’m sure things have changed since. But at the time WeWork had three big groups internally that kind of built or created the three aspects of the WeWork product. They were called digital, physical and community. And Adam, the CEO, felt, and he was very right, that while each group is doing a great job, sometimes if you look – if you’re going to think or look from how things are from the perspective of the customer, the member, there are gaps between those groups that we’re not even aware of. And the goal was to identify those gaps, to me that translates to research, and then solve them – solve the problems there. I’m thinking of an example. Think about conference rooms. So, conference rooms is something that WeWork offers. Members pay for it. So, somebody physically designed the conference room. An architect decided on the size and location. An interior designer decided on the mood and what will be in the room. Somebody from IT picked and AV system for that room. Somebody in digital developed a system to book this room. Somebody in community designed a policy of how to use this room. And community team members in the buildings enforce this policy. Everything is good. Everything is working well, but then situations happen. Such as members come to a meeting, they book the room and then another member is squatting the room and refusing to go out. Even if they walk in they realize that I’m a startup, I booked a room for a meeting with a potential investor and then I see a room that is designed as a music room with bean bags and no projector, clearly inappropriate for my meeting. Or, that person who’s squatting, I’m going to the community manager, but they are dealing with a leak of water from the ceiling on another member’s head. They’re all very nice, but they can’t solve my problem right now. So, this is what I’m talking about when I said the gap between one of those groups. So, we’re trying to identify those gaps – because in many cases we didn’t even know about this – and try and solve them. That was the premise back then.
Steve: You’ve written books. You give a lot of talks.
Tomer: Less now.
Steve: You have a good sort of history of creating material. You’ve interviewed a lot of people. People listening, what would you send them to, to buy, read, watch?
Tomer: Well, our publisher would not be a publisher – would be happy if I say that they should buy my book, yours too. And that would be – the name of the book is Validating Product Ideas Through Lean User Research. That’s a book for – I would only mention that because my first book is for researchers, and a lot of researchers do listen, so maybe I’ll mention both. But the second book, the one I mentioned, is for – it’s a step by step guide into answering different research questions that people have. Each chapter is a research question, step by step, on how to answer it through research that anybody can do quickly. The first book is called It’s Our Research. It’s to solve a problem that, at least at the time – I now have different thoughts about that – but, at least at the time a lot of researchers had, maybe today as well, and that is a problem that a lot of people don’t want to do research because they feel they have the answers already, or they have good intuition. And also, once they agree to do research in some cases they don’t want to act on it.
Steve: Wait, act on?
Tomer: On their research results.
Steve: What they’ve learned? Yes
Tomer: So, that’s a book that’s supposed to help with that problem. And I’m kind of having different thoughts now because my answer in that book was – and that’s why I called it It’s Our Research – make them feel that it’s their research as much as you feel it’s yours and then they would want it and they would do something about it. That was my point. Today, I’m having kind of different thoughts about how to get to a point where research is wanted and acted on. And I will try it at some point. We just need to grow a little bit at Goldman. But, my thoughts are when you – and I posted something about that recently – when you plug your own charger into the wall, do you really care how electricity gets there? What’s happening in the power plant? Why it’s working? How is it efficient? Is not efficient, and so on. You don’t really care. You want your phone charged. My thoughts about research is why wouldn’t it be the same. People have questions. Research provides answers. Yes, a lot is going on to get to those answers, but if you have a question, why do we need to bother you with all the details? Just do the thing, trust us to do the thing, and we’ll give you an answer to your question. I’m going to try that at some point. I’m thinking – it’s not political, but building a wall between stakeholders and researchers. That wall could be Slack or something like that through which stakeholders ask questions and researchers provide answers. If we have an answer immediately, if we have a system like Polaris or something like that, we can provide an answer. If we don’t, we will just ask a few questions, follow-up questions, and then do the research and get the answer. Just thoughts. I haven’t tried it yet.
Steve: Which makes me think of the research ops piece a little bit where – like building up participant recruitment infrastructure…
Tomer: Yeah.
Steve: …is an interesting one because back in the old days where we had to do everything ourselves, you’re learning about your problem by figuring how to – by recruiting. You also learn about your problem by dealing with your stakeholder and seeing what – it’s that art piece vs. this kind of process infrastructure piece and it’s interesting to think about like what is lost and what is gained? Or how is changed when you create infrastructure that – like if you’re a researcher and you’re completely decoupled from participant recruiting, that may change how you deal with people that you meet, or how you deal with framing the problem. So, for everything that we build up a process, that’s efficiency, that kind of is a query system, how does that change what we do? And who is coming to this field?
Tomer: Yeah.
Steve: These are not necessarily my own thoughts, but just things I’m hearing from people as well.
Tomer: One other thing that I would send people to is a series of Medium posts that I’ve published in the past year, maybe less, about measuring user experience. A lot of people like to talk about metrics these days. I took the HEART framework from Google and then we have a post per letter about happiness, engagement, adoption, retention and task success. And for each one, what it is, what’s important to measure, why, how, mistakes and what actions you can take from each one? So, this is something that I’m interested in these days, measurements. And I’m trying to figure out the “H” part, the happiness part, specifically. There are a ton of challenges with that. How to measure satisfaction and happiness. I’m also posting – kind of tracking my – not tracking my own life but paying more attention to when I’m exposed to requests to rate satisfaction and happiness and I share them with people, with my thoughts about them.
Steve: Okay. Great. Anything else that we should talk about in this conversation?
Tomer: I said I am speaking kind of publicly a lot less, but I do do that from time to time. I’ll be speaking in two conferences, the Face of Finance in April in New York and in London, User Research London in June, in London.
Steve: Alright. Well, thanks for taking the time to chat and sharing all the information and stories and everything. I really appreciate it.
Tomer: That was fun. Thank you.
Steve: Thanks. And so concludes another episode of Dollars to Donuts. Follow the podcast on Twitter, and subscribe to the podcast at portigal.com/podcast, or iTunes, or Spotify, or Stitcher, or anyplace you get your podcasts. Also online at portigal.com/podcast is the transcript and links for this episode (and of course all the previous episodes). At Amazon and rosenfeldmedia.com you can buy Tomer’s books and my books. Our rocking theme music was written and performed by Bruce Todd.
The post 17. Tomer Sharon of Goldman Sachs first appeared on Portigal Consulting.
In this episode of Dollars to Donuts, I talk with Tomer Sharon, the Head of User Research and Metrics at Goldman Sachs. We talk about how to assess potential hires for user research positions, infrastructure for capturing and searching a body of data, and developing a practice inside a willing, yet large, organization.
Some parts of kind of pure research are creative. Probably the biggest one is translating a set of questions that a team has into okay, what are we going to do to get answers? If it was that easy to come up with an answer to that, then anybody could do that well. That’s not the case. A lot of people are having a lot of trouble with that part. So, I think that’s a creative part. You’re not going to see a beautiful painting coming out of that, but it is creative. – Tomer Sharon
Show Links* Tomer on LinkedIn * Tomer on Twitter * Goldman Sachs * WeWork * It’s OUR Research on Twitter * It’s OUR Research on Amazon * Validating Product Ideas Through Lean User Research * Goldman Sachs Private Wealth Management * Marcus by Goldman Sachs * UserZoom * UserTesting * OKRs * ResearchOps * Democratizing UX (and Polaris) * Masters In Human Factors at Bentley * Adam Neumann, WeWork CEO * Key Experience Indicators: How to decide what to measure? (Medium) * Google’s HEART Framework for Measuring UX * Face of Finance NYC 2019 * User Research London 2019
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Greetings, humans! Thanks for listening to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization.
Over the past while I’ve been putting together a household emergency kit. It’s primarily shopping exercise, and I’ve ordered a hand crank and solar powered radio, a replacement for matches, latex gloves, bandages, and air filter masks (which we made use of during a period of dangerously poor air quality recently). The last step was getting some food that will last – cans of soup and stew, crackers, single-serve breakfast cereals. There’s something satisfying about acquiring a bunch of stuff and storing it away, somewhat organized. And that led to a stray thought that I noticed – “Oh, I can’t wait to use all this great stuff!” And then I realized how crazy that sounded. I don’t want to use it! I don’t want there to be some emergency that is bad enough that I’m drinking the emergency water stored in the garage and eating canned stew, also stored in the garage. I mean, yes, we’ll eat or donate the food before it expires and replace it, but it’s a whole set of preparations that I hope I’ll not use, which leaves me with the hope for no shopping gratification, kind of a confusing way to feel.
But it did remind of the recent workshop I led with researchers in Sydney, Australia. We looked at a lot of the user research war stories I’ve been collecting, like those published in Doorbells, Danger, and Dead Batteries, and we pulled out lessons and best practices. There was – as there always is – a lot of discussion about safety and preparation. It seemed to me that people who worked in organizations with established safety cultures already had a strong baseline of safety procedures for user research and fieldwork, like texting a contact before and after a session, not going out alone, and so on. Their work cultures were strong on processes, especially for safety, so thinking about this for research was obvious. But not everyone works in that kind of environment, and plenty of researchers work for themselves, without that corporate structure to support them in creating best practices for safety. Anyway it led to a lot of discussion beyond just safety about running through possibilities ahead of time so that when any situation comes up it’s not a surprise and there’s at least a starting point already established about how to respond. I think this is a great idea, but I think we have to acknowledge the limitations – you can’t plan for every possible situation, there are always going to be things that come up that you probably haven’t ever considered. I think that some planning for the unexpected will help you to adapt in the moment to surprises, but that’s different than the false comfort of assuming you have every contingency planned for.
I hope I never have to make use of our large cache of sterile latex gloves, but maybe just having acquired them I’m in a slightly better situation for some other unexpected situation?
You can help me continue to produce this podcast for you. I run my own small business, and you can hire me! I lead in-depth user research that helps drive product and strategy decisions, that helps drive internal change. I provide guidance to teams while they are learning about their customers, and I deliver training that teaches people how to be better at user research and analysis. You can buy either of my two books – the classic Interviewing Users and Doorbells, Danger, and Dead Batteries, a book of stories from other researchers about the kinds of things that happen when they go out into the field. You can also review this podcast on iTunes and review the books on Amazon. With your support, I can keep doing this podcast for you.
All right! Time for my interview with Tomer. Tomer Sharon is the Head of User Research and Metrics at Goldman Sachs. He’s worked at Google and WeWork, and written two books – It’s Our Research, and Validating Product Ideas Through Lean User Research.
Thanks for being on the podcast. It’s great to have you here – virtually here in audio space that we’re all sharing. Why don’t you start off – just do a little introduction. Who are you? Where do you work? What are you doing?
Tomer Sharon: Okay. Thank you for having me, first. My name is Tomer Sharon. I am currently Head of User Research and Metrics at Goldman Sachs. I do have a second day job. I’m also heading a design group for a product called PWM (Private Wealth Management). Yeah, this is where I’m at in the past – well, almost a year.
Steve: Alright. So, what is Goldman Sachs?
Tomer: Goldman Sachs is, I would say, an investment bank. Probably one of the more important banks in the world. Big corporate. Definitely not one you would associated with design and research, at least not that type of research. But they are changing and they’re celebrating 150 years this year and they’re moving towards what’s called outside digital transformation and that includes learning more from their audiences and investing a lot more in design.
Steve: What’s the relationship between the existence of your role and this larger shift that’s going on?
Tomer: I think there’s a strong relationship. They have been realizing that they can’t just be living in their own box and they have to open up and try and understand audiences that they’re engaged with already and new audiences. I’ll give an example. Goldman has a commercial bank that’s called Marcus. It’s been around for a couple of years, but still these are consumers that Goldman is now trying to attract. So, it’s definitely not the kind of typical audience that they’re used to. So, they understand that they need to open up, learn from them, and design for them and with them, and that is a shift that has been happening in the past few years. And my role – I didn’t replace anyone. I’m the first one. – is a part of that shift.
Steve: Is there any sort of one point or one incident, or key moment, I guess, which sort of marks that transition in a company like Goldman to say yeah, we’ve been doing it this way, we need to do it this way.
Tomer: I think it happened – I don’t know if there was one event, but I think it happened in the past maybe two years. The user experience team there was very small and then suddenly they decided that it’s time to invest more in that. And from zero it went to several dozens, many dozens, within a year. And the more people do their work, show their work, share their work, and their work is very successful, the more teams and leaders talk about that and then it’s like a cycle that feeds itself and then it grows and grows. So, that’s been happening a lot in the past few years.
Steve: And do you have a perspective on – this awareness that you’re describing at Goldman seems to align with what I’ve heard and observed in financial services in general. That it’s an industry that maybe wasn’t seen as paying attention to consumers, users, design, and over the past number of years.
Tomer: Yeah. I will admit, this is my first financial services job. So, I’m not really familiar with that world other than Goldman. But, so I hear. I don’t really know from first experience.
Steve: But I’m inferring – correct me if I’m wrong – I’m inferring that that also is not a conversation that you’re having inside Goldman?
Tomer: No. It’s very, I would say, kind of practical and tactical. We’re not talking about the concept of having me and people like me there. We’re just focusing on doing the work that we know how to do, we’ve been doing for many years, and just bringing that insight and understanding of that world to an organization that wasn’t aware of that previously.
Steve: That world is the process and tools of learning about people?
Tomer: Um, yeah. I would say process, tools, people. They’re used to hiring different people. I saw that in Google at the time. I saw that at WeWork at the time where even formally you don’t have in the – I don’t know HR systems – you don’t have names for roles for what we do, for who we are. I can’t remember the names, but we were engineers or UI engineers, or things like that. Until you get recognized, and I experienced that at Google, and then you do have a job family for design and for research and so on. It’s a process.
Steve: So, at Goldman you’re trying to then like – that’s part of the…
Tomer: Well, I’m not – we’re too, especially in research, we’re too few people to start having a job family in the HR systems, but we’ll get there. What I’m trying to do is first a lot of evangelism. A lot of conversations with people to plant seeds in their minds that they might need somebody like that. That they might engage in a project, a one-off project, and learn more about it. And with several groups and divisions it works already. So, we have people there and they start doing their work.
Steve: It’s interesting that you’re sort of highlighting evangelism vs. like conducting research.
Tomer: Right. Right. Sometimes it’s integrated. Sometimes I’m asking people to – some kind of a leap of faith. And then we actually, we do the research and then the work talks for itself. I don’t need to evangelize anymore. They get it. They understand. They want more of it and from there the doors are open. I prefer it this way. Even in the past, I prefer to kind of do the work show then kind of wave my hands and talk about it. I found it always be more meaningful to people and more persuading.
Steve: I’ve heard this. I don’t know if this is what’s playing out for you in these situations, but that you help someone take a leap of faith. You do this work. The results speak for themselves for them. Do those results – or how can you help those results spread elsewhere in the organization?
Tomer: Um, I use that example when I talk with others and I invite people to connect. I mean in the same company you can hear from them. You can meet them. I sometimes facilitate those meetings. And then they hear from others. I know it will take time, that it’s not something that happens over a day or a week or a month. Sometimes a year. But these conversations happen, whether I’m aware of them, or not. Whether I facilitate them or not. I’m confident that the more work we do the more people we’ll have, and we’ll be more impactful.
Steve: How does that work across – I don’t know if they’re called business units. Goldman as you said speaks to different kinds of users, different kinds of customers, with its products and services.
Tomer: So, what I do – so, as soon as we have researchers joining the group, we assign them to a – let’s call it a business unit, or a product team. And then they’re theirs 100% of the time. I only support them with infrastructure, career growth and things like that. And then they do the work with the team. I do a lot of legwork kind of before that happens to make sure that they have a team that wants them, that needs them, and that knows what to expect. So, it’s not the first time that they hear about that research thing with the appearance of the person. So, that’s how I intend to kind of continue doing rather than hire to – like work in an agency model. I have a pool of people that kind of come and go kind of based on projects.
Steve: Okay, so, just to reiterate, you talk about help someone to take a leap of faith, and I’m maybe changing your words a little bit. And a next step from that is to hire someone and put them in that team.
Tomer: Yes.
Steve: And then they do the work and then the work, as you said, the work speaks for itself, but it’s the work that you’ve staffed someone into that team?
Tomer: Yeah. I’ll just add to that. In many cases it’s not really a leap of faith because there are people who are – I call myself an outsider – who didn’t grow up in Goldman, that know about this field, know about research, know about people like us. So, they’re very open to having them. So, that’s much easier.
Steve: So, you said you’re not looking to do an agency model.
Tomer: No.
Steve: From your face, I think there’s a point of view.
Tomer: Yes, definitely.
Steve: What’s the point of view behind that?
Tomer: The point of view is that I want people to feel they belong; the researchers feel they belong to a team. And I feel that if I just send them off to short term projects they can’t grow with the team. They can’t have any history with the product. They’re not familiar. They’re really like consultants that come and go. I want them to feel a part of the team, understand all the history, sit with them. I don’t look for them to sit next to me. I want them to sit with their teams. And then going to be a part of that team and then be more impactful. I strongly believe that this is the way to go.
Steve: Yeah.
Tomer: And I’ve seen that happen before with my own eyes. Not WeWork, because we didn’t do that at WeWork, but definitely at Google where we experienced, at some point, a decentralization. So, there was one UX group and they were decentralized into the different business units. Not much has changed because people were assigned to teams already, but it just felt like the right thing to do, to have people sit with their teams, work with the same team, same product, for long periods of time.
Steve: So what implications, if any, does that have for what kind of skillset someone that you’re hiring needs to have?
Tomer: Yeah. That’s a good one. So, I’m looking for more experienced people. More senior people. There will be a time where we will hire more junior people and that’s not the time when we’re just starting. And I had the same thing at WeWork when we were building a group from scratch. You need experienced people. So, I’m looking for people who are eager and passionate about building something from scratch, taking a team that maybe doesn’t know anything about user research and try and kind of build that relationship and build those results. We talked about that earlier. With that team and grow with them, grow the practice with them. And maybe if it’s extremely successful grow a team there at some point.
Steve: There’s always a thing with questions and answers, at least for me. I ask a question, I have an idea what answer I’m going to get. And it’s interesting when the answer is different. I know it doesn’t always cleave this way, but there’s sort of the soft skills and the hard skills thing. I imagined I was going to get a hard skills answer because I was thinking about some of the pragmatic constraints of this, but your answer emphasized more, I think, softer skills. Growth, advocacy, leadership from a research point of view.
Tomer: I think it kind of almost goes without saying that once you’re – I don’t know, maybe I’m wrong. But once you’re – I always like to – I tell that to my people when we hire. We look for resumes that scream researcher. So, if your resume screams researcher, then that’s covered. I’m more interested in the soft skills.
Steve: Yeah. Can I push on that just a little bit more?
Tomer: Sure.
Steve: And I think – my bias here, or the perspective I come from is the consultant. So, I come in and out. So, I know what the limitations of that are and so just from where I sit, I see the challenges of what you’re describing because – I mean there’s a life cycle of when a team is developing – there’s probably a smarter way to say that. But something is being built and it goes through stages over weeks/months/years and so the research needs change and so the way the researcher has to respond to that, or lead that, or support that changes. So, that’s kind of the – I’m just revealing what I was probing on here, what I thought the answer was going to be. When you say screams researcher in a resume does that sort of imply that the ability to – are you inferring from that resume the ability to support that team and all these different areas of its evolution?
Tomer: Yeah. A resume that screams researcher is a resume that you see all the methods, you see the flexibility in kind of not sticking to one method or one approach. Doing that in multiple cultures, multiple companies. Have a trajectory of growth. So, you’re doing kind of from less meaningful things to more meaningful things, and so on. So, that to me screams researcher. Not, you know, you look at the job titles and you ask yourself why am I even reading this resume. And then you read a cover letter that says something along the lines of “I’ve always wanted to be.” Maybe at some point, but not now.
Steve: Yes. There is an archetype of a person I think that identifies strongly with researchers and sort of what they’re about, but their experience doesn’t match that.
Tomer: And I’m smiling because there’s always an exception. I remember one at WeWork where – and I was talking – speaking the same way – only senior people. I’m not looking for more junior people. And then I hired two junior people because there’s always an exception. Because you see somebody who makes me ignore everything I just said and hire them.
Steve: What’s an example of something that overrides that?
Tomer: I don’t even know how to explain it. It’s just – sometimes it’s the spark in the eyes that you see immediately with people and you see that they are going to be like a sponge. That’s a good metaphor. And you just know it’s going to work.
Steve: And you said passion early on.
Tomer: Yeah, yeah, yeah.
Steve: And I think evidence of passion is past performance. But that’s not the only evidence of passion if you’re seeing that with people that you’re meeting.
Tomer: Right, right. Although in most cases I’m very reluctant to hire based on passion if you don’t have experience. Because you can have a lot of passion and motivation, but you know nothing. It doesn’t mean you’re going to be bad, but it means a lot of people are going to need to support. So, I think kind of my approach is that I’ll try and do zero to very little of that because we’re just starting. We’re just a few people. We don’t really have the time that we need to support our teams. We don’t have the time to support another person. Not now. When we grow, yeah, but that’s definitely – you know, if we were 10/20 senior people on the team I would say we have to have more junior people because we can’t have a team of only senior people. But right now, we’re not even close to that.
Steve: Can you say the size that you are now?
Tomer: Yeah, sure. We are four. In a couple of weeks, we’ll be five. Worldwide, both here in New York and in London.
Steve: And is there like a roadmap for number of people to get to that 10 or 20?
Tomer: Yeah. I don’t want to mention numbers, but I think we’re on a path of growth.
Steve: Okay. So, you talk about sort of the time commitment required to support people at different levels. I want to go back and clarify one thing. It was almost an aside. You said that you don’t want to do the agency model. You want people with these teams, but you’re trying to provide infrastructure. And then you used a phrase like career growth. So, given the size you’re at now, acknowledging that there’s bandwidth challenges here, so what does infrastructure look like? What does career growth look like now?
Tomer: Um, infrastructure. So, that’s easy. So, there’s a lot of – when I look around me there’s a lot of motivation to do research, but there are needs in terms of – and gaps in terms of tools, knowledge, guidance, process. So, research was happening before there were researchers, but it was kind of very kind of based on sporadic motivation from different people who are really trying to do the best they could do. So, we need to support them with services that are out there, industry standard services. So, we need to sign agreements and get those licenses on board, different vendors.
Steve: What’s an example?
Tomer: UserTesting, UserZoom. These would be two, but there are more. And, um – let’s see what else. We are kind of creating process for, okay what happens if you want to do research, but you don’t have a researcher and you probably won’t have a researcher in the next few years? But you still – you acknowledge the fact that you need research now. So, um – so we have some – we have established some kind of a way to ask for that and then sign up for office hours or something like that, and then get some help from us. We will advise you and help you kind of get going without the researcher. What else? We started working with OKRs. So, if that’s – to me that’s a part of kind of infrastructure. Or the cool kids now call it research ops. So, research ops.
Steve: So, explain what OKRs are and explain what research ops is.
Tomer: OKRs is Objectives and Key Results. It’s a goal setting approach. There are many. This is just the one that I’m used to, familiar with. And that’s just a way to set goals and see how you’re doing. So, we do that as a research group as well. Research ops – I would take the easy route and say this is everything that helps research happen without the actual research.
Steve: Are the OKRs an example of research ops? That’s what you were saying?
Tomer: So, the tools that I mentioned – the OKRs, hiring, knowledge management.
Steve: All of the infrastructure?
Tomer: Yes. If somebody wants to do – I don’t know, whatever – a usability test, and they don’t know how. So, you want them to have the tool. We want them to have a knowledge base that they can access and see okay, what do I need to do to run the usability test? And we want them to have guidance and support from a person, from a researcher who knows what they’re doing. And I would say all of that is research ops. For researchers, a big part of research ops is participant recruitment. So, finding people to learn from. That’s a big part. And something that I’m truly passionate about is kind of insight repositories. Building some kind of a repository that we can pull from later on. I would add to that any infrastructure involving measuring the user experience. So, building that. I would also include that under research ops.
Steve: So, you said you’re passionate about insight repository?
Tomer: Yeah.
Steve: Do I have to ask an actual question?
Tomer: What do you want to know?
Steve: What are you trying to get to? At Goldman, what are you working towards with that?
Tomer: It’s the same as everywhere. I, for many years, I realized I was kind of bothered by how wasteful research is. That even I felt that I’m “learning the same things over and over again.” I know that other people did research, in the same company, did research that I’m about to do and even if I get their insights I’m going to do that again. And I know it’s really, really messy and hard to retain all the knowledge that you gather from research. The second I had an opportunity to do something about it, I did. That was at WeWork. And we built a system that we called Polaris for that, to solve these problems. We identified the – it’s going to sound funny, but we identified that the main problem, the main root cause for these problems are – or is, the research report, that was what I called back then the atomic unit of a research insight and we changed that unit into – that’s why the metaphor breaks – a smaller atom, that we called a nugget, a research nugget. And that’s what we stored in this repository. So, a nugget was a combination of an observation, evidence and tags.
Steve: What was the last one?
Tomer: Tags. And then due to these tags you can then search through this database and find answers to questions that you didn’t design studies around. So, it happened many times at WeWork where people came to us and said, “what do we know about…?” or, “can we do a study about blah blah?” And we said let’s try Polaris first and realized that we have all the answers without even needing to do more research. This will only happen – you have to change your ways a little bit, not just work with a system like that – this will only happen if you do kind of continuous research – continuous and open-ended research. Back then at WeWork we did exit interviews. So, every WeWork member, customer that decided to leave, we pinged them and talked with them, interviewed them, and asked them kind of very open-ended questions such as why are you leaving? What worked well at WeWork? What didn’t work so well? If you had 15 minutes with the CEO, what would you tell him? Things like that. And then that allowed us to have answers to many, many questions because these research participants, these exiting members decided what they wanted to talk about. If they wanted to talk about – I don’t know, whatever – the price that was too high for them, or the coffee that was too great for them to leave to another place, or whatever it is that they chose, went into the system and then we could pull it out later on and then see okay we heard – and combining with – combining that with a user experience measurement system would lead you to a situation where you can say okay, I saw that satisfaction with coffee in our WeWork buildings in the Netherlands has gone down in the past month. Here is a play list of three Dutch members bitch about coffee from the past month. And then you have the what happened – the numbers. You have the why it happened from these videos. And if you’re going to “serve that” to the person that buys or decides how to brew coffee in the Netherlands then that’s half way through to the solution. So, we imagine something like that at Goldman as well.
Steve: So, it only works, you said, if you have kind of ongoing open-ended research?
Tomer: Yeah.
Steve: Why is that?
Tomer: Because if you always – so, the other type of research I had to give it a name. So, I would call it dedicated research. Dedicated research is research that you do, and you know what research questions you have beforehand, and you answer those questions. And then you can create nuggets and it’s all good. But then you’ll only have answers to those questions. When you do open-ended you have answers to questions you never imagined that you might have, or may have in the future.
Steve: What if my research question is what are the highs and lows of a WeWork member?
Tomer: So, if you do that once you’ll get a snapshot of that point in time. But if you do that continuously, all the time – and at WeWork we, at some point, interviewed -all the UX team members did that on a regular frequency. Then you have thousands and thousands of data points.
Steve: Okay. So, there’s sort of a scale here – scope or scale. I don’t even know which one it is, but there’s an amount of data that covers a breadth of topics and that is also refreshed.
Tomer: Yes.
Steve: Okay. So, what’s – I would hate if anyone asked me this question, but I’m going to ask you. What’s an insight? Because you were kind of saying a nugget is this sort of stuff, but you brought up insight. So, what’s an insight.
Tomer: I actually have a definition for that. I’m thinking about that these days. An insight to me is a deep understanding of the situation. So, you – I’m trying to think of an example. I’ll go to WeWork again. So, imagine a researcher that walks in a WeWork building, looking around, and then sees WeWork has shared open spaces, but also private offices. Okay. So, let’s say they notice in private offices that a lot of them have printers that they’ve brought in and that looks odd because WeWork offers printing. So, we have printers and we offer that as a part of your WeWork membership and if you’re going to go over you pay for more. It’s a nice stream of revenue for WeWork and when they counted it – let’s say the count it and saw that half of the private offices brought in their own printers. They go to a second building and a third building and they count, and they see the same. Half of the members that have private offices brought in their own printer. So, let’s say they stop here, go back to the office and add an insight, nugget, whatever we call it, to the system saying half of WeWork members in those buildings brought in their own printer. That to me is not a deep understanding of the situation. It’s very interesting. It may be indicative of something that’s going on that we’re not aware of, but that’s not enough. We have to understand why? So, I would encourage that researcher to knock on the doors and ask why? And then we may hear things like, “oh, you know you have a 15-page manual on how to install printers and I’m not going to waste my time on that.” Or, “you have to log in each time you go to a printer and that’s taking more time.” And so on and so forth. “We do steal your paper, so we enjoy that.” That’s what I mean by deeper understanding, so we can better understand the situation, know to answer why that is happening, have some evidence and then I would say that’s an insight. That’s a deep understanding of the situation. And to me the system that we built is going to enforce that, so you cannot submit nuggets or insights without that why part? Just facts are not enough. We don’t need facts. We need facts plus why they are what they are.
Steve: Right. There’s a behavior and what’s the reason for that behavior. I struggle when I hear people talk about insights because sometimes they talk about why a single person is doing something, as opposed to sort of why users of a system are experiencing something. Like, in your scenario, you knock on somebody’s door and say what’s up with your own printer? “It’s too hard to install.” And you come back and say – like there’s a difference between we understand why this person was doing it and then sort of the generalized conclusion. I don’t know, I’m putting my own language into your framework. It’s probably not working, but…
Tomer: That’s alright. What we would do with Polaris is gather those individual insights. So, each one would be a nugget. And then if you have 100 of these, the only difference would be the video, the person in front of the camera explaining why they brought in a printer, or whatever it is. And then you can create a playlist and show it to IT, or whoever decided that we’re going to go with this system for printing, and have them decide what they’re going to do about it.
Steve: Okay, so let me push on that a little bit. Because when you talk to say 5 people or something, about this behavior, bringing in your own printers, people are going to give related, but sort of seemingly individualized explanations. And there’s an act of interpretation, analysis and synthesis – words that we often use – to sort of say well let’s look at all of those. Like what’s the overarching reason? Talk about deep understanding. It’s not that we loose track of the fact that the installation manual is messy. There are not enough plugs. There are a lot of sort of reasons, but the larger issue is something like complexity, or not seen as adding to my business or something. There’s a higher order thing. To me, that’s what the insight is. I think you were talking about doing that.
Tomer: Yeah.
Steve: The words are slippery here when we’re talking about insights and nuggets and sort of explanations.
Tomer: Nugget is more the kind of the technical way we called it. But, I agree. And the way it happened in Polaris is through those playlists. So, we encourage people, both in – you know people who belong to the UX team, and ones that are not, to collect these nuggets into playlists and then prove a point, or do this analysis, get to an insight and share it. So, it depends on what you find and what you collect there, but let’s say you search for printers and then you get 73 results. You sift through them and you see that 15 are not really related to what you’re trying to communicate here. The rest is too many, so you’ll pick maybe 7 that the videos are really good, like “good participants” that eloquently explain the point and then you can add your analysis in writing and describe that higher insight, or deeper insight.
Steve: Right. So, Polaris kind of sets you up to make that interpretation.
Tomer: Yeah, yeah.
Steve: It gives you some structure or some way to quickly…
Tomer: Yes, yes.
Steve: Does Polaris capture that thing it facilitates you to make?
Tomer: What do you mean?
Steve: So, create this playlist. You watch the playlist and you come up with sort of new articulation. The biggest issue around printers for us is that we are doing this, and they are thinking that. Right? That’s a new piece of knowledge that’s created by reviewing what Polaris gives you.
Tomer: Yeah. Polaris was not smart. It’s just a tool. It would do whatever – it allows you to do whatever you want to do with it. So, it allows you to add text to do that. So, if that’s a yes then it allows you to do that. Polaris, in its kind of essence, is very simple. Just a tool that facilitates the kind of storage of those nuggets and creation of those playlists.
Steve: Yeah. So, that really helps me understand sort of what you’re aiming at when you talk about insight repository. It’s to me something you can re-query to come up with new conclusions.
Tomer: Yes. Exactly.
Steve: As opposed to sort of here’s all the things that we’ve concluded.
Tomer: Yes. Exactly. Exactly.
Steve: Okay. Wow. Here’s the GIF of the mind blowing up a little bit.
Tomer: You want more?
Steve: If you have more, yes.
Tomer: Now it’s just ideas and that’s not something I’m kind of working towards in Goldman, but I’m thinking – so imagine every company that should have a Polaris has it. That’s also a waste because then every company has its own repository and then I’m sure there’s a lot of overlap. So, maybe in the future there should be a kind of open…
Steve: Panopticon.
Tomer: And open Polaris, or whatever we call it, that anybody can contribute to and anybody can pull from. I’m not doing anything about that.
Steve: A friend of mine – so, this is like a third hand quote that I’m sure I’m misquoting, but a friend of mine told me this and he was quoting the head of knowledge management at NASA. This guy says, “the best knowledge management tool is lunch.”
Tomer: I can see where – I can see why that was said. Yet, I wholeheartedly reject the idea.
Steve: Yeah
Tomer: I mean, that’s not scalable. That’s not what if I went to the bathroom when that happened? What if I started working in NASA the day after that important insight was shared? I can understand the kind of anecdotal part of it, how it’s useful. But to me there has to be something – I don’t know what to call it – more solid.
Steve: Yeah. Analogously, you started off our conversation by describing the lunching that you’re doing at Goldman, connecting people, meeting people. That’s different I think than transmitting nuggets, but you are using sort of lunch, and I mean very vaguely lunch, time with people, talking to them as a way to – I don’t know, as a way to do what? What’s the difference between sort of the NASA lunch thing and what you’re doing with socializing and connecting people?
Tomer: I think what I’m trying to do is kind of socialize a discipline. And if I understand it correctly, the NASA lunch is to socialize an insight maybe. So, yeah, I don’t think we’re there yet in terms of socializing insights.
Steve: And who knows what the context of that quote, which has been quoted, which I’m requoting. It may be exactly coherent with what you’re talking about.
Tomer: Maybe
Steve: I want to loop back to something that you talked about as part of this infrastructure. You said there’s groups that don’t have a researcher and may never have a researcher. So, what are sort of tools, knowledgebase, that can help them do things? I feel like that’s – there’s a thing in our field about who does research? And I’m not even sure what the label for that is?
Tomer: Job security.
Steve: Yeah. Is that what it is?
Tomer: Should we let them do research?
Steve: Right.
Tomer: Yeah. It makes me laugh. Of course, we should let them. As if we’re the authority. But, yeah, of course. I mean why would anyone not be allowed to do research? Because they didn’t go to school? I don’t think so. If somebody wants to do it, to me that’s huge. So, we should give them everything we can to let them do it. Are they going to be doing a bad job? Maybe. But to me bad research is better than no research. It’s a first step and we are good about the tools, the socializing what we do, socializing best practices, things will get better. Yeah, there will probably be crap added in the first few times. Maybe the first 20 times. But if they really want to and they have this passion, then why kill it by saying that it’s our job, or something like that. So, yes, I’m all for “letting them” do research. Definitely.
Steve: I mean I think you highlighted exactly – so there’s – I think job security is a fear, but I think bad research is also a fear, as you said.
Tomer: Yeah. I’m okay with that.
Steve: And you said bad research is better than no research.
Tomer: To, me, yeah. 100%. Yes
Steve: I like how definitive you are.
Tomer: I’m…
Steve: Because that’s a hot topic, I think. I’ve heard people go back and forth on it.
Tomer: I know. I heard that too. I’m definitely on that side.
Steve: I also will say you’re describing ways to limit or mitigate bad research.
Tomer: Help make it better.
Steve: Yeah.
Tomer: Yeah, yeah. I mean first they need to know about it. What happens – there are so many people who develop, let’s say software, at Goldman. They’re not even aware of our existence. It’s not that they think about it and say, “uh, no, I’m not going to do it.” They don’t even know that this is happening, that we are there. So, I think there’s a long way to go. We need to kind of be more popular and be more known and then provide all the tools, help, guidance, knowledge that we can. Knowing that we can’t support everybody. It’s not going to happen. It happened even at Google that had, at the time 100s, and today probably a lot more researchers. There are teams that build stuff. Why not allow them to do research to help them?
Steve: You made the comment that research was happening before there were researchers.
Tomer: Yeah.
Steve: In the history of the world that’s also true, right.
Tomer: Yeah. True. We can’t stop that or control that.
Steve: So, maybe just a slight shift. We’re sort of talking about who’s allowed to do what, or what are researchers and what do we do? You mentioned early on that you also take on an additional role. Do you want to give some context to that? And what does that mean for you?
Tomer: Yeah, I also lead a design group for a product, Private Wealth Management. It allows very rich people to manage their money. It’s a part of a service that Goldman offers. And the digital aspect of it is not the primary aspect of it. It’s just kind of a supporting role. It’s mostly based on a relationship between an advisor, a Goldman Sachs advisor, and a client. And there’s the usual suspect of website apps and so on. I’m now leading a group of people that design that. And I mean design with the expanded way we define it. So, it’s not just designers. It’s also researchers and data people and a writer and prototyping and so on.
Steve: Are there differences between managing researchers – we talked before about people coming in with resumes that scream researchers – vs. all the different kinds of functions you’re working with on that team?
Tomer: Honestly, no. I don’t think so. I would be the first to admit it. I’m personally not the best designer. Not in the world definitely and not that I could be. But, I think there are things that are similar no matter what kind of group you’re leading. It’s good that you know something about what the group is doing, but I think it’s mostly about empowering the right people, giving them what they need, releasing them of things that are just stupid, that they don’t need to do, and they don’t need to be involved in, and focusing them on what they are passionate about. This doesn’t have any direct relationship with design or research or whatever it is.
Steve: Yeah. There’s an email list that I think you and I are both on that’s about design and user research. And there was a thread, or maybe people were having a conference call. I can’t even remember how it manifested, but the topic was researchers managing designers, which seems like it’s a newer thing. If you look historically, like research was sort of the accessory or adjacency to design, so design teams kind of managed researchers. But as research has grown there’s other people in the situation like yours where their label for themselves would lean more towards researcher, but they’re managing designers. So, it’s interesting that you sort of don’t see a difference because I feel like the thrust of this group needing to talk was hey, there’s something different here and so how are we going to deal with it?
Tomer: I know I have kind of my internal bias toward research. So, I’m probably more kind of attentive to mostly when that is not happening. Maybe than a person who would be a designer that manages designers. But I’m just guessing. I don’t know. I know that I’m definitely – I care about research and I notice and say something when it doesn’t happen. I don’t know – does it have to be a designer – designers need to know if that’s their thing.
Steve: I think I might switch gears entirely here.
Tomer: Go for it.
Steve: I’d love to just go way back, like as far back as you want to go and maybe give the story of things you did in your life to kind of get here, whether those are work or school or other things?
Tomer: That got me here?
Steve: Yeah. What’s your sort of background, or your narrative arc, if you will?
Tomer: So, I’ll tell you something from a long time ago that probably really was the tipping point, that I wasn’t even aware of that that was the tipping point at the time. I mean it’s not a secret I’m originally from Israel and I relocated, it was 12 years ago. And while in Israel I served in the Army. I signed for a career in the Army. So, the whole shebang. I was going to be a career officer for the long term. But then when I was 24, long story short, I was a paraglider. And I took a course and I got injured badly and I was out of the Army for a year. I was at home recovering. And that was a bad thing. Bad injury, but that opened my eyes. And during that year I came back to the Army and said I want to cancel the whole thing; I don’t want to stay; I want out. I will do my next job because we planned for one more job, but that’s it. I’m out. And that’s what happened. I think without that accident I would probably – I’d probably be retired by now, but I would be a career officer and not what I am today. And kind of looking back, I’m happy that that’s what happened. I would say that’s the biggest thing that affected what I’m doing today.
Steve: So, opening your eyes was realizing that you didn’t want to go on the path that you were on. Was there any hints for you of what path you did want to pursue?
Tomer: I knew it was creative. I was in a wheelchair for four months and then crutches and then learn how to walk again. And I made my way, once I was able to kind of get up, to a local artist that gave kind of very open-ended lessons in his basement, like a couple of blocks from my house at the time. So, I started painting and tried all kinds of ways to paint. And I know – I didn’t know to say that I will be an artist and honestly, I wasn’t really good at it. But I knew it would be something creative. I didn’t know exactly what.
Steve: If you look at your work today, does it match that?
Tomer: Um, not 100% overlap, but I feel some of it is, yeah.
Steve: I mean I have wrestled, mostly privately, with just the idea is research a creative field? Or are we creative?
Tomer: Some of it is.
Steve: I found myself in a collaboration with people that I think more traditionally fit that job description and I kind of had my hair blown back, just on sort of the speed and breadth of making stuff. It was definitely intimidating.
Tomer: So, I would call myself a researcher, but still I was heavily involved in shaping Polaris. That’s a product. It’s not research work. I’m now involved in creating a system that – I also lead a small team of engineers that build a system to measure the user experience. So, that is definitely more creative than maybe research. But some parts of kind of pure research are creative. To me probably the biggest one is translating one or a set of questions that a team has into okay, what are we going to do to get answers. That’s not always – if it was that easy to come up with an answer to that, then anybody could do that well. That’s not the case. A lot of people are having a lot of trouble with that part. So, I think that’s a creative part. You’re not going to see a beautiful painting coming out of that, but it is creative.
Steve: Right. I think, for me, creating the new story out of a bunch of experiences or nuggets, or whatever you’re pulling from…
Tomer: Realizing, getting to an insight. Yeah.
Steve: So, what did you do after the art class? What do you end up doing?
Tomer: I applied – I applied to what we call today, probably a visual communications program. Got accepted to one of the best ones, if not the best one in Israel at the time, and last minute decided that it’s not for me. And then I learned – I studied copywriting. So, I’m a certified copywriter, in Hebrew though. And then I took my first job – or, I worked. I worked for 3 years as – I didn’t really know what I was going to do, so I did something that I knew how to do and that was I worked in a very small consultancy for military oriented industries. So, I did that for – I was a project manager. I did that for I think 3 years. That was the time where I kind of learned all these things and really set my mind that anything army related is not for me. And then my first real job in that direction was I was – funny how names were at that time. I was an internet copywriter, which we would probably call today a content strategist, for a website that I would compare it to Monster or Indeed, or something like that today. And there I got exposed to – probably at the time it was Jakob Nielsen and people in that area. I started reading more. I did some, again what we would call today, product management work there. And then I asked them to switch to what we would call today a researcher. They said, “no.” And I was like, okay, and I looked for a company who would take me. And there was one company that took me as – again, it wasn’t researcher. It was called a usability something. And that’s it. That’s how it started. They were very brave, I should admit, because I didn’t know much.
Steve: But they took you as a researcher?
Tomer: Yeah.
Steve: So, that was sort of your first time with the title.
Tomer: Yeah. You want an even funnier story. The only person who actually knew what it was was the CEO. He was the one who interviewed me. And then he hired me and then a month later he decided to leave. So, the only person who really knew what I was doing left about a month in. But that went well. That’s how it started.
Steve: What was the point at which you came to the U.S.?
Tomer: Um, so, I had a couple more jobs and I realized very quickly that, at the time at least, there weren’t enough, or even at all, opportunities to grow into managing a group or a team of people who do that. I was always the only person in the company that did that. And I realized, or I got to the conclusion that the only place for me to work in a company that had a lot of people of my tribe would be here in the states. And I also realized the army there is not like the army here. I didn’t have any academic degree. I also realized that I needed the right degree because these companies that I was thinking about would not even read my resume. My resume didn’t scream researcher. So, I went to school. I continued working and went to school. I completed my Bachelor’s and then applied to Bentley University here and then moved. And as I was studying there I – during school there, in Massachusetts, I contacted Google and things rolled from there.
Steve: We talked about WeWork a little bit. Can you – like what was the – what was your role at Google and maybe what was your role at WeWork?
Tomer: Google, I was a user researcher, a senior user researcher. First in advertising. They’ve changed all the names by now, but at the time it was DoubleClick for Publishers. That was the product I was involved in. Ironically, I was the only researcher there in that group of hundreds of people. And after, I think, 2 ½ years I switched transferred to Search and in Search, I was first Voice Search and then we called it, at the time, Core Search – the bag of 10 blue links and how they developed from that point to what you see today as all the visual aspects of the result and so on. So, that started back then, vertical by vertical – TV, movies, music. I was doing a lot of research into search results for sports. I know a lot more than I should about all kinds of sports, cricket and so on. Yeah.
Steve: And then what was the role you took at WeWork?
Tomer: WeWork, I was head of user experience. So, started a group from scratch. The goal there, and this is kind of following several conversations with the CEO and co-founder that hired me. At the time – again, I’m sure things have changed since. But at the time WeWork had three big groups internally that kind of built or created the three aspects of the WeWork product. They were called digital, physical and community. And Adam, the CEO, felt, and he was very right, that while each group is doing a great job, sometimes if you look – if you’re going to think or look from how things are from the perspective of the customer, the member, there are gaps between those groups that we’re not even aware of. And the goal was to identify those gaps, to me that translates to research, and then solve them – solve the problems there. I’m thinking of an example. Think about conference rooms. So, conference rooms is something that WeWork offers. Members pay for it. So, somebody physically designed the conference room. An architect decided on the size and location. An interior designer decided on the mood and what will be in the room. Somebody from IT picked and AV system for that room. Somebody in digital developed a system to book this room. Somebody in community designed a policy of how to use this room. And community team members in the buildings enforce this policy. Everything is good. Everything is working well, but then situations happen. Such as members come to a meeting, they book the room and then another member is squatting the room and refusing to go out. Even if they walk in they realize that I’m a startup, I booked a room for a meeting with a potential investor and then I see a room that is designed as a music room with bean bags and no projector, clearly inappropriate for my meeting. Or, that person who’s squatting, I’m going to the community manager, but they are dealing with a leak of water from the ceiling on another member’s head. They’re all very nice, but they can’t solve my problem right now. So, this is what I’m talking about when I said the gap between one of those groups. So, we’re trying to identify those gaps – because in many cases we didn’t even know about this – and try and solve them. That was the premise back then.
Steve: You’ve written books. You give a lot of talks.
Tomer: Less now.
Steve: You have a good sort of history of creating material. You’ve interviewed a lot of people. People listening, what would you send them to, to buy, read, watch?
Tomer: Well, our publisher would not be a publisher – would be happy if I say that they should buy my book, yours too. And that would be – the name of the book is Validating Product Ideas Through Lean User Research. That’s a book for – I would only mention that because my first book is for researchers, and a lot of researchers do listen, so maybe I’ll mention both. But the second book, the one I mentioned, is for – it’s a step by step guide into answering different research questions that people have. Each chapter is a research question, step by step, on how to answer it through research that anybody can do quickly. The first book is called It’s Our Research. It’s to solve a problem that, at least at the time – I now have different thoughts about that – but, at least at the time a lot of researchers had, maybe today as well, and that is a problem that a lot of people don’t want to do research because they feel they have the answers already, or they have good intuition. And also, once they agree to do research in some cases they don’t want to act on it.
Steve: Wait, act on?
Tomer: On their research results.
Steve: What they’ve learned? Yes
Tomer: So, that’s a book that’s supposed to help with that problem. And I’m kind of having different thoughts now because my answer in that book was – and that’s why I called it It’s Our Research – make them feel that it’s their research as much as you feel it’s yours and then they would want it and they would do something about it. That was my point. Today, I’m having kind of different thoughts about how to get to a point where research is wanted and acted on. And I will try it at some point. We just need to grow a little bit at Goldman. But, my thoughts are when you – and I posted something about that recently – when you plug your own charger into the wall, do you really care how electricity gets there? What’s happening in the power plant? Why it’s working? How is it efficient? Is not efficient, and so on. You don’t really care. You want your phone charged. My thoughts about research is why wouldn’t it be the same. People have questions. Research provides answers. Yes, a lot is going on to get to those answers, but if you have a question, why do we need to bother you with all the details? Just do the thing, trust us to do the thing, and we’ll give you an answer to your question. I’m going to try that at some point. I’m thinking – it’s not political, but building a wall between stakeholders and researchers. That wall could be Slack or something like that through which stakeholders ask questions and researchers provide answers. If we have an answer immediately, if we have a system like Polaris or something like that, we can provide an answer. If we don’t, we will just ask a few questions, follow-up questions, and then do the research and get the answer. Just thoughts. I haven’t tried it yet.
Steve: Which makes me think of the research ops piece a little bit where – like building up participant recruitment infrastructure…
Tomer: Yeah.
Steve: …is an interesting one because back in the old days where we had to do everything ourselves, you’re learning about your problem by figuring how to – by recruiting. You also learn about your problem by dealing with your stakeholder and seeing what – it’s that art piece vs. this kind of process infrastructure piece and it’s interesting to think about like what is lost and what is gained? Or how is changed when you create infrastructure that – like if you’re a researcher and you’re completely decoupled from participant recruiting, that may change how you deal with people that you meet, or how you deal with framing the problem. So, for everything that we build up a process, that’s efficiency, that kind of is a query system, how does that change what we do? And who is coming to this field?
Tomer: Yeah.
Steve: These are not necessarily my own thoughts, but just things I’m hearing from people as well.
Tomer: One other thing that I would send people to is a series of Medium posts that I’ve published in the past year, maybe less, about measuring user experience. A lot of people like to talk about metrics these days. I took the HEART framework from Google and then we have a post per letter about happiness, engagement, adoption, retention and task success. And for each one, what it is, what’s important to measure, why, how, mistakes and what actions you can take from each one? So, this is something that I’m interested in these days, measurements. And I’m trying to figure out the “H” part, the happiness part, specifically. There are a ton of challenges with that. How to measure satisfaction and happiness. I’m also posting – kind of tracking my – not tracking my own life but paying more attention to when I’m exposed to requests to rate satisfaction and happiness and I share them with people, with my thoughts about them.
Steve: Okay. Great. Anything else that we should talk about in this conversation?
Tomer: I said I am speaking kind of publicly a lot less, but I do do that from time to time. I’ll be speaking in two conferences, the Face of Finance in April in New York and in London, User Research London in June, in London.
Steve: Alright. Well, thanks for taking the time to chat and sharing all the information and stories and everything. I really appreciate it.
Tomer: That was fun. Thank you.
Steve: Thanks. And so concludes another episode of Dollars to Donuts. Follow the podcast on Twitter, and subscribe to the podcast at portigal.com/podcast, or iTunes, or Spotify, or Stitcher, or anyplace you get your podcasts. Also online at portigal.com/podcast is the transcript and links for this episode (and of course all the previous episodes). At Amazon and rosenfeldmedia.com you can buy Tomer’s books and my books. Our rocking theme music was written and performed by Bruce Todd.
The post 17. Tomer Sharon of Goldman Sachs first appeared on Portigal Consulting.
In this episode of Dollars to Donuts, I speak with Marianne Berkovich, Head of User Research & Consumer Insights at Glooko. We talk about doing research through leadership changes, setting up opportunities for self-critique, and how to build empathy, especially in health technology, by experiencing some aspect of the condition and treatment yourself.
It really bothers me when smart people go out and build things and spend a lot of time and energy to build things that are not for humans. And I’m like, erh, why didn’t they do that? For me it’s more about empowering people who have that energy and who have that entrepreneurial spirit to make the things that are right. Not just make stuff, but actually, make the right thing. It’s a set of skills that I think maybe everybody should have and maybe once everybody has those skills and can do it well, maybe the role of researcher doesn’t need to exist. Until then, I feel like it’s my duty to go out and spread the gospel, as it were, of this is how you talk to users. – Marianne Berkovich
Show Links* World IA Day San Francisco * Marianne on LinkedIn * Marianne on Twitter * Glooko * Google * Adobe * Blue Ridge Labs * The Human-Computer Interaction Institute at Carnegie Mellon University * Lockheed Martin * Bonnie John * Honing your user research craft: Doing a critique through a “Don’t Leave Data on the Table” class (Medium) * UX Interviewing: Personalized Coaching to Avoid Leaving Data on the Table (from From CHI EA ’17) * Interviewing Workshop: Don’t Leave Data on the Table (from Mixed Methods podcast) * Net Promoter Score * Three Ways to Build Empathy when working in Health Tech (Medium) * Rufus the Bear at the JDRF * 9Round Fitness * More human: designing a world where people come first * Venture for America * Nasdaq Entrepreneurial Center
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Well, hi, and welcome to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization.
I was reading the New York Times today and I noticed something unusual, although I’m seeing this sort of thing more and more. After the byline, but before the article starts, is some italicized text in square brackets. It reads “What you need to know to start the day: Get New York Today in your inbox.” This is copy that only belongs online, in the app or the website or an email newsletter. Presumably you would click on something. But I’m looking at the newspaper – I feel like I’m starring in one of those YouTube videos where a toddler is trying to swipe a magazine and can’t figure out why it’s not a touch screen. I see the New York Times making this kind of error in their print edition every few weeks, and it’s kind of appalling, because it suggests a lack of detail that I don’t expect from a high quality product like the Times. When you get an email that has the wrong name in the salutation, even though we’ve all done it ourselves, it brings your appreciation down a notch or two. They clearly aren’t taking the care that they used to, and that we would hope for.
And it’s especially interesting because I remember when the opposite used to be true; when the experience we had online, with news especially, was a not-quite-there translation of a print experience. And now it’s flopped. The print edition readers are not the primary customers. The organization has identified a different key user and it’s not us.
I can’t help but wonder about any of the users we learn about, are they in a less desirable category or perceived that way because of changes in internal processes or organizational structure? Do we care about their experience, or are we making them into what we call “edge cases” which is a fancy way of dismissing them. It’s hard to imagine the print reader of the New York Times as an edge case, but hey, that’s where we are.
I want to remind you that I’m looking for ways to be able to keep making this podcast for you. Here’s how you can help. You can hire me! I plan and lead user research projects, I coach teams who are working to learn from their customers, and I run training workshops to teach people how to be better at research and analysis. I’ve got two books you can buy- the classic Interviewing Users and Doorbells, Danger, and Dead Batteries, a book of stories from other researchers about the kinds of things that happen when they go out into the field. You rating review this podcast on iTunes, and you can review both books on Amazon. With your support, I can keep doing this podcast for you.
Let’s get to my interview with Marianne Berkovich. We did this a little differently – as part of a live event. We closed out San Francisco’s local edition of “World Information Architecture Day” with our interview, live on stage, in front of an audience. Then when we got off stage, we sat down in a somewhat noisy room and talked a little more to cover some of the things we didn’t have time for. We’ve cleaned it up as best we can but the audio may be a little bit different from how things normally sound. It was really fun for the two of us to speak on stage, and I hope to have more opportunities to record episodes of the podcast in similar settings.
Marianne is the Head of User Research + Customer Insights at Glooko. She’s worked as a consultant and for Google and Adobe.
Thanks for agreeing to talk with me, Marianne. Why don’t we just start by having you introduce yourself. What do you do? Where do you work? Tell us about that.
Marianne Berkovich: My name is Marianne Berkovich and I am Head of User Research & Consumer Insights at Glooko and we’re an online diabetes management platform.
Steve: What is an online diabetes management platform?
Marianne: That’s a great question. So, diabetes is a condition that’s a lot to do with numbers. You want to keep your blood sugar not too low, not too high. So, it’s really conducive and lends itself well to technology and checking your numbers. We have an app for the person with diabetes. They can track all sorts of things there that could affect their blood sugar and then they can also send that information to their clinician who can further see patterns of you’re high in the mornings and what are we going to do about that? So, it’s a platform that both the clinician can access and look at those things as well as for the patient themselves.
Steve: Where in the history of this company and its product did you come in, to bring in user research?
Marianne: So, I am the first user researcher. The company was founded in 2010. I’ve been there for about a year and a half and I am learning all sorts of things of what it’s like to be, first in a healthcare company, or a health tech company, and also being the first researcher.
Steve: Okay, let’s see how easy this is. What are some of the things you’re learning about being the first researcher and working in a health tech company?
Marianne: First of all, as a researcher, it just feels very weird to be answering the questions and not asking the questions. Maybe I answer questions in a way that makes it easier for the researcher to ask the next follow-up question. So, but you know what a great researcher Steve is, so this is showcasing his talent too, right! So I think some of the things that I’m learning is the role of advocacy. That even though there was a need for hey it’s time for a researcher, we need someone full-time – before they were hiring vendors or kind of doing it ad hoc. And so, it was time to have somebody who could be dedicated and who’s trained in this. But at the same time there’s a lot of – I don’t want to say – in some cases there was resistance, but I think in a lot of cases it’s more of just not knowing what good research looks like, or part of the reason my title is so long is that people had an impression that oh, user research is just usability studies. And so, I talked with my manager and we put in the “& Consumer Insights.” Head of User Research & Consumer Insights – so it’s like it’s everything. It’s going to be going out and doing field visits. It might be surveys. It might be a lot of different things. So, to kind of help with that advocacy. And I feel like I do spend – I mean, I think as researchers we always spend a lot of our time in advocacy mode, but I’m not surprised – or maybe surprised – it’s just taking a lot more of my effort to do that advocacy work of what research is and how can it be used.
Steve: I just want to clarify. You’re advocating for user research?
Marianne: Yeah. So, I think even though the company talks about being patient centered and user centered, that what does that really mean? I think one of the things that I’m finding out – so, there’s a lot of people who have been in the field for a long time and they’re like we understand diabetes. We’ve been in this for many, many years. And to have somebody come in – I don’t have any experience with diabetes; I don’t have diabetes myself – to come in with these, you know, different insights or different ways of doing things and people are like, we’ve been there, we know how to do this. And so, to advocate for, hey, we went out and talked to some people. We learned this thing that’s different and knew and it’s maybe against the conventional wisdom, how might we use this? And maybe some of that resistance of like, well we’ve always done it this way. Or conventional wisdom says this other thing.
Steve: So, the other part that you were learning was working in healthcare tech. It’s a new industry for you. So, talk about maybe what that’s been like.
Marianne: Yeah, so – and before that I was – I did a fellowship and we were working in the legal space. And the legal space, the health tech space, are slower moving than just straight up consumer products. It’s like hey let’s build something and we’ll launch an app and do these things. You know regulated spaces of like what is the FDA like? I had a little bit of background in human factors testing, but we have a product that actually got FDA approved. But what does human factors testing look like? How is that different from just regular usability testing? What does it mean to understand the whole ecosystem of payers and employers and insurance companies and PDMs and all that type of thing of moving that needle and kind of inching things forward when you’re in this space that can’t be disrupted so easily because there’s good reasons why these constraints exist.
Steve: So, the company was about 8 years in existence when you joined. So, how much of that were you building from scratch? Describe a little bit about what you encountered and what you had to build around sort of the tech and regulatory aspects maybe.
Marianne: So, actually, luckily the regulatory aspect is just one part of what we offer. It’s the Mobile Insulin Dosing System. So, when you start on insulin, how do you come up to taking the right dose of a certain type of insulin. So, the product does a lot more than that. So, this was just one small part of it. And it was actually – that was already in the works when I joined. So, the app was already out. The website was already out. MIDS was already sort of in progress. So, I think for me it’s been a role of a little bit of back to basics of like let’s learn about our users. How are people with Type 1 diabetes different from Type 2 diabetes. So, in fact, I was telling Steve before this, I’m doing an ethnography right now and so I was in Fresno this week talking to folks with Type 1 diabetes. I’m in Phoenix on Monday, but kind of building that basic understanding and actually building personas and really having a deeper understanding rather than like, yeah well, we have people now who are Type 1s, they know that they need. To move beyond that, to really have some foundational knowledge, and it’s causing us to go back and be like okay, well, is this – is there things in the app that are working the best way. Maybe we can make the clinician experience more efficient. Maybe there’s things that we learn when we go to visit a clinician – be like oh, that’s not how we thought that worked, can we rethink that. So, I think it’s a little bit of back to basics and rethinking certain things rather than to build anything from scratch.
Steve: Before you talked about advocacy which was kind of saying hey, we need to learn these kinds of things, but in some ways doing that reveals the most scary thing of all – oh we need to rethink assumptions and yeah, a 10-year-old company, it’s entrenched. So, what happens when the things that you’re uncovering are inviting, challenging – they’re inviting the challenging of established belief structures around the product and what it looks like and everything. How are you doing that?
Marianne: I think it leads to another kind of wrinkle of working in a startup is that we’ve had a lot of turnover at the leadership level, actually. So, the CEO who was the CEO when I joined was not the founder. He was the second CEO. And he reached a point where he was like, “you know what I really like getting the company to where it is now, but now that we’re really – it’s time to scale and grow things, it’s not what I want to do.” So, we got a new CEO. We got a new head of product. We got a new commercialization officer. So, all that advocacy I had done and sort of like when we did they Type 2 ethnography when I first started, like all that – you know taking people on the road and showing them and doing these empathy workshops and all that – whoosh, out the window because we’ve got a whole new cast of characters now. So, I think that was part of it too. It was a little bit humbling to have to re-sort of create my credibility again with – it’s a whole new cast of characters and a whole new set of people to influence. Luckily my manager has been a great champion of that. And so, I think it’s finding ways that we can leverage and sort of maybe it’s not the right time for things. So, we’ll go out and do this research and we have this foundational stuff. We sort of did a big aha, then those people went away. Can we save that and find another time to sort of bring that to the fore when it might be a little bit more accepted? So, I think it’s both sort of pushing for it and finding the right time to sort of introduce those things.
Steve: So, what does a manager do? How does that championing work?
Marianne: I think, – and I don’t know if this is true, but I feel like as a woman you know I can say a little of things if she’s also a woman, but like women in general, like it’s much more effective if instead of me saying it, I say like, “yeah, what Steve said.” Or, you know, what somebody else said. So, I think because of that, I think it becomes more effective. So, she’s not tooting her own horn. I’m not tooting my own horn, but she is sort of pointing to the work that I’m doing. Plus I think we have different kind of communication styles, and sometimes she can sort of, – I’ll say a bunch of stuff and she can sort of synthesize it in a really nice way and it sort of becomes a little bit more impactful that way, that like I can spend all my time sort of yelling and screaming – metaphorically speaking – and then she can sort of bring it home in a very succinct way.
Steve: So, part of what she’s doing then is also – you talk about finding the right moments. Or, what should we be doing to have impact on the organization as the layers above us are changing. I don’t know. I’m wondering about when new people come in, and this is going to vary on a case by case basis, but you’ve gone through this whole series of trying to open people’s minds up a little bit and say hey things are different. Those people leave. New people come in, but is there an opportunity there? Do those people – what baggage do they have, or do they bring in? I’m not asking you to like slag anybody individually, but as you see these kinds of changes happen in an organization and you’re trying to craft a story that’s about how the world really is vs. maybe what we hope or assume – how does the – what changes when new people come in, with or without baggage, around what the truth is?
Marianne: I think part of it is understanding where people are coming from. Because I think in my last role – I was at Google for a long time and I think like certain assumptions I made that like everybody knew what user research was and everybody knew this and I kind of got used to that, that that was the norm. Like product managers of course know what I do, and everybody knows how this works. And so, I had to figure out like, oh, okay, I can’t assume that people know what these things are, or that when I say user research and what you think in your experience with user research, like maybe that was focus groups. And they’re like yeah, we did those and it was great and I was like okay, great that’s a start. So, I kind of know where I’m starting from with folks. So, I think that’s part of it. And I think also, because I am the only researcher and things move a little bit more slowly, like we did the stuff around Type 2s and now we’re doing the Type 1s. And so now I’m using all that stuff that I learned last time of like what was effective in terms of running a workshop and getting people to come on visits with me. And it’s like it’s a little bit smoother again this time, but we’re doing that again this time around. So, it seems that having that opportunity to redo that foundational research has resurfaced itself. And so, to take advantage of that opportunity, that it’s like well, let’s look at Type 1s now.
Steve: There’s something here about you’ve had lots of experience. This is not your first job. You’ve been at lots of different organizations, you’ve done lots of research and influenced stakeholders and product teams and so on, but it sounds like that there’s a good measure of learning in this job which is about how do I do the thing that I know how to do to be effective in this context. Does that ever go away for researchers, do you think?
Marianne: I hope not. I think it’s – and I think that that’s part of the reason that like, you know – one of the reasons I left Google is like it took me a while to realize the thing that I was doing there and the things that – that, kind of really I’m passionate about are not things that I could do at Google. But like who leaves Google, right. So, I think finding ways to find my path and things that are not sort of the traditional way of moving up in, you know, you’re a junior research, then you’re a senior researcher and then you’re a manager and all these things. To find my own path and sort of be okay with that and find different ways to learn in different contexts. Like the fellowship that I did with Blue Ridge Labs which is a social impact incubator. And I was again, the only researcher, and so I got to do more mentoring. And so that was interesting. So, I think finding ways to both follow my passion and find ways that like what does this organization look like? And I think we’ve probably heard this all before in terms of doing sort of user research or bringing that lens to our stakeholders or our people that we’re working with as well. I think that part never goes away and it’s always changing because it’s always a different set of people.
Steve: Right, more so – is the landscape changing more in a place like Glooko than in a place like Google?
Marianne: That’s a good question. I think it’s very different. Google’s obviously a very – there’s just many people. And so, I think the dynamics of what it’s like to have an organization that big and when I joined I knew all the researchers and now the scale – each product’s team is much larger than that. So, I think it’s a different type of – it’s a different set of issues. So, I think wherever you go – like I’m from the East Coast and people ask – they’re like well what’s better, east coast or west coast, or all this stuff? And I’m like they’re different, right. I don’t think there are things that you can really compare. So, for it’s, – and this is my first time working in a startup and so I don’t have another startup to compare it to. So, maybe if I went to a different startup I could do more of that comparing and contrasting, but in some ways, it feels like apples and oranges of a large organization that’s established and, you know, is well funded and all that vs. a smaller one that is working in a very different space.
Steve: Just hypothetical – I know we’re not supposed to ask – apparently you aren’t supposed to ask hypothetical projection questions in user research, so good thing this is not user research. If you were to look for a job at another startup, I’m just thinking about you joining an 8-year-old company, is there a maturity that you would look for, or a timespan that you would think differently about in the next stages of your career?
Marianne: I think it’s less for me about that. It’s more about really getting jazzed about the problem that I’m solving. So, for me diabetes is a very big issue. Thirty million people in the U.S. and growing. And so that feels like a real meaty issue and that gets me excited. So, I’m like even if some days it feels like I’m pushing a boulder uphill, it seems like a worthwhile thing and that’s what gets me up in the morning. So, for me it’s much more about that and going in with eyes wide open of like well what kind of organization in it – is it, and like okay do I want to take that on. And if the answer is yes then working around those constraints because that’s just the nature of the beast.
Steve: You talked before about figuring out the right job title that would describe, to the rest of the organization, the way that you were going to work. As you – can you say more about sort of what that process was, what those conversations were that identified an opportunity, that helped get you excited that this was something you wanted to do?
Marianne: Well honestly, I think part of it was naivety, that like this was – I think it was actually the way it was posted or sort of written up was Manager of Consumer Insights and I’m like okay, whatever. You know, I’m not on a ladder. I’m the only one, so it doesn’t really matter. So, for me, I was so excited – in fact I was consulting before and so Glooko was one of my clients and I got hired. So, we crafted the role a little bit based on what I was doing. And so for me it was less about negotiating the right title and in fact I think only after I joined – because I was like, sounds great, like it sounds like this is a good fit, sign me up, and then only when I started and I was like hey, I want to make business cards, but I don’t really like this Manager of Consumer Insights – we had that conversation after I joined. And so that’s when my manager started telling me about like hey, this is some of the perceptions because I had gone in very naively also of like everybody knows what user research is. Like, we’re in Silicon Valley, like everybody knows what this is. And so, it didn’t even really occur to me that that advocacy, or the extent of that advocacy that would need to be done.
Steve: So, what – you worked with the organization as a consultant and then came in-house to kind of lead the effort. What was similar and different about – you know, the before and after that transition?
Marianne: So I think one thing is, it was very intimidating. I didn’t know anything about diabetes and so I ran my first study and like everybody showed up and so they were listening in on the call and I’m like how’s this going to go? Like I hope I don’t say anything stupid. And so, you know I think as a consultant, coming up to speed on a different domain very quickly, and I think to me it’s about asking those open-ended questions and asking questions in a way that it doesn’t really matter if you know the domain. I mean it’s certainly much better if you do, but there are ways that you can sort of cover that up, and especially if you’re going in with an apprentice mindset and all those types of things, it kind of helps the situation along a little bit. So, I think for me that was definitely intimidating to not know the domain and have everybody show up.
I think another difference is as a consultant, and I wasn’t a consultant for very long. It was about a year or so before I decided to take the role with Glooko , but I was very cognizant of how I was spending every single hour and whether it was going to lead to making money or not, because is this going to generate a lead? Is it going to generate a sale? All those types of questions were very much top of mind whereas when you’re in-house I think you can spend the sort of hours – like you’re spending your capital in a different way. You’re building relationships and that’s kind of how you’re sort of making money. So, to me it’s sort of a different way that you don’t have to be so aware of every hour leading to money.
Steve: I’m going to switch gears a little bit and maybe we could just rewind. I’d love to hear you describe maybe your path. How did you get into this stuff? What brought you to where we are sitting today?
Marianne: So, I was an English major for undergrad and after I graduated I didn’t know what I wanted to do. So, I spent a couple of years – I actually at that point did work. I was in D.C. and I worked at the National Museum of American Art. We were digitizing the collections. It was a very long time ago and so I was actually cleaning up all the photos and I’m like eh, I’m not really into this technology thing. My Dad also was a computer science professor. I’m like definitely not into that technology thing. Like that’s stuff that my Dad does. Boring. So, I spent a couple of years kind of bumming around a little bit, decided to move out to Denver and I started working as a technical writer. And I was working at a financial services company and so my role was to write the help text for this complex financial software. So, I would sit with the developers. They would explain to me how the complex software worked, and I would write it up. And then I had a thought. I’m like if we just made the software easier to use I wouldn’t have to write anything. So, that led me to first get a certificate in technical communication. And in that I started kind of looking around a little bit more and found the whole field of human centered design and human computer interaction. I looked around and I found a program at Carnegie Mellon. And at that point I had actually switched over to Lockheed Martin, which is a great role – you can ask me about that in a second. Sorry, I just keep getting the questions to ask. And, but I – you know I decided that I really needed a degree in this thing. That just kind of reading books about it or whatever wasn’t enough. So, I decided to go to grad school and I remember just seeing that description of the master’s program and I was like that’s exactly what I want to do and like all my life had been leading to that point to be like that’s exactly what I want to do and having that feeling of like, yup, this is the direction I want to head in.
Steve: Which master’s program? And then keep going. Yes, please.
Marianne: It was a master’s in Human-Computer Interaction at Carnegie Mellon University. So, the role at Lockheed Martin. So, I’m old. I’m definitely over 35. And this was sort of in the olden days and what we were doing is taking – so, you know how we have wildfires and all that type of stuff? So, the resources to manage all those things, like sending the air tankers and the trucks and the crews and all that – that was being done by hand. So, Lockheed Martin got a contact to turn that into – it wasn’t even online. It was just kind of a digital program to do that. So, we would interview subject matter experts and I was like a requirements analyst. Like we didn’t have roles of like designer researcher. Like that wasn’t a thing yet. And so, for me it was like super exciting to solve a real problem. We got to actually go to Boulder and like see the command center and see some of those things. I mean I really felt like I was making a difference and I knew what this was about. But I felt very much at a disadvantage of like, I don’t know what should the icons be? I don’t know. Nobody has best practices around icons yet. And so, it felt like early days and that’s why I really wanted to go back to school and like learn some stuff because I knew there was a lot that had been done already.
Steve: So, you come out of that program and where do you go?
Marianne: Consulting. Yeah, I actually stuck around Pittsburgh for a while and I again ran my own consulting thing. I do that to sort of – when I’m in exploration phase. Or, when I’m dating somebody who just started a PhD program and I can’t leave Pittsburgh. One or the other. So, yeah, and the guy I was dating, he was very entrepreneurial, and he was like such a great cheerleader. He was like yeah, we can figure this out. He had been running his own startup before he started the PhD program and he was like, yeah you can do this. I’m like, yeah, I can do this. So, he did help me a lot and so I was finding projects and was kind of getting my feet wet with that. But then I found that it was time to leave Pittsburgh. That it wasn’t what I wanted to be doing. I really wanted to learn from other people and so I started looking for other opportunities.
Steve: What were those opportunities that you found?
Marianne: You can take it in a different direction. So, I actually interviewed at Google at that point and they didn’t want me, I didn’t want them. It was a very different company at that point. But I also interviewed at Adobe. There’s actually a pipeline from Carnegie Mellon straight here to Silicon Valley. I mean seriously, just pull up a bus, just put the graduates on it and bus us all out here. So, there were a bunch of people who I had gone to grad school with who were at Adobe and I went to interview there, and it felt like a really nice fit. I remember actually – after the Google interview I came home to my hotel and I just turned on the Simpsons and like all I could do was just sit there and like not move. But after the Abode interview, I actually – it was one of my first times out here in Silicon Valley and like I went hiking. I was like I was that inspired and that energized that I went hiking. I’m like this is a sign that maybe I should take this role, that this is a good role for me.
Steve: We can cue up some Simpsons for you after this conversation. You know, just put you in a dark room. Um- so, when you talk about the work that you’re doing now, you’re doing field work, you’re doing ethnography, you’re taking people out. When did you learn how to do that?
Marianne: That’s a good question. I feel like I should credit CMU with setting me on the path and teaching me about contextual inquiry – thank you Bonnie John. And I think for me one of the things that they told us at Carnegie Mellon is – it’s a one-year program, so if you didn’t come in as a computer scientist you’re not leaving as a computer scientist. If you didn’t come in as a designer, you’re not leaving a designer. But we’re going to give you enough exposure to coding and designing and having your stuff on a wall and getting critiqued and to do some social science stuff. And so, for me, having been an English major and I minored in art history and theology, I mean I was as liberal arts as you could get. I wasn’t going to come out as any of those things and I didn’t gravitate towards it. I was like I’m not a designer. So, I think I just gravitated towards research. And I think maybe the English degree kind of prepped me for that in terms of asking good questions and thinking about things and looking for patterns. So, I think just naturally I’m a pattern seeker and I think honing my ability to sort of ask questions just came with time. So, I think it was having a good solid foundation and then just kind of having a natural affinity for it. And I read your book – much, much later, but I did read your book.
Steve: Right. By the time my book came out you’d been doing this for a while. Can you talk about the environment at Adobe? What was there there to develop the practice of user research? In the practitioners like yourself that are coming in with X amount of experience, what did they do well that let you get to the next level of your craft?
Marianne: I think actually I was really lucky at both Adobe and Google, being surrounded by really smart people. So, that made it okay of like I don’t know how to do this, and can someone help me? Like I learned how to do surveys through some of the really awesome people at Google. So, I think for me it was just being surrounded by really supportive people and I think that was one of the things that was part of the culture at Abode. Like we would go to my manager’s house. She would have like all the researchers at her house and we would do these offsite – and it really felt like family. I mean it sounds a little cheesy, but – and so it felt like it was a place to grow and learn. So, I think for me that was a great stepping stone as my first sort of real job after graduate school, to be like oh, okay, this is what being a researcher is like. Oh, this is how you interact with product teams. So, I think it was just learning all those basics. But I think one of the things that – kind of coming back to the question of learning to do ethnographic work and all that is once you’re out – and especially you’re probably one researcher. You’re rarely lucky enough to work with another researcher who can sort of observe you can say like hey, why are you asking that question? Or, maybe we could do it differently. One of the things that I found, when I did have some opportunities at Google to work with other researchers, I’m like we don’t all have the same skills. Like what happens between you and a participant is different. And so, one of the things that having noticed that is I started a class at Google to critique ourselves. So, give researchers an opportunity to observe each other moderating and to give feedback to each other. And then I read a Medium article about it. It’s called Don’t Leave Data on the Table, if you want to look it up. I think that’s one of the things too is that we don’t get that critical eye anymore and we assume that like every researcher is the same and it’s like well actually, you know – and those things can also creep up on us. Like I see myself all the time, especially now that I’m sort of teaching other designers at my company more of like, ah, leading question. Yes or no question. You know. So, I find myself still doing that, even if I’m aware of it. But I think having those things pointed out to us is also really helpful too.
Steve: What’s the structure of the critiquing class?
Marianne: So, it was slightly different when I taught it externally at CHI, and internally. So, internally all our videos are available to each other. So, basically people would send in clips from research that they did. We would make small groups, so it lets you work in a group of five, and we would watch a segment of somebody conducting a study. And at any point people could stop the video and be like hey, I noticed this. It was like when we were actually practicing to teach it, one of my co-teachers, she actually had her pen and she would do a lot of like pointing and sort of like do a lot of things that indicated high status. And she was like, I never noticed that – until she saw herself, until we talked about it, she wasn’t aware of that body language and what it was conveying. And so basically we’d spend 5/10 minutes, or however the time divides up, to watch that video and stop if we see anything. And we asked people to rent that idea that like, again, we don’t know the context of what came before. We don’t know the context after. But this is what I’m noticing. And that’s what we do as researchers is we notice. So, it could be a pattern, it could not, but at least invite people to consider something that they’re doing. And also, by seeing other people, just having that conversation, seeing what other people – and be like okay, I didn’t show that, I didn’t do that in my clip, but like I’ve definitely done that before too. So, learning from each other as well has been really valuable.
Steve: It sounds like a way to create kind of a safe space where it’s okay to critique is that everybody is kind of up for that critique and we’re all being exposed together. Is that kind of the way to make it safe? Because this is our workplace and now we’re talking about how we’re not living up to sort of the high standards of our profession.
Marianne: Yeah. And I think that’s one of the things that we tried really hard to make it a safe space. And one of the things that we did was actually we had – spoiler alert – we’d give out like balloons, which are also like these cords that you pull to sort of stop the production line. But just having a room full of balloons made it a little bit more fun. When I taught at CHI we had to create the videos and so I had these little stuffed animal guys that people could put their phones on when they were doing the videos. So, then we had little stuffed animals on all the tables. And so, – and also modeling it ourselves that – I don’t remember if we actually showed some of the videos, but we definitely told some of the stories of like when we did this ourselves – it’s like I’ve been a practitioner for how many years? Like I’m still learning. I still screw this stuff up. So, making ourselves vulnerable was part of making that safe space too. And also, I think what we tried to do was, when we did it within Google, was not have everybody who was on the same team be together. So, it wasn’t like anybody who you worked with directly.
Steve: You used the phrase “sort of screw this up” and I think like the pen example and I’m sure there’s others that are clearly things we shouldn’t do as researchers. To me it seems like there’s a certain amount of stuff that’s subjective that you and I would do it differently because we’re different and we have different personalities and different energies. For me, critique about that would be really interesting because I like to see how other people handle things. My question here I guess is, is there – does the critique approach look at alternative approaches? How do you talk about that vs. best and worst practices?
Marianne: I think we also gave an opportunity for people to stop the cord on themselves. To be like I didn’t know what to do here. Like help me brainstorm. So, there’s that opportunity too of like for the person themselves to be like how will we handle this? And to hear different perspectives on – there’s, you know, I could have done this, or I could have done this, or in situations in the past I’ve handled it differently. So, I think having just that variety of there’s not one right way to do that, that it depends on the context. And maybe sometimes somebody explains like well this is what happened earlier in the session and so it led me to do it in this way and we’re like, oh, okay, well that makes a lot of sense. And so, I think having a little bit more of that context helps too.
Steve: Are you able to bring any of this into the environment you’re in now where, if anything, you’re teaching people and leading them.
Marianne: Yeah. And I think I’m learning a lot more about screwing it up also. While I’m out in the field doing these Type 1 ethnography, like I’m trying to help designers to be able to do their own research. And so, one of our more junior researchers is actually doing one on one interviews. So, I helped, and we talked about it and I wrote the discussion guide and the study plan. We talked about it and she was all on board and off to the races we went. And then I actually had a cancellation, so I was able to watch one of the sessions and I was like oh no, I didn’t – I screwed up. I didn’t prepare her well enough. It was a little bit more than she could handle and that was squarely on me, you know. That because I’m so used to doing it and it comes so naturally to me that I didn’t recognize that she was in a different space and she would need a different level of support and that I just hadn’t prepared her. And so, we role played a little bit and we practiced a little bit. But I still feel like – I feel guilty. I’m like she’s got more sessions this week and like I don’t know if she’s really prepared. So, I think that part is on me as well to make sure that the things that I’m modeling and the things that I’m sort of teaching are doable for where the designers are and not just hey it’s easy, anybody can do this.
Steve: In my opinion there are some things in life that we do that we maybe learn more from screwing up than we do by succeeding. I don’t know what falls into that category and what doesn’t. I wonder if, as much as you feel for this person and feel responsible for them, I wonder just ultimate perspective, did you set them up for a lot of learning they might not have otherwise got to by – of seeing the edges of what they’re able to control or what they’re able to execute on?
Marianne: I can’t speak for her.
Steve: Good. Alright. Well we hit a dead end. Let’s see what other questions I have written for you here. If we were to have this conversation again in a year or two years, what kinds of things would you want to talk about that you would have achieved in this organization?
Marianne: I hope that there’s less sort of friction. I think we’re already starting to see the seeds of it, that you know product managers who have been at the company for a long time are saying like oh, I’m doing this new thing, you know I need user research. They don’t know what it is. They don’t know why exactly, but they’re sort of like it’s a good thing generally. So, I think to have more of that. Honestly, I think like some of the things that are more – a little boring, kind of more infrastructure stuff – like I hate recruiting. I’m a terrible recruiter. I’m terrible with like time zones and all those details, that type of stuff. So, like I’d love to have a recruiter. I’d love to have another researcher on board. And some of the things that I feel like I have a great success under my belt, that it looks like we’re moving away from NPS (Net Promoter Score), hooray, thank you, thank you, and moving towards satisfaction. You know asking satisfaction question and how often we’re going to do that. You know, that’s a really interesting thing that I’ve been thinking about is can we pop that up in an app. I mean this is an app that people use for health and like they might be having a low blood sugar. You know this is a stressful time that people are using the app. Like nobody is excited about managing their diabetes. So, to pop up something in the app, to be like how are we doing, that’s not the right thing. So, I haven’t figured out like how we might introduce those satisfaction questions and when is the right time to do those things. But at least moving away from net promoter and to asking the satisfaction questions and asking that on a regular cadence so that we can see over time like are we getting better and kind of what the trajectory is, I think that would be a huge thing to have accomplished as well
Steve: That’s a great note to end it on. So, thank you very much. It was a great conversation. Thank you. Thanks, everybody.
Okay that was the end of the interview on stage, and now here’s the remainder of our conversation. So, welcome back Marianne.
Marianne: Thank you. It’s the bonus track, huh?
Steve: Right, this is the director’s cut version of the podcast. So, one of the questions I wanted to follow-up with you on is you talked about the work you’re doing on the Type 2 and the Type 1 and really trying to drive change in the organization. I’m wondering, sort of tactically or practically, what are ways to bring those experiences that patients have into the organization itself?
Marianne: I think actually one of the most fundamental ways that it started is using our app itself and so – I don’t have diabetes myself, but for the first 2 weeks that I was – or first couple of weeks I was at Glooko, I sat down with our certified diabetes educator and we pretended that I had just been diagnosed. And so, she talked me through what somebody would hear. How they would be introduced to pricking their finger and measuring their blood glucose. And so, I spent 2 weeks and I logged all the food that I ate, all the exercise that I did. I measured my blood glucose twice a day just to get that experience of what it was like and like yeah, it was pretty crappy, even after a couple of days, to prick your finger. And to imagine that’s for the rest of your life. Like, I had the luxury of stopping. And so, for me it was a really small way to understand what it was like for somebody who was newly diagnosed with diabetes to do that. And I also presented that back to the team and I sort of, encouraged other people to try. So, we instituted a program. So, anybody who starts at Glooko now can get a blood glucose meter and to try it out themselves and to see what that experience is like. So, it’s one small way to at least start building that empathy.
Steve: Are there others that you want to describe?
Marianne: Yeah, actually, so I wrote a Medium article about different ways to build empathy when working in health tech. So, one of them was experience it yourself. So, actually to try the app and try measuring your blood glucose all the time. See for yourself. So, when we did the ethnographic visits, to bring people out so they could actually talk to and see people. And listen to others. So, some people that I work with actually do have diabetes and so I wanted to hear their stories. So, we started something called Life with Diabetes Storytelling. So, just a couple of folks each time and they’ll either tell their own story of how they were diagnosed and what it was like. Or some people have children who have diabetes and that whole experience. Or some people have parents who have diabetes. So, it humanizes it even further because these are people that we actually work with. So, it’s one thing to visit somebody’s home and you never see that stranger again, but these people that we see all the time and we think nothing of somebody at the lunch counter, or lunch table, pricking their finger and taking their blood glucose, or pulling up their shirt and dosing insulin right at the table. Like that’s just kind of what happens. And to ask those questions and making it okay to ask like well when did you tell your wife? And what is it like to do sports and kind of do those things that you used to do? So, it’s been a really great way to create empathy and even bond as a team, to hear each other’s stories. We put them up all on the website so people who start new can watch the videos themselves and learn about people stories.
Steve: And what’s – the format is videos. Are there are materials that are – I guess describe what this looks like?
Marianne: So, I was a little ambitious at first. I was like we’re going to have three people who speak once a month and I realized the company’s not that big and a lot of people actually don’t want to talk. And even I was encouraging people to – even if you don’t have diabetes, you don’t know anyone with diabetes, like there’s lot of this stuff on the Internet. Like watch some videos. Read some people’s blogs. Like bring that in. Haven’t had any takers on that yet. So, now kind of what we’re doing is we just have two speakers and we do it about every other month. And I just give people the floor. I say you’ve got 15-20 minutes, tell your story. Some people take less time. Some people are a little bit more hands – they take a little bit more time. But all I’m doing is giving them the floor, just creating that space. And it’s been wildly popular. Like engineers will come. Like people – we actually fill the room where it is. So, it’s just an opportunity for people to tell their story. And I also wanted to make it a little bit special, so the JDRF has these little bears called Rufus the Bear that teach kids who are diagnosed like how to do their insulin and all that. So, I’m riffing off that. So, I have these little bears that I give people after they speak, and it says, dear so and so, thanks for sharing a diabetes story. So, it’s personalized and people sit them on their desk so that when you walk around you can see who shared a story. So, I wanted to make it a little special of like, yeah, thanks for sharing and we know whose stories are out there too.
Steve: That’s great. So maybe a different question, following up on some of the things we talked about earlier. You know thinking – we talked about how you learn research and kind of your process through these different environments that you were in. You know at this point, do you have a super power?
Marianne: I think my super power is complexity busting. So, I think that based on what I’m hearing – and I think qualitative research gets poo-pooed a little bit, but I found that being able to suss out those patterns from just a few interviews and turn that into a robust framework of – you know I did a thing when I worked at Google around ads and what are the characteristics in ads and that framework is still being used. So, some of the things that I’m doing now in terms of persona research – or developing personas and also the Bingo card framework of the needs of people with diabetes. So, there’s a couple of columns, there’s a couple of rows and people have different needs and sort of fill that down individually. And those types of things that lead to action and sort of like synthesizing, encapsulating like here’s the things you need to know. Like, I’ve thought about it and I’ve taken all this chaos and I’ve put it into something that’s usable and actionable that the teams can use. And I think that’s one of the things that really makes me happy when I’ve figured something out and then turned it into a product that the team is like aha, we can run with it. This is actually useful for making decisions.
Steve: Right. I think those two pieces are really interesting because I’m with you on the finding patterns rapidly and being able to tell a new story about them, but I think you’re productizing them and communicating some kind of communication design around them so that other people can understand that, I think that’s two superpowers in one, maybe the way you’re describing it. Yeah. Do you have a way of thinking about your own brand, your own identify as someone that – as a researcher, or someone that works in tech, works in these kinds of spaces? How do you think about yourself that way?
Marianne: I think for a long time I thought I’m sort of like everybody else and like we’re all sort of doing the same things. And I have an opportunity now – I’m part of this incubator for women leaders and it’s having me sort of rethink what is my brand? And that one needs to have a brand. I think I also maybe started thinking about it when I started having my own consulting practice and I’m like how do I stand out and what do I offer and how do I package these super powers, and kind of bring that to the fore? So, – and part of this incubator that I’m in, it just started somewhat recently – but the women who are in it, like it’s not just people who are in tech. I think a lot of us are in tech, but we have like doctor and we have some lawyers and we have some other people who are doing other things. And so, it’s great to be with that different community. And so, one of the things that we did in our first session was really think about ourselves as a company. It’s like we all work for companies. We’re all like what’s the vision statement? What’s the mission statement? Every company, you’ve done SWOT analysis. So, we’re turning that lens on ourselves. And so, part of our homework for this month is to figure that out. And I think one of the things that I was thinking about is my mission is really user centered everything. That user centered stuff can be applied to any domain and having worked in finance and wildland firefighting and consumer stuff and with creative professionals, I think I’ve seen it there. But that’s all been within tech. But really, applied to things outside of tech. So, it can be applied to parenting. It can be applied to government. It can be applied to just a lot of different things. Like in fact I joined a gym recently. It’s called 9Round Kickboxing. And I realize how user centered they are. So, I’ve got to riff on this for a little bit. So, some insight somebody had along the way was people don’t want to spend a lot of time at the gym. They want to go at any time they want to. They don’t want to wait for a particular class and they like the special attention from a trainer. So, what does 9Round offer? You can literally go at any time. There’s 9 rounds that you do in order, but you can start as soon as you get there. Each round is 3 minutes. So, you’re done in 30 minutes and it’s one of these high intensity workouts. And there’s a trainer there who can like help you and spar with you at a particular moment. I’m like this is a fabulous example of being user centered and really understanding what your customers need. And I was like I signed up for the year. I’m like done. This is exactly – I’m your target user. You’ve figured out my needs and this is fantastic. So, thinking a lot more about human centered everything and how can we apply that to domains and sort of just everyday life because when things are human centered it’s more humane.
Steve: Is there a source or a reference or an inspiration for that human label, kind of in what you’re looking at?
Marianne: I think it came from you know having worked in a lot of different domains. I think what really crystalized it for me was I read a book called More Human by Steve Hilton and he’s married to a Silicon Valley exec, but he was in the UK and then he moved here and he was like – went to – started hanging out at Stanford and came upon design thinking and he’s like aha, all these things I’ve been doing and thinking about policy and applying it to business and all these things. And so, he really crystalized it and I’m like, yeah, I believe that. Like what if our businesses were more focused that way and what if like the policy of how do we get rid of homelessness actually like prototyped for us, rather than just spending tons of money and doing a particular project for 5 years. So, I think he really captured that human centered everything for me.
Steve: So, as you think about that as your brand, how are you going to operationalize that brand – to use a terrible phrase. But how does this go beyond a concept for you? What do you think you’re going to be doing?
Marianne: So, I think it’s something that I’ve been doing in terms of mentoring, like working with entrepreneurs – so when I left Google, I was sort of exploring two paths. One was more mission oriented something or other and the entrepreneur side of everything. So, right now my full-time job is much more in that mission driven kind of aspect of it, but I’ve also been doing mentoring for entrepreneurs. I’m mentoring a woman in Venture for America right now. I’ve done kind of office hours for a bunch of things. I’ve taught at Nasdaq Entrepreneurial Center. And so, I think those are ways where I can find different ways to apply it and find ways that – how do you find intrapreneurs and entrepreneurs to give them the tools to be able to ask questions and go out and talk to users to apply it in a lot of different ways? That is a way to make that happen. And so, I think that for me, whether it’s my full-time job, or kind of like the balance of those things may change in the future. So, maybe I’ll be spending more time and I’ll find a role that lets me do more of that working with entrepreneurs and helping them develop that and maybe less in this oriented stuff, or maybe I’ll be able to find ways to combine the two. But I think for me, in sort of the next phase, it’s kind of how do I find the overlap, or sort of finding the balance of those two things?
Steve: You know sort of the thesis of this podcast is around – I don’t know thesis, but just the people we want to have as guests are people who are user research types and I’m wondering, as you describe this sort of human centered everything, and these mentorship roles, are you being a researcher, as a way you would define it in doing that work? We have labels and so I want to ask you to kind of unpack the labels and see what you’re sort of framing on this work you’re exploring vs. maybe the work that connects you and I and why we’re having this conversation?
Marianne: Yeah, that’s a really interesting question. I think for me, it’s – I think once I get sort of hot under the collar about something, like everything else goes out the window. You know I was talking earlier about once I found Glooko and I’m like wow, they’re working on a really big, juicy problem. This is something I wanted to do, who cares what the actual label is, or what the title is. So, I think for me, like it really bothers me when people – smart people – go out and build things and spend a lot of time and energy to build things that are not for humans. And I’m like, erh, why didn’t they do that? So, I think for me it’s more about empowering people who have that energy and who have that entrepreneurial spirit to make the things that are right. So, not just make stuff,
but actually, make the right thing. So, for me it’s a set of skills that I think maybe everybody should have and maybe once everybody has those skills and can do it well, maybe the role of researcher doesn’t need to exist. But until then, I feel like it’s my duty to go out and spread the gospel, as it were, of this is how you talk to users. This is how the type of information you can get. These are types of things that you can’t get and the limitations of the things that we can do as well.
Steve: So, it’s a really articulate lovely reminder of why we do research? It’s what we’re trying to have happen and you’re looking at achieving that goal through research, or through mentorship, or through advocacy and all the things that you’re doing are putting stuff out in the world that helps people – I don’t want to put – now I’m putting a lot of words in your mouth, but there’s kind of a producing of things ideal that you are a champion of. So, I see research – as you explain it, I can see research as a part of that, but not the only one.
Marianne: I think you summed that up really nicely. So, thank you for making me sound more articulate.
Steve: That’s the reflecting back technique. Okay, so now maybe wrapping up our epilogue, anything to add in this? Anything I should have asked you about?
Marianne: I was just thinking another user centered everything. I’m doing some volunteering and working with a foster child and I’ve been able to turn that into a user centered thing too. Because she doesn’t live with me, I just see her for a few hours each week, so I think very intently about the activities I’m going to do and being a crafty person also, I have spent a lot of time on YouTube videos and Pinterest and whatnot and making activities that I’m like well we’ve got to work on like fine motor skills. We’ve got to work on like counting or matching and stuff like that. And I’m having such a great time designing these little activities for her and also seeing how she responds and like, oh, she doesn’t like those types of things. Or, she needs more of these types of things. So, I feel like bringing it to – like I think I just naturally bring it to all aspects of my life and I see the power that it has, and I just feel I’m like – I just need to be a champion for those things because I see the value of it and I just want the world to know.
Steve: That’s a lovely note to wrap up on. So, thank you so much, Marianne.
Marianne: Thank you so much.
Steve: Well! That wraps up this episode of Dollars to Donuts. Go to portigal.com/podcast for the transcript as well as links for this episode. You can follow us on Twitter, and subscribe to the podcast at portigal.com, or iTunes, or Spotify, or Stitcher, or anyplace excellent podcasts are distributed. You can buy my books are available at Amazon and rosenfeldmedia.com. The amazing theme music was written and performed by Bruce Todd.
The post 16. Marianne Berkovich of Glooko first appeared on Portigal Consulting.
In this episode of Dollars to Donuts, I speak with Marianne Berkovich, Head of User Research & Consumer Insights at Glooko. We talk about doing research through leadership changes, setting up opportunities for self-critique, and how to build empathy, especially in health technology, by experiencing some aspect of the condition and treatment yourself. It really […]
The post 16. Marianne Berkovich of Glooko first appeared on Portigal Consulting.
In this episode of Dollars to Donuts, I speak with Marianne Berkovich, Head of User Research & Consumer Insights at Glooko. We talk about doing research through leadership changes, setting up opportunities for self-critique, and how to build empathy, especially in health technology, by experiencing some aspect of the condition and treatment yourself.
It really bothers me when smart people go out and build things and spend a lot of time and energy to build things that are not for humans. And I’m like, erh, why didn’t they do that? For me it’s more about empowering people who have that energy and who have that entrepreneurial spirit to make the things that are right. Not just make stuff, but actually, make the right thing. It’s a set of skills that I think maybe everybody should have and maybe once everybody has those skills and can do it well, maybe the role of researcher doesn’t need to exist. Until then, I feel like it’s my duty to go out and spread the gospel, as it were, of this is how you talk to users. – Marianne Berkovich
Show Links* World IA Day San Francisco * Marianne on LinkedIn * Marianne on Twitter * Glooko * Google * Adobe * Blue Ridge Labs * The Human-Computer Interaction Institute at Carnegie Mellon University * Lockheed Martin * Bonnie John * Honing your user research craft: Doing a critique through a “Don’t Leave Data on the Table” class (Medium) * UX Interviewing: Personalized Coaching to Avoid Leaving Data on the Table (from From CHI EA ’17) * Interviewing Workshop: Don’t Leave Data on the Table (from Mixed Methods podcast) * Net Promoter Score * Three Ways to Build Empathy when working in Health Tech (Medium) * Rufus the Bear at the JDRF * 9Round Fitness * More human: designing a world where people come first * Venture for America * Nasdaq Entrepreneurial Center
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Well, hi, and welcome to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization.
I was reading the New York Times today and I noticed something unusual, although I’m seeing this sort of thing more and more. After the byline, but before the article starts, is some italicized text in square brackets. It reads “What you need to know to start the day: Get New York Today in your inbox.” This is copy that only belongs online, in the app or the website or an email newsletter. Presumably you would click on something. But I’m looking at the newspaper – I feel like I’m starring in one of those YouTube videos where a toddler is trying to swipe a magazine and can’t figure out why it’s not a touch screen. I see the New York Times making this kind of error in their print edition every few weeks, and it’s kind of appalling, because it suggests a lack of detail that I don’t expect from a high quality product like the Times. When you get an email that has the wrong name in the salutation, even though we’ve all done it ourselves, it brings your appreciation down a notch or two. They clearly aren’t taking the care that they used to, and that we would hope for.
And it’s especially interesting because I remember when the opposite used to be true; when the experience we had online, with news especially, was a not-quite-there translation of a print experience. And now it’s flopped. The print edition readers are not the primary customers. The organization has identified a different key user and it’s not us.
I can’t help but wonder about any of the users we learn about, are they in a less desirable category or perceived that way because of changes in internal processes or organizational structure? Do we care about their experience, or are we making them into what we call “edge cases” which is a fancy way of dismissing them. It’s hard to imagine the print reader of the New York Times as an edge case, but hey, that’s where we are.
I want to remind you that I’m looking for ways to be able to keep making this podcast for you. Here’s how you can help. You can hire me! I plan and lead user research projects, I coach teams who are working to learn from their customers, and I run training workshops to teach people how to be better at research and analysis. I’ve got two books you can buy- the classic Interviewing Users and Doorbells, Danger, and Dead Batteries, a book of stories from other researchers about the kinds of things that happen when they go out into the field. You rating review this podcast on iTunes, and you can review both books on Amazon. With your support, I can keep doing this podcast for you.
Let’s get to my interview with Marianne Berkovich. We did this a little differently – as part of a live event. We closed out San Francisco’s local edition of “World Information Architecture Day” with our interview, live on stage, in front of an audience. Then when we got off stage, we sat down in a somewhat noisy room and talked a little more to cover some of the things we didn’t have time for. We’ve cleaned it up as best we can but the audio may be a little bit different from how things normally sound. It was really fun for the two of us to speak on stage, and I hope to have more opportunities to record episodes of the podcast in similar settings.
Marianne is the Head of User Research + Customer Insights at Glooko. She’s worked as a consultant and for Google and Adobe.
Thanks for agreeing to talk with me, Marianne. Why don’t we just start by having you introduce yourself. What do you do? Where do you work? Tell us about that.
Marianne Berkovich: My name is Marianne Berkovich and I am Head of User Research & Consumer Insights at Glooko and we’re an online diabetes management platform.
Steve: What is an online diabetes management platform?
Marianne: That’s a great question. So, diabetes is a condition that’s a lot to do with numbers. You want to keep your blood sugar not too low, not too high. So, it’s really conducive and lends itself well to technology and checking your numbers. We have an app for the person with diabetes. They can track all sorts of things there that could affect their blood sugar and then they can also send that information to their clinician who can further see patterns of you’re high in the mornings and what are we going to do about that? So, it’s a platform that both the clinician can access and look at those things as well as for the patient themselves.
Steve: Where in the history of this company and its product did you come in, to bring in user research?
Marianne: So, I am the first user researcher. The company was founded in 2010. I’ve been there for about a year and a half and I am learning all sorts of things of what it’s like to be, first in a healthcare company, or a health tech company, and also being the first researcher.
Steve: Okay, let’s see how easy this is. What are some of the things you’re learning about being the first researcher and working in a health tech company?
Marianne: First of all, as a researcher, it just feels very weird to be answering the questions and not asking the questions. Maybe I answer questions in a way that makes it easier for the researcher to ask the next follow-up question. So, but you know what a great researcher Steve is, so this is showcasing his talent too, right! So I think some of the things that I’m learning is the role of advocacy. That even though there was a need for hey it’s time for a researcher, we need someone full-time – before they were hiring vendors or kind of doing it ad hoc. And so, it was time to have somebody who could be dedicated and who’s trained in this. But at the same time there’s a lot of – I don’t want to say – in some cases there was resistance, but I think in a lot of cases it’s more of just not knowing what good research looks like, or part of the reason my title is so long is that people had an impression that oh, user research is just usability studies. And so, I talked with my manager and we put in the “& Consumer Insights.” Head of User Research & Consumer Insights – so it’s like it’s everything. It’s going to be going out and doing field visits. It might be surveys. It might be a lot of different things. So, to kind of help with that advocacy. And I feel like I do spend – I mean, I think as researchers we always spend a lot of our time in advocacy mode, but I’m not surprised – or maybe surprised – it’s just taking a lot more of my effort to do that advocacy work of what research is and how can it be used.
Steve: I just want to clarify. You’re advocating for user research?
Marianne: Yeah. So, I think even though the company talks about being patient centered and user centered, that what does that really mean? I think one of the things that I’m finding out – so, there’s a lot of people who have been in the field for a long time and they’re like we understand diabetes. We’ve been in this for many, many years. And to have somebody come in – I don’t have any experience with diabetes; I don’t have diabetes myself – to come in with these, you know, different insights or different ways of doing things and people are like, we’ve been there, we know how to do this. And so, to advocate for, hey, we went out and talked to some people. We learned this thing that’s different and knew and it’s maybe against the conventional wisdom, how might we use this? And maybe some of that resistance of like, well we’ve always done it this way. Or conventional wisdom says this other thing.
Steve: So, the other part that you were learning was working in healthcare tech. It’s a new industry for you. So, talk about maybe what that’s been like.
Marianne: Yeah, so – and before that I was – I did a fellowship and we were working in the legal space. And the legal space, the health tech space, are slower moving than just straight up consumer products. It’s like hey let’s build something and we’ll launch an app and do these things. You know regulated spaces of like what is the FDA like? I had a little bit of background in human factors testing, but we have a product that actually got FDA approved. But what does human factors testing look like? How is that different from just regular usability testing? What does it mean to understand the whole ecosystem of payers and employers and insurance companies and PDMs and all that type of thing of moving that needle and kind of inching things forward when you’re in this space that can’t be disrupted so easily because there’s good reasons why these constraints exist.
Steve: So, the company was about 8 years in existence when you joined. So, how much of that were you building from scratch? Describe a little bit about what you encountered and what you had to build around sort of the tech and regulatory aspects maybe.
Marianne: So, actually, luckily the regulatory aspect is just one part of what we offer. It’s the Mobile Insulin Dosing System. So, when you start on insulin, how do you come up to taking the right dose of a certain type of insulin. So, the product does a lot more than that. So, this was just one small part of it. And it was actually – that was already in the works when I joined. So, the app was already out. The website was already out. MIDS was already sort of in progress. So, I think for me it’s been a role of a little bit of back to basics of like let’s learn about our users. How are people with Type 1 diabetes different from Type 2 diabetes. So, in fact, I was telling Steve before this, I’m doing an ethnography right now and so I was in Fresno this week talking to folks with Type 1 diabetes. I’m in Phoenix on Monday, but kind of building that basic understanding and actually building personas and really having a deeper understanding rather than like, yeah well, we have people now who are Type 1s, they know that they need. To move beyond that, to really have some foundational knowledge, and it’s causing us to go back and be like okay, well, is this – is there things in the app that are working the best way. Maybe we can make the clinician experience more efficient. Maybe there’s things that we learn when we go to visit a clinician – be like oh, that’s not how we thought that worked, can we rethink that. So, I think it’s a little bit of back to basics and rethinking certain things rather than to build anything from scratch.
Steve: Before you talked about advocacy which was kind of saying hey, we need to learn these kinds of things, but in some ways doing that reveals the most scary thing of all – oh we need to rethink assumptions and yeah, a 10-year-old company, it’s entrenched. So, what happens when the things that you’re uncovering are inviting, challenging – they’re inviting the challenging of established belief structures around the product and what it looks like and everything. How are you doing that?
Marianne: I think it leads to another kind of wrinkle of working in a startup is that we’ve had a lot of turnover at the leadership level, actually. So, the CEO who was the CEO when I joined was not the founder. He was the second CEO. And he reached a point where he was like, “you know what I really like getting the company to where it is now, but now that we’re really – it’s time to scale and grow things, it’s not what I want to do.” So, we got a new CEO. We got a new head of product. We got a new commercialization officer. So, all that advocacy I had done and sort of like when we did they Type 2 ethnography when I first started, like all that – you know taking people on the road and showing them and doing these empathy workshops and all that – whoosh, out the window because we’ve got a whole new cast of characters now. So, I think that was part of it too. It was a little bit humbling to have to re-sort of create my credibility again with – it’s a whole new cast of characters and a whole new set of people to influence. Luckily my manager has been a great champion of that. And so, I think it’s finding ways that we can leverage and sort of maybe it’s not the right time for things. So, we’ll go out and do this research and we have this foundational stuff. We sort of did a big aha, then those people went away. Can we save that and find another time to sort of bring that to the fore when it might be a little bit more accepted? So, I think it’s both sort of pushing for it and finding the right time to sort of introduce those things.
Steve: So, what does a manager do? How does that championing work?
Marianne: I think, – and I don’t know if this is true, but I feel like as a woman you know I can say a little of things if she’s also a woman, but like women in general, like it’s much more effective if instead of me saying it, I say like, “yeah, what Steve said.” Or, you know, what somebody else said. So, I think because of that, I think it becomes more effective. So, she’s not tooting her own horn. I’m not tooting my own horn, but she is sort of pointing to the work that I’m doing. Plus I think we have different kind of communication styles, and sometimes she can sort of, – I’ll say a bunch of stuff and she can sort of synthesize it in a really nice way and it sort of becomes a little bit more impactful that way, that like I can spend all my time sort of yelling and screaming – metaphorically speaking – and then she can sort of bring it home in a very succinct way.
Steve: So, part of what she’s doing then is also – you talk about finding the right moments. Or, what should we be doing to have impact on the organization as the layers above us are changing. I don’t know. I’m wondering about when new people come in, and this is going to vary on a case by case basis, but you’ve gone through this whole series of trying to open people’s minds up a little bit and say hey things are different. Those people leave. New people come in, but is there an opportunity there? Do those people – what baggage do they have, or do they bring in? I’m not asking you to like slag anybody individually, but as you see these kinds of changes happen in an organization and you’re trying to craft a story that’s about how the world really is vs. maybe what we hope or assume – how does the – what changes when new people come in, with or without baggage, around what the truth is?
Marianne: I think part of it is understanding where people are coming from. Because I think in my last role – I was at Google for a long time and I think like certain assumptions I made that like everybody knew what user research was and everybody knew this and I kind of got used to that, that that was the norm. Like product managers of course know what I do, and everybody knows how this works. And so, I had to figure out like, oh, okay, I can’t assume that people know what these things are, or that when I say user research and what you think in your experience with user research, like maybe that was focus groups. And they’re like yeah, we did those and it was great and I was like okay, great that’s a start. So, I kind of know where I’m starting from with folks. So, I think that’s part of it. And I think also, because I am the only researcher and things move a little bit more slowly, like we did the stuff around Type 2s and now we’re doing the Type 1s. And so now I’m using all that stuff that I learned last time of like what was effective in terms of running a workshop and getting people to come on visits with me. And it’s like it’s a little bit smoother again this time, but we’re doing that again this time around. So, it seems that having that opportunity to redo that foundational research has resurfaced itself. And so, to take advantage of that opportunity, that it’s like well, let’s look at Type 1s now.
Steve: There’s something here about you’ve had lots of experience. This is not your first job. You’ve been at lots of different organizations, you’ve done lots of research and influenced stakeholders and product teams and so on, but it sounds like that there’s a good measure of learning in this job which is about how do I do the thing that I know how to do to be effective in this context. Does that ever go away for researchers, do you think?
Marianne: I hope not. I think it’s – and I think that that’s part of the reason that like, you know – one of the reasons I left Google is like it took me a while to realize the thing that I was doing there and the things that – that, kind of really I’m passionate about are not things that I could do at Google. But like who leaves Google, right. So, I think finding ways to find my path and things that are not sort of the traditional way of moving up in, you know, you’re a junior research, then you’re a senior researcher and then you’re a manager and all these things. To find my own path and sort of be okay with that and find different ways to learn in different contexts. Like the fellowship that I did with Blue Ridge Labs which is a social impact incubator. And I was again, the only researcher, and so I got to do more mentoring. And so that was interesting. So, I think finding ways to both follow my passion and find ways that like what does this organization look like? And I think we’ve probably heard this all before in terms of doing sort of user research or bringing that lens to our stakeholders or our people that we’re working with as well. I think that part never goes away and it’s always changing because it’s always a different set of people.
Steve: Right, more so – is the landscape changing more in a place like Glooko than in a place like Google?
Marianne: That’s a good question. I think it’s very different. Google’s obviously a very – there’s just many people. And so, I think the dynamics of what it’s like to have an organization that big and when I joined I knew all the researchers and now the scale – each product’s team is much larger than that. So, I think it’s a different type of – it’s a different set of issues. So, I think wherever you go – like I’m from the East Coast and people ask – they’re like well what’s better, east coast or west coast, or all this stuff? And I’m like they’re different, right. I don’t think there are things that you can really compare. So, for it’s, – and this is my first time working in a startup and so I don’t have another startup to compare it to. So, maybe if I went to a different startup I could do more of that comparing and contrasting, but in some ways, it feels like apples and oranges of a large organization that’s established and, you know, is well funded and all that vs. a smaller one that is working in a very different space.
Steve: Just hypothetical – I know we’re not supposed to ask – apparently you aren’t supposed to ask hypothetical projection questions in user research, so good thing this is not user research. If you were to look for a job at another startup, I’m just thinking about you joining an 8-year-old company, is there a maturity that you would look for, or a timespan that you would think differently about in the next stages of your career?
Marianne: I think it’s less for me about that. It’s more about really getting jazzed about the problem that I’m solving. So, for me diabetes is a very big issue. Thirty million people in the U.S. and growing. And so that feels like a real meaty issue and that gets me excited. So, I’m like even if some days it feels like I’m pushing a boulder uphill, it seems like a worthwhile thing and that’s what gets me up in the morning. So, for me it’s much more about that and going in with eyes wide open of like well what kind of organization in it – is it, and like okay do I want to take that on. And if the answer is yes then working around those constraints because that’s just the nature of the beast.
Steve: You talked before about figuring out the right job title that would describe, to the rest of the organization, the way that you were going to work. As you – can you say more about sort of what that process was, what those conversations were that identified an opportunity, that helped get you excited that this was something you wanted to do?
Marianne: Well honestly, I think part of it was naivety, that like this was – I think it was actually the way it was posted or sort of written up was Manager of Consumer Insights and I’m like okay, whatever. You know, I’m not on a ladder. I’m the only one, so it doesn’t really matter. So, for me, I was so excited – in fact I was consulting before and so Glooko was one of my clients and I got hired. So, we crafted the role a little bit based on what I was doing. And so for me it was less about negotiating the right title and in fact I think only after I joined – because I was like, sounds great, like it sounds like this is a good fit, sign me up, and then only when I started and I was like hey, I want to make business cards, but I don’t really like this Manager of Consumer Insights – we had that conversation after I joined. And so that’s when my manager started telling me about like hey, this is some of the perceptions because I had gone in very naively also of like everybody knows what user research is. Like, we’re in Silicon Valley, like everybody knows what this is. And so, it didn’t even really occur to me that that advocacy, or the extent of that advocacy that would need to be done.
Steve: So, what – you worked with the organization as a consultant and then came in-house to kind of lead the effort. What was similar and different about – you know, the before and after that transition?
Marianne: So I think one thing is, it was very intimidating. I didn’t know anything about diabetes and so I ran my first study and like everybody showed up and so they were listening in on the call and I’m like how’s this going to go? Like I hope I don’t say anything stupid. And so, you know I think as a consultant, coming up to speed on a different domain very quickly, and I think to me it’s about asking those open-ended questions and asking questions in a way that it doesn’t really matter if you know the domain. I mean it’s certainly much better if you do, but there are ways that you can sort of cover that up, and especially if you’re going in with an apprentice mindset and all those types of things, it kind of helps the situation along a little bit. So, I think for me that was definitely intimidating to not know the domain and have everybody show up.
I think another difference is as a consultant, and I wasn’t a consultant for very long. It was about a year or so before I decided to take the role with Glooko , but I was very cognizant of how I was spending every single hour and whether it was going to lead to making money or not, because is this going to generate a lead? Is it going to generate a sale? All those types of questions were very much top of mind whereas when you’re in-house I think you can spend the sort of hours – like you’re spending your capital in a different way. You’re building relationships and that’s kind of how you’re sort of making money. So, to me it’s sort of a different way that you don’t have to be so aware of every hour leading to money.
Steve: I’m going to switch gears a little bit and maybe we could just rewind. I’d love to hear you describe maybe your path. How did you get into this stuff? What brought you to where we are sitting today?
Marianne: So, I was an English major for undergrad and after I graduated I didn’t know what I wanted to do. So, I spent a couple of years – I actually at that point did work. I was in D.C. and I worked at the National Museum of American Art. We were digitizing the collections. It was a very long time ago and so I was actually cleaning up all the photos and I’m like eh, I’m not really into this technology thing. My Dad also was a computer science professor. I’m like definitely not into that technology thing. Like that’s stuff that my Dad does. Boring. So, I spent a couple of years kind of bumming around a little bit, decided to move out to Denver and I started working as a technical writer. And I was working at a financial services company and so my role was to write the help text for this complex financial software. So, I would sit with the developers. They would explain to me how the complex software worked, and I would write it up. And then I had a thought. I’m like if we just made the software easier to use I wouldn’t have to write anything. So, that led me to first get a certificate in technical communication. And in that I started kind of looking around a little bit more and found the whole field of human centered design and human computer interaction. I looked around and I found a program at Carnegie Mellon. And at that point I had actually switched over to Lockheed Martin, which is a great role – you can ask me about that in a second. Sorry, I just keep getting the questions to ask. And, but I – you know I decided that I really needed a degree in this thing. That just kind of reading books about it or whatever wasn’t enough. So, I decided to go to grad school and I remember just seeing that description of the master’s program and I was like that’s exactly what I want to do and like all my life had been leading to that point to be like that’s exactly what I want to do and having that feeling of like, yup, this is the direction I want to head in.
Steve: Which master’s program? And then keep going. Yes, please.
Marianne: It was a master’s in Human-Computer Interaction at Carnegie Mellon University. So, the role at Lockheed Martin. So, I’m old. I’m definitely over 35. And this was sort of in the olden days and what we were doing is taking – so, you know how we have wildfires and all that type of stuff? So, the resources to manage all those things, like sending the air tankers and the trucks and the crews and all that – that was being done by hand. So, Lockheed Martin got a contact to turn that into – it wasn’t even online. It was just kind of a digital program to do that. So, we would interview subject matter experts and I was like a requirements analyst. Like we didn’t have roles of like designer researcher. Like that wasn’t a thing yet. And so, for me it was like super exciting to solve a real problem. We got to actually go to Boulder and like see the command center and see some of those things. I mean I really felt like I was making a difference and I knew what this was about. But I felt very much at a disadvantage of like, I don’t know what should the icons be? I don’t know. Nobody has best practices around icons yet. And so, it felt like early days and that’s why I really wanted to go back to school and like learn some stuff because I knew there was a lot that had been done already.
Steve: So, you come out of that program and where do you go?
Marianne: Consulting. Yeah, I actually stuck around Pittsburgh for a while and I again ran my own consulting thing. I do that to sort of – when I’m in exploration phase. Or, when I’m dating somebody who just started a PhD program and I can’t leave Pittsburgh. One or the other. So, yeah, and the guy I was dating, he was very entrepreneurial, and he was like such a great cheerleader. He was like yeah, we can figure this out. He had been running his own startup before he started the PhD program and he was like, yeah you can do this. I’m like, yeah, I can do this. So, he did help me a lot and so I was finding projects and was kind of getting my feet wet with that. But then I found that it was time to leave Pittsburgh. That it wasn’t what I wanted to be doing. I really wanted to learn from other people and so I started looking for other opportunities.
Steve: What were those opportunities that you found?
Marianne: You can take it in a different direction. So, I actually interviewed at Google at that point and they didn’t want me, I didn’t want them. It was a very different company at that point. But I also interviewed at Adobe. There’s actually a pipeline from Carnegie Mellon straight here to Silicon Valley. I mean seriously, just pull up a bus, just put the graduates on it and bus us all out here. So, there were a bunch of people who I had gone to grad school with who were at Adobe and I went to interview there, and it felt like a really nice fit. I remember actually – after the Google interview I came home to my hotel and I just turned on the Simpsons and like all I could do was just sit there and like not move. But after the Abode interview, I actually – it was one of my first times out here in Silicon Valley and like I went hiking. I was like I was that inspired and that energized that I went hiking. I’m like this is a sign that maybe I should take this role, that this is a good role for me.
Steve: We can cue up some Simpsons for you after this conversation. You know, just put you in a dark room. Um- so, when you talk about the work that you’re doing now, you’re doing field work, you’re doing ethnography, you’re taking people out. When did you learn how to do that?
Marianne: That’s a good question. I feel like I should credit CMU with setting me on the path and teaching me about contextual inquiry – thank you Bonnie John. And I think for me one of the things that they told us at Carnegie Mellon is – it’s a one-year program, so if you didn’t come in as a computer scientist you’re not leaving as a computer scientist. If you didn’t come in as a designer, you’re not leaving a designer. But we’re going to give you enough exposure to coding and designing and having your stuff on a wall and getting critiqued and to do some social science stuff. And so, for me, having been an English major and I minored in art history and theology, I mean I was as liberal arts as you could get. I wasn’t going to come out as any of those things and I didn’t gravitate towards it. I was like I’m not a designer. So, I think I just gravitated towards research. And I think maybe the English degree kind of prepped me for that in terms of asking good questions and thinking about things and looking for patterns. So, I think just naturally I’m a pattern seeker and I think honing my ability to sort of ask questions just came with time. So, I think it was having a good solid foundation and then just kind of having a natural affinity for it. And I read your book – much, much later, but I did read your book.
Steve: Right. By the time my book came out you’d been doing this for a while. Can you talk about the environment at Adobe? What was there there to develop the practice of user research? In the practitioners like yourself that are coming in with X amount of experience, what did they do well that let you get to the next level of your craft?
Marianne: I think actually I was really lucky at both Adobe and Google, being surrounded by really smart people. So, that made it okay of like I don’t know how to do this, and can someone help me? Like I learned how to do surveys through some of the really awesome people at Google. So, I think for me it was just being surrounded by really supportive people and I think that was one of the things that was part of the culture at Abode. Like we would go to my manager’s house. She would have like all the researchers at her house and we would do these offsite – and it really felt like family. I mean it sounds a little cheesy, but – and so it felt like it was a place to grow and learn. So, I think for me that was a great stepping stone as my first sort of real job after graduate school, to be like oh, okay, this is what being a researcher is like. Oh, this is how you interact with product teams. So, I think it was just learning all those basics. But I think one of the things that – kind of coming back to the question of learning to do ethnographic work and all that is once you’re out – and especially you’re probably one researcher. You’re rarely lucky enough to work with another researcher who can sort of observe you can say like hey, why are you asking that question? Or, maybe we could do it differently. One of the things that I found, when I did have some opportunities at Google to work with other researchers, I’m like we don’t all have the same skills. Like what happens between you and a participant is different. And so, one of the things that having noticed that is I started a class at Google to critique ourselves. So, give researchers an opportunity to observe each other moderating and to give feedback to each other. And then I read a Medium article about it. It’s called Don’t Leave Data on the Table, if you want to look it up. I think that’s one of the things too is that we don’t get that critical eye anymore and we assume that like every researcher is the same and it’s like well actually, you know – and those things can also creep up on us. Like I see myself all the time, especially now that I’m sort of teaching other designers at my company more of like, ah, leading question. Yes or no question. You know. So, I find myself still doing that, even if I’m aware of it. But I think having those things pointed out to us is also really helpful too.
Steve: What’s the structure of the critiquing class?
Marianne: So, it was slightly different when I taught it externally at CHI, and internally. So, internally all our videos are available to each other. So, basically people would send in clips from research that they did. We would make small groups, so it lets you work in a group of five, and we would watch a segment of somebody conducting a study. And at any point people could stop the video and be like hey, I noticed this. It was like when we were actually practicing to teach it, one of my co-teachers, she actually had her pen and she would do a lot of like pointing and sort of like do a lot of things that indicated high status. And she was like, I never noticed that – until she saw herself, until we talked about it, she wasn’t aware of that body language and what it was conveying. And so basically we’d spend 5/10 minutes, or however the time divides up, to watch that video and stop if we see anything. And we asked people to rent that idea that like, again, we don’t know the context of what came before. We don’t know the context after. But this is what I’m noticing. And that’s what we do as researchers is we notice. So, it could be a pattern, it could not, but at least invite people to consider something that they’re doing. And also, by seeing other people, just having that conversation, seeing what other people – and be like okay, I didn’t show that, I didn’t do that in my clip, but like I’ve definitely done that before too. So, learning from each other as well has been really valuable.
Steve: It sounds like a way to create kind of a safe space where it’s okay to critique is that everybody is kind of up for that critique and we’re all being exposed together. Is that kind of the way to make it safe? Because this is our workplace and now we’re talking about how we’re not living up to sort of the high standards of our profession.
Marianne: Yeah. And I think that’s one of the things that we tried really hard to make it a safe space. And one of the things that we did was actually we had – spoiler alert – we’d give out like balloons, which are also like these cords that you pull to sort of stop the production line. But just having a room full of balloons made it a little bit more fun. When I taught at CHI we had to create the videos and so I had these little stuffed animal guys that people could put their phones on when they were doing the videos. So, then we had little stuffed animals on all the tables. And so, – and also modeling it ourselves that – I don’t remember if we actually showed some of the videos, but we definitely told some of the stories of like when we did this ourselves – it’s like I’ve been a practitioner for how many years? Like I’m still learning. I still screw this stuff up. So, making ourselves vulnerable was part of making that safe space too. And also, I think what we tried to do was, when we did it within Google, was not have everybody who was on the same team be together. So, it wasn’t like anybody who you worked with directly.
Steve: You used the phrase “sort of screw this up” and I think like the pen example and I’m sure there’s others that are clearly things we shouldn’t do as researchers. To me it seems like there’s a certain amount of stuff that’s subjective that you and I would do it differently because we’re different and we have different personalities and different energies. For me, critique about that would be really interesting because I like to see how other people handle things. My question here I guess is, is there – does the critique approach look at alternative approaches? How do you talk about that vs. best and worst practices?
Marianne: I think we also gave an opportunity for people to stop the cord on themselves. To be like I didn’t know what to do here. Like help me brainstorm. So, there’s that opportunity too of like for the person themselves to be like how will we handle this? And to hear different perspectives on – there’s, you know, I could have done this, or I could have done this, or in situations in the past I’ve handled it differently. So, I think having just that variety of there’s not one right way to do that, that it depends on the context. And maybe sometimes somebody explains like well this is what happened earlier in the session and so it led me to do it in this way and we’re like, oh, okay, well that makes a lot of sense. And so, I think having a little bit more of that context helps too.
Steve: Are you able to bring any of this into the environment you’re in now where, if anything, you’re teaching people and leading them.
Marianne: Yeah. And I think I’m learning a lot more about screwing it up also. While I’m out in the field doing these Type 1 ethnography, like I’m trying to help designers to be able to do their own research. And so, one of our more junior researchers is actually doing one on one interviews. So, I helped, and we talked about it and I wrote the discussion guide and the study plan. We talked about it and she was all on board and off to the races we went. And then I actually had a cancellation, so I was able to watch one of the sessions and I was like oh no, I didn’t – I screwed up. I didn’t prepare her well enough. It was a little bit more than she could handle and that was squarely on me, you know. That because I’m so used to doing it and it comes so naturally to me that I didn’t recognize that she was in a different space and she would need a different level of support and that I just hadn’t prepared her. And so, we role played a little bit and we practiced a little bit. But I still feel like – I feel guilty. I’m like she’s got more sessions this week and like I don’t know if she’s really prepared. So, I think that part is on me as well to make sure that the things that I’m modeling and the things that I’m sort of teaching are doable for where the designers are and not just hey it’s easy, anybody can do this.
Steve: In my opinion there are some things in life that we do that we maybe learn more from screwing up than we do by succeeding. I don’t know what falls into that category and what doesn’t. I wonder if, as much as you feel for this person and feel responsible for them, I wonder just ultimate perspective, did you set them up for a lot of learning they might not have otherwise got to by – of seeing the edges of what they’re able to control or what they’re able to execute on?
Marianne: I can’t speak for her.
Steve: Good. Alright. Well we hit a dead end. Let’s see what other questions I have written for you here. If we were to have this conversation again in a year or two years, what kinds of things would you want to talk about that you would have achieved in this organization?
Marianne: I hope that there’s less sort of friction. I think we’re already starting to see the seeds of it, that you know product managers who have been at the company for a long time are saying like oh, I’m doing this new thing, you know I need user research. They don’t know what it is. They don’t know why exactly, but they’re sort of like it’s a good thing generally. So, I think to have more of that. Honestly, I think like some of the things that are more – a little boring, kind of more infrastructure stuff – like I hate recruiting. I’m a terrible recruiter. I’m terrible with like time zones and all those details, that type of stuff. So, like I’d love to have a recruiter. I’d love to have another researcher on board. And some of the things that I feel like I have a great success under my belt, that it looks like we’re moving away from NPS (Net Promoter Score), hooray, thank you, thank you, and moving towards satisfaction. You know asking satisfaction question and how often we’re going to do that. You know, that’s a really interesting thing that I’ve been thinking about is can we pop that up in an app. I mean this is an app that people use for health and like they might be having a low blood sugar. You know this is a stressful time that people are using the app. Like nobody is excited about managing their diabetes. So, to pop up something in the app, to be like how are we doing, that’s not the right thing. So, I haven’t figured out like how we might introduce those satisfaction questions and when is the right time to do those things. But at least moving away from net promoter and to asking the satisfaction questions and asking that on a regular cadence so that we can see over time like are we getting better and kind of what the trajectory is, I think that would be a huge thing to have accomplished as well
Steve: That’s a great note to end it on. So, thank you very much. It was a great conversation. Thank you. Thanks, everybody.
Okay that was the end of the interview on stage, and now here’s the remainder of our conversation. So, welcome back Marianne.
Marianne: Thank you. It’s the bonus track, huh?
Steve: Right, this is the director’s cut version of the podcast. So, one of the questions I wanted to follow-up with you on is you talked about the work you’re doing on the Type 2 and the Type 1 and really trying to drive change in the organization. I’m wondering, sort of tactically or practically, what are ways to bring those experiences that patients have into the organization itself?
Marianne: I think actually one of the most fundamental ways that it started is using our app itself and so – I don’t have diabetes myself, but for the first 2 weeks that I was – or first couple of weeks I was at Glooko, I sat down with our certified diabetes educator and we pretended that I had just been diagnosed. And so, she talked me through what somebody would hear. How they would be introduced to pricking their finger and measuring their blood glucose. And so, I spent 2 weeks and I logged all the food that I ate, all the exercise that I did. I measured my blood glucose twice a day just to get that experience of what it was like and like yeah, it was pretty crappy, even after a couple of days, to prick your finger. And to imagine that’s for the rest of your life. Like, I had the luxury of stopping. And so, for me it was a really small way to understand what it was like for somebody who was newly diagnosed with diabetes to do that. And I also presented that back to the team and I sort of, encouraged other people to try. So, we instituted a program. So, anybody who starts at Glooko now can get a blood glucose meter and to try it out themselves and to see what that experience is like. So, it’s one small way to at least start building that empathy.
Steve: Are there others that you want to describe?
Marianne: Yeah, actually, so I wrote a Medium article about different ways to build empathy when working in health tech. So, one of them was experience it yourself. So, actually to try the app and try measuring your blood glucose all the time. See for yourself. So, when we did the ethnographic visits, to bring people out so they could actually talk to and see people. And listen to others. So, some people that I work with actually do have diabetes and so I wanted to hear their stories. So, we started something called Life with Diabetes Storytelling. So, just a couple of folks each time and they’ll either tell their own story of how they were diagnosed and what it was like. Or some people have children who have diabetes and that whole experience. Or some people have parents who have diabetes. So, it humanizes it even further because these are people that we actually work with. So, it’s one thing to visit somebody’s home and you never see that stranger again, but these people that we see all the time and we think nothing of somebody at the lunch counter, or lunch table, pricking their finger and taking their blood glucose, or pulling up their shirt and dosing insulin right at the table. Like that’s just kind of what happens. And to ask those questions and making it okay to ask like well when did you tell your wife? And what is it like to do sports and kind of do those things that you used to do? So, it’s been a really great way to create empathy and even bond as a team, to hear each other’s stories. We put them up all on the website so people who start new can watch the videos themselves and learn about people stories.
Steve: And what’s – the format is videos. Are there are materials that are – I guess describe what this looks like?
Marianne: So, I was a little ambitious at first. I was like we’re going to have three people who speak once a month and I realized the company’s not that big and a lot of people actually don’t want to talk. And even I was encouraging people to – even if you don’t have diabetes, you don’t know anyone with diabetes, like there’s lot of this stuff on the Internet. Like watch some videos. Read some people’s blogs. Like bring that in. Haven’t had any takers on that yet. So, now kind of what we’re doing is we just have two speakers and we do it about every other month. And I just give people the floor. I say you’ve got 15-20 minutes, tell your story. Some people take less time. Some people are a little bit more hands – they take a little bit more time. But all I’m doing is giving them the floor, just creating that space. And it’s been wildly popular. Like engineers will come. Like people – we actually fill the room where it is. So, it’s just an opportunity for people to tell their story. And I also wanted to make it a little bit special, so the JDRF has these little bears called Rufus the Bear that teach kids who are diagnosed like how to do their insulin and all that. So, I’m riffing off that. So, I have these little bears that I give people after they speak, and it says, dear so and so, thanks for sharing a diabetes story. So, it’s personalized and people sit them on their desk so that when you walk around you can see who shared a story. So, I wanted to make it a little special of like, yeah, thanks for sharing and we know whose stories are out there too.
Steve: That’s great. So maybe a different question, following up on some of the things we talked about earlier. You know thinking – we talked about how you learn research and kind of your process through these different environments that you were in. You know at this point, do you have a super power?
Marianne: I think my super power is complexity busting. So, I think that based on what I’m hearing – and I think qualitative research gets poo-pooed a little bit, but I found that being able to suss out those patterns from just a few interviews and turn that into a robust framework of – you know I did a thing when I worked at Google around ads and what are the characteristics in ads and that framework is still being used. So, some of the things that I’m doing now in terms of persona research – or developing personas and also the Bingo card framework of the needs of people with diabetes. So, there’s a couple of columns, there’s a couple of rows and people have different needs and sort of fill that down individually. And those types of things that lead to action and sort of like synthesizing, encapsulating like here’s the things you need to know. Like, I’ve thought about it and I’ve taken all this chaos and I’ve put it into something that’s usable and actionable that the teams can use. And I think that’s one of the things that really makes me happy when I’ve figured something out and then turned it into a product that the team is like aha, we can run with it. This is actually useful for making decisions.
Steve: Right. I think those two pieces are really interesting because I’m with you on the finding patterns rapidly and being able to tell a new story about them, but I think you’re productizing them and communicating some kind of communication design around them so that other people can understand that, I think that’s two superpowers in one, maybe the way you’re describing it. Yeah. Do you have a way of thinking about your own brand, your own identify as someone that – as a researcher, or someone that works in tech, works in these kinds of spaces? How do you think about yourself that way?
Marianne: I think for a long time I thought I’m sort of like everybody else and like we’re all sort of doing the same things. And I have an opportunity now – I’m part of this incubator for women leaders and it’s having me sort of rethink what is my brand? And that one needs to have a brand. I think I also maybe started thinking about it when I started having my own consulting practice and I’m like how do I stand out and what do I offer and how do I package these super powers, and kind of bring that to the fore? So, – and part of this incubator that I’m in, it just started somewhat recently – but the women who are in it, like it’s not just people who are in tech. I think a lot of us are in tech, but we have like doctor and we have some lawyers and we have some other people who are doing other things. And so, it’s great to be with that different community. And so, one of the things that we did in our first session was really think about ourselves as a company. It’s like we all work for companies. We’re all like what’s the vision statement? What’s the mission statement? Every company, you’ve done SWOT analysis. So, we’re turning that lens on ourselves. And so, part of our homework for this month is to figure that out. And I think one of the things that I was thinking about is my mission is really user centered everything. That user centered stuff can be applied to any domain and having worked in finance and wildland firefighting and consumer stuff and with creative professionals, I think I’ve seen it there. But that’s all been within tech. But really, applied to things outside of tech. So, it can be applied to parenting. It can be applied to government. It can be applied to just a lot of different things. Like in fact I joined a gym recently. It’s called 9Round Kickboxing. And I realize how user centered they are. So, I’ve got to riff on this for a little bit. So, some insight somebody had along the way was people don’t want to spend a lot of time at the gym. They want to go at any time they want to. They don’t want to wait for a particular class and they like the special attention from a trainer. So, what does 9Round offer? You can literally go at any time. There’s 9 rounds that you do in order, but you can start as soon as you get there. Each round is 3 minutes. So, you’re done in 30 minutes and it’s one of these high intensity workouts. And there’s a trainer there who can like help you and spar with you at a particular moment. I’m like this is a fabulous example of being user centered and really understanding what your customers need. And I was like I signed up for the year. I’m like done. This is exactly – I’m your target user. You’ve figured out my needs and this is fantastic. So, thinking a lot more about human centered everything and how can we apply that to domains and sort of just everyday life because when things are human centered it’s more humane.
Steve: Is there a source or a reference or an inspiration for that human label, kind of in what you’re looking at?
Marianne: I think it came from you know having worked in a lot of different domains. I think what really crystalized it for me was I read a book called More Human by Steve Hilton and he’s married to a Silicon Valley exec, but he was in the UK and then he moved here and he was like – went to – started hanging out at Stanford and came upon design thinking and he’s like aha, all these things I’ve been doing and thinking about policy and applying it to business and all these things. And so, he really crystalized it and I’m like, yeah, I believe that. Like what if our businesses were more focused that way and what if like the policy of how do we get rid of homelessness actually like prototyped for us, rather than just spending tons of money and doing a particular project for 5 years. So, I think he really captured that human centered everything for me.
Steve: So, as you think about that as your brand, how are you going to operationalize that brand – to use a terrible phrase. But how does this go beyond a concept for you? What do you think you’re going to be doing?
Marianne: So, I think it’s something that I’ve been doing in terms of mentoring, like working with entrepreneurs – so when I left Google, I was sort of exploring two paths. One was more mission oriented something or other and the entrepreneur side of everything. So, right now my full-time job is much more in that mission driven kind of aspect of it, but I’ve also been doing mentoring for entrepreneurs. I’m mentoring a woman in Venture for America right now. I’ve done kind of office hours for a bunch of things. I’ve taught at Nasdaq Entrepreneurial Center. And so, I think those are ways where I can find different ways to apply it and find ways that – how do you find intrapreneurs and entrepreneurs to give them the tools to be able to ask questions and go out and talk to users to apply it in a lot of different ways? That is a way to make that happen. And so, I think that for me, whether it’s my full-time job, or kind of like the balance of those things may change in the future. So, maybe I’ll be spending more time and I’ll find a role that lets me do more of that working with entrepreneurs and helping them develop that and maybe less in this oriented stuff, or maybe I’ll be able to find ways to combine the two. But I think for me, in sort of the next phase, it’s kind of how do I find the overlap, or sort of finding the balance of those two things?
Steve: You know sort of the thesis of this podcast is around – I don’t know thesis, but just the people we want to have as guests are people who are user research types and I’m wondering, as you describe this sort of human centered everything, and these mentorship roles, are you being a researcher, as a way you would define it in doing that work? We have labels and so I want to ask you to kind of unpack the labels and see what you’re sort of framing on this work you’re exploring vs. maybe the work that connects you and I and why we’re having this conversation?
Marianne: Yeah, that’s a really interesting question. I think for me, it’s – I think once I get sort of hot under the collar about something, like everything else goes out the window. You know I was talking earlier about once I found Glooko and I’m like wow, they’re working on a really big, juicy problem. This is something I wanted to do, who cares what the actual label is, or what the title is. So, I think for me, like it really bothers me when people – smart people – go out and build things and spend a lot of time and energy to build things that are not for humans. And I’m like, erh, why didn’t they do that? So, I think for me it’s more about empowering people who have that energy and who have that entrepreneurial spirit to make the things that are right. So, not just make stuff,
but actually, make the right thing. So, for me it’s a set of skills that I think maybe everybody should have and maybe once everybody has those skills and can do it well, maybe the role of researcher doesn’t need to exist. But until then, I feel like it’s my duty to go out and spread the gospel, as it were, of this is how you talk to users. This is how the type of information you can get. These are types of things that you can’t get and the limitations of the things that we can do as well.
Steve: So, it’s a really articulate lovely reminder of why we do research? It’s what we’re trying to have happen and you’re looking at achieving that goal through research, or through mentorship, or through advocacy and all the things that you’re doing are putting stuff out in the world that helps people – I don’t want to put – now I’m putting a lot of words in your mouth, but there’s kind of a producing of things ideal that you are a champion of. So, I see research – as you explain it, I can see research as a part of that, but not the only one.
Marianne: I think you summed that up really nicely. So, thank you for making me sound more articulate.
Steve: That’s the reflecting back technique. Okay, so now maybe wrapping up our epilogue, anything to add in this? Anything I should have asked you about?
Marianne: I was just thinking another user centered everything. I’m doing some volunteering and working with a foster child and I’ve been able to turn that into a user centered thing too. Because she doesn’t live with me, I just see her for a few hours each week, so I think very intently about the activities I’m going to do and being a crafty person also, I have spent a lot of time on YouTube videos and Pinterest and whatnot and making activities that I’m like well we’ve got to work on like fine motor skills. We’ve got to work on like counting or matching and stuff like that. And I’m having such a great time designing these little activities for her and also seeing how she responds and like, oh, she doesn’t like those types of things. Or, she needs more of these types of things. So, I feel like bringing it to – like I think I just naturally bring it to all aspects of my life and I see the power that it has, and I just feel I’m like – I just need to be a champion for those things because I see the value of it and I just want the world to know.
Steve: That’s a lovely note to wrap up on. So, thank you so much, Marianne.
Marianne: Thank you so much.
Steve: Well! That wraps up this episode of Dollars to Donuts. Go to portigal.com/podcast for the transcript as well as links for this episode. You can follow us on Twitter, and subscribe to the podcast at portigal.com, or iTunes, or Spotify, or Stitcher, or anyplace excellent podcasts are distributed. You can buy my books are available at Amazon and rosenfeldmedia.com. The amazing theme music was written and performed by Bruce Todd.
The post 16. Marianne Berkovich of Glooko first appeared on Portigal Consulting.
Welcome back to Dollars to Donuts. This episode features Leanne Waldal, Senior Director of Product Research at New Relic. We talk about establishing research in an organization for the first time, building up a diverse set of research collaborators, and the pleasure of taking on certain types of challenges. I’ve seen hopeful examples in startups […]
The post 15. Leanne Waldal of New Relic first appeared on Portigal Consulting.
Welcome back to Dollars to Donuts. This episode features Leanne Waldal, Senior Director of Product Research at New Relic. We talk about establishing research in an organization for the first time, building up a diverse set of research collaborators, and the pleasure of taking on certain types of challenges.
I’ve seen hopeful examples in startups recently where even though there are only 3-5 people they bring in someone for research and that’s highly unusual. Usually, if you’re starting up a company you have seed money, or you have your first round, the first people you’re hiring are engineers. You’re not making one of your first 10 hires a researcher. I know some example now around the San Francisco Bay Area where they’ve actually brought someone in whose role is to do research really early in the company. And that either points to a certain amount of humility around, oh, just because I am the user it doesn’t mean I know the users. Or, because they’re going into a space they don’t know. – Leanne Waldal
Show Links* The Secrets of Lyndon Johnson’s Archives, by Robert A. Caro * The Turnaround * Leanne on LinkedIn * Leanne on Twitter * New Relic * Autodesk * Dropbox * Site Reliability Engineering * EPIC * LUMA Institute * RACI model * Berkeley School of Information
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Well, hi, and if you’re a new listener, welcome to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization. otherwise, welcome back to Dollars to Donuts, the podcast where – okay, you got it.
It’s been a little while since the last episode, and I’m really happy to be back making new episodes for you. In a little bit I’ll talk about how you can support me, and the podcast, but first I wanted to mention an interesting article I’m reading. It’s from The New Yorker magazine, from the January 28, 2019 issue. It’s an article by Robert Caro who is an author of among other things, a multi-volume biography of Lyndon Johnson. This article is adapted from his memoir called Working: Researching Interviewing Writing, and his recollections about those activities are what caught my eye. He describes going to do research at the Johnson Library and Museum and going through boxes and boxes of papers. It’s the hard and tedious work of investigative journalism and what struck me was that amount of inferring, and cross-referencing and delving he was doing – he was coming up with facts and perspective that were not there for the taking but were from analyzing and synthesizing the information that he did have as well as what he didn’t have. Later in the article he talks about the importance of “place” in his research, that to understand Johnson he had to leave New York and really spend time in the Texas Hill Country. He writes “As soon as we moved there, as soon as the people of the Hill Country realized we were there to stay, their attitude toward us softened; they started to talk to me in a different way. I began to hear the details they had not included in the anecdotes they had previous told me.” And finally, he talks about silence in interviewing, citing two characters from fiction – Inspector Maigret and George Smiley, and their tactics to quote keep themselves from talking and let silence do its work unquote, where Maigret fiddles with his pipe and Smiley uses his tie to polish his glasses. Caro has his own brilliant technique: “When I’m waiting for the person I’m interviewing to break a silence by giving me a piece of information I want, I write SU (for Shut Up!) in my notebook. If anyone were ever to look through my notebooks, he would find a lot of SUs.”
Speaking of the connections between journalistic interviewing and user research, I want to recommend the podcast from 2017 “The Turnaround” – it’s interviews with different kinds of journalists about doing interviews, about getting to the story. I learned as much from the contrasts between our different objectives as I did from their best practices that apply directly. Check it out!
Now, obviously, I’m back with more episodes of Dollars to Donuts, I’m taking more of an open-ended approach to new episodes, and they’ll appear as they’re ready, without any specific frequency. It takes a lot to do this podcast and rather than taking advertising, or doing crowdfunding, I want to ask you to support the podcast – and me. You can hire me! I plan and lead user research projects, I coach teams who are working to learn from their customers, and I run training workshops to teach people how to be better at research and analysis. You can buy my books for yourself and your friends and your colleagues – I’ve got two books – the classic Interviewing Users and more recently, a book of stories from other researchers about the kinds of things that happen when they go out into the field – it’s called Doorbells, Danger, and Dead Batteries. You can also give this podcast a star rating and a review on iTunes, and you can review either book on Amazon. With your support, I can keep doing this podcast for you.
Let’s get to my interview with Leanne Waldal. She is the Senior Director of Product Research at New Relic. She has led research at Autodesk and Dropbox as well as running her own research agency for almost 17 years!
Thanks for being on the podcast.
Leanne Waldal: Absolutely
Steve: Let’s start with a basic introduction. Do you want to tell us who you are? What kind of work you do? What your company is?
Leanne: I’m Leanne Waldal. I am Senior Director of Product Research at New Relic. New Relic does application performance monitoring. It usually makes people’s eyes glaze over. Basically, we have a tool that developers and engineers, primarily that group, uses to monitor the traffic on websites and mobile apps. So, if you think about streaming media, online retail, financial services – tons and tons of data and traffic – their site reliability team, or engineering team, might use us to monitor that to make sure nothing goes down. I’m based in San Francisco and I’m a part of the design team which is a part of the product team. The product team is mostly in Portland, Oregon. We have a smaller portion of it in San Francisco and we also have a team in Phoenix and in Barcelona.
Steve: Can we talk a little more about New Relic and who the customers are? You said to make sure that it’s not down, but I’m assuming it’s not just that binary state, like is Amazon up or is Amazon down?
Leanne: It’s to monitor it. So, be watching to see if something happens that’s slower. Usually it doesn’t matter if something is faster. It means you’ve optimized performance and things are running really well. So, if things are going slower or you notice that people are having longer wait times for page loads or checkout loads or media streaming, or whatever sort of thing they’re trying to do in your app. You would have spent a considerable amount of time setting our products up, basically, to then do all that monitoring and raise alerts when something goes wrong.
Steve: Okay. So, it’s not up and down. It’s slow and it could be any part.
Leanne: Exactly. Lag time. Checkout’s not happening in the amount of time that you expect it to.
Steve: Did you use the phrase reliability engineers?
Leanne: Yes. So, the new role that basically Google created about 10ish years ago is the site reliability engineer. They wrote a book about site reliability engineering. So, our target user is usually called a dev ops engineer or a site reliability engineer. If you are a more modern company with the way that you’re keeping track of your website and your apps, then you probably actually have a development operations team, or a site reliability team. If you’re not quite there yet, then these people might be a part of an engineering team, or IT operations or something.
Steve: That’s good context. Then you said, you’re part of the design team which sits in the product team?
Leanne: Yeah. So, at New Relic, engineering and product and design all report up to the chief product officer. Different companies do it different ways. Engineering might be alongside product or design. Design might be alongside. But at our company, if you’re in the product org, you belong to the design team, the product team or the engineering team, or a few different teams that are in the org.
Steve: How long have you been in the role that you’re in?
Leanne: I’ve been at New Relic for six months. I did a similar job at Autodesk, before New Relic, for about a year and a half. And before that I had a similar role at Dropbox for a couple of years.
Steve: How do you compare and contrast those three organizations and what you saw and what the trajectory was in those roles?
Leanne: We could start with Dropbox. Dropbox was private, pre-IPO. It was very much a unicorn. When I started there it had 400-600 people. I don’t remember exactly how many. There was no research. They had a small design team. They had, as a lot of tech companies at that time, being a unicorn, had tons of engineers. And also, because Dropbox is a consumer app as well as a business app, everybody used it, so they felt like they knew who the user was.
Steve: Everybody that worked there?
Leanne: Everybody that worked there, yeah. So, I started setting up a research team there and grew it as the design team was growing. That’s different from Autodesk where I went next. A very old company, very fascinating products with fascinating customers and use cases. And people there who were researchers there, who had been researchers there for a long time. And what I was doing there was combining research and analytics together. At Dropbox, analytics was separate. And I had a small team that was more globally disbursed. At Dropbox, my team was all in San Francisco, to start. Here at New Relic, when I joined, there was one researcher here in San Francisco and one in Portland and a similar sort of tone around a lot of the people who work here at the company. Our site reliability engineers (or have been), so they know a lot about the market and a lot about the users. So, we’re basically here to help the company understand all of the new users it’s acquired since it started because it’s a different company now than it was 10 years ago. Ten years ago, New Relic was a company that was mostly reaching out to developers and engineers to use its product and now we’re mostly focused on enterprise and we have lots of enterprise customers now.
Steve: There’s that classic – you referenced this, right. People think they know they’re customer because they are users.
Leanne: Yeah, we all do.
Steve: But you’re kind of saying yes, to a certain extent. I don’t hear you shutting that down?
Leanne: It’s like yes and the marketplace out there has changed and maybe there’s something new that we’re not looking at right now. And also I’ve noticed that for companies where they maybe used to do contextual inquiry and go out and visit their customers and users and understand them deeply, in the last 10-15 years companies have tended to move to surveys and remote interviews and stepped away from that sort of like deep understanding you get from being in a room with someone, or on a construction site, or on the road with them when they’re using your mobile app, or whatever it is. So, you can see everything around them while they’re using your product or your app. And when you move away from that you lose sight of that holistic story of the customer experience. And then research can come in and do that for you.
Steve: Why do you think there is that movement towards the remote and movement away from the contextual work?
Leanne: Contextual work takes time and effort and can be exhausting. Exhausting to do it and exhausting to come back and know what to do with all that data, how to tell a story out of it and how to decide how it can have impact or value? Or sort of the like, now what do we do? I think also as humans we get really familiar with where we are, and we neglect to notice that the world around us has changed.
Steve: So, that’s back to your point then. We started this company based on work that we’d done and skills that we had and users that we knew, that we were, and the world moves on and shifts and new customers…
Leanne: And it’s not just New Relic or Autodesk or Dropbox. That happens at any company. You can be in banking or in legal. Ask anybody who has been there a long time, compared with a person who just started. They’re having much different experiences of it.
Steve: You described, especially with Dropbox and New Relic, this point at which the company had little or no research, and that is not unique. We know lots of companies like that.
Leanne: Oh yes.
Steve: Do you have a hypothesis about what’s going on? There’s a point at which someone like you starts talking to these companies and starts saying, “hey, here’s what I would do if you had me come in and work with you.” What’s going on beforehand? What’s the point of need or pain that’s identified inside these companies when they realize…
Leanne: Sure. So, what I’ve noticed at the companies I’ve worked at, as well as companies I was interviewing at for this sort of position, and companies I know the story around when they hired someone in sort of like a director or higher role in research, usually something changes in revenue and that change in revenue sets forward sort of like how do we figure out what’s going on. And if they have someone who is currently working there in a role who has had experience in the past with doing market research, or doing churn research, then those people will start to raise the flag of like oh, we have to do research. And then it’s how do we do it. So, that’s one reason that someone brings someone in to do it. Another reason is that there’s sort of like a groundswell of engineers, product managers and designers who are telling the people who make hiring decisions, “we need someone to be in charge of design.” Or, “we need someone to be in charge of research. We’re doing a lot of this ourselves and we don’t feel like we’re doing it well enough.” Or, “we, for example, do lots and lots of interviews. We don’t have time to synthesize the data. So, could you please hire a researcher to work with me.” So, sometimes that happens. Another way it happens, which is how it happened here at New Relic, is that the company decides to hire like a VP or SVP of design and then that person who knows, oh this is what a design team looks like – you have someone who is focusing on content design and language and someone who is focusing on interaction design and visual design. And I also need someone who is focused on research. Sometimes it’s from the top down. Sometimes it’s from the ground up. Sometimes it’s there used to be this role at the company before and someone left or was laid off and then they just didn’t backfill it and then after a while realized oh that’s really a role that’s really important.
Steve: We need that.
Leanne: Yeah, and it’s not – I used to think that maybe it was sort of like a trend, like everybody is starting to be customer centric, or user centric, or whatever that means. So, then they were like, “oh, well Joe down the block is doing that, so we need to do this too.” And what do we do, we hire someone to run research. But, as I have been interviewing for jobs over the year and talking to different companies and doing these jobs, I’ve realized, no, there’s actually something that happens inside the company that causes that change. It’s not so much like copying a competitor or another company that you care about. You might do that with engineering or something, but research is sort of like the first thing to go if you start running out of money. And, I’ve seen hopeful examples in startups recently where even though there are only 3-5 people they bring in someone for research and that’s highly unusual. Usually, if you’re starting up a company you have seed money, or you have your first round, the first people you’re hiring are engineers. You’re not making one of your first 10 hires a researcher. So, I know some example now around the San Francisco Bay Area where they’ve actually brought someone in whose role is to do research really early in the company. And that either points to a certain amount of humility around, oh, just because I am the user it doesn’t mean I know the users. Or, because they’re going into a space they don’t know. They just had a really good idea about it and they really need to understand the space.
Steve: That’s an encouraging sign.
Leanne: Yeah.
Steve: Is there a distinction between – you gave a number of different scenarios where research comes in – do you see a distinction in sort of the context, or the action that’s being taken when bringing in a person to do research vs. bringing in a person to lead/manage/run/build research?
Leanne: Yeah. So, if you’re brought in as a person to do research you’re usually reacting to the things you’re being asked to do. You usually don’t get to pull your head up and do strategic work. You’re doing lots of like compare this design with this, do this survey to answer this question, help me with like interviewing these six users in remote interviews. If you’re being brought in as someone to say like where should research fit into the company, it’s more challenging, but more interesting and exciting because then you’re being told like tell us what we should know? Like where should we go with this? Are you going to focus more on the market research side, or the sales research side? Are you going to focus more on the product design side? What are all the things that we need? What kind of programs do we need? How do we get access to users? How do we interact with users and engage users? There’s just all these little pieces of how a research process and practice runs inside of a company that is great fun to put together because you have to do it based on who you’re working with and the culture of that organization. It’s not just a cookie cutter that you put into a company.
Steve: Is there a trajectory from one to the other? Like someone that’s hired because the need is to do research. Is there a trajectory for that person to start answering the kinds of questions that you’re talking about?
Leanne: Yeah, yeah. I think if that’s sort of like the way you work, or your personality, or the way you’re motivated, you can get there. For example, I have someone on my team right now who I mentioned to the other day like – she’s a researcher and I said, “you could make some day, if you wanted to, a really amazing research operations manager because you’re really good at all these pieces around managing the research and communicating it.” Because half of research is PR and sales. You have to find the people to engage with it, to sell it to at the end. You know you want your research to have impact on product or a marketing campaign or whatever you’re doing. So, you have to gather the people around you. And you have to keep track of, particularly in a B2B company, who are all the people who have to be involved if I want to do visit customers. So, in a consumer company you just go out and find people. You talk to them and you do whatever you want with them. At a B2B company you have to be engaging with customer success, with sales roles – whatever they’re called – account execs, account managers – whatever that company calls them. You have to often also be engaging with product managers, designers, design managers. Engineers, didn’t used to be, just as a stereotype, interested in research and now at some companies they are very interested. And so managing all those types of people that you either have to get approval from, or you need to make sure that they’re bought into what you’re doing, or you need them to come along with you and do it, takes a lot of work. So, there’s this whole new role of research operations that usually helps keep track of that. And I’ve noticed some researchers have a real knack for sort of like managing all those details around their project and then shining a light on it and sort of going out and selling it, sort of like a marketing campaign. They just know who to schmooze with and talk with. And some researchers are really good at doing the research and writing the report and sharing it. I pointed out to her, “this is something you are doing for our team without being asked. Like, that’s a real superpower.”
Steve: Does that start to change the – I think you said this, at least indirectly – does someone who is shining around operations, like your colleague here, does that start to change how research is perceived or experienced by others in the organization?
Leanne: Yeah. Especially if you’re an organization that’s not used to research. Or they’re used to doing it themselves. So, they’re not used to partnering with someone else and they’re not used to having someone say like, “oh that’s not really a survey. That’s more of an interview.” Or, “oh, that’s not really an interview. We need to actually go watch people to do this.” Or, “this can’t be answered in an interview survey, or watching people. We need them to be in a diary study.” They aren’t used to having someone bring in all these different methods. And they aren’t used to having someone sort of like find some insights and then tell them. So, that’s why it’s so important to have a collaboration. I want you with me while I’m doing this research so that when I tell you the pens need to be blue, not red, you were there to hear everyone say they like blue better than red, instead of throwing the report over the wall.
Steve: I hadn’t really thought before of operations as a – I’m not sure you’re saying this – but, almost a Trojan Horse. That there are these sort of tactical objective problems to be solved under operations. How are we going to find these people? What is it going to take logistically? What is it going to take legally and so on? That starts to change some of the conversations maybe that can happen internally. Like you said, “this is your problem. We recommend you go about it this way.”
Leanne: Yeah.
Steve: I hadn’t thought of that under an operations lens myself.
Leanne: It’s really more in a B2B company than a consumer company because you have to manage so many relationships and oftentimes a researcher wants to focus on sort of like the details of the research. Like ask any researcher, they don’t want to do all the details around recruiting and scheduling people – or, most don’t.
Steve: Right.
Leanne: And so, you want someone who can do that. And then sometimes researchers also aren’t the people who know how to best sell their work or edit it down into like the three bullet points for an executive presentation. And it’s helpful to have somebody who knows how to do that. You know just sort of like editing and presenting and PR and marketing. And also knowing who to market it to? Like, oh, I know that there’s this sales VP in this office in Toronto who is going to be interested in this. That mentality of like keeping all these dots rolling together. And if you have someone who knows how to do that and also understands how to do research and what goes into research, that’s what I see makes a really good manager from an operations perspective. There’s the people management side too. Being able to mentor and coach and take care of people. But there’s more and more in B2B research a need for someone specifically in this operations role to just sort of like help everybody and raise the brand of the team and the company, or in the product team or the design team.
Steve: It’s so cool that we’re at a point in this field and this practice where we acknowledge, as we always did, that there’s a number of very different kinds of skills and expertises that are required. And you just outlined a whole bunch of them. But that we’re able to sort of say, and not be laughed out of the room, like oh, that might be different people. You talked about collaboration and collaboration with people with different strengths or superpowers.
Leanne: Yeah. And another expertise and skill that’s being introduced more into companies is the academic anthropologist who is now working inside companies on a research team. Like EPIC used to be primarily all academics and now there’s a bunch of people from business involved in EPIC, and also people who used to only be academic anthropologists and ethnographers working on research teams, inside tech companies or consulting agencies, or whatever. So, I recently hired a few anthropologists on my team. One who is a former professor, one who had applied anthropological research practices for market research. Another who had done research with an NGO, but also worked inside of a company recently. And I think that when you have that academic background and you bring it into a company, it’s like a special sauce. You also bring in a special sauce if you’ve been working in tech for 15 years and you know the ins and outs of how a product team works, but I like mixing those two things together because people learn off of each other.
Steve: You like to mix the special sauce.
Leanne: Yeah.
Steve: So, it’s an extra special sauce.
Leanne: Yeah. So, for example, we decided as a team that we wanted to look at all of the internal blog posts and other things that had been written about customers and users, from the point of view of people inside the company, because my team is new to this company. We were like okay, let’s take a look at everything that everybody else wrote. And I was trying to figure out how we would do that. Like, who knows how many internal blog posts there are and how many things that have been written about customer visits and things that different people know about users that they report out internally. And then I realized, I have some former academics on my team, academics who are now working as researchers in industry. They have skills in literature review. So, brilliant. I took them and had them put together like how are we going to review this and code it – special sauce, like they absolutely know how to do that. It was really easy for them to put it together. And then took other people who had more industry experience, or application focused experience, and said, “okay, now you can do what was set up. You can do the review of following the system.” And I have people who know how to set up systems set up the system.
Steve: So, that’s an example of the special sauce. They know how to deal with this.
Leanne: Yeah. And if you have a team who has mixed experiences like that, then they just sort of like, they lift each other up and they teach each other things. And that’s something that you can’t learn if you’re the only researcher, or you’re just one of a few researchers who all have exactly the same background. So, you all come out of the – whatever the programs are now, at Carnegie Mellon, Stanford and Berkeley and University of Washington and on and on. And you go and get a job and you do remote interviewing and some focus groups and some other things But everybody around you is doing exactly the same thing. Then you don’t get a chance to sort of like up level yourself. And you also, like if you’re the person managing that team, you don’t get a chance to learn new things from people on your team that you hired, that like you don’t have the same background as them. So, it just improves humanity basically.
Steve: So how do you think about like what the – if the team is kind of eclectic now and you have all these different mixes of skills and backgrounds and aptitudes, how do you think about what the collective should be? How do you think about the mix?
Leanne: Of a whole team?
Steve: Yeah
Leanne: So, I do miss the team I used to have which had both research and analytics. I think being able to have people on your team who do data science, or who are more like business analysts who know how to count things up and make charts, or someone who knows how to mix data together and come up with reports out of it, you get a much faster feedback between the analytics people and the research people. Analytics is like oh I did this, I want to know why? Research is like I can go out and find out why. Or research says, “I need some access to people who have certain characteristics, can you find me a list of them through analytics? Thank you so much, I’ll go out and talk to them and then I’ll come back to you.” In most companies those roles are separate. So, it is here. Analytics is a separate team. We work super well together, but sometimes when you have them both on the team it’s really nice because you’re both at the same team meeting and you’re both seeing what they’re doing. From a research perspective, having people who are all quant or all qual, or halfway in between or know both, is useful because anybody who’s in the job market right now, when they look for their next job the requirements are going up and up and up. We don’t want you to just be somebody who does interviewing. We need you to know how to design a survey. We need you do know how to do SQL queries. We need you to know how to do – like there’s more and more requirements showing up in job descriptions now. And so, if you can be a part of a team where somebody else knows that thing you don’t know, it just – we’re all going to like leave and get another job someday. Nobody stays somewhere forever. So, it will just help you when you’re going out to look for your next job.
Steve: So, skill development, kind of through proximity to your colleagues.
Leanne: Yeah. And it’s also diversity. Like any time, you have a diverse team, whatever you’re defining diversity as, just makes for a better team environment because all these different perspectives around the table from all the different places they came from, just makes for like different things to feed into feedback about a project you’re working on, or something that’s going on in the company. You know, we didn’t all come from the same school or the same program, so we don’t have a difference of thought in the way we approach a problem.
Steve: So, for you, thinking about the teams that you’ve put together – because you’re talking about different kinds of – I don’t know if diversity is the right word, the way you’re using it, but qual and quant are different orientations, or…
Leanne: And also, academic and industry. Or in tech, because I’ve worked in tech for 20+ years, I look for people to come work at the tech company I’m working at who haven’t necessarily worked in tech before. So, look for adjacencies. Like you did research, but you did it for a car company. Or, you did research, but you did it for an agency. So, that they can bring different things into the team. That’s what I mean by diversity. You know research, but you used to be a professor. You know research, but you were working on an astrophysics PhD and decided that’s not what you wanted to do anymore, and you realize things that you learned as a part of your graduate program – so you totally know how to do surveys and how to talk to people and how to do research.
Steve: I had this Twitter conversation with someone the other day and they asked me where is a writing course that I can learn to be better at writing the kinds of things I have to create as a researcher. They can’t find things like that. They can find persuasive writing which is for sales. I kind of scratched my head and said, “I don’t know that that exists.” I went into this little sort of pontification, as Twitter invites you to do, about learning from adjacencies like take a creative writing class. Take a journalism writing class. And in doing that, you would start to see, oh here’s how the lessons of doing this adjacent thing well could apply. That was the best advice that I had. I feel sort of affirmed, from what you were saying, that adjacencies for certain kinds of things bring a certain kind of value.
Leanne: So, the Luma Institute, who put this framework together around design thinking methods so that it’s easy to teach and easy to learn, calls that alternative worlds. And I think that’s a useful way to think about it, that for your person who is trying to figure out how to write a research report for whatever audience they’re trying to serve, well that’s storytelling. It’s sales, it’s marketing. It’s slide development. It’s all these other things. And when you start thinking of it as how would I think of this from the point of view of a salesperson? Or how would I approach this from the point of view of a marketing person who has to put customer stories on the web and sell that to somebody? Or how would I do this as the point of view of the person I am presenting this research to? Like what do I want to hear? It gets you out of the sort of like oh I have to have 10 slides and one has to have my methodology and the end one has to have like recommendations. You can start thinking about it more as like I want to tell this story so that they land on that naturally. Or, they ask me questions about it when I’m done that will answer all those things, so I don’t have to have it all in my report.
Steve: I feel like this is a transgressive notion of expertise though, especially in tech.
Leanne: Yeah. It would be nice for all researchers if they could just come in and do their research and research was valued and anything that researchers said they were learning from customers/users was taken and put into the roadmap. I don’t know anywhere that happens. And because there isn’t anywhere that happens and the way the culture in these teams work, you have to have this sort of like marketing and collaboration and partnership mentality around your work. If you’re a consultant, you get a project, you do it, you collaborate, you sell it a little bit – you’re done. You don’t have to still hang out and make sure it’s being used and follow-up.
Steve: It’s just all sunshine and roses being a consultant, right!
Leanne: Exactly. That’s why I don’t do it anymore.
Steve: I had a coach advising me, in my consulting business, and what they said was – they said many things, but one of them was anytime you start talking with a prospective client, show them – show me how you’ve helped someone like me. That adjacency kind of framing – the advice was that wasn’t as persuasive. So, that could be what have you done in my industry, in my vertical?
Leanne: Yeah.
Steve: You know, and I think researchers, or anyone that just likes to sort of – that gets deep into different kinds of problems, you know we see connections between things that are – you know we know why this problem is like this other problem, even though it’s different.
Leanne: And you want the other people to see that and sometimes it’s hard to show them a lens that will help them see that the way you saw it. Yeah.
Steve: I think the advice was that’s not where you start the relationship. You start the relationship a little more close in. “Oh, this thing that you’re doing. Yes, I have done that.” Even though every problem is new in ways that – it’s very hard to do that.
Leanne: Yes. Yes.
Steve: Sort of starting the – showing relevancy right at the outset of the conversation.
Leanne: When I ran a consulting agency, that’s how I started it. Somebody asked me if I knew how to do something and I’d never done it before and I said I knew how to do it. And all of a sudden, I had a consulting.
Steve: So, your career is just built on lies.
Leanne: Yes.
Steve: Is what we’re saying here, okay.
Leanne: Yes.
Steve: And we’re leaving this in. Yeah, well I think that’s true too. There’s the confidence to explore. I mean can you do this? Yes. Have you done it? Well, no, but I can do it.
Leanne: Yeah. And sometimes the question never comes up, have you done this? It’s just the can you?
Steve: So, that’s – so, we’re talking sort of requests in the consultant side of things. Inside organizations, in any of the ones you’ve worked in, like how are projects formed? Who decides?
Leanne: So many different ways. So, I would say that if you’re beginning research in a company, you’re taking requests and you’re probably using your boss or some other key stakeholders you recognize to decide which to work on. And you’re probably prioritizing those in some sort of backlog, if that’s in a spreadsheet or it’s a Jira or whatever. So, you’re showing people sort of like here’s what’s in my backlog. Here’s what’s coming up next. If you’re a team, the way I prefer to run a research team, which is the way I’m gradually getting it set up here – how it was set up at my last couple of jobs – is to have researchers who are paired or embedded, or whatever you want to call them, but they’re primarily working with a product area. A product team, a feature team, a product area, however your organization is set up. And they are servicing that team and partnering and collaborating with that team. And they own the research for that. But because they’re a part of the research team they also have a holistic view across the organization, because they’re also meeting with other researchers who are doing the same thing and seeing where their work, or where their product area or features sort of overlap, or interlock with each other. And then having some people who are more senior, principal researchers, just working on special strategic projects. So, that could be, depending on a company, that you have someone who is more focused on market research or competitive research. It might be someone who is focused more on research to understand where we should go next? So, sort of like future casting and sort of like looking at what’s going on, finding edge cases, seeing spots for innovations and other sorts of insights. And then also the operations side. People to manage panels of users and recruit people. And then there’s always, depending on how many people there are, some sort of management tier for people management and coaching and that sort of thing.
Steve: And when you talk about embedded you – it’s certainly a frame that you hear described with some regularity, but your words are a little different. Some people talk about embedded, that that researcher is on that team, sort of fully on that team. You’re almost describing like a workstream or a customer or something.
Leanne: Yeah, because the way people work on teams now is not the way that HR software recognizes. So, HR software, for example, recognizes I have a boss. He’s a VP of design. Recognizes I have people who report to me, all the people on my team. Um, maybe recognizes that I have people that might be good for 360 reviews. So, my direct colleagues who also report to my boss. But, that’s such a limited view of what I call my team because – I’ll just talk for me. Like I work with lots of different people around the company who, if I was going to be reviewed, I would want them to weigh in. People on my team – actually, we had a research team meeting this morning. I had them all bring a stakeholder map because I wanted to see – now that I have people that are sort of like embedded or paired within teams, like who are they working with and what does it look like, compared with people on my team whose work is sort of nebulous around all the different people that they’re managing and all the different work that they’re doing. And it turned out that way. So, if you’re a researcher on a team – a lot of people refer to the three-legged stool, but if you have a researcher it’s a four-legged stool. So, you have someone from research, from engineering, from product and from design, all working together with like a common focus on how are we going to get this out the door and make sure it meets customer needs, solving a problem, all that sort of stuff. If you are somebody in operations or a principal researcher working on a strategic project, you just have a mess of people around you. And so, all these people sort of like vacillate in between teams. Designers do the same thing. Designers are a part of the scrum team and they’re a part of the design team. Engineers are a part of a team for some sort of product or feature. They’re also a part of an engineering team. And if gets even more complicated when your manager is in one city and you’re in another city. Because if you’re enough time zones away you probably have somebody who is sort of like your dotted line manager in that city to help you with like all the things around your job. For example, I have a researcher in Barcelona. We can’t look over each other’s shoulders. There’s only two hours a day when we’re both at work. So, there’s a director of design in that office who is her sort of – who is her manager there. So, if she has questions about how does this work? Or how do I do this? Or who is in charge of this? Or help me review this? He can do that. But no HR software recognizes that. It doesn’t recognize she basically has two bosses. It doesn’t recognize that I have one boss, but actually these three other people are kind of my bosses too. It doesn’t recognize that that designer has a boss for whoever is running that scrum team, but that designer also reports to a design manager or director. So, most people work that way now. There’s very few people inside companies anymore who will say they belong to one team and that’s all they belong to. Everybody belongs to like multiple teams and we have this archaic structure around this person decides like your performance review and hopefully gets input from all these people. But, just lines people up in like lines and doesn’t realize that it’s a little more messy than that.
Steve: So, does that gap between the reality and the structure of the software, what’s – are there consequences to that for how we work?
Leanne: I don’t think there’s consequences. I think it’s just the way it is. I think there’s more 360 reviews now, just in my experience, then there were 20 years ago, because you work with so many other people now. I can remember jobs from early to mid-90’s. Nobody was doing 360 reviews of any of the jobs I had then. Like I worked with a set of people in a geographical space. I didn’t work with anybody else. I think it’s just more like it is what it is and things have come in to help manage and understand that, but we, the people who sort of like do to the sort of organizational management, haven’t figured out how to accurately represent that in a structure of the software that manages the company. Some companies try to go – what was it, like Zappos, that decided nobody had a manager. That seems like part of a reaction to that – Holacracy, that’s what it was called.
Steve: So, we got into this because we were talking about what does it mean to be – there sometimes seems a tension between researchers are embedded, or researchers are centralized. And I think what you’re saying is they are neither, or both. And that that’s how work happens.
Leanne: Yeah, and you have to look at the culture of your org. How does your org work? Who wants to participate and who wants – it’s that old RACI model. Like who wants to be involved? Who wants to be informed? Who wants to partner with you?
Steve: So, wait.
Leanne: Do you know the RACI model?
Steve: I know it, but I need to clarify it.
Leanne: It’s where you define – I might not get this right. It’s like where you define who is responsible/accountable, who is interested in consulting on it and who just wants to be informed. So, if you’re thinking about a project, you always have people who sort of fall into those categories. So, I just think it depends – some cultures are very top down and as you go up the ladder they only want to be informed and not as involved. Some cultures are more participatory and as you go up the career ladder, everybody wants to participate and know and be involved. So, it just depends where you land and get a sense of who just needs an inform, who actually wants to come along and sit with you while you do a remote interview, and travel 3 blocks away to visit the customer. You know how far up the ladder does that go?
Steve: So, coming into these organizations that have done very little with research to date – like understanding the culture, it seems like that’s got to be an initial step to get the lay of the land?
Leanne: Yeah. Who’s making decisions and what data are they using to make those decisions? Do they want any more data? Usually the answer is yes. Anybody wants to know more to make better decisions and, in a corporation, make more money. So, figuring out who those people are, where the gaps are, what they need to know. And then another thing you can do when you start at a company is be the researcher yourself of that org because what you’ll start to notice, just like any research project, is that you’ll start hearing the same thing from multiple people. So, you’ll start hearing like, “the one thing we’ve never figured out is whether people really like blue or red better.” You know there’s always something that’s like the one thing where people aren’t aware that everybody has the same question, but because you got them to open up and talk and had casual conversation and used all your interview tips and tricks to interview multiple, different people who don’t work with each other, you’ll start to see a theme around something that everybody wants to know, but nobody knows, but nobody is like trying to figure it out because they don’t know everybody else wants to know. I’ve seen that at every company. We’re working on a project like that here right now where there’s basically two things that I heard across the entire company when I first came here. So, I was like okay, that’s where I’m going to start some key research projects and then I’m going to bring in some people who are interested in it. And then as I hire people in, I’m just going to start attaching researchers to different product areas and work with the people who run design to sort of think about which designers need the most help right now. Or, what meets the business directives, or the key priorities, or the goals, or whatever – the product org of the company.
Steve: Are there stages of evolution? I don’t know if it’s milestones or some framework that at these points you’re kind of coming in where there’s very little and you start building in a way that is specific to what you’re learning about the culture, are there stages that you could identify that you’re passing through or moving towards?
Leanne: Oh, yes. So, once you get sort of like programs and tools set up you don’t really have to do as much anymore. So, programs and tools like is there decent survey software? Do we have access to users? How do we access them? Is there a panel? When we do research, do we have standard agreements that we sign with the humans that we’re doing research with? Repositories – where are we storing everything? How are we storing it? How are we accessing it? Do we need transcripts? Do we have a tool for that? Do we need to do card sorting? Do we have a tool for that? Just sort of like sorting out all the tools and processes for the practice. What was the question? I got lost on a rabbit hole of process and tools.
Steve: You were answering the question. So, what are the stages or milestones?
Leanne: Oh yeah, okay. So, looking at what’s needed that is more like capital improvements. So, like we need a new roof. Okay, we’ll get a roof, it will last for 20 years. Done. So, things like how you pay incentives and research agreements, that’s sort of like the roof. Then you need like everything that pays the electric bill and the gas bill all the time. So, those are projects. So, what projects need attention right now. And you learn that from interviewing sort of like all the key stakeholders you’re going to work with, which in my example earlier was once you interview a whole bunch of them you start to find out that there are some key things that everybody wants to know. And then figuring out, either with your boss, if she or he knows, or with key stakeholders, like if I was to start bringing in researchers one at a time, where would we start? Who would benefit the most? What would be most impactful? Or said another way, sort of like what are the things launching soon, or later? What are the business goals, or what’s in the strategy? Or what are the business directives? Or all those sort of things that make a company run. And that helps you because then you’re making sure you’re focusing on something that is actually going to eventually affect the company’s bottom line. So, when the research comes out and it helps that, then you’ve been like okay, this is how we do it at this company. We’re not academic. We don’t go sit back and sort of think about things and just, you know, go research things out of curiosity. We look at where the company is going to make sure we’re helping it and helping it do better with the research that we’re doing.
Steve: And then what’s a five year – that’s a presumptive question. They’re all presumptive questions. What’s sort of the horizon for which you would have vision? The time horizon?
Leanne: For the times that I’ve come into a company and there’s been no research, or one or two, like here, individual researchers, what I’ve told people is it’s going to take 12-18 months, with budget and with head count, to get this into a modern research practice. And here’s how we’ll do it incrementally and here’s how we’ll check in and benchmark and measure it along the way. When I was at Autodesk and I inherited a team and then I sort of hired in and added on to it, it was much different. These were really old products that had a really amazing install base. And some of them were market leaders. It wasn’t like companies that were two years old, five years old or ten years old, or fifteen years old. Like, once you’re at 30 years you’re settled in a certain way. So, I did different things. I tried to better understand the products and better understand the company to see where we would could add value. And then also, because that was a much larger company, see where research was happening across the company to see if we could sort of like combine efforts.
For example, I went to some company that’s like 25 years old, that has say 1,000 researchers. Because there are some companies out there now that have that many. And I was plopped in as a director of research with 20 to 50 direct reports. It’s a much, much different role than coming into the company and setting something up. They’re you’re coming in and you’re saying how is this done? Is there any way it could be improved? Is there any way I could sort of like do – no, there isn’t? Okay. So, how do we prioritize things? How do we have impact? How do we collaborate and share out and be good corporate citizens and be a part of the team?
I’ve realized, with this third job that I’m on now, after doing consulting for many years, that I do really like growing things. There’s a challenge to it that like gets me into work every day. Like, oh, we have to figure this out. Oh, there’s lots of problems to solve. Oh, how are we going to prioritize this? Or how are we going to solve for this? You know, how am I going to find someone to fill this role?
Steve: You just listed a bunch of things as positives that someone – if you’re in a different emotional state or a different sort of mindset – could frame those all as negatives.
Leanne: Yeah. Yeah. You have to understand the challenge you like to work on. Is the challenge I want to be the person who works with the team to help them figure out what they need to solve? Or do I want to solve an organizational challenge while also working on business priorities and solving problems?
Steve: So, this is something that kind of sparks you?
Leanne: Yeah. And I didn’t realize that until – it was like 3 months ago or so. We hosted a breakfast here and somebody was asking me what I was doing, and they were like, “Well it sounds like you’re really good at that because you’ve done that three times now, and you started your own company.” And I was like, oh, that’s the first time I realized like this is my thing that I like to do.
Steve: Yeah. That’s great.
Leanne: Sometimes you have to be almost 50 to have finally found yourself.
Steve: Right. It’s the journey, not a destination, right.
Leanne: Yes.
Steve: I’m sure you’re going to keep finding yourself.
Leanne: Yes.
Steve: Anything else that I should have asked you about that you want to share with us? The collective us. I know it’s just you and I in a room.
Leanne: Well, there is one other thing I’ve noticed in the past few years. I have used a community platform, that’s sold by a company and meant for marketing and brand teams, and have used it for product and design research. There’s something about – so, support always has its help forums. Marketing and brand usually do focus groups, but now sometimes they run online communities where you can get badges and win things for participating in their research, or telling stories, or whatever. There’s a space in between that people who are on research teams at companies sometimes use and sometimes don’t. So, a typical team might have a panel, particularly if you’re B2B and it’s hard to get access to people. Or even if you’re a consumer-focused company and you just want to make sure you have like your 10,000 people who you can access at any time to easily invite to research. But I think there’s something about the way that people on social media interact with each other in a group in one-to-many fashion and many-to-many fashion that is happening in some ways in research teams, and sometimes not. So, if you’re a researcher and you can, at any time, poke into a community that belongs to you – this is my panel, but your panel, they can’t all see each other. In a community they can see each other. And so, it acts a bit like a support forum, a bit like this marketing and branding, sort of like what you think of our new colors sort of thing. Or this is our new video for our marketing campaign. But instead it’s for product and design. I’ve noticed there’s a certain fear of it – oh my God, if we get our users and customers to talk to each other, what will they say about us? But I think if you get over that fear and you open it up and put together a private space where they’re all talking together and you and your product design, engineering, marketing sales, whatever partners are also in there, you create this sort of like brand loyalty and product loyalty that you don’t necessarily get just from marketing brand and sales and just from support. So, I’ve done that in my last two companies, created this sort of community where people could see each other, and we could interact with them in little like pop-up sort of projects. Haven’t done it here yet. But when I talk to the vendor that I’ve used for doing this, the last time they demoed it for me, I saw that the examples they were giving me were for a product and design team. So, I was like huh, you’re now selling to product and design teams. So, there’s something going on with people picking that up and doing that.
Steve: How do you set people’s expectations, the people that are participants in this community, for what – as you said, it has elements of other kinds of communities that are maybe more familiar…
Leanne: Well, you have to make sure, particularly in business settings, that – you know don’t share anything that you wouldn’t want your competitors to see, because you don’t know if your competitor is in this community, or not. We’re focusing on sort of like your work and your use cases of these sorts of technology. Don’t disclose anything that you shouldn’t be disclosing. But, if you’re in a space where people are using a technology where they get a lot out of finding out how other people are using it too, then you create relationships with each other and then you create your own relationship with them. Another downside is you can get bullies, so you need to have guidelines. We can kick you out if you behave in one of these ways. And with panels, because it’s the company to the user or customer, and that’s it, it’s harder to find sort of like an intrinsic incentive for someone to participate in research. In a community you usually don’t have to have any incentives that are hoodies or cash or gift cards or whatever because the intrinsic incentive is that they’re getting value from other people being in there. And then every once in a while being able to talk to a product manager or talk to a designer or talk to somebody else. So, I think that – I don’t know if that’s something that will keep going, but I see it as a way, particularly in B2B research, to provide value for your customers that actually helps out your sales people and gives your product people and design people a way to interact with users and then also give the user support that they’re probably also getting over in the support forum, but a different kind of support because you feel like you’re part of a group.
Steve: It’s compelling to me to think about just blurring the different kinds of interactions that companies have with people.
Leanne: Yeah
Steve: Like you said, there’s marketing, there’s research, there’s support and we structure the companies around those kinds of functions and the tools and so on.
Leanne: Yeah. And it can make it hard to see the end-to-end customer experience. Yeah.
Steve: What – maybe we can rewind. Like how did you end up in research as a profession?
Leanne: Back, when I graduated from college, in the early 90s, I went to work for a startup cellphone company as a statistician. My background is in statistical computing and economics from my university education. So, I was analyzing what were then considered very, very large sets of data – terabytes of data. I was running a neural net program on a Sun SPARCstation. I don’t even think it had a gigabyte of RAM. Yeah. It had two processors. It was amazing. It would take like 4 to 6 days to run something, to spit out a model. And if I got it wrong I had to do it all over again. What I was doing was I was working in a marketing organization and I was there to help them like figure out things like who should we sell call waiting to? Like should we just sort of try to sell it to everyone? Or should we target people? So, I was figuring out targeted lists based on behavior that we saw in cellphone use data, to help the marketing team understand who to sell things to. When they did that, what I realized was I didn’t know why the people were buying it? Like, just because I could predict like because you have these like usage analytics, it makes you more likely to buy, but why is that? And a couple of things happened then. The company I was working for got bought by AT&T, became what is now AT&T Wireless Services. When you get bought by a much larger company, a bunch of chaos ensues. Your job often changes. Your boss often changes. I was very young. I was in my early 20s. And at that exact time Mosaic came out and the web became graphical.
So, there were all these things happening that occurred at this point in my life where I was like, huh, this is my first job out of college, but now I’m interested in knowing why these things happen? I just taught myself HTML. I figured out how to put together a webpage. Oh my goodness, there’s this thing called Match.com and I met this girl and she lives in San Francisco and I want to date her. Oh, I’ll just go take this job at this company in San Francisco, this web company that has like 20 people and they’re making websites and they’re from all different sorts of places. And I became a QA manager there. Once again, sort of being asked, “can you do this?” Not, have you done this. And I said, “yes.” And then from QA realizing that we didn’t really know what the users were doing. And I, at this point, had never heard of user research. But I was friends with people who were doing that at different companies. So, I asked them about it. And then I left that company and went to a startup. Was also a QA manager. Got laid off when the company didn’t get its next round of funding and went and did some consulting. Doing like web development for the summer. Set up a company that I had intended to be a company that did QA for web apps and somebody came to me and said, “can you do some researcher interviews with users while you’re doing browser testing?” And I said, “yes, we can.” And I was like here’s my chance. I’ve always wanted to know why people are doing things. And so, I happened into it that way. And in the mid to late 90s anybody could do that. You could teach yourself PHP. You could declare yourself to be a web marketer. You could say I’m an information architect. Like you just said what you were, and you could just do it. So, I quickly went and found some friends and said, “I have this consulting project. I don’t know what to do, help me out.” And they were like well you do AB…like the things I just mentioned. You have to do A and B and C and D. I was like okay. So, I did A, B, C and D. I did the project. I got paid. I thought it was amazing. I was like I want to do this again. I was like a kid who gets ice cream for the first time. Like, I will have more of that. And that’s how it got started. In consulting, what you need, from my experience, is you need one or two projects as reference projects and then that helps you get the next project and the next project and the next project. And then it was 17 years later.
Steve: Wow. You’re saying that there was a time when you could sort of declare yourself to be this thing.
Leanne: Yeah. When the web was new. When this new technology was coming out and consumers were grabbing onto it and companies were seeing that money could be made there and – people think now that there are more jobs than people, but then there really was more jobs than the labor market could supply because the labor market didn’t have the skills and the education system wasn’t educating people with those skills. For example, in looking for research interns for this coming summer, I interviewed people who were getting degrees in user research. Like a Bachelors of Science in user research. And I was like wow, like companies are now preparing people and there’s a whole education system out there in these D Schools and I Schools and everything now that didn’t exist in the 90s.
Steve: Right. I sometimes feel like oh I’m – ‘cuz I came up through the same system, or lack of systems, that you did.
Leanne: Yeah. If I wanted to do this job now – say, if I had gotten my Masters at the Berkeley I School, or whatever, and I went and got a job as like a senior researcher at a tech company, and I worked myself up, way up, it would be much, much different than my experience of teaching myself and figuring it out.
Steve: What’s lost or gained in that evolution, if anything?
Leanne: Well, I can clearly see that my sort of like – YouTube didn’t exist then, but it would have been like the university of YouTube, the way that I went about teaching myself how to do research and starting up a whole consulting agency around it – it would have been much different if I had been schooled in a particular way in how to do research. I think some things would have been lost because it wouldn’t have been sort of like me and a couple of other people just sort of figuring out how to do something. You’d come up with new stuff. But I think some things might have been gained. I might have more confidence – like I have a Masters in this, so I know how to do it. Or, I did my thesis on this and I am – yeah. There’s a certain amount of imposter syndrome that goes around, even if you have that education. But sometimes when you don’t, and you make it up yourself, you can doubt whether or not you really know what you’re doing. Fake it ‘til you make it.
Steve: I remember just being in a consultancy kind of around that period of time and that every prospective client that came in we would have to write a proposal for what we were going to do. And it just felt like writing a Master’s thesis every time. Or some just impossible thing to give birth to. Like explaining what it was, deciding the steps, trying to articulate those steps. We just didn’t have any point of reference and so we were sort of figuring it out every time until I think at some point we started to settle in and like oh, here’s kind of what – what it looks like. And now like writing a proposal is not hard. There’s many other hard things.
Leanne: But now you have templates and you can copy it.
Steve: And you know – I mean I have a narrative that runs through my head that oh yeah it kind of goes like this. We have more tribal knowledge as well as…
Leanne: Yes. More people to reach out to, to get examples from too, than used to exist.
Steve: Right. In this era, this consultancy, not only were we figuring it out, we were doing so in pretty much isolation. Because you knew a few people, but most of us at that era – there wasn’t a community of researchers to kind of connect to.
Leanne: Yahoo Groups did not exist yet. Google Groups was not there. Google wasn’t even here.
Steve: Yeah. This is great. It’s a very interesting conversation. I learned a lot.
Leanne: It was great chatting with you Steve.
Steve: I appreciate you taking the time.
Leanne: Absolutely.
Steve: Anything else? Are we done? Any last thing?
Leanne: It’s fantastic. Yes. Perfect ending.
Steve: Mic drop. Alright, thank you.
That’s it for this episode of Dollars to Donuts Go to portigal.com/podcast for the transcript as well as links for this episode. At the website you can also check out all the other episodes, and subscribe to the podcast on iTunes, Spotify, and all the usual places. My books are available at Amazon and rosenfeldmedia.com. Our theme music was written and performed by Bruce Todd.
The post 15. Leanne Waldal of New Relic first appeared on Portigal Consulting.
Welcome back to Dollars to Donuts. This episode features Leanne Waldal, Senior Director of Product Research at New Relic. We talk about establishing research in an organization for the first time, building up a diverse set of research collaborators, and the pleasure of taking on certain types of challenges.
I’ve seen hopeful examples in startups recently where even though there are only 3-5 people they bring in someone for research and that’s highly unusual. Usually, if you’re starting up a company you have seed money, or you have your first round, the first people you’re hiring are engineers. You’re not making one of your first 10 hires a researcher. I know some example now around the San Francisco Bay Area where they’ve actually brought someone in whose role is to do research really early in the company. And that either points to a certain amount of humility around, oh, just because I am the user it doesn’t mean I know the users. Or, because they’re going into a space they don’t know. – Leanne Waldal
Show Links* The Secrets of Lyndon Johnson’s Archives, by Robert A. Caro * The Turnaround * Leanne on LinkedIn * Leanne on Twitter * New Relic * Autodesk * Dropbox * Site Reliability Engineering * EPIC * LUMA Institute * RACI model * Berkeley School of Information
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Well, hi, and if you’re a new listener, welcome to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization. otherwise, welcome back to Dollars to Donuts, the podcast where – okay, you got it.
It’s been a little while since the last episode, and I’m really happy to be back making new episodes for you. In a little bit I’ll talk about how you can support me, and the podcast, but first I wanted to mention an interesting article I’m reading. It’s from The New Yorker magazine, from the January 28, 2019 issue. It’s an article by Robert Caro who is an author of among other things, a multi-volume biography of Lyndon Johnson. This article is adapted from his memoir called Working: Researching Interviewing Writing, and his recollections about those activities are what caught my eye. He describes going to do research at the Johnson Library and Museum and going through boxes and boxes of papers. It’s the hard and tedious work of investigative journalism and what struck me was that amount of inferring, and cross-referencing and delving he was doing – he was coming up with facts and perspective that were not there for the taking but were from analyzing and synthesizing the information that he did have as well as what he didn’t have. Later in the article he talks about the importance of “place” in his research, that to understand Johnson he had to leave New York and really spend time in the Texas Hill Country. He writes “As soon as we moved there, as soon as the people of the Hill Country realized we were there to stay, their attitude toward us softened; they started to talk to me in a different way. I began to hear the details they had not included in the anecdotes they had previous told me.” And finally, he talks about silence in interviewing, citing two characters from fiction – Inspector Maigret and George Smiley, and their tactics to quote keep themselves from talking and let silence do its work unquote, where Maigret fiddles with his pipe and Smiley uses his tie to polish his glasses. Caro has his own brilliant technique: “When I’m waiting for the person I’m interviewing to break a silence by giving me a piece of information I want, I write SU (for Shut Up!) in my notebook. If anyone were ever to look through my notebooks, he would find a lot of SUs.”
Speaking of the connections between journalistic interviewing and user research, I want to recommend the podcast from 2017 “The Turnaround” – it’s interviews with different kinds of journalists about doing interviews, about getting to the story. I learned as much from the contrasts between our different objectives as I did from their best practices that apply directly. Check it out!
Now, obviously, I’m back with more episodes of Dollars to Donuts, I’m taking more of an open-ended approach to new episodes, and they’ll appear as they’re ready, without any specific frequency. It takes a lot to do this podcast and rather than taking advertising, or doing crowdfunding, I want to ask you to support the podcast – and me. You can hire me! I plan and lead user research projects, I coach teams who are working to learn from their customers, and I run training workshops to teach people how to be better at research and analysis. You can buy my books for yourself and your friends and your colleagues – I’ve got two books – the classic Interviewing Users and more recently, a book of stories from other researchers about the kinds of things that happen when they go out into the field – it’s called Doorbells, Danger, and Dead Batteries. You can also give this podcast a star rating and a review on iTunes, and you can review either book on Amazon. With your support, I can keep doing this podcast for you.
Let’s get to my interview with Leanne Waldal. She is the Senior Director of Product Research at New Relic. She has led research at Autodesk and Dropbox as well as running her own research agency for almost 17 years!
Thanks for being on the podcast.
Leanne Waldal: Absolutely
Steve: Let’s start with a basic introduction. Do you want to tell us who you are? What kind of work you do? What your company is?
Leanne: I’m Leanne Waldal. I am Senior Director of Product Research at New Relic. New Relic does application performance monitoring. It usually makes people’s eyes glaze over. Basically, we have a tool that developers and engineers, primarily that group, uses to monitor the traffic on websites and mobile apps. So, if you think about streaming media, online retail, financial services – tons and tons of data and traffic – their site reliability team, or engineering team, might use us to monitor that to make sure nothing goes down. I’m based in San Francisco and I’m a part of the design team which is a part of the product team. The product team is mostly in Portland, Oregon. We have a smaller portion of it in San Francisco and we also have a team in Phoenix and in Barcelona.
Steve: Can we talk a little more about New Relic and who the customers are? You said to make sure that it’s not down, but I’m assuming it’s not just that binary state, like is Amazon up or is Amazon down?
Leanne: It’s to monitor it. So, be watching to see if something happens that’s slower. Usually it doesn’t matter if something is faster. It means you’ve optimized performance and things are running really well. So, if things are going slower or you notice that people are having longer wait times for page loads or checkout loads or media streaming, or whatever sort of thing they’re trying to do in your app. You would have spent a considerable amount of time setting our products up, basically, to then do all that monitoring and raise alerts when something goes wrong.
Steve: Okay. So, it’s not up and down. It’s slow and it could be any part.
Leanne: Exactly. Lag time. Checkout’s not happening in the amount of time that you expect it to.
Steve: Did you use the phrase reliability engineers?
Leanne: Yes. So, the new role that basically Google created about 10ish years ago is the site reliability engineer. They wrote a book about site reliability engineering. So, our target user is usually called a dev ops engineer or a site reliability engineer. If you are a more modern company with the way that you’re keeping track of your website and your apps, then you probably actually have a development operations team, or a site reliability team. If you’re not quite there yet, then these people might be a part of an engineering team, or IT operations or something.
Steve: That’s good context. Then you said, you’re part of the design team which sits in the product team?
Leanne: Yeah. So, at New Relic, engineering and product and design all report up to the chief product officer. Different companies do it different ways. Engineering might be alongside product or design. Design might be alongside. But at our company, if you’re in the product org, you belong to the design team, the product team or the engineering team, or a few different teams that are in the org.
Steve: How long have you been in the role that you’re in?
Leanne: I’ve been at New Relic for six months. I did a similar job at Autodesk, before New Relic, for about a year and a half. And before that I had a similar role at Dropbox for a couple of years.
Steve: How do you compare and contrast those three organizations and what you saw and what the trajectory was in those roles?
Leanne: We could start with Dropbox. Dropbox was private, pre-IPO. It was very much a unicorn. When I started there it had 400-600 people. I don’t remember exactly how many. There was no research. They had a small design team. They had, as a lot of tech companies at that time, being a unicorn, had tons of engineers. And also, because Dropbox is a consumer app as well as a business app, everybody used it, so they felt like they knew who the user was.
Steve: Everybody that worked there?
Leanne: Everybody that worked there, yeah. So, I started setting up a research team there and grew it as the design team was growing. That’s different from Autodesk where I went next. A very old company, very fascinating products with fascinating customers and use cases. And people there who were researchers there, who had been researchers there for a long time. And what I was doing there was combining research and analytics together. At Dropbox, analytics was separate. And I had a small team that was more globally disbursed. At Dropbox, my team was all in San Francisco, to start. Here at New Relic, when I joined, there was one researcher here in San Francisco and one in Portland and a similar sort of tone around a lot of the people who work here at the company. Our site reliability engineers (or have been), so they know a lot about the market and a lot about the users. So, we’re basically here to help the company understand all of the new users it’s acquired since it started because it’s a different company now than it was 10 years ago. Ten years ago, New Relic was a company that was mostly reaching out to developers and engineers to use its product and now we’re mostly focused on enterprise and we have lots of enterprise customers now.
Steve: There’s that classic – you referenced this, right. People think they know they’re customer because they are users.
Leanne: Yeah, we all do.
Steve: But you’re kind of saying yes, to a certain extent. I don’t hear you shutting that down?
Leanne: It’s like yes and the marketplace out there has changed and maybe there’s something new that we’re not looking at right now. And also I’ve noticed that for companies where they maybe used to do contextual inquiry and go out and visit their customers and users and understand them deeply, in the last 10-15 years companies have tended to move to surveys and remote interviews and stepped away from that sort of like deep understanding you get from being in a room with someone, or on a construction site, or on the road with them when they’re using your mobile app, or whatever it is. So, you can see everything around them while they’re using your product or your app. And when you move away from that you lose sight of that holistic story of the customer experience. And then research can come in and do that for you.
Steve: Why do you think there is that movement towards the remote and movement away from the contextual work?
Leanne: Contextual work takes time and effort and can be exhausting. Exhausting to do it and exhausting to come back and know what to do with all that data, how to tell a story out of it and how to decide how it can have impact or value? Or sort of the like, now what do we do? I think also as humans we get really familiar with where we are, and we neglect to notice that the world around us has changed.
Steve: So, that’s back to your point then. We started this company based on work that we’d done and skills that we had and users that we knew, that we were, and the world moves on and shifts and new customers…
Leanne: And it’s not just New Relic or Autodesk or Dropbox. That happens at any company. You can be in banking or in legal. Ask anybody who has been there a long time, compared with a person who just started. They’re having much different experiences of it.
Steve: You described, especially with Dropbox and New Relic, this point at which the company had little or no research, and that is not unique. We know lots of companies like that.
Leanne: Oh yes.
Steve: Do you have a hypothesis about what’s going on? There’s a point at which someone like you starts talking to these companies and starts saying, “hey, here’s what I would do if you had me come in and work with you.” What’s going on beforehand? What’s the point of need or pain that’s identified inside these companies when they realize…
Leanne: Sure. So, what I’ve noticed at the companies I’ve worked at, as well as companies I was interviewing at for this sort of position, and companies I know the story around when they hired someone in sort of like a director or higher role in research, usually something changes in revenue and that change in revenue sets forward sort of like how do we figure out what’s going on. And if they have someone who is currently working there in a role who has had experience in the past with doing market research, or doing churn research, then those people will start to raise the flag of like oh, we have to do research. And then it’s how do we do it. So, that’s one reason that someone brings someone in to do it. Another reason is that there’s sort of like a groundswell of engineers, product managers and designers who are telling the people who make hiring decisions, “we need someone to be in charge of design.” Or, “we need someone to be in charge of research. We’re doing a lot of this ourselves and we don’t feel like we’re doing it well enough.” Or, “we, for example, do lots and lots of interviews. We don’t have time to synthesize the data. So, could you please hire a researcher to work with me.” So, sometimes that happens. Another way it happens, which is how it happened here at New Relic, is that the company decides to hire like a VP or SVP of design and then that person who knows, oh this is what a design team looks like – you have someone who is focusing on content design and language and someone who is focusing on interaction design and visual design. And I also need someone who is focused on research. Sometimes it’s from the top down. Sometimes it’s from the ground up. Sometimes it’s there used to be this role at the company before and someone left or was laid off and then they just didn’t backfill it and then after a while realized oh that’s really a role that’s really important.
Steve: We need that.
Leanne: Yeah, and it’s not – I used to think that maybe it was sort of like a trend, like everybody is starting to be customer centric, or user centric, or whatever that means. So, then they were like, “oh, well Joe down the block is doing that, so we need to do this too.” And what do we do, we hire someone to run research. But, as I have been interviewing for jobs over the year and talking to different companies and doing these jobs, I’ve realized, no, there’s actually something that happens inside the company that causes that change. It’s not so much like copying a competitor or another company that you care about. You might do that with engineering or something, but research is sort of like the first thing to go if you start running out of money. And, I’ve seen hopeful examples in startups recently where even though there are only 3-5 people they bring in someone for research and that’s highly unusual. Usually, if you’re starting up a company you have seed money, or you have your first round, the first people you’re hiring are engineers. You’re not making one of your first 10 hires a researcher. So, I know some example now around the San Francisco Bay Area where they’ve actually brought someone in whose role is to do research really early in the company. And that either points to a certain amount of humility around, oh, just because I am the user it doesn’t mean I know the users. Or, because they’re going into a space they don’t know. They just had a really good idea about it and they really need to understand the space.
Steve: That’s an encouraging sign.
Leanne: Yeah.
Steve: Is there a distinction between – you gave a number of different scenarios where research comes in – do you see a distinction in sort of the context, or the action that’s being taken when bringing in a person to do research vs. bringing in a person to lead/manage/run/build research?
Leanne: Yeah. So, if you’re brought in as a person to do research you’re usually reacting to the things you’re being asked to do. You usually don’t get to pull your head up and do strategic work. You’re doing lots of like compare this design with this, do this survey to answer this question, help me with like interviewing these six users in remote interviews. If you’re being brought in as someone to say like where should research fit into the company, it’s more challenging, but more interesting and exciting because then you’re being told like tell us what we should know? Like where should we go with this? Are you going to focus more on the market research side, or the sales research side? Are you going to focus more on the product design side? What are all the things that we need? What kind of programs do we need? How do we get access to users? How do we interact with users and engage users? There’s just all these little pieces of how a research process and practice runs inside of a company that is great fun to put together because you have to do it based on who you’re working with and the culture of that organization. It’s not just a cookie cutter that you put into a company.
Steve: Is there a trajectory from one to the other? Like someone that’s hired because the need is to do research. Is there a trajectory for that person to start answering the kinds of questions that you’re talking about?
Leanne: Yeah, yeah. I think if that’s sort of like the way you work, or your personality, or the way you’re motivated, you can get there. For example, I have someone on my team right now who I mentioned to the other day like – she’s a researcher and I said, “you could make some day, if you wanted to, a really amazing research operations manager because you’re really good at all these pieces around managing the research and communicating it.” Because half of research is PR and sales. You have to find the people to engage with it, to sell it to at the end. You know you want your research to have impact on product or a marketing campaign or whatever you’re doing. So, you have to gather the people around you. And you have to keep track of, particularly in a B2B company, who are all the people who have to be involved if I want to do visit customers. So, in a consumer company you just go out and find people. You talk to them and you do whatever you want with them. At a B2B company you have to be engaging with customer success, with sales roles – whatever they’re called – account execs, account managers – whatever that company calls them. You have to often also be engaging with product managers, designers, design managers. Engineers, didn’t used to be, just as a stereotype, interested in research and now at some companies they are very interested. And so managing all those types of people that you either have to get approval from, or you need to make sure that they’re bought into what you’re doing, or you need them to come along with you and do it, takes a lot of work. So, there’s this whole new role of research operations that usually helps keep track of that. And I’ve noticed some researchers have a real knack for sort of like managing all those details around their project and then shining a light on it and sort of going out and selling it, sort of like a marketing campaign. They just know who to schmooze with and talk with. And some researchers are really good at doing the research and writing the report and sharing it. I pointed out to her, “this is something you are doing for our team without being asked. Like, that’s a real superpower.”
Steve: Does that start to change the – I think you said this, at least indirectly – does someone who is shining around operations, like your colleague here, does that start to change how research is perceived or experienced by others in the organization?
Leanne: Yeah. Especially if you’re an organization that’s not used to research. Or they’re used to doing it themselves. So, they’re not used to partnering with someone else and they’re not used to having someone say like, “oh that’s not really a survey. That’s more of an interview.” Or, “oh, that’s not really an interview. We need to actually go watch people to do this.” Or, “this can’t be answered in an interview survey, or watching people. We need them to be in a diary study.” They aren’t used to having someone bring in all these different methods. And they aren’t used to having someone sort of like find some insights and then tell them. So, that’s why it’s so important to have a collaboration. I want you with me while I’m doing this research so that when I tell you the pens need to be blue, not red, you were there to hear everyone say they like blue better than red, instead of throwing the report over the wall.
Steve: I hadn’t really thought before of operations as a – I’m not sure you’re saying this – but, almost a Trojan Horse. That there are these sort of tactical objective problems to be solved under operations. How are we going to find these people? What is it going to take logistically? What is it going to take legally and so on? That starts to change some of the conversations maybe that can happen internally. Like you said, “this is your problem. We recommend you go about it this way.”
Leanne: Yeah.
Steve: I hadn’t thought of that under an operations lens myself.
Leanne: It’s really more in a B2B company than a consumer company because you have to manage so many relationships and oftentimes a researcher wants to focus on sort of like the details of the research. Like ask any researcher, they don’t want to do all the details around recruiting and scheduling people – or, most don’t.
Steve: Right.
Leanne: And so, you want someone who can do that. And then sometimes researchers also aren’t the people who know how to best sell their work or edit it down into like the three bullet points for an executive presentation. And it’s helpful to have somebody who knows how to do that. You know just sort of like editing and presenting and PR and marketing. And also knowing who to market it to? Like, oh, I know that there’s this sales VP in this office in Toronto who is going to be interested in this. That mentality of like keeping all these dots rolling together. And if you have someone who knows how to do that and also understands how to do research and what goes into research, that’s what I see makes a really good manager from an operations perspective. There’s the people management side too. Being able to mentor and coach and take care of people. But there’s more and more in B2B research a need for someone specifically in this operations role to just sort of like help everybody and raise the brand of the team and the company, or in the product team or the design team.
Steve: It’s so cool that we’re at a point in this field and this practice where we acknowledge, as we always did, that there’s a number of very different kinds of skills and expertises that are required. And you just outlined a whole bunch of them. But that we’re able to sort of say, and not be laughed out of the room, like oh, that might be different people. You talked about collaboration and collaboration with people with different strengths or superpowers.
Leanne: Yeah. And another expertise and skill that’s being introduced more into companies is the academic anthropologist who is now working inside companies on a research team. Like EPIC used to be primarily all academics and now there’s a bunch of people from business involved in EPIC, and also people who used to only be academic anthropologists and ethnographers working on research teams, inside tech companies or consulting agencies, or whatever. So, I recently hired a few anthropologists on my team. One who is a former professor, one who had applied anthropological research practices for market research. Another who had done research with an NGO, but also worked inside of a company recently. And I think that when you have that academic background and you bring it into a company, it’s like a special sauce. You also bring in a special sauce if you’ve been working in tech for 15 years and you know the ins and outs of how a product team works, but I like mixing those two things together because people learn off of each other.
Steve: You like to mix the special sauce.
Leanne: Yeah.
Steve: So, it’s an extra special sauce.
Leanne: Yeah. So, for example, we decided as a team that we wanted to look at all of the internal blog posts and other things that had been written about customers and users, from the point of view of people inside the company, because my team is new to this company. We were like okay, let’s take a look at everything that everybody else wrote. And I was trying to figure out how we would do that. Like, who knows how many internal blog posts there are and how many things that have been written about customer visits and things that different people know about users that they report out internally. And then I realized, I have some former academics on my team, academics who are now working as researchers in industry. They have skills in literature review. So, brilliant. I took them and had them put together like how are we going to review this and code it – special sauce, like they absolutely know how to do that. It was really easy for them to put it together. And then took other people who had more industry experience, or application focused experience, and said, “okay, now you can do what was set up. You can do the review of following the system.” And I have people who know how to set up systems set up the system.
Steve: So, that’s an example of the special sauce. They know how to deal with this.
Leanne: Yeah. And if you have a team who has mixed experiences like that, then they just sort of like, they lift each other up and they teach each other things. And that’s something that you can’t learn if you’re the only researcher, or you’re just one of a few researchers who all have exactly the same background. So, you all come out of the – whatever the programs are now, at Carnegie Mellon, Stanford and Berkeley and University of Washington and on and on. And you go and get a job and you do remote interviewing and some focus groups and some other things But everybody around you is doing exactly the same thing. Then you don’t get a chance to sort of like up level yourself. And you also, like if you’re the person managing that team, you don’t get a chance to learn new things from people on your team that you hired, that like you don’t have the same background as them. So, it just improves humanity basically.
Steve: So how do you think about like what the – if the team is kind of eclectic now and you have all these different mixes of skills and backgrounds and aptitudes, how do you think about what the collective should be? How do you think about the mix?
Leanne: Of a whole team?
Steve: Yeah
Leanne: So, I do miss the team I used to have which had both research and analytics. I think being able to have people on your team who do data science, or who are more like business analysts who know how to count things up and make charts, or someone who knows how to mix data together and come up with reports out of it, you get a much faster feedback between the analytics people and the research people. Analytics is like oh I did this, I want to know why? Research is like I can go out and find out why. Or research says, “I need some access to people who have certain characteristics, can you find me a list of them through analytics? Thank you so much, I’ll go out and talk to them and then I’ll come back to you.” In most companies those roles are separate. So, it is here. Analytics is a separate team. We work super well together, but sometimes when you have them both on the team it’s really nice because you’re both at the same team meeting and you’re both seeing what they’re doing. From a research perspective, having people who are all quant or all qual, or halfway in between or know both, is useful because anybody who’s in the job market right now, when they look for their next job the requirements are going up and up and up. We don’t want you to just be somebody who does interviewing. We need you to know how to design a survey. We need you do know how to do SQL queries. We need you to know how to do – like there’s more and more requirements showing up in job descriptions now. And so, if you can be a part of a team where somebody else knows that thing you don’t know, it just – we’re all going to like leave and get another job someday. Nobody stays somewhere forever. So, it will just help you when you’re going out to look for your next job.
Steve: So, skill development, kind of through proximity to your colleagues.
Leanne: Yeah. And it’s also diversity. Like any time, you have a diverse team, whatever you’re defining diversity as, just makes for a better team environment because all these different perspectives around the table from all the different places they came from, just makes for like different things to feed into feedback about a project you’re working on, or something that’s going on in the company. You know, we didn’t all come from the same school or the same program, so we don’t have a difference of thought in the way we approach a problem.
Steve: So, for you, thinking about the teams that you’ve put together – because you’re talking about different kinds of – I don’t know if diversity is the right word, the way you’re using it, but qual and quant are different orientations, or…
Leanne: And also, academic and industry. Or in tech, because I’ve worked in tech for 20+ years, I look for people to come work at the tech company I’m working at who haven’t necessarily worked in tech before. So, look for adjacencies. Like you did research, but you did it for a car company. Or, you did research, but you did it for an agency. So, that they can bring different things into the team. That’s what I mean by diversity. You know research, but you used to be a professor. You know research, but you were working on an astrophysics PhD and decided that’s not what you wanted to do anymore, and you realize things that you learned as a part of your graduate program – so you totally know how to do surveys and how to talk to people and how to do research.
Steve: I had this Twitter conversation with someone the other day and they asked me where is a writing course that I can learn to be better at writing the kinds of things I have to create as a researcher. They can’t find things like that. They can find persuasive writing which is for sales. I kind of scratched my head and said, “I don’t know that that exists.” I went into this little sort of pontification, as Twitter invites you to do, about learning from adjacencies like take a creative writing class. Take a journalism writing class. And in doing that, you would start to see, oh here’s how the lessons of doing this adjacent thing well could apply. That was the best advice that I had. I feel sort of affirmed, from what you were saying, that adjacencies for certain kinds of things bring a certain kind of value.
Leanne: So, the Luma Institute, who put this framework together around design thinking methods so that it’s easy to teach and easy to learn, calls that alternative worlds. And I think that’s a useful way to think about it, that for your person who is trying to figure out how to write a research report for whatever audience they’re trying to serve, well that’s storytelling. It’s sales, it’s marketing. It’s slide development. It’s all these other things. And when you start thinking of it as how would I think of this from the point of view of a salesperson? Or how would I approach this from the point of view of a marketing person who has to put customer stories on the web and sell that to somebody? Or how would I do this as the point of view of the person I am presenting this research to? Like what do I want to hear? It gets you out of the sort of like oh I have to have 10 slides and one has to have my methodology and the end one has to have like recommendations. You can start thinking about it more as like I want to tell this story so that they land on that naturally. Or, they ask me questions about it when I’m done that will answer all those things, so I don’t have to have it all in my report.
Steve: I feel like this is a transgressive notion of expertise though, especially in tech.
Leanne: Yeah. It would be nice for all researchers if they could just come in and do their research and research was valued and anything that researchers said they were learning from customers/users was taken and put into the roadmap. I don’t know anywhere that happens. And because there isn’t anywhere that happens and the way the culture in these teams work, you have to have this sort of like marketing and collaboration and partnership mentality around your work. If you’re a consultant, you get a project, you do it, you collaborate, you sell it a little bit – you’re done. You don’t have to still hang out and make sure it’s being used and follow-up.
Steve: It’s just all sunshine and roses being a consultant, right!
Leanne: Exactly. That’s why I don’t do it anymore.
Steve: I had a coach advising me, in my consulting business, and what they said was – they said many things, but one of them was anytime you start talking with a prospective client, show them – show me how you’ve helped someone like me. That adjacency kind of framing – the advice was that wasn’t as persuasive. So, that could be what have you done in my industry, in my vertical?
Leanne: Yeah.
Steve: You know, and I think researchers, or anyone that just likes to sort of – that gets deep into different kinds of problems, you know we see connections between things that are – you know we know why this problem is like this other problem, even though it’s different.
Leanne: And you want the other people to see that and sometimes it’s hard to show them a lens that will help them see that the way you saw it. Yeah.
Steve: I think the advice was that’s not where you start the relationship. You start the relationship a little more close in. “Oh, this thing that you’re doing. Yes, I have done that.” Even though every problem is new in ways that – it’s very hard to do that.
Leanne: Yes. Yes.
Steve: Sort of starting the – showing relevancy right at the outset of the conversation.
Leanne: When I ran a consulting agency, that’s how I started it. Somebody asked me if I knew how to do something and I’d never done it before and I said I knew how to do it. And all of a sudden, I had a consulting.
Steve: So, your career is just built on lies.
Leanne: Yes.
Steve: Is what we’re saying here, okay.
Leanne: Yes.
Steve: And we’re leaving this in. Yeah, well I think that’s true too. There’s the confidence to explore. I mean can you do this? Yes. Have you done it? Well, no, but I can do it.
Leanne: Yeah. And sometimes the question never comes up, have you done this? It’s just the can you?
Steve: So, that’s – so, we’re talking sort of requests in the consultant side of things. Inside organizations, in any of the ones you’ve worked in, like how are projects formed? Who decides?
Leanne: So many different ways. So, I would say that if you’re beginning research in a company, you’re taking requests and you’re probably using your boss or some other key stakeholders you recognize to decide which to work on. And you’re probably prioritizing those in some sort of backlog, if that’s in a spreadsheet or it’s a Jira or whatever. So, you’re showing people sort of like here’s what’s in my backlog. Here’s what’s coming up next. If you’re a team, the way I prefer to run a research team, which is the way I’m gradually getting it set up here – how it was set up at my last couple of jobs – is to have researchers who are paired or embedded, or whatever you want to call them, but they’re primarily working with a product area. A product team, a feature team, a product area, however your organization is set up. And they are servicing that team and partnering and collaborating with that team. And they own the research for that. But because they’re a part of the research team they also have a holistic view across the organization, because they’re also meeting with other researchers who are doing the same thing and seeing where their work, or where their product area or features sort of overlap, or interlock with each other. And then having some people who are more senior, principal researchers, just working on special strategic projects. So, that could be, depending on a company, that you have someone who is more focused on market research or competitive research. It might be someone who is focused more on research to understand where we should go next? So, sort of like future casting and sort of like looking at what’s going on, finding edge cases, seeing spots for innovations and other sorts of insights. And then also the operations side. People to manage panels of users and recruit people. And then there’s always, depending on how many people there are, some sort of management tier for people management and coaching and that sort of thing.
Steve: And when you talk about embedded you – it’s certainly a frame that you hear described with some regularity, but your words are a little different. Some people talk about embedded, that that researcher is on that team, sort of fully on that team. You’re almost describing like a workstream or a customer or something.
Leanne: Yeah, because the way people work on teams now is not the way that HR software recognizes. So, HR software, for example, recognizes I have a boss. He’s a VP of design. Recognizes I have people who report to me, all the people on my team. Um, maybe recognizes that I have people that might be good for 360 reviews. So, my direct colleagues who also report to my boss. But, that’s such a limited view of what I call my team because – I’ll just talk for me. Like I work with lots of different people around the company who, if I was going to be reviewed, I would want them to weigh in. People on my team – actually, we had a research team meeting this morning. I had them all bring a stakeholder map because I wanted to see – now that I have people that are sort of like embedded or paired within teams, like who are they working with and what does it look like, compared with people on my team whose work is sort of nebulous around all the different people that they’re managing and all the different work that they’re doing. And it turned out that way. So, if you’re a researcher on a team – a lot of people refer to the three-legged stool, but if you have a researcher it’s a four-legged stool. So, you have someone from research, from engineering, from product and from design, all working together with like a common focus on how are we going to get this out the door and make sure it meets customer needs, solving a problem, all that sort of stuff. If you are somebody in operations or a principal researcher working on a strategic project, you just have a mess of people around you. And so, all these people sort of like vacillate in between teams. Designers do the same thing. Designers are a part of the scrum team and they’re a part of the design team. Engineers are a part of a team for some sort of product or feature. They’re also a part of an engineering team. And if gets even more complicated when your manager is in one city and you’re in another city. Because if you’re enough time zones away you probably have somebody who is sort of like your dotted line manager in that city to help you with like all the things around your job. For example, I have a researcher in Barcelona. We can’t look over each other’s shoulders. There’s only two hours a day when we’re both at work. So, there’s a director of design in that office who is her sort of – who is her manager there. So, if she has questions about how does this work? Or how do I do this? Or who is in charge of this? Or help me review this? He can do that. But no HR software recognizes that. It doesn’t recognize she basically has two bosses. It doesn’t recognize that I have one boss, but actually these three other people are kind of my bosses too. It doesn’t recognize that that designer has a boss for whoever is running that scrum team, but that designer also reports to a design manager or director. So, most people work that way now. There’s very few people inside companies anymore who will say they belong to one team and that’s all they belong to. Everybody belongs to like multiple teams and we have this archaic structure around this person decides like your performance review and hopefully gets input from all these people. But, just lines people up in like lines and doesn’t realize that it’s a little more messy than that.
Steve: So, does that gap between the reality and the structure of the software, what’s – are there consequences to that for how we work?
Leanne: I don’t think there’s consequences. I think it’s just the way it is. I think there’s more 360 reviews now, just in my experience, then there were 20 years ago, because you work with so many other people now. I can remember jobs from early to mid-90’s. Nobody was doing 360 reviews of any of the jobs I had then. Like I worked with a set of people in a geographical space. I didn’t work with anybody else. I think it’s just more like it is what it is and things have come in to help manage and understand that, but we, the people who sort of like do to the sort of organizational management, haven’t figured out how to accurately represent that in a structure of the software that manages the company. Some companies try to go – what was it, like Zappos, that decided nobody had a manager. That seems like part of a reaction to that – Holacracy, that’s what it was called.
Steve: So, we got into this because we were talking about what does it mean to be – there sometimes seems a tension between researchers are embedded, or researchers are centralized. And I think what you’re saying is they are neither, or both. And that that’s how work happens.
Leanne: Yeah, and you have to look at the culture of your org. How does your org work? Who wants to participate and who wants – it’s that old RACI model. Like who wants to be involved? Who wants to be informed? Who wants to partner with you?
Steve: So, wait.
Leanne: Do you know the RACI model?
Steve: I know it, but I need to clarify it.
Leanne: It’s where you define – I might not get this right. It’s like where you define who is responsible/accountable, who is interested in consulting on it and who just wants to be informed. So, if you’re thinking about a project, you always have people who sort of fall into those categories. So, I just think it depends – some cultures are very top down and as you go up the ladder they only want to be informed and not as involved. Some cultures are more participatory and as you go up the career ladder, everybody wants to participate and know and be involved. So, it just depends where you land and get a sense of who just needs an inform, who actually wants to come along and sit with you while you do a remote interview, and travel 3 blocks away to visit the customer. You know how far up the ladder does that go?
Steve: So, coming into these organizations that have done very little with research to date – like understanding the culture, it seems like that’s got to be an initial step to get the lay of the land?
Leanne: Yeah. Who’s making decisions and what data are they using to make those decisions? Do they want any more data? Usually the answer is yes. Anybody wants to know more to make better decisions and, in a corporation, make more money. So, figuring out who those people are, where the gaps are, what they need to know. And then another thing you can do when you start at a company is be the researcher yourself of that org because what you’ll start to notice, just like any research project, is that you’ll start hearing the same thing from multiple people. So, you’ll start hearing like, “the one thing we’ve never figured out is whether people really like blue or red better.” You know there’s always something that’s like the one thing where people aren’t aware that everybody has the same question, but because you got them to open up and talk and had casual conversation and used all your interview tips and tricks to interview multiple, different people who don’t work with each other, you’ll start to see a theme around something that everybody wants to know, but nobody knows, but nobody is like trying to figure it out because they don’t know everybody else wants to know. I’ve seen that at every company. We’re working on a project like that here right now where there’s basically two things that I heard across the entire company when I first came here. So, I was like okay, that’s where I’m going to start some key research projects and then I’m going to bring in some people who are interested in it. And then as I hire people in, I’m just going to start attaching researchers to different product areas and work with the people who run design to sort of think about which designers need the most help right now. Or, what meets the business directives, or the key priorities, or the goals, or whatever – the product org of the company.
Steve: Are there stages of evolution? I don’t know if it’s milestones or some framework that at these points you’re kind of coming in where there’s very little and you start building in a way that is specific to what you’re learning about the culture, are there stages that you could identify that you’re passing through or moving towards?
Leanne: Oh, yes. So, once you get sort of like programs and tools set up you don’t really have to do as much anymore. So, programs and tools like is there decent survey software? Do we have access to users? How do we access them? Is there a panel? When we do research, do we have standard agreements that we sign with the humans that we’re doing research with? Repositories – where are we storing everything? How are we storing it? How are we accessing it? Do we need transcripts? Do we have a tool for that? Do we need to do card sorting? Do we have a tool for that? Just sort of like sorting out all the tools and processes for the practice. What was the question? I got lost on a rabbit hole of process and tools.
Steve: You were answering the question. So, what are the stages or milestones?
Leanne: Oh yeah, okay. So, looking at what’s needed that is more like capital improvements. So, like we need a new roof. Okay, we’ll get a roof, it will last for 20 years. Done. So, things like how you pay incentives and research agreements, that’s sort of like the roof. Then you need like everything that pays the electric bill and the gas bill all the time. So, those are projects. So, what projects need attention right now. And you learn that from interviewing sort of like all the key stakeholders you’re going to work with, which in my example earlier was once you interview a whole bunch of them you start to find out that there are some key things that everybody wants to know. And then figuring out, either with your boss, if she or he knows, or with key stakeholders, like if I was to start bringing in researchers one at a time, where would we start? Who would benefit the most? What would be most impactful? Or said another way, sort of like what are the things launching soon, or later? What are the business goals, or what’s in the strategy? Or what are the business directives? Or all those sort of things that make a company run. And that helps you because then you’re making sure you’re focusing on something that is actually going to eventually affect the company’s bottom line. So, when the research comes out and it helps that, then you’ve been like okay, this is how we do it at this company. We’re not academic. We don’t go sit back and sort of think about things and just, you know, go research things out of curiosity. We look at where the company is going to make sure we’re helping it and helping it do better with the research that we’re doing.
Steve: And then what’s a five year – that’s a presumptive question. They’re all presumptive questions. What’s sort of the horizon for which you would have vision? The time horizon?
Leanne: For the times that I’ve come into a company and there’s been no research, or one or two, like here, individual researchers, what I’ve told people is it’s going to take 12-18 months, with budget and with head count, to get this into a modern research practice. And here’s how we’ll do it incrementally and here’s how we’ll check in and benchmark and measure it along the way. When I was at Autodesk and I inherited a team and then I sort of hired in and added on to it, it was much different. These were really old products that had a really amazing install base. And some of them were market leaders. It wasn’t like companies that were two years old, five years old or ten years old, or fifteen years old. Like, once you’re at 30 years you’re settled in a certain way. So, I did different things. I tried to better understand the products and better understand the company to see where we would could add value. And then also, because that was a much larger company, see where research was happening across the company to see if we could sort of like combine efforts.
For example, I went to some company that’s like 25 years old, that has say 1,000 researchers. Because there are some companies out there now that have that many. And I was plopped in as a director of research with 20 to 50 direct reports. It’s a much, much different role than coming into the company and setting something up. They’re you’re coming in and you’re saying how is this done? Is there any way it could be improved? Is there any way I could sort of like do – no, there isn’t? Okay. So, how do we prioritize things? How do we have impact? How do we collaborate and share out and be good corporate citizens and be a part of the team?
I’ve realized, with this third job that I’m on now, after doing consulting for many years, that I do really like growing things. There’s a challenge to it that like gets me into work every day. Like, oh, we have to figure this out. Oh, there’s lots of problems to solve. Oh, how are we going to prioritize this? Or how are we going to solve for this? You know, how am I going to find someone to fill this role?
Steve: You just listed a bunch of things as positives that someone – if you’re in a different emotional state or a different sort of mindset – could frame those all as negatives.
Leanne: Yeah. Yeah. You have to understand the challenge you like to work on. Is the challenge I want to be the person who works with the team to help them figure out what they need to solve? Or do I want to solve an organizational challenge while also working on business priorities and solving problems?
Steve: So, this is something that kind of sparks you?
Leanne: Yeah. And I didn’t realize that until – it was like 3 months ago or so. We hosted a breakfast here and somebody was asking me what I was doing, and they were like, “Well it sounds like you’re really good at that because you’ve done that three times now, and you started your own company.” And I was like, oh, that’s the first time I realized like this is my thing that I like to do.
Steve: Yeah. That’s great.
Leanne: Sometimes you have to be almost 50 to have finally found yourself.
Steve: Right. It’s the journey, not a destination, right.
Leanne: Yes.
Steve: I’m sure you’re going to keep finding yourself.
Leanne: Yes.
Steve: Anything else that I should have asked you about that you want to share with us? The collective us. I know it’s just you and I in a room.
Leanne: Well, there is one other thing I’ve noticed in the past few years. I have used a community platform, that’s sold by a company and meant for marketing and brand teams, and have used it for product and design research. There’s something about – so, support always has its help forums. Marketing and brand usually do focus groups, but now sometimes they run online communities where you can get badges and win things for participating in their research, or telling stories, or whatever. There’s a space in between that people who are on research teams at companies sometimes use and sometimes don’t. So, a typical team might have a panel, particularly if you’re B2B and it’s hard to get access to people. Or even if you’re a consumer-focused company and you just want to make sure you have like your 10,000 people who you can access at any time to easily invite to research. But I think there’s something about the way that people on social media interact with each other in a group in one-to-many fashion and many-to-many fashion that is happening in some ways in research teams, and sometimes not. So, if you’re a researcher and you can, at any time, poke into a community that belongs to you – this is my panel, but your panel, they can’t all see each other. In a community they can see each other. And so, it acts a bit like a support forum, a bit like this marketing and branding, sort of like what you think of our new colors sort of thing. Or this is our new video for our marketing campaign. But instead it’s for product and design. I’ve noticed there’s a certain fear of it – oh my God, if we get our users and customers to talk to each other, what will they say about us? But I think if you get over that fear and you open it up and put together a private space where they’re all talking together and you and your product design, engineering, marketing sales, whatever partners are also in there, you create this sort of like brand loyalty and product loyalty that you don’t necessarily get just from marketing brand and sales and just from support. So, I’ve done that in my last two companies, created this sort of community where people could see each other, and we could interact with them in little like pop-up sort of projects. Haven’t done it here yet. But when I talk to the vendor that I’ve used for doing this, the last time they demoed it for me, I saw that the examples they were giving me were for a product and design team. So, I was like huh, you’re now selling to product and design teams. So, there’s something going on with people picking that up and doing that.
Steve: How do you set people’s expectations, the people that are participants in this community, for what – as you said, it has elements of other kinds of communities that are maybe more familiar…
Leanne: Well, you have to make sure, particularly in business settings, that – you know don’t share anything that you wouldn’t want your competitors to see, because you don’t know if your competitor is in this community, or not. We’re focusing on sort of like your work and your use cases of these sorts of technology. Don’t disclose anything that you shouldn’t be disclosing. But, if you’re in a space where people are using a technology where they get a lot out of finding out how other people are using it too, then you create relationships with each other and then you create your own relationship with them. Another downside is you can get bullies, so you need to have guidelines. We can kick you out if you behave in one of these ways. And with panels, because it’s the company to the user or customer, and that’s it, it’s harder to find sort of like an intrinsic incentive for someone to participate in research. In a community you usually don’t have to have any incentives that are hoodies or cash or gift cards or whatever because the intrinsic incentive is that they’re getting value from other people being in there. And then every once in a while being able to talk to a product manager or talk to a designer or talk to somebody else. So, I think that – I don’t know if that’s something that will keep going, but I see it as a way, particularly in B2B research, to provide value for your customers that actually helps out your sales people and gives your product people and design people a way to interact with users and then also give the user support that they’re probably also getting over in the support forum, but a different kind of support because you feel like you’re part of a group.
Steve: It’s compelling to me to think about just blurring the different kinds of interactions that companies have with people.
Leanne: Yeah
Steve: Like you said, there’s marketing, there’s research, there’s support and we structure the companies around those kinds of functions and the tools and so on.
Leanne: Yeah. And it can make it hard to see the end-to-end customer experience. Yeah.
Steve: What – maybe we can rewind. Like how did you end up in research as a profession?
Leanne: Back, when I graduated from college, in the early 90s, I went to work for a startup cellphone company as a statistician. My background is in statistical computing and economics from my university education. So, I was analyzing what were then considered very, very large sets of data – terabytes of data. I was running a neural net program on a Sun SPARCstation. I don’t even think it had a gigabyte of RAM. Yeah. It had two processors. It was amazing. It would take like 4 to 6 days to run something, to spit out a model. And if I got it wrong I had to do it all over again. What I was doing was I was working in a marketing organization and I was there to help them like figure out things like who should we sell call waiting to? Like should we just sort of try to sell it to everyone? Or should we target people? So, I was figuring out targeted lists based on behavior that we saw in cellphone use data, to help the marketing team understand who to sell things to. When they did that, what I realized was I didn’t know why the people were buying it? Like, just because I could predict like because you have these like usage analytics, it makes you more likely to buy, but why is that? And a couple of things happened then. The company I was working for got bought by AT&T, became what is now AT&T Wireless Services. When you get bought by a much larger company, a bunch of chaos ensues. Your job often changes. Your boss often changes. I was very young. I was in my early 20s. And at that exact time Mosaic came out and the web became graphical.
So, there were all these things happening that occurred at this point in my life where I was like, huh, this is my first job out of college, but now I’m interested in knowing why these things happen? I just taught myself HTML. I figured out how to put together a webpage. Oh my goodness, there’s this thing called Match.com and I met this girl and she lives in San Francisco and I want to date her. Oh, I’ll just go take this job at this company in San Francisco, this web company that has like 20 people and they’re making websites and they’re from all different sorts of places. And I became a QA manager there. Once again, sort of being asked, “can you do this?” Not, have you done this. And I said, “yes.” And then from QA realizing that we didn’t really know what the users were doing. And I, at this point, had never heard of user research. But I was friends with people who were doing that at different companies. So, I asked them about it. And then I left that company and went to a startup. Was also a QA manager. Got laid off when the company didn’t get its next round of funding and went and did some consulting. Doing like web development for the summer. Set up a company that I had intended to be a company that did QA for web apps and somebody came to me and said, “can you do some researcher interviews with users while you’re doing browser testing?” And I said, “yes, we can.” And I was like here’s my chance. I’ve always wanted to know why people are doing things. And so, I happened into it that way. And in the mid to late 90s anybody could do that. You could teach yourself PHP. You could declare yourself to be a web marketer. You could say I’m an information architect. Like you just said what you were, and you could just do it. So, I quickly went and found some friends and said, “I have this consulting project. I don’t know what to do, help me out.” And they were like well you do AB…like the things I just mentioned. You have to do A and B and C and D. I was like okay. So, I did A, B, C and D. I did the project. I got paid. I thought it was amazing. I was like I want to do this again. I was like a kid who gets ice cream for the first time. Like, I will have more of that. And that’s how it got started. In consulting, what you need, from my experience, is you need one or two projects as reference projects and then that helps you get the next project and the next project and the next project. And then it was 17 years later.
Steve: Wow. You’re saying that there was a time when you could sort of declare yourself to be this thing.
Leanne: Yeah. When the web was new. When this new technology was coming out and consumers were grabbing onto it and companies were seeing that money could be made there and – people think now that there are more jobs than people, but then there really was more jobs than the labor market could supply because the labor market didn’t have the skills and the education system wasn’t educating people with those skills. For example, in looking for research interns for this coming summer, I interviewed people who were getting degrees in user research. Like a Bachelors of Science in user research. And I was like wow, like companies are now preparing people and there’s a whole education system out there in these D Schools and I Schools and everything now that didn’t exist in the 90s.
Steve: Right. I sometimes feel like oh I’m – ‘cuz I came up through the same system, or lack of systems, that you did.
Leanne: Yeah. If I wanted to do this job now – say, if I had gotten my Masters at the Berkeley I School, or whatever, and I went and got a job as like a senior researcher at a tech company, and I worked myself up, way up, it would be much, much different than my experience of teaching myself and figuring it out.
Steve: What’s lost or gained in that evolution, if anything?
Leanne: Well, I can clearly see that my sort of like – YouTube didn’t exist then, but it would have been like the university of YouTube, the way that I went about teaching myself how to do research and starting up a whole consulting agency around it – it would have been much different if I had been schooled in a particular way in how to do research. I think some things would have been lost because it wouldn’t have been sort of like me and a couple of other people just sort of figuring out how to do something. You’d come up with new stuff. But I think some things might have been gained. I might have more confidence – like I have a Masters in this, so I know how to do it. Or, I did my thesis on this and I am – yeah. There’s a certain amount of imposter syndrome that goes around, even if you have that education. But sometimes when you don’t, and you make it up yourself, you can doubt whether or not you really know what you’re doing. Fake it ‘til you make it.
Steve: I remember just being in a consultancy kind of around that period of time and that every prospective client that came in we would have to write a proposal for what we were going to do. And it just felt like writing a Master’s thesis every time. Or some just impossible thing to give birth to. Like explaining what it was, deciding the steps, trying to articulate those steps. We just didn’t have any point of reference and so we were sort of figuring it out every time until I think at some point we started to settle in and like oh, here’s kind of what – what it looks like. And now like writing a proposal is not hard. There’s many other hard things.
Leanne: But now you have templates and you can copy it.
Steve: And you know – I mean I have a narrative that runs through my head that oh yeah it kind of goes like this. We have more tribal knowledge as well as…
Leanne: Yes. More people to reach out to, to get examples from too, than used to exist.
Steve: Right. In this era, this consultancy, not only were we figuring it out, we were doing so in pretty much isolation. Because you knew a few people, but most of us at that era – there wasn’t a community of researchers to kind of connect to.
Leanne: Yahoo Groups did not exist yet. Google Groups was not there. Google wasn’t even here.
Steve: Yeah. This is great. It’s a very interesting conversation. I learned a lot.
Leanne: It was great chatting with you Steve.
Steve: I appreciate you taking the time.
Leanne: Absolutely.
Steve: Anything else? Are we done? Any last thing?
Leanne: It’s fantastic. Yes. Perfect ending.
Steve: Mic drop. Alright, thank you.
That’s it for this episode of Dollars to Donuts Go to portigal.com/podcast for the transcript as well as links for this episode. At the website you can also check out all the other episodes, and subscribe to the podcast on iTunes, Spotify, and all the usual places. My books are available at Amazon and rosenfeldmedia.com. Our theme music was written and performed by Bruce Todd.
The post 15. Leanne Waldal of New Relic first appeared on Portigal Consulting.
In the final episode of the season I speak with Monal Chokshi, Head of User Experience Research at Lyft. We discuss traditional paths to a user research career, creating routines for meeting different types of users, and the emergence of leadership roles in user research.
The post 14. Monal Chokshi of Lyft first appeared on Portigal Consulting.
In the final episode of the season I speak with Monal Chokshi, Head of User Experience Research at Lyft. We discuss traditional paths to a user research career, creating routines for meeting different types of users, and the emergence of leadership roles in user research.
As researchers we love having curiosity and it’s fulfilling to do the hands-on work. That’s what makes us passionate about research, but in a business one of the dangers is getting too far into that fascination, but then not taking that next step. Maybe that’s where managers can help with making sure that the data and the analysis and then these insights and actionable recommendations are affecting product. We’re not just doing the research and being fascinated for our own curiosity and fulfillment, we’re making sure it’s put to good use. – Monal Chokshi
Show Links* Monal Chokshi * Follow Monal on Twitter * Lyft * Lyft partners with Didi * John Zimmer and Logan Green * Commodore 64 * VIC-20 * Trilogy * John Morkes * Jakob Nielsen * Sun Microsystems * SoundCloud * Terry Winograd * Don Norman at UCSD * General Assembly * Lyft’s partnership with National MedTrans Network
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Thanks, Monal for being a guest on Dollars to Donuts.
Monal Chokshi:Yeah, sure. Well first Steve thanks for having me on your podcast episode. Really excited to talk more about user experience research at Lyft. So my name is Monal Chokshi and I’m the head of user experience research at Lyft.
Steve:And if I didn’t know what Lyft was, what would you explain to me?
Monal:Yeah, so Lyft is an on demand transportation service. So for those not familiar, basically all you have to do is download the Lyft app, open it and tap a button and basically you have a driver ready to drive you wherever you want within minutes at your doorstep.
Steve:Where does Lyft operate?
Monal:So the headquarters are in San Francisco. We also have offices in Seattle, Nashville and New York.
Steve:And where do you have drivers?
Monal:Drivers all throughout the country. I believe we are now in over 190 cities.
Steve:This is an American operation?
Monal:Yes.
Steve:So drivers in the United States right now. Is there an international goals for the company?
Monal:Well right now we recently in the last 6 to 9 months announced some partnerships with international ride sharing companies. For example with Didi which is the largest ride sharing company, similar to Lyft, in China. And Grab which is in Southeast Asia. I believe they’re based in Singapore. So the idea being that travelers from the U.S. would be able to travel to one of these places, open the Lyft app and call a car. It wouldn’t be a Lyft car. It would be one of these folks from our partner companies coming to pick them up. So we haven’t launched that yet, but that is a partnership that we’ve announced and we’re really looking forward to getting that going.
Steve:It makes me think of like code sharing and airlines where you can purchase through the sort of front end that you’ve always used, but the service is delivered through a partnership with somebody else. Is that a, is that a – does that comparison apply to something like this?
Monal:I think it might. I think it’s a pretty good way to think about it from an analogy.
Steve:I’m sure there’s probably better analogies for how products and services in one market end up in another, but that’s the best I could think of on the fly. That’s really interesting.
Monal:Yeah.
Steve:And I’m sure there’s going to be design challenges – what does that experience look like in that different situation.
Monal:Yeah, we’re really excited about designing this new experience just because of the various challenges that come with different cultures and different transportation schemes for various cities, especially international. As we all know the cultures, the modes of transportation are different from city to city in the US and then overseas there are various challenges with language as well. So there’s a lot of really exciting things to think about.
Steve:I love like the really specific use case you’re looking at now which is I’m from the United States, I use Lyft here. I’m in a different context so how can I bring access to that service with me. It makes me think about what people that use Netflix complain about it or I think I hear this with the iTunes store. For example, you can’t get the same access when you’re in a different place. So this gives you – I’m sort of imagining it gives you – you can sort of bring your Lyft travel experience with you into these other environments.
Monal:Yeah. And people use and are comfortable with Lyft here. We want to be able to make sure that they have the comfort of the same kind of service over there, something that they can trust and feel good about. Something that’s familiar in an area where everything seems unfamiliar to them.
Steve:Can you talk about the work that you do here and what is research like at Lyft?
Monal:Sure. So research at Lyft has always been part of our core DNA in a sense. The founders, Logan and John, are really focused on treating people better and that includes all of our users, our drivers, our passengers and so really understanding who these people are and figuring out how to create the best user experience for them has always been top of mind here. Before I joined we did do various research activities and there were several initiatives happening at Lyft. Once I joined, I joined to sort of make that part of our regular product and design processes and so the kinds of research that our team does really runs the whole gamut of what UX research can do. We do everything from field studies and surveys to user and usability testing of new designs and products. So yeah, it’s been a really fun place to work, just also because user experience is so well regarded and supported here.
Steve:When did you come to Lyft?
Monal:I’ve been at Lyft for about a year.
Steve:So you’re describing a little bit about how – you know what you were brought in to do, to sort of – I don’t think you used the word formalize, but that’s sort of the word that comes to mind. I guess is the question about – you know what that change has been that you’ve helped drive.
Monal:Sure. I think a lot of the research previous to when I joined hadn’t been formalized in a sense that now it’s much more a part of the regular, daily, weekly process. And I’ve also joined to basically build a world-class team of user researchers so that – you know Lyft has grown so much in the last year and we are continually growing at a very increasingly rapid rate. So there are a lot more needs that we have from a research perspective and I think as we grow the company there are more needs for insights. So, bringing that to the table and helping build a company that’s very data and insights driven is a priority for the company.
Steve: I’ve seen sort of two different modes of how insights – I’m sure there’s many more, but two kind of primary modes in terms of how insights can impact decisions. One is sort of the reactive and one is sort of the proactive. And you know where reactive is somebody comes to researchers with a question and says ‘Hey, can you help us make a decision about this?’ and the other is researchers often who have done research are evangelizing those insights to try to bring them out to find the people that can use them to make decisions. I don’t know if you even buy into that kind of framework, but I’m wondering a) does that ring true and b) if so like how does that map to what you’ve seen in your work?
Monal:Yeah, sure. I think we do a mix of both. When I first joined I think it was more the former in the sense that people have specific questions and we’re always aiming to try and answer business questions using various methodologies, whatever is going to work best to get those questions answered. But there are definitely times when we actually see something that is interesting and we’ll probe further on it in order to say actually there’s an opportunity here, or actually there’s some pain points or something that we need to look at further here and we’ll bring that to the attention of stakeholders in the company for further investigation.
Steve:So it sounds like part of that evolution, the growing of the company and the growing of the role of what you and your team are doing is – I mean developing the ability and developing the context to have those conversations. Yes we can answer the questions you’re asking and hey everybody else here’s things that we’re already doing or can do to support you.
Monal:Um-hmm, definitely.
Steve:So maybe that’s – maybe what I sort of asked inarticulately a moment ago, maybe that is the vector of a growing research practice is that you shift from reactive to more kind of reaching out.
Steve:Okay. So we’ve talked a little bit about sort of some of what you’ve been working on in the last year, but maybe we can just go back, back, back and maybe talk about how did you end up being a researcher? What are some of the things that you’ve done that have led to today?
Monal:Yeah, definitely. It’s funny, I often joke that my background and my entrance to becoming a UX researcher is very non-traditional just because it is in fact on paper very traditional. I’ve been focused on this area since the beginning of my career, even back in my days at Stanford as an undergraduate. So I think that’s pretty rare for folks in our area. So yeah, when I was growing up I always had a fascination for computers and technology and played a lot of video games as a kid. Started programming on my own for fun when I was in middle school, on my Commodore 64 – dates me a bit. And when I went to school in the mid-90s I was lucky enough to be at Stanford where a lot of stuff with the Internet was just coming, seeing light, and getting out into the world. So very lucky to have been there and found my major which is called symbolic systems. So symbolic systems is an interdisciplinary major made up of core classes from computer science, philosophy, psychology and linguistics. And I graduated with a concentration in human computer interaction. So a lot of what I do today is rooted all the way back from that time and one of my first – I guess the – my start of my career, I’d say I did – I was more of a – what we would call today a UX generalist. So back in the dotcom times, as you remember – sorry to call you out…
Steve: Hey, I had a VIC-20. I had an older Commodore computer than yours. I’ll own it.
Monal:Cool. So as a UX generalist – I’d maybe call someone like this today – I did everything from front end coding and UI design to information architecture and user research. And back then, I mean I was really seeking more objective ways as a designer to know how do I go about my design. And unfortunately I think most user research at that point was limited to usability testing. So while I had done a summer internship while I was in school at Boeing in a usability lab, I grew more interested with various kinds of methods of research. Like what else can we do, how else can we learn about users? And I was very lucky to have had a mentor at this company called Trilogy – it was my first company that I worked for – someone named John Morkes who now has a consulting company in Austin, Texas. He’s had it for many years now and back in the late 90s he actually worked with Jakob Nielsen at Sun. So he was probably the best mentor I could ask for and we together built a usability lab at Trilogy and he mentored me while I learned more about doing things like contextual inquiries, focus groups, all kinds of methodologies and that’s what got me really excited about focusing my career in this area. And I had some really big successes from doing multiple method research in terms of helping the company understand what was actually happening with users, even at that time and this was back in like 1999. So I saw the power and the value of what user research could provide. From thereon out it got a little more challenging as the fall of the dotcom era occurred and at that time user experience research or usability they called it – I guess UX was sort of coming to bear around that time, the term. It was much harder to find a job doing this because at that time it was seen as somewhat of a luxury kind of a position. Companies didn’t necessarily see the value in having it beyond something that if they could afford to do it, great. It wasn’t table stakes. So I focused on continuing with design, but I kept coming back to research in my mind, this is what I want to do. So I eventually went back to grad school to focus on understanding more different kinds of methodologies, how can I apply this? How can I become a UX research as for a career. So I went to UCSD and studied cognitive science there and eventually started working as a UX researcher and I have been ever since. So prior to Lyft I was at SoundCloud in Berlin for 2 years, starting a research team there. And prior to that Intuit and Yahoo for several years. And you know as I mentioned around the dotcom time I worked in some various startups.
Steve:You prefaced this whole story with – that your story, it’s a traditional path which makes it sort of – as you said, that’s the exception.
Monal:Exactly.
Steve:So I was listening as you were talking and think well what is the traditional path. So I don’t know. I’ll just throw it out as a question. What about your story and the path you’ve gone through, what makes it traditional?
Monal:Traditional on paper.
Steve:Yes, I’m sorry, yes. That’s an important disclaimer.
Monal:Right, right. Well it’s because I actually, you know back in 1994 I took a class at Stanford called introduction to Human-Computer Interaction with Terry Winograd where we were actually designing and looking at user interfaces for software. So this to me is something that most people, maybe going back that far, didn’t necessarily look at at that time. And even when I talk to folks today, you know 20+ years later, folks are still like wow, what you do is really cool. And I say yeah I know. They’re like how do I do that? How do I get into this industry? I want to do that, but I don’t have the background. I didn’t go to school for it. Should I go to school for it? Should I take classes? How do I break in? And I think there are a lot of folks who – most folks I know who are researchers today, even designers, people in the UX field, they’ve come from all different paths and I think that’s part of what makes it a really interesting set of folks because we are so diverse and we have lots of different paths that we come from, but all of ‘em include doing something with people, trying to understand people and enjoying that.
Steve:So what’s your answer to those people that ask you how do I get into this? What would you tell them?
Monal:Well I think it depends on the circumstance. My advice – I mean, and it depends on what kind of career they’re looking for, but there are more classes today which I encourage people to take. There’s lots of stuff online and I mean obviously the best thing would be to just get the experience, but I think it can be very difficult to do that cold. I know there have been folks who have just gone out and done research on their own and said hey, look, I’ve done this about your company. I’ve used your app. I tested it. Sent it over to the company and sometimes with really great success. I’ve even had some people send me stuff like that about Lyft. So it’s really interesting to see how folks are basically trying to break into the industry. But I mean definitely a traditional on paper route is very helpful, but I don’t think it’s necessary.
Steve:So in your traditional on paper route – it sounds like I’m talking about a paper route – I mean, you know the – it seems like there was this culmination for you with UC San Diego. It seems like that’s where you said oh I’m going to go become one of these things and that’s the way that you went about doing it. And of course you had all this background, going all the way back to Terry Winograd’s class, but like a graduate degree in – what was it, cognitive science?
Monal:Um-hmm.
Steve:See, I don’t know. My perception, that seems not traditional on paper to me. That sort of seems like more in the HCI realm and less in the – although having said that I’m not sure is it a social science degree or a design degree. Like what’s my perception of the traditional on paper thing? So the fact that you and I don’t even share definitions of traditional on paper even though – we have – I’m a little older than you but we have – we’ve seen some of the same eras of this work. I don’t know I think it only goes to bolster your really important point that the diversity of backgrounds and disciplinary orientations and all that that make up the work is just part in parcel of what this practice is about.
Monal:Um-hmm, definitely. I mean I chose to go to UCSD because – well first it was on my radar because of Don Norman and you know the Design of Everyday Things, previously called the Psychology of Everyday Things, really resonated with me and that was sort of my bible for – for you know when it first came out. And then there were multiple labs there that were focused on using ethnographic techniques in order to design technology for everyday use and so that was one of the things that really resonated with me in terms of going there. It wasn’t so much the cognitive science academic courses. It was more about the folks there and the opportunities to do research there where people were thinking very similarly in terms of you know how to do research and it was very applied in that sense. So yeah, so I – even though it does sound somewhat orthogonal to UX it was actually very close to it.
Steve:Especially once you explain it. Yeah, seeing an academic use of ethnography kind of in terms of using technology that seems very – that’s still cutting edge for the work world and it’s extremely cutting edge for academic back a few years even.
Monal:Yeah and back then I think in the early 2000s – I mean when I graduated from Stanford I always wanted to go back to school ‘cuz I’m always like wanting to learn and a student at heart and I wanted to work first. And so I already had on my mind getting a graduate degree and craving the desire to go back to academia at some point. I just wanted a break and some real practical experience, but that said it wasn’t that I hadn’t tried becoming a researcher. Back then you really needed an advanced degree to get into that field because I had tried. And part of it was the fall of the dotcom era. So it was really difficult to have those opportunities, but the companies that were still looking for folks like that, like some of the bigger folks – I mean I don’t even think Google –Google definitely didn’t have folks then looking at that. But maybe Yahoo or eBay. That you definitely had to have more of an academic background to enter.
Steve:Right and such a contrast to now where as you said there’s a lot of classes out there where these classes, and General Assembly springs to mind, but I think there’s probably dozens and dozens of other examples, but they seem to be maybe the most common brand that I come across where that – like a UX Intensive from General Assembly, like that’s the that’s sort of the symbol of qualification where an advanced degree was something. So that’s – if you just compare and contrast what it takes to get an advanced degree vs. what these kinds of programs are providing and sort of the coin to the realm has really shifted. I don’t know part of me things the population has – of researchers has just exploded between the time that you’re talking about and now. I’m now sure what the – do we know what the cause is and what’s the effect? Or what has led to what?
Monal:Yeah, it’s a good question and actually before I left for Berlin in 2012 I really feel like there’s been a shift. I mean – when I finished my degree and started working again I thought wow I’m always going to have to work at a big company because I still had this mindset of what I explained back in the dotcom days where this was a luxury. You’re the last to come into the company and you’re the first out. But that’s no longer the case and I’ve thought about this a bit and I sometimes think that maybe, you know back then it was all about the technology – wow, this is really tough to build. There’s a lot that needs to go into this. Just getting that done was a really big accomplishment. And I think now – not to say that building code is easy, or writing code, but building an app it doesn’t have to be that difficult. And so it’s become really important in the marketplace to have a great experience. It’s not just about having the functionality now, it’s the form as well. It’s a lot of times a differentiator as to why people choose your product vs. another. And I think that’s part of the reason why people are jumping on the bandwagon of oh yeah, actually this is really impactful. We can learn how to get our product to be successful in the marketplace.
Steve:And that think about the importance of the experience, I agree with you totally and it’s – we keep getting it demonstrated to us over and over again – I mean as consumers I guess we see this. It’s really hard to do that. And you know we have lots of good examples of a good experience and it’s just amazing to see how it’s impossible to sort of replicate. I think it you look at sort of the battles between mobile phones over the last 8 years or something, like it’s hard to – just because someone does one that has a good experience you just see these terrible things come out of other companies. And if we were two people wearing – you know with design in our title, we might talk about that issue in a certain way. I wonder if you have a perspective, wearing a research hat, like why do you – what’s the conversation a researcher should be having about why it’s hard to – we know that a good experience is important, but the creation of that experience is like amazingly elusive.
Monal:Yeah, I don’t know if it’s necessarily something that researchers can always pinpoint too. I think a culmination of factors have to come together. Various things and some of them are just very emotive and users can’t describe them. Researchers might be able to observe it happening, but I think it takes a really good researcher to be able to dig in and understand all those things that are happening in this person’s subconscious so to speak.
Steve:Yeah. That makes me think of a comment you made early on where you used the phrase ‘world-class research team’. Can you define what that is?
Monal:Well I’ll tell you a little bit about what I look for in a researcher. So there’s three main traits that I sort of boil down to what I think are most important for researchers – user researchers to have. First people need to be empathetic, so have user empathy – and part of that is being a really good listener. So being a good listener, being a very observant person, also being personable and approachable because any time you’re with a user or someone that you want to learn from and about they need to feel comfortable in your presence. And along with the user empathy piece and the definition of being empathetic is you need to be able to look at things from that person’s perspective. So if someone is really judgmental that’s going to detract from their ability to be a researcher just because you have to be able to consider their point of view.
The second thing is analytical thinking. So obviously strong one. One of the things that I think can be the most impactful for the quality of research is being able not just to collect all the data, which you might use the empathy skills to do in a sense, but to then take those insights – well take the data and the findings and turn them into insights and actionable recommendations that will drive good results for the business and for product design. So there’s a lot of other areas also where analytical thinking can come into play, but I think that’s what’s unique to this field and this profession.
And lastly, this one is probably overlooked by many, but I would say having really great project management skills and being organized because oftentimes as researchers we are managing so much when it comes to participants, sessions, multiple projects and you have to be organized and keep all of your data very organized. It’s just part of the territory in terms of having those skills to manage all of that. And oftentimes manage the teams and stakeholders that come along with that so that you can ensure that people are there to observe, to participate. You know there’s some administration work – administrative work going on here, but a lot of it revolves – your success as a researcher resolves around you being able to stay on top of it and be very detail oriented.
Steve:So those are things you look for, right. How do people exhibit those characteristics in a way that you can assess?
Monal:Um-hmm. I mean it’s definitely hard to do in a single interview. During the interview process I definitely like to be able to see them in action. We have folks do – typically do like some kind of homework during the process and I often like – like very many companies like to have them do a challenge in the office where we recreate, you know kind of a real life scenario, some role playing, and so we can kind of see how they tackle problems as well as interact with folks and understand how they may work with stakeholders. I mean teamwork, all these other things are also very important, but the three that I pointed out were specific to, I think, user experience researchers that are not necessarily applicable to other fields for specific reasons.
Steve:I like that your three kind of span a research project, for lack of a better noun. I mean you’re not looking at people’s data gathering abilities. Or not just that, but what you do with the data and then how do you help it – how do you help that result succeed in the environment in which you’re working. So sort of managing the logistics of your stakeholders and their participation and kind of their engagement and so on. I like your list quite a bit.
Monal:Thank you.
Steve:I think those three things make sense, but is there anything controversial there or anything missing or where you would dispute how other people approach this?
Monal:There’s definitely a lot missing from the list. I think these are just three things I keep in mind, but of course there’s plenty of other traits that are important. You know, as I mentioned, teamwork, being a fast learner, being adaptable to change. So many things that we probably look for on a holistic level. You know if you want somebody on your team you want to like them. There are lots of things that I think can be applied to many different roles when you join a company. I mean at Lyft we definitely get excited about folks who are also excited about our vision and the vision at Lyft you know is something that attracted me at first. Having lived in Berlin for 2 years, coming back to San Francisco I didn’t know much about Lyft, just because we didn’t have it over there and I’d heard a lot of friends and other folks talking about this company. So I finally decided to give it a try when I was on a visit here and thought it was so interesting, like the whole sharing economy and talking with my driver and I started thinking about Lyft as a company and realized I didn’t know much about them, let me check ‘em out and searched them online and started reading about their story and that’s part of what got me really excited because our founders were really interested – they weren’t just interested in building a better taxi. They were really interested in kind of the long term, big picture vision of creating better communities and reconnecting people through transportation. You know the long term vision is redesigning cities. It’s really cool. I mean this is such a hot space right now and we’re working on so many exciting projects as a company. One of the things I love is that we’re constantly executing. We’re getting things out the door. There’s always something big happening, but the vision is always what keeps us focused on what it is we’re trying to do. And that vision is something that I really can get behind because it makes my job feel a lot more meaningful. I mean I think as UX researchers, you know having empathy is something that we just innately have within us and we desire helping others and so when you look at the meta, for this company that’s what we’re aiming to do as well. So, you know we’re looking at reducing the carbon footprint, getting fewer cars on the road, putting more people in vehicles, making rides cheaper through ride sharing, Lyft Line, and building communities through that experience. I’ve had so many amazing conversations with other passengers or drivers in the car and I always just leave with a smile. So it’s really fun to work on something in the office, take a Lyft home, and actually talk to my driver about oh, wow, you have this issue, we’re actually fixing that right now. I can’t actually tell them that, but it’s really neat to know that we’re making a difference on a daily basis.
Steve:It’s fascinating to hear you talk about the relationship between, especially between you as a researcher, but with your researcher sort of identity on, between being a UX researcher and the vision of the company. So you talked a little bit earlier on about the different users. You kind of said, like you talked about passengers. You talked about people using the app which I guess might not be the same as the passenger. You talked about drivers. Tell me about, who do you study? What do you look at?
Monal:Well as you said, definitely passengers. People who are looking to get a ride, who are using Lyft to take rides. And then the drivers who give rides. So currently the Lyft app that we have today, the single app, serves both of those types of folks. And for people – most people probably don’t ever see the driver’s side, but once you apply to be a driver, are approved as a driver, there’s a whole nother world on the other side of the app. So we look at both those users and more recently we announced at the beginning of this year a partnership with a company called National MedTrans Network. So their users are a whole new set of users for us to investigate and learn about. I can tell you a little bit of a story about them. So they – National MedTrans Network is a company in New York. They work with insurance companies like Medicare and Medicaid whose patients are these elderly folks who basically need routine medical rides, rides to doctor’s appointments. A lot of times – all the time non-emergency rides. And they often don’t need to be transported in an ambulance or any special kind of vehicle. So National MedTrans has all these relationships with various car service companies, including taxis in New York City. One of the interesting things that came to bear was that whenever they have a taxi or car that doesn’t show up for a patient or cancels for one reason or another, which is pretty common, it takes – in New York City it takes them about 45 minutes to get that passenger, that patient, another ride through the taxi company. And so they started using Lyft and found that wow you can get a ride in just a matter of moments using Lyft and get this patient home so they can take their medication, so they can do whatever it is they need to do and not have to wait there on the street or whatever. And these folks have a lot of times more special needs then some of our normal types of passengers. So the great thing about this is in terms of building this new product research was highly involved. And so I went over to see National MedTrans, brought a designer with me and as anyone who’s done contextual inquiries know there’s so much more to learn when you’re in that environment and learning who the users are, what their daily tasks and processes are and the entire environment. And this company where we visited them was like people in cubes functioning as a call center. They’re scheduling rides. They’re working with taxi companies. They’re talking to patients, caretakers, people who are managing transportation needs for these patients and it was really, really insightful for us to be there and use the insights there to figure out how to build the best possible product for these users. It’s something totally different than what we have in terms of our native app offerings today.
Steve:And who will- within that use case are there multiple users there that have different experiences with Lyft?
Monal:Well the primary user in this case is the person at Nat MedTrans who is requesting a ride and managing the ride. So right now a lot of these elderly folks, they don’t even have smartphones. So it is a little bit of a juggle in the sense that Lyft wasn’t necessarily designed for this use case from the first and so we’re getting to the point where how can we build the best possible product. There’s the driver, there’s the passenger and now there’s this middle person who functions as if – kind of like the app does for you or me who might be taking a ride today.
Steve:It just speaks to a really, really (in capital letters for me) BIG idea. A lot of the technologies that we see are apps and there’s interesting – I don’t know let’s just call them democracy or democratic access I guess is probably a better word for this. If you don’t have the income or the technology, sort of know how to use those things you’re shut out of a whole bunch of ways to participate. And what excites me about what you’re talking about is that it’s a fascinating research question then is how do we make non-mobile device users Lyft users? Those principals can apply to anybody but you guys are digging into that. It’s really fascinating.
Monal:It is actually really fascinating and just being there and thinking through these whole – you know this set of issues and this problem, it really brought me back to my roots where this feels like a true user experience problem to solve, you know. Lots of different players, various tasks and processes, how do we make this work?
Steve:I hear the sort of analytical aspect of it as you talk about it, but I also hear your sort of – your point one about empathy. Like there’s something in your voice which won’t come through in our transcript, but there’s something in your voice as you talk about it. It seems like this touched you or this struck you in a certain way.
Monal:Yeah, definitely. I mean the thing that I think struck both me and the designer when we were there was like wow we’re there in this call center, someone called in, a patient who had been waiting and they couldn’t get a ride and how quickly they were able to resolve the issue with just using Lyft. And it just blew our minds. We were like wow. It was just really cool. Like we made this person’s day so much better by just getting them home.
Steve:So that project sounds like a sort of a special one, maybe sort of outside. Or it’s a new area for you to be looking at beyond the traditional set of users that you’ve been thinking about. I don’t know, in the life cycle of doing the research, whether it’s getting your data or helping the teams make use of it, I don’t know is there any spots that you find particularly challenging that you’ve had to dig into?
Monal:Well another thing that actually drew me to Lyft was the fact that I think there are a lot of really interesting research challenge with our product. We’re really on the cutting edge with regard to mobile devices, being completely mobile, having to try and do research in a very constrained environment which is a car. And I think that there are a lot of methodological challenges that come with that. So for example if you think about it the best research comes from real life situations. You want to be able to observe – from of course from life qualitative user research. You want to observe what’s happening in the real world, or simulate it as closely as possible. And if you want to find out what’s happening in ride experience, if you’re in the car you’re already biasing what’s happening there because you need to announce yourself as the researcher. Everyone knows you’re from Lyft. People are going to be a little bit more cautious about how they act. Let’s say take the researcher out of the car. Let’s just put a camera in there. Well then there’s legal constraints as well. You need their permission and if you have their permission then of course they’re again going to be a little bit on guard and this will really bias the results if this is what you’re interested in learning about. So we’ve had to come up with some creative and innovative ways to study some of the things that we’re interested in learning more about and I think along with that you know recording, conducting research, and huge amounts of data are always challenges for researchers across all industries. So specific to Lyft though, and specific to this area in particular, I think we have some really interesting challenges and it’s fun because it’s – you come to work every day thinking about – you’re really stretching your brain and you feel really good when you come up with something new and get to be creative in those senses.
Steve:Are there any process innovations or anything that you can share?
Monal:Not exactly (laughs). I can tell you about one of the first programs that I started when I joined. So we wanted to ensure that we had user testing for most of our projects coming through on a regular basis. So I started a program called Drivers in the Office, so D-I-T-O, which we call “deeto”. And along with that PACSITO, which is passengers, which we often abbreviate as PACS, in the office. So DITO and PACSITO, it’s like a weekly user testing cycle and it’s been a lot of fun. We have people come into the office here in San Francisco. Occasionally travel to other cities and occasionally do some remote studies wherever appropriate. But what we do is we have, and especially for drivers, for DITO, what we’ll do is we’ll actually have drivers come and bring their car, park in our lot and then we’ll have them use a prototype of a new design or product that we are testing and this allows us to capture everything that’s going on while they drive. So it’s one thing to be a passenger, in your house, on the street, looking at the app. You can stop, you can look at it for as long as you want in a maybe normal situation let’s say. But as a driver you have so many other distractions and concerns. I mean it’s a very cognitively complex activity. You are doing so many different things at once and now we’ve just added one more thing to pay attention to which is the app. So it’s a huge design challenge for our team and also from a research perspective it’s really important for us to be able to get the data in the right environment, in the contextual way that they will be using it because they’re using the app at an arm’s length. They can only look at it upon glance, most of the time when they’re driving for safety reasons. And we have to be really cautious about ensuring that it’s not overly distracting. So one of the things that’s been really great about this program is that times when we’ve tested like more low fidelity prototypes where we can’t go in the car, versus going in the car, we’ve learned quite a bit.
Steve: There seems like there’s an interesting, maybe sort of a hybrid – there’s like a semi-contextual aspect here where they’re not in a lab, there’s a high fidelity simulation of what the thing might be. You’re dealing with their actual car, you have realistic use cases, but you’re not also sending them off into the wild to live with this thing for a day. It’s a constrained experiment, but you’re sort of creating I guess as much context within an experiment as you can.
Monal:Um-hmm, right. And I think that’s important for us to – it’s a big part of how it’s going to be used in the real world so we need to capture that – how they do so in that environment.
Steve:What about other examples of kinds of, within what you can talk about, what are some other kinds of research that your team has done?
Monal:We basically try to do as much mixed methodology as needed. We do surveys, as I mentioned contextual inquiries. We’re basically employing everything that we need to design the right kinds of studies to answer the business questions. You know we have some user interviews coming up. We have – what we also try to do is work very closely with our analytics team.
Steve:Can you talk about anything that has – that people that use Lyft has experienced that’s changed as a result of some of the research you’ve done?
Monal:Definitely. One of the most exciting things that I feel user research has impacted was our Lyft app redesign that launched at the very end of last year. So we did a lot of iterative user studies, user testing especially around various versions of designs and the product moving forward and most of the product and design decisions relied heavily on the research that we did. So by the time we launched the app, the new redesign, we felt and our team felt very confident that it would be successful once it launched and it definitely was. We’ve seen really great results and have heard a lot of great positive reviews and comments about what we did and I think a huge part of that, I feel really proud to say, was due to the research that we did. So whenever you can affect the flagship app on such a big scale with this design you can feel really good about it. Of course we don’t take all the credit. There’s such incredibly talented people here that you know we were just lucky to be part of that process.
Steve:I think even just this phrase about sort of being proud of what you accomplished, it makes me think about how do you lead a research team. I mean giving out kudos, I think that’s one part of it. I’m just projecting on you here a little bit, but what are other things. Talk to me about being a leader in research. What does that mean?
Monal:Yeah, I mean it’s interesting because I think until recently there weren’t a lot of leaders in research. Primarily I feel like research has had a lot of individual contributors and that’s how we’ve been seen in a lot of ways, people who are providing data analysis, insights, and hey, great, thanks you did your job. And now with the growth of research and especially user research we’re seeing more roles where there are leaders. And I think it’s great for all individual contributors because a lot of times – and this still happens today – the individual contributors who have worked as researchers in the past haven’t always reported up through someone who understood what they were doing. And so it’s great to see that that’s changing to a degree now and I think just in that sense of having the empathy (back to empathy again) for understanding like what these researchers are trying to accomplish, what their personal goals are – I think a lot of what researchers in general – I’m just thinking back through my career as well – have felt is, I mean we really want to make a difference. We really want to have influence on the product or the design. We did this great research, we want to make sure that it gets in there. And this hasn’t always been the case for me in past companies. And one of the great things here at Lyft is that what we’ve produced has been so highly utilized, probably more than any other company I’ve worked at. So coming back to your question in terms of leading, I think ensuring that the work that we do does get implemented or is seen as respectable, credible, and that people understand what we do and how we do it. And bringing them along for the ride too, it brings us to a point where within the organization you’re seen as oh wow, like they do really great work and they’re going to affect really positive change for us. So sort of helping the team be known and understood as experts and ensuring that the work that they do has a positive effect on the company and for themselves.
Steve:And you’re right. That fell on the individual contributor to sort of – you had to do all that you had to do and you had to be able to do that.
Monal:Um-hmm.
Steve:And so you’re talking about like what a good manager does. They sort of, they advocate for their team, they…
Monal:Um-hmm.
Steve:…protect them and champion them.
Monal:Definitely.
Steve:You talked early on about this really important mentorship relationship that you had in your career. Are there commonalities between what you experienced kind of as the recipient of mentorship and what you try to provide as a leader now at this point?
Monal:Definitely. I mean John who was my mentor back then, he gave me so much and I feel like I’m at the point in my career where I definitely can do that for other folks as well. So right now it’s primarily here at Lyft, but I’d love to do that outside of Lyft as well.
Steve:How do we define mentorship? We talked about leadership, but what’s the difference between mentorship and leadership?
Monal:Yeah, I think mentorship is a little bit more focused on – well in my mind helping folks develop new skills as well. Leadership doesn’t necessarily have to do that. I think leadership can do that. I mean if there was a Venn diagram it would probably be overlapping in the circles, but I definitely believe that a good mentor will help someone on a personal level develop into whatever career goals they have. Helping them along that path to get to where they want to be. And for me I learned a lot of methods from John. I think I’m trying to do that as well on a mentorship level, but also just be someone always who’s there to talk through, you know, career related growth.
Steve:You know in some ways – we’ve talked a little about sort of the evolution of the field and it makes sense that at that period of time developing methods, not that that was all you were learning, but that was kind of a key. And I think about your three areas – I mean the first one you listed, you kind of titled it as empathy and then you listed like 18 other, like really important sort of soft skills that are crucial to being a researcher. So when you’re there for people as you say I think that’s starting to model, yes the methods, but also now these other pieces that are so important to being a good researcher.
Monal:Um-hmm.
Steve:That are just very, to me very human skills that we all can – we all can do to work on.
Monal:Um-hmm, definitely
Steve:Are there other things about you and your background that we should understand to know what kind of researcher you are? Any weird dots you want to connect for us?
Monal:Yeah, actually I think one of the things that’s really unique to me is that I previously had another life as a professional athlete. So I was very active in team sports and that’s a big part of like my philosophy too. Very team oriented. I started playing soccer when I was six years old and I got a scholarship for running when I was at Stanford. So that’s been a really big part of my development and who I am and the really cool thing about running at Stanford is that when I started there and started running on the cross country team there we weren’t nationally ranked. And by the time I was a senior my teammates elected me as the captain of the team and we won the NCAA national title that year which was less than four years later and so it’s been really interesting to draw a lot of comparisons and parallels to my athletics career and how I’ve envisioned like sports and teamwork and accomplishments and my working career. And it’s funny, I don’t even realize I’m making these analogies. I started talking about it to a coworker the other day and I’m like wow I do this actually quite a lot. But there are so many parallels to draw with regard to pushing yourself hard, getting to the next level, goal setting, you know the competition and teams, relating to people, supporting one another. And one of the reasons why I think we won the national title back in 1996 was because we had such a tight, supportive team and I really feel that we have that now at Lyft as well. User research is part of the product design team and we are such a close team. Actually we just came back from a weekend retreat in Tahoe. So we’re kind of like a tight-knit family and I really feel that that support and that kind of environment fuels – helps fuel really great success.
Steve:So some of these parallels you’re describing, they seem about what athletics teaches you about working and collaborating. Are there any threads that you can pull from athletics that go into things that are specific to UX research?
Monal:That’s a really good question. I think I need a minute to think about it. It’s something I haven’t thought about before. I’m sure there are. I think I need to noodle on this, but I like the question a lot. I think because a lot of what – the parallels I’ve drawn have been more meta, like high level. Yeah, nothing springs to mind, but at some point I’ll get back to you. I’ll be like ‘Steve guess what, I figured it out and now my life’s questions have been complete Like I know. I see the light!’
Steve:Do you have questions for me?
Monal:Yeah, actually, so I was curious because you’re one of the few UX research veterans in the field and I was curious what kind of trends you’re currently observing in our little field that’s growing?
Steve:Yeah, I think – I liked what you said about mixed methods. I think that seems to be, you know, where traditionally on paper there was a lot of us and them. I feel like I’m hearing more harmonious stories and I think it behooves us as a practice to tell harmonious stories, but because that speaks to a successful, integrated, mature, not a diva, you’re wrong, this focus group sucked, blah, blah, blah. Like that’s sort of – that’s what we’ve been saying and I think true or not it’s sort of disharmonious and I feel like the stories of how groups have come together, to me that’s like oh we’re a little more mature as a world now because we can coexist well with everybody. And the qual/quant things used to be cats and dogs, but as you said you guys integrate well. I’ve heard that from lots of groups that I’ve talked to and I’ve seen that. You know clients I work with start to say like oh yeah, we have this group, this group, this group and we can kind of poll lots of – so that’s mixed methods. That’s also, you know – I don’t know sort of mixed disciplines I guess. So it seems like that – to me there’s just a maturity level there with that. Yeah, that’s my big one I’ll leave as the answer for that.
Monal:I agree and I really think that in order to have the most robust insights we do need a good triangulation of different types of data to be able to tell that story. I was also curious, I mean as we talked about UX research has come a long way from you know the mid-90s and where the worldwide web was at that time. I mean just technology has come a long way and UX research has alongside of that. So that said, where do you think UX research needs to develop most given our current landscape?
Steve:These are the questions I should be asking you but I didn’t.
Monal:The tables have turned.
Steve:Yes. Yeah, what’s the sort of opportunity? And this came up as a topic in an earlier episode of this podcast series where I don’t know that we answered it, but we talked about just this idea of research sort of evolving to the point where it disappears. That research right now there are leaders that are trying to develop in a business, but – and you’ve talked about the successful collaborations and so on. Is this an activity or is this a discipline. You know I hear so much about teams trying to – I mean one way to deal with the overwhelming demand of research is that research leaders are empowering teams to go do their own research and so then they have to deal with this question of well what do we do as the researchers vs. what do we kind of let everyone do. I mean that’s the consequence of the evolution of research and we’ve been saying, as part of that evolution, like everyone should be doing it, everyone can do it, right. There’s books that teach people how to do research, like that’s a change and so we – I think in the early days we struggled with do we let go of this or do we hold on to it and we chose to do a bit of both. So how does that play out I think is an interesting question? How should it play out? And research doesn’t exist in a vacuum, so how technology has developed. You know we are in a lot of the way – in a lot of environments that research is being used it’s a close colleague of design, right. I mean you talked about being part of the product design organization, not some other – part of the market research organization. That’s who our sort of, our teammates are. And design is changing and you talked about this already too, right. That companies used to just do technology and sort of experience was secondary. So I think where research goes we’re sort of subject to these larger changes in how – in how experiences are created. So I don’t have an answer to that and I’m not sure it’s a need. I don’t feel like oh we better get this sorted out or we’re dead. I just feel like there’s some interesting, sort of existential questions to reflect on. You know I mean over the course of our careers you and I have seen what kinds of organizations and what kind of roles are there for us. It’s shifted a lot and as you said there weren’t leaders like yourself a few years ago and not in the numbers that there are no. So even just for our own careers, like it’s going to be different in a few years. I don’t know. It’s tough to predict the future, but I think those are the – those are interesting, if abstract areas for me to keep looking at.
Monal:Um-hmm, definitely. I mean I think it’s – I agree with you and I also think it’s really great news that the discipline is growing because there is a lot of value that we can provide. And you know the Lyft research team is also growing so I think it’s a really good sign of, you know how much we’re able to impact the organization in a positive way and I love seeing now, after having come back from Berlin – previously it was like oh we can only work at big companies. And now I feel like every startup has a researcher. Maybe not every startup, but it’s becoming more and more common which is amazing.
Steve:You know the designer as founder sort of thing, it’s like a – it’s a thing that gets recognized and talked about and I feel like maybe we identified with – I feel like I was on some Slack channel or some list and people were saying well when do we get our first researcher as founder. I’m probably misquoting someone that said that to me and I apologize for that and that company may exist, but certainly that idea is nascent whereas designer as founder in the startup world at least is well established. Maybe that’s the, you know sort of next wave for us.
Monal:Um-hmm. Cool. I have one more question for you.
Steve:Alright.
Monal:So I’m really curious, and I know you’ve had a lot of questions about this podcast, but I was curious what you’ve found to be the most striking similarities between all of whom you’ve talked to so far.
Steve:Don’t you love when you go out in the field and the person at the end says like “hey what else have you heard?” Or “am I normal?” They want you to kind of normalize them as part of it. That’s not exactly your question here, but yeah, what do researchers have in common? I think the emphasis on product is really, it’s like a thing that we should be proud of as the field and it goes back again to your list of three things that you talked about. As you kind of researchers needing to do good data collection – I’m paraphrasing you here – make sense of it and then sort of help it live in the organization. I am definitely twisting what you said, but that’s sort of thrust I took away from it. And I don’t know, I carry some scars from my early days of coexisting with people that – where research was their end – they loved – and you know I guess scars, but I’m also definitely guilty of this myself. Like research is fascinating. If you do it well the experience is great and you learn things that have no relevance. And I mean I’m a big fan of not knowing if it has relevance, but you learn things that you don’t know if it’s what you’re going to do with it, but you also learn things that just are great and they’re not applicable, but are part of what you have to do on that journey. And then you make sense of things and you find some really interesting takeaways, or things that are potential takeaways that aren’t necessarily usable by the company. Of course then you find here’s the things that we learned and here’s what we need to do about it and here’s all – but all the way along there’s a lot of fascination, right, like…
Monal:Um-hmm.
Steve:…it’s super engaging and to me very creative and I don’t know. I can be seduced at many points along the way. I mean, I don’t know – I think I’m good at making sure I get to that end point, but there’s a lot of seduction along the way with just stuff that is super inspiring, super thought provoking, that you kind of dig into and start synthesizing and organizing. So the scars are around, you know maybe sort of early days in the field, or my early days, where that’s where people kind of dwelled and someone had to come along and kind of pull them and say like okay you can’t deliver that. You have to show actionability. That sort of discussion was needed. And I mean it makes sense that the people that I speak to, like you who are leaders – I mean in order to be a leader you have to be beyond that. There may still be practitioners and contributors who kind of need to learn more of that, but I feel like if I take the temperature of the practice in terms of how it’s being led it’s about what we do with research.
Monal:Um-hmm.
Steve:And that is a massive change again over when I started in my younger days. And so to see that in common – again, maybe you go well of course Steve look who you’re talking to and what kind of jobs they have and look at what kind of companies they work for. You know again, yeah, you might find people at that skill level, like yourself, and experience level – if we were to like shift back 10 years you would be maybe working in an R&D lab or an advanced products group or something and not shipping stuff. So yeah, I think one of the commonalities is like these are people making products, or helping companies make services, whatever the thing that their company does they’re doing that and obviously that’s – I think that’s really, really good.
Monal:Um-hmm, yeah. One of the things that struck me about what you’re saying is that fascination and I think as researchers like a lot of what we love is having that curiosity and it’s so fulfilling just to do the hands-on work. My role is managerial, but I’m always still – you know I do some hands-on, individual contributor work as well. I just love doing it, right. That’s what makes us passionate about research, we love doing it, but I think in a business one of the dangers is getting too far into that fascination, but then not taking that next step. And maybe that’s where managerials, or managers can help with making sure that the data and the analysis and then these insights and actionable recommendations happen and are affecting product. That we’re not just doing the research and being fascinated for our own curiosity and fulfillment of that, but that we’re actually making sure it’s being put to good use.
Steve:I think that’s really important and it reminds me of just a period in my own career where I was managing my own team in my consultancy and I think like – we always had many different projects for different clients and I want to say maybe for a period of 18 months, which is a long time in sort of consulting years right, I didn’t go in the field at all. I was doing what you’re talking about, right, sort of managing the problem, helping with the synthesis, kind of trying to tie everything back. Sort of a creative lead and it was really cool to sort of – you know you remove one part of the puzzles and the other parts – I guess the other things kind of rise in prominence. So it was real interesting for me to step out of certain tasks and step into other tasks in a larger way. And I actually remember the first time that I went – it just worked out that like oh yeah I was going to do some of these interviews and I almost had a panic attack. Like it had just been so long. And I was not on my own. I just remember like kind of coming up to the door of somebody’s and just about freaking out. And you know the way I work now I’m sort of – I do more of everything, but having had that experience of sort of again playing just part of the role was really, really interesting. It’s good to hear that you – it sounds like you’re in the right spot. You still have your hands in things, but you can lead and kind of manage in that spot.
Monal:Um-hmm, yeah.
Steve:Alright well that’s maybe everything we have time for to talk about today. But this was really interesting. It’s one of these conversations where now I’m curious about a million other things. I’ll have my imaginary next 90 minute conversation with you. But yeah, thank you very much. You’ve really shared a lot today.
Monal:Thank you Steve. This was a lot of fun and yeah, I really enjoyed it.
The post 14. Monal Chokshi of Lyft first appeared on Portigal Consulting.
In the final episode of the season I speak with Monal Chokshi, Head of User Experience Research at Lyft. We discuss traditional paths to a user research career, creating routines for meeting different types of users, and the emergence of leadership roles in user research.
As researchers we love having curiosity and it’s fulfilling to do the hands-on work. That’s what makes us passionate about research, but in a business one of the dangers is getting too far into that fascination, but then not taking that next step. Maybe that’s where managers can help with making sure that the data and the analysis and then these insights and actionable recommendations are affecting product. We’re not just doing the research and being fascinated for our own curiosity and fulfillment, we’re making sure it’s put to good use. – Monal Chokshi
Show Links* Monal Chokshi * Follow Monal on Twitter * Lyft * Lyft partners with Didi * John Zimmer and Logan Green * Commodore 64 * VIC-20 * Trilogy * John Morkes * Jakob Nielsen * Sun Microsystems * SoundCloud * Terry Winograd * Don Norman at UCSD * General Assembly * Lyft’s partnership with National MedTrans Network
Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.
TranscriptSteve Portigal: Thanks, Monal for being a guest on Dollars to Donuts.
Monal Chokshi:Yeah, sure. Well first Steve thanks for having me on your podcast episode. Really excited to talk more about user experience research at Lyft. So my name is Monal Chokshi and I’m the head of user experience research at Lyft.
Steve:And if I didn’t know what Lyft was, what would you explain to me?
Monal:Yeah, so Lyft is an on demand transportation service. So for those not familiar, basically all you have to do is download the Lyft app, open it and tap a button and basically you have a driver ready to drive you wherever you want within minutes at your doorstep.
Steve:Where does Lyft operate?
Monal:So the headquarters are in San Francisco. We also have offices in Seattle, Nashville and New York.
Steve:And where do you have drivers?
Monal:Drivers all throughout the country. I believe we are now in over 190 cities.
Steve:This is an American operation?
Monal:Yes.
Steve:So drivers in the United States right now. Is there an international goals for the company?
Monal:Well right now we recently in the last 6 to 9 months announced some partnerships with international ride sharing companies. For example with Didi which is the largest ride sharing company, similar to Lyft, in China. And Grab which is in Southeast Asia. I believe they’re based in Singapore. So the idea being that travelers from the U.S. would be able to travel to one of these places, open the Lyft app and call a car. It wouldn’t be a Lyft car. It would be one of these folks from our partner companies coming to pick them up. So we haven’t launched that yet, but that is a partnership that we’ve announced and we’re really looking forward to getting that going.
Steve:It makes me think of like code sharing and airlines where you can purchase through the sort of front end that you’ve always used, but the service is delivered through a partnership with somebody else. Is that a, is that a – does that comparison apply to something like this?
Monal:I think it might. I think it’s a pretty good way to think about it from an analogy.
Steve:I’m sure there’s probably better analogies for how products and services in one market end up in another, but that’s the best I could think of on the fly. That’s really interesting.
Monal:Yeah.
Steve:And I’m sure there’s going to be design challenges – what does that experience look like in that different situation.
Monal:Yeah, we’re really excited about designing this new experience just because of the various challenges that come with different cultures and different transportation schemes for various cities, especially international. As we all know the cultures, the modes of transportation are different from city to city in the US and then overseas there are various challenges with language as well. So there’s a lot of really exciting things to think about.
Steve:I love like the really specific use case you’re looking at now which is I’m from the United States, I use Lyft here. I’m in a different context so how can I bring access to that service with me. It makes me think about what people that use Netflix complain about it or I think I hear this with the iTunes store. For example, you can’t get the same access when you’re in a different place. So this gives you – I’m sort of imagining it gives you – you can sort of bring your Lyft travel experience with you into these other environments.
Monal:Yeah. And people use and are comfortable with Lyft here. We want to be able to make sure that they have the comfort of the same kind of service over there, something that they can trust and feel good about. Something that’s familiar in an area where everything seems unfamiliar to them.
Steve:Can you talk about the work that you do here and what is research like at Lyft?
Monal:Sure. So research at Lyft has always been part of our core DNA in a sense. The founders, Logan and John, are really focused on treating people better and that includes all of our users, our drivers, our passengers and so really understanding who these people are and figuring out how to create the best user experience for them has always been top of mind here. Before I joined we did do various research activities and there were several initiatives happening at Lyft. Once I joined, I joined to sort of make that part of our regular product and design processes and so the kinds of research that our team does really runs the whole gamut of what UX research can do. We do everything from field studies and surveys to user and usability testing of new designs and products. So yeah, it’s been a really fun place to work, just also because user experience is so well regarded and supported here.
Steve:When did you come to Lyft?
Monal:I’ve been at Lyft for about a year.
Steve:So you’re describing a little bit about how – you know what you were brought in to do, to sort of – I don’t think you used the word formalize, but that’s sort of the word that comes to mind. I guess is the question about – you know what that change has been that you’ve helped drive.
Monal:Sure. I think a lot of the research previous to when I joined hadn’t been formalized in a sense that now it’s much more a part of the regular, daily, weekly process. And I’ve also joined to basically build a world-class team of user researchers so that – you know Lyft has grown so much in the last year and we are continually growing at a very increasingly rapid rate. So there are a lot more needs that we have from a research perspective and I think as we grow the company there are more needs for insights. So, bringing that to the table and helping build a company that’s very data and insights driven is a priority for the company.
Steve: I’ve seen sort of two different modes of how insights – I’m sure there’s many more, but two kind of primary modes in terms of how insights can impact decisions. One is sort of the reactive and one is sort of the proactive. And you know where reactive is somebody comes to researchers with a question and says ‘Hey, can you help us make a decision about this?’ and the other is researchers often who have done research are evangelizing those insights to try to bring them out to find the people that can use them to make decisions. I don’t know if you even buy into that kind of framework, but I’m wondering a) does that ring true and b) if so like how does that map to what you’ve seen in your work?
Monal:Yeah, sure. I think we do a mix of both. When I first joined I think it was more the former in the sense that people have specific questions and we’re always aiming to try and answer business questions using various methodologies, whatever is going to work best to get those questions answered. But there are definitely times when we actually see something that is interesting and we’ll probe further on it in order to say actually there’s an opportunity here, or actually there’s some pain points or something that we need to look at further here and we’ll bring that to the attention of stakeholders in the company for further investigation.
Steve:So it sounds like part of that evolution, the growing of the company and the growing of the role of what you and your team are doing is – I mean developing the ability and developing the context to have those conversations. Yes we can answer the questions you’re asking and hey everybody else here’s things that we’re already doing or can do to support you.
Monal:Um-hmm, definitely.
Steve:So maybe that’s – maybe what I sort of asked inarticulately a moment ago, maybe that is the vector of a growing research practice is that you shift from reactive to more kind of reaching out.
Steve:Okay. So we’ve talked a little bit about sort of some of what you’ve been working on in the last year, but maybe we can just go back, back, back and maybe talk about how did you end up being a researcher? What are some of the things that you’ve done that have led to today?
Monal:Yeah, definitely. It’s funny, I often joke that my background and my entrance to becoming a UX researcher is very non-traditional just because it is in fact on paper very traditional. I’ve been focused on this area since the beginning of my career, even back in my days at Stanford as an undergraduate. So I think that’s pretty rare for folks in our area. So yeah, when I was growing up I always had a fascination for computers and technology and played a lot of video games as a kid. Started programming on my own for fun when I was in middle school, on my Commodore 64 – dates me a bit. And when I went to school in the mid-90s I was lucky enough to be at Stanford where a lot of stuff with the Internet was just coming, seeing light, and getting out into the world. So very lucky to have been there and found my major which is called symbolic systems. So symbolic systems is an interdisciplinary major made up of core classes from computer science, philosophy, psychology and linguistics. And I graduated with a concentration in human computer interaction. So a lot of what I do today is rooted all the way back from that time and one of my first – I guess the – my start of my career, I’d say I did – I was more of a – what we would call today a UX generalist. So back in the dotcom times, as you remember – sorry to call you out…
Steve: Hey, I had a VIC-20. I had an older Commodore computer than yours. I’ll own it.
Monal:Cool. So as a UX generalist – I’d maybe call someone like this today – I did everything from front end coding and UI design to information architecture and user research. And back then, I mean I was really seeking more objective ways as a designer to know how do I go about my design. And unfortunately I think most user research at that point was limited to usability testing. So while I had done a summer internship while I was in school at Boeing in a usability lab, I grew more interested with various kinds of methods of research. Like what else can we do, how else can we learn about users? And I was very lucky to have had a mentor at this company called Trilogy – it was my first company that I worked for – someone named John Morkes who now has a consulting company in Austin, Texas. He’s had it for many years now and back in the late 90s he actually worked with Jakob Nielsen at Sun. So he was probably the best mentor I could ask for and we together built a usability lab at Trilogy and he mentored me while I learned more about doing things like contextual inquiries, focus groups, all kinds of methodologies and that’s what got me really excited about focusing my career in this area. And I had some really big successes from doing multiple method research in terms of helping the company understand what was actually happening with users, even at that time and this was back in like 1999. So I saw the power and the value of what user research could provide. From thereon out it got a little more challenging as the fall of the dotcom era occurred and at that time user experience research or usability they called it – I guess UX was sort of coming to bear around that time, the term. It was much harder to find a job doing this because at that time it was seen as somewhat of a luxury kind of a position. Companies didn’t necessarily see the value in having it beyond something that if they could afford to do it, great. It wasn’t table stakes. So I focused on continuing with design, but I kept coming back to research in my mind, this is what I want to do. So I eventually went back to grad school to focus on understanding more different kinds of methodologies, how can I apply this? How can I become a UX research as for a career. So I went to UCSD and studied cognitive science there and eventually started working as a UX researcher and I have been ever since. So prior to Lyft I was at SoundCloud in Berlin for 2 years, starting a research team there. And prior to that Intuit and Yahoo for several years. And you know as I mentioned around the dotcom time I worked in some various startups.
Steve:You prefaced this whole story with – that your story, it’s a traditional path which makes it sort of – as you said, that’s the exception.
Monal:Exactly.
Steve:So I was listening as you were talking and think well what is the traditional path. So I don’t know. I’ll just throw it out as a question. What about your story and the path you’ve gone through, what makes it traditional?
Monal:Traditional on paper.
Steve:Yes, I’m sorry, yes. That’s an important disclaimer.
Monal:Right, right. Well it’s because I actually, you know back in 1994 I took a class at Stanford called introduction to Human-Computer Interaction with Terry Winograd where we were actually designing and looking at user interfaces for software. So this to me is something that most people, maybe going back that far, didn’t necessarily look at at that time. And even when I talk to folks today, you know 20+ years later, folks are still like wow, what you do is really cool. And I say yeah I know. They’re like how do I do that? How do I get into this industry? I want to do that, but I don’t have the background. I didn’t go to school for it. Should I go to school for it? Should I take classes? How do I break in? And I think there are a lot of folks who – most folks I know who are researchers today, even designers, people in the UX field, they’ve come from all different paths and I think that’s part of what makes it a really interesting set of folks because we are so diverse and we have lots of different paths that we come from, but all of ‘em include doing something with people, trying to understand people and enjoying that.
Steve:So what’s your answer to those people that ask you how do I get into this? What would you tell them?
Monal:Well I think it depends on the circumstance. My advice – I mean, and it depends on what kind of career they’re looking for, but there are more classes today which I encourage people to take. There’s lots of stuff online and I mean obviously the best thing would be to just get the experience, but I think it can be very difficult to do that cold. I know there have been folks who have just gone out and done research on their own and said hey, look, I’ve done this about your company. I’ve used your app. I tested it. Sent it over to the company and sometimes with really great success. I’ve even had some people send me stuff like that about Lyft. So it’s really interesting to see how folks are basically trying to break into the industry. But I mean definitely a traditional on paper route is very helpful, but I don’t think it’s necessary.
Steve:So in your traditional on paper route – it sounds like I’m talking about a paper route – I mean, you know the – it seems like there was this culmination for you with UC San Diego. It seems like that’s where you said oh I’m going to go become one of these things and that’s the way that you went about doing it. And of course you had all this background, going all the way back to Terry Winograd’s class, but like a graduate degree in – what was it, cognitive science?
Monal:Um-hmm.
Steve:See, I don’t know. My perception, that seems not traditional on paper to me. That sort of seems like more in the HCI realm and less in the – although having said that I’m not sure is it a social science degree or a design degree. Like what’s my perception of the traditional on paper thing? So the fact that you and I don’t even share definitions of traditional on paper even though – we have – I’m a little older than you but we have – we’ve seen some of the same eras of this work. I don’t know I think it only goes to bolster your really important point that the diversity of backgrounds and disciplinary orientations and all that that make up the work is just part in parcel of what this practice is about.
Monal:Um-hmm, definitely. I mean I chose to go to UCSD because – well first it was on my radar because of Don Norman and you know the Design of Everyday Things, previously called the Psychology of Everyday Things, really resonated with me and that was sort of my bible for – for you know when it first came out. And then there were multiple labs there that were focused on using ethnographic techniques in order to design technology for everyday use and so that was one of the things that really resonated with me in terms of going there. It wasn’t so much the cognitive science academic courses. It was more about the folks there and the opportunities to do research there where people were thinking very similarly in terms of you know how to do research and it was very applied in that sense. So yeah, so I – even though it does sound somewhat orthogonal to UX it was actually very close to it.
Steve:Especially once you explain it. Yeah, seeing an academic use of ethnography kind of in terms of using technology that seems very – that’s still cutting edge for the work world and it’s extremely cutting edge for academic back a few years even.
Monal:Yeah and back then I think in the early 2000s – I mean when I graduated from Stanford I always wanted to go back to school ‘cuz I’m always like wanting to learn and a student at heart and I wanted to work first. And so I already had on my mind getting a graduate degree and craving the desire to go back to academia at some point. I just wanted a break and some real practical experience, but that said it wasn’t that I hadn’t tried becoming a researcher. Back then you really needed an advanced degree to get into that field because I had tried. And part of it was the fall of the dotcom era. So it was really difficult to have those opportunities, but the companies that were still looking for folks like that, like some of the bigger folks – I mean I don’t even think Google –Google definitely didn’t have folks then looking at that. But maybe Yahoo or eBay. That you definitely had to have more of an academic background to enter.
Steve:Right and such a contrast to now where as you said there’s a lot of classes out there where these classes, and General Assembly springs to mind, but I think there’s probably dozens and dozens of other examples, but they seem to be maybe the most common brand that I come across where that – like a UX Intensive from General Assembly, like that’s the that’s sort of the symbol of qualification where an advanced degree was something. So that’s – if you just compare and contrast what it takes to get an advanced degree vs. what these kinds of programs are providing and sort of the coin to the realm has really shifted. I don’t know part of me things the population has – of researchers has just exploded between the time that you’re talking about and now. I’m now sure what the – do we know what the cause is and what’s the effect? Or what has led to what?
Monal:Yeah, it’s a good question and actually before I left for Berlin in 2012 I really feel like there’s been a shift. I mean – when I finished my degree and started working again I thought wow I’m always going to have to work at a big company because I still had this mindset of what I explained back in the dotcom days where this was a luxury. You’re the last to come into the company and you’re the first out. But that’s no longer the case and I’ve thought about this a bit and I sometimes think that maybe, you know back then it was all about the technology – wow, this is really tough to build. There’s a lot that needs to go into this. Just getting that done was a really big accomplishment. And I think now – not to say that building code is easy, or writing code, but building an app it doesn’t have to be that difficult. And so it’s become really important in the marketplace to have a great experience. It’s not just about having the functionality now, it’s the form as well. It’s a lot of times a differentiator as to why people choose your product vs. another. And I think that’s part of the reason why people are jumping on the bandwagon of oh yeah, actually this is really impactful. We can learn how to get our product to be successful in the marketplace.
Steve:And that think about the importance of the experience, I agree with you totally and it’s – we keep getting it demonstrated to us over and over again – I mean as consumers I guess we see this. It’s really hard to do that. And you know we have lots of good examples of a good experience and it’s just amazing to see how it’s impossible to sort of replicate. I think it you look at sort of the battles between mobile phones over the last 8 years or something, like it’s hard to – just because someone does one that has a good experience you just see these terrible things come out of other companies. And if we were two people wearing – you know with design in our title, we might talk about that issue in a certain way. I wonder if you have a perspective, wearing a research hat, like why do you – what’s the conversation a researcher should be having about why it’s hard to – we know that a good experience is important, but the creation of that experience is like amazingly elusive.
Monal:Yeah, I don’t know if it’s necessarily something that researchers can always pinpoint too. I think a culmination of factors have to come together. Various things and some of them are just very emotive and users can’t describe them. Researchers might be able to observe it happening, but I think it takes a really good researcher to be able to dig in and understand all those things that are happening in this person’s subconscious so to speak.
Steve:Yeah. That makes me think of a comment you made early on where you used the phrase ‘world-class research team’. Can you define what that is?
Monal:Well I’ll tell you a little bit about what I look for in a researcher. So there’s three main traits that I sort of boil down to what I think are most important for researchers – user researchers to have. First people need to be empathetic, so have user empathy – and part of that is being a really good listener. So being a good listener, being a very observant person, also being personable and approachable because any time you’re with a user or someone that you want to learn from and about they need to feel comfortable in your presence. And along with the user empathy piece and the definition of being empathetic is you need to be able to look at things from that person’s perspective. So if someone is really judgmental that’s going to detract from their ability to be a researcher just because you have to be able to consider their point of view.
The second thing is analytical thinking. So obviously strong one. One of the things that I think can be the most impactful for the quality of research is being able not just to collect all the data, which you might use the empathy skills to do in a sense, but to then take those insights – well take the data and the findings and turn them into insights and actionable recommendations that will drive good results for the business and for product design. So there’s a lot of other areas also where analytical thinking can come into play, but I think that’s what’s unique to this field and this profession.
And lastly, this one is probably overlooked by many, but I would say having really great project management skills and being organized because oftentimes as researchers we are managing so much when it comes to participants, sessions, multiple projects and you have to be organized and keep all of your data very organized. It’s just part of the territory in terms of having those skills to manage all of that. And oftentimes manage the teams and stakeholders that come along with that so that you can ensure that people are there to observe, to participate. You know there’s some administration work – administrative work going on here, but a lot of it revolves – your success as a researcher resolves around you being able to stay on top of it and be very detail oriented.
Steve:So those are things you look for, right. How do people exhibit those characteristics in a way that you can assess?
Monal:Um-hmm. I mean it’s definitely hard to do in a single interview. During the interview process I definitely like to be able to see them in action. We have folks do – typically do like some kind of homework during the process and I often like – like very many companies like to have them do a challenge in the office where we recreate, you know kind of a real life scenario, some role playing, and so we can kind of see how they tackle problems as well as interact with folks and understand how they may work with stakeholders. I mean teamwork, all these other things are also very important, but the three that I pointed out were specific to, I think, user experience researchers that are not necessarily applicable to other fields for specific reasons.
Steve:I like that your three kind of span a research project, for lack of a better noun. I mean you’re not looking at people’s data gathering abilities. Or not just that, but what you do with the data and then how do you help it – how do you help that result succeed in the environment in which you’re working. So sort of managing the logistics of your stakeholders and their participation and kind of their engagement and so on. I like your list quite a bit.
Monal:Thank you.
Steve:I think those three things make sense, but is there anything controversial there or anything missing or where you would dispute how other people approach this?
Monal:There’s definitely a lot missing from the list. I think these are just three things I keep in mind, but of course there’s plenty of other traits that are important. You know, as I mentioned, teamwork, being a fast learner, being adaptable to change. So many things that we probably look for on a holistic level. You know if you want somebody on your team you want to like them. There are lots of things that I think can be applied to many different roles when you join a company. I mean at Lyft we definitely get excited about folks who are also excited about our vision and the vision at Lyft you know is something that attracted me at first. Having lived in Berlin for 2 years, coming back to San Francisco I didn’t know much about Lyft, just because we didn’t have it over there and I’d heard a lot of friends and other folks talking about this company. So I finally decided to give it a try when I was on a visit here and thought it was so interesting, like the whole sharing economy and talking with my driver and I started thinking about Lyft as a company and realized I didn’t know much about them, let me check ‘em out and searched them online and started reading about their story and that’s part of what got me really excited because our founders were really interested – they weren’t just interested in building a better taxi. They were really interested in kind of the long term, big picture vision of creating better communities and reconnecting people through transportation. You know the long term vision is redesigning cities. It’s really cool. I mean this is such a hot space right now and we’re working on so many exciting projects as a company. One of the things I love is that we’re constantly executing. We’re getting things out the door. There’s always something big happening, but the vision is always what keeps us focused on what it is we’re trying to do. And that vision is something that I really can get behind because it makes my job feel a lot more meaningful. I mean I think as UX researchers, you know having empathy is something that we just innately have within us and we desire helping others and so when you look at the meta, for this company that’s what we’re aiming to do as well. So, you know we’re looking at reducing the carbon footprint, getting fewer cars on the road, putting more people in vehicles, making rides cheaper through ride sharing, Lyft Line, and building communities through that experience. I’ve had so many amazing conversations with other passengers or drivers in the car and I always just leave with a smile. So it’s really fun to work on something in the office, take a Lyft home, and actually talk to my driver about oh, wow, you have this issue, we’re actually fixing that right now. I can’t actually tell them that, but it’s really neat to know that we’re making a difference on a daily basis.
Steve:It’s fascinating to hear you talk about the relationship between, especially between you as a researcher, but with your researcher sort of identity on, between being a UX researcher and the vision of the company. So you talked a little bit earlier on about the different users. You kind of said, like you talked about passengers. You talked about people using the app which I guess might not be the same as the passenger. You talked about drivers. Tell me about, who do you study? What do you look at?
Monal:Well as you said, definitely passengers. People who are looking to get a ride, who are using Lyft to take rides. And then the drivers who give rides. So currently the Lyft app that we have today, the single app, serves both of those types of folks. And for people – most people probably don’t ever see the driver’s side, but once you apply to be a driver, are approved as a driver, there’s a whole nother world on the other side of the app. So we look at both those users and more recently we announced at the beginning of this year a partnership with a company called National MedTrans Network. So their users are a whole new set of users for us to investigate and learn about. I can tell you a little bit of a story about them. So they – National MedTrans Network is a company in New York. They work with insurance companies like Medicare and Medicaid whose patients are these elderly folks who basically need routine medical rides, rides to doctor’s appointments. A lot of times – all the time non-emergency rides. And they often don’t need to be transported in an ambulance or any special kind of vehicle. So National MedTrans has all these relationships with various car service companies, including taxis in New York City. One of the interesting things that came to bear was that whenever they have a taxi or car that doesn’t show up for a patient or cancels for one reason or another, which is pretty common, it takes – in New York City it takes them about 45 minutes to get that passenger, that patient, another ride through the taxi company. And so they started using Lyft and found that wow you can get a ride in just a matter of moments using Lyft and get this patient home so they can take their medication, so they can do whatever it is they need to do and not have to wait there on the street or whatever. And these folks have a lot of times more special needs then some of our normal types of passengers. So the great thing about this is in terms of building this new product research was highly involved. And so I went over to see National MedTrans, brought a designer with me and as anyone who’s done contextual inquiries know there’s so much more to learn when you’re in that environment and learning who the users are, what their daily tasks and processes are and the entire environment. And this company where we visited them was like people in cubes functioning as a call center. They’re scheduling rides. They’re working with taxi companies. They’re talking to patients, caretakers, people who are managing transportation needs for these patients and it was really, really insightful for us to be there and use the insights there to figure out how to build the best possible product for these users. It’s something totally different than what we have in terms of our native app offerings today.
Steve:And who will- within that use case are there multiple users there that have different experiences with Lyft?
Monal:Well the primary user in this case is the person at Nat MedTrans who is requesting a ride and managing the ride. So right now a lot of these elderly folks, they don’t even have smartphones. So it is a little bit of a juggle in the sense that Lyft wasn’t necessarily designed for this use case from the first and so we’re getting to the point where how can we build the best possible product. There’s the driver, there’s the passenger and now there’s this middle person who functions as if – kind of like the app does for you or me who might be taking a ride today.
Steve:It just speaks to a really, really (in capital letters for me) BIG idea. A lot of the technologies that we see are apps and there’s interesting – I don’t know let’s just call them democracy or democratic access I guess is probably a better word for this. If you don’t have the income or the technology, sort of know how to use those things you’re shut out of a whole bunch of ways to participate. And what excites me about what you’re talking about is that it’s a fascinating research question then is how do we make non-mobile device users Lyft users? Those principals can apply to anybody but you guys are digging into that. It’s really fascinating.
Monal:It is actually really fascinating and just being there and thinking through these whole – you know this set of issues and this problem, it really brought me back to my roots where this feels like a true user experience problem to solve, you know. Lots of different players, various tasks and processes, how do we make this work?
Steve:I hear the sort of analytical aspect of it as you talk about it, but I also hear your sort of – your point one about empathy. Like there’s something in your voice which won’t come through in our transcript, but there’s something in your voice as you talk about it. It seems like this touched you or this struck you in a certain way.
Monal:Yeah, definitely. I mean the thing that I think struck both me and the designer when we were there was like wow we’re there in this call center, someone called in, a patient who had been waiting and they couldn’t get a ride and how quickly they were able to resolve the issue with just using Lyft. And it just blew our minds. We were like wow. It was just really cool. Like we made this person’s day so much better by just getting them home.
Steve:So that project sounds like a sort of a special one, maybe sort of outside. Or it’s a new area for you to be looking at beyond the traditional set of users that you’ve been thinking about. I don’t know, in the life cycle of doing the research, whether it’s getting your data or helping the teams make use of it, I don’t know is there any spots that you find particularly challenging that you’ve had to dig into?
Monal:Well another thing that actually drew me to Lyft was the fact that I think there are a lot of really interesting research challenge with our product. We’re really on the cutting edge with regard to mobile devices, being completely mobile, having to try and do research in a very constrained environment which is a car. And I think that there are a lot of methodological challenges that come with that. So for example if you think about it the best research comes from real life situations. You want to be able to observe – from of course from life qualitative user research. You want to observe what’s happening in the real world, or simulate it as closely as possible. And if you want to find out what’s happening in ride experience, if you’re in the car you’re already biasing what’s happening there because you need to announce yourself as the researcher. Everyone knows you’re from Lyft. People are going to be a little bit more cautious about how they act. Let’s say take the researcher out of the car. Let’s just put a camera in there. Well then there’s legal constraints as well. You need their permission and if you have their permission then of course they’re again going to be a little bit on guard and this will really bias the results if this is what you’re interested in learning about. So we’ve had to come up with some creative and innovative ways to study some of the things that we’re interested in learning more about and I think along with that you know recording, conducting research, and huge amounts of data are always challenges for researchers across all industries. So specific to Lyft though, and specific to this area in particular, I think we have some really interesting challenges and it’s fun because it’s – you come to work every day thinking about – you’re really stretching your brain and you feel really good when you come up with something new and get to be creative in those senses.
Steve:Are there any process innovations or anything that you can share?
Monal:Not exactly (laughs). I can tell you about one of the first programs that I started when I joined. So we wanted to ensure that we had user testing for most of our projects coming through on a regular basis. So I started a program called Drivers in the Office, so D-I-T-O, which we call “deeto”. And along with that PACSITO, which is passengers, which we often abbreviate as PACS, in the office. So DITO and PACSITO, it’s like a weekly user testing cycle and it’s been a lot of fun. We have people come into the office here in San Francisco. Occasionally travel to other cities and occasionally do some remote studies wherever appropriate. But what we do is we have, and especially for drivers, for DITO, what we’ll do is we’ll actually have drivers come and bring their car, park in our lot and then we’ll have them use a prototype of a new design or product that we are testing and this allows us to capture everything that’s going on while they drive. So it’s one thing to be a passenger, in your house, on the street, looking at the app. You can stop, you can look at it for as long as you want in a maybe normal situation let’s say. But as a driver you have so many other distractions and concerns. I mean it’s a very cognitively complex activity. You are doing so many different things at once and now we’ve just added one more thing to pay attention to which is the app. So it’s a huge design challenge for our team and also from a research perspective it’s really important for us to be able to get the data in the right environment, in the contextual way that they will be using it because they’re using the app at an arm’s length. They can only look at it upon glance, most of the time when they’re driving for safety reasons. And we have to be really cautious about ensuring that it’s not overly distracting. So one of the things that’s been really great about this program is that times when we’ve tested like more low fidelity prototypes where we can’t go in the car, versus going in the car, we’ve learned quite a bit.
Steve: There seems like there’s an interesting, maybe sort of a hybrid – there’s like a semi-contextual aspect here where they’re not in a lab, there’s a high fidelity simulation of what the thing might be. You’re dealing with their actual car, you have realistic use cases, but you’re not also sending them off into the wild to live with this thing for a day. It’s a constrained experiment, but you’re sort of creating I guess as much context within an experiment as you can.
Monal:Um-hmm, right. And I think that’s important for us to – it’s a big part of how it’s going to be used in the real world so we need to capture that – how they do so in that environment.
Steve:What about other examples of kinds of, within what you can talk about, what are some other kinds of research that your team has done?
Monal:We basically try to do as much mixed methodology as needed. We do surveys, as I mentioned contextual inquiries. We’re basically employing everything that we need to design the right kinds of studies to answer the business questions. You know we have some user interviews coming up. We have – what we also try to do is work very closely with our analytics team.
Steve:Can you talk about anything that has – that people that use Lyft has experienced that’s changed as a result of some of the research you’ve done?
Monal:Definitely. One of the most exciting things that I feel user research has impacted was our Lyft app redesign that launched at the very end of last year. So we did a lot of iterative user studies, user testing especially around various versions of designs and the product moving forward and most of the product and design decisions relied heavily on the research that we did. So by the time we launched the app, the new redesign, we felt and our team felt very confident that it would be successful once it launched and it definitely was. We’ve seen really great results and have heard a lot of great positive reviews and comments about what we did and I think a huge part of that, I feel really proud to say, was due to the research that we did. So whenever you can affect the flagship app on such a big scale with this design you can feel really good about it. Of course we don’t take all the credit. There’s such incredibly talented people here that you know we were just lucky to be part of that process.
Steve:I think even just this phrase about sort of being proud of what you accomplished, it makes me think about how do you lead a research team. I mean giving out kudos, I think that’s one part of it. I’m just projecting on you here a little bit, but what are other things. Talk to me about being a leader in research. What does that mean?
Monal:Yeah, I mean it’s interesting because I think until recently there weren’t a lot of leaders in research. Primarily I feel like research has had a lot of individual contributors and that’s how we’ve been seen in a lot of ways, people who are providing data analysis, insights, and hey, great, thanks you did your job. And now with the growth of research and especially user research we’re seeing more roles where there are leaders. And I think it’s great for all individual contributors because a lot of times – and this still happens today – the individual contributors who have worked as researchers in the past haven’t always reported up through someone who understood what they were doing. And so it’s great to see that that’s changing to a degree now and I think just in that sense of having the empathy (back to empathy again) for understanding like what these researchers are trying to accomplish, what their personal goals are – I think a lot of what researchers in general – I’m just thinking back through my career as well – have felt is, I mean we really want to make a difference. We really want to have influence on the product or the design. We did this great research, we want to make sure that it gets in there. And this hasn’t always been the case for me in past companies. And one of the great things here at Lyft is that what we’ve produced has been so highly utilized, probably more than any other company I’ve worked at. So coming back to your question in terms of leading, I think ensuring that the work that we do does get implemented or is seen as respectable, credible, and that people understand what we do and how we do it. And bringing them along for the ride too, it brings us to a point where within the organization you’re seen as oh wow, like they do really great work and they’re going to affect really positive change for us. So sort of helping the team be known and understood as experts and ensuring that the work that they do has a positive effect on the company and for themselves.
Steve:And you’re right. That fell on the individual contributor to sort of – you had to do all that you had to do and you had to be able to do that.
Monal:Um-hmm.
Steve:And so you’re talking about like what a good manager does. They sort of, they advocate for their team, they…
Monal:Um-hmm.
Steve:…protect them and champion them.
Monal:Definitely.
Steve:You talked early on about this really important mentorship relationship that you had in your career. Are there commonalities between what you experienced kind of as the recipient of mentorship and what you try to provide as a leader now at this point?
Monal:Definitely. I mean John who was my mentor back then, he gave me so much and I feel like I’m at the point in my career where I definitely can do that for other folks as well. So right now it’s primarily here at Lyft, but I’d love to do that outside of Lyft as well.
Steve:How do we define mentorship? We talked about leadership, but what’s the difference between mentorship and leadership?
Monal:Yeah, I think mentorship is a little bit more focused on – well in my mind helping folks develop new skills as well. Leadership doesn’t necessarily have to do that. I think leadership can do that. I mean if there was a Venn diagram it would probably be overlapping in the circles, but I definitely believe that a good mentor will help someone on a personal level develop into whatever career goals they have. Helping them along that path to get to where they want to be. And for me I learned a lot of methods from John. I think I’m trying to do that as well on a mentorship level, but also just be someone always who’s there to talk through, you know, career related growth.
Steve:You know in some ways – we’ve talked a little about sort of the evolution of the field and it makes sense that at that period of time developing methods, not that that was all you were learning, but that was kind of a key. And I think about your three areas – I mean the first one you listed, you kind of titled it as empathy and then you listed like 18 other, like really important sort of soft skills that are crucial to being a researcher. So when you’re there for people as you say I think that’s starting to model, yes the methods, but also now these other pieces that are so important to being a good researcher.
Monal:Um-hmm.
Steve:That are just very, to me very human skills that we all can – we all can do to work on.
Monal:Um-hmm, definitely
Steve:Are there other things about you and your background that we should understand to know what kind of researcher you are? Any weird dots you want to connect for us?
Monal:Yeah, actually I think one of the things that’s really unique to me is that I previously had another life as a professional athlete. So I was very active in team sports and that’s a big part of like my philosophy too. Very team oriented. I started playing soccer when I was six years old and I got a scholarship for running when I was at Stanford. So that’s been a really big part of my development and who I am and the really cool thing about running at Stanford is that when I started there and started running on the cross country team there we weren’t nationally ranked. And by the time I was a senior my teammates elected me as the captain of the team and we won the NCAA national title that year which was less than four years later and so it’s been really interesting to draw a lot of comparisons and parallels to my athletics career and how I’ve envisioned like sports and teamwork and accomplishments and my working career. And it’s funny, I don’t even realize I’m making these analogies. I started talking about it to a coworker the other day and I’m like wow I do this actually quite a lot. But there are so many parallels to draw with regard to pushing yourself hard, getting to the next level, goal setting, you know the competition and teams, relating to people, supporting one another. And one of the reasons why I think we won the national title back in 1996 was because we had such a tight, supportive team and I really feel that we have that now at Lyft as well. User research is part of the product design team and we are such a close team. Actually we just came back from a weekend retreat in Tahoe. So we’re kind of like a tight-knit family and I really feel that that support and that kind of environment fuels – helps fuel really great success.
Steve:So some of these parallels you’re describing, they seem about what athletics teaches you about working and collaborating. Are there any threads that you can pull from athletics that go into things that are specific to UX research?
Monal:That’s a really good question. I think I need a minute to think about it. It’s something I haven’t thought about before. I’m sure there are. I think I need to noodle on this, but I like the question a lot. I think because a lot of what – the parallels I’ve drawn have been more meta, like high level. Yeah, nothing springs to mind, but at some point I’ll get back to you. I’ll be like ‘Steve guess what, I figured it out and now my life’s questions have been complete Like I know. I see the light!’
Steve:Do you have questions for me?
Monal:Yeah, actually, so I was curious because you’re one of the few UX research veterans in the field and I was curious what kind of trends you’re currently observing in our little field that’s growing?
Steve:Yeah, I think – I liked what you said about mixed methods. I think that seems to be, you know, where traditionally on paper there was a lot of us and them. I feel like I’m hearing more harmonious stories and I think it behooves us as a practice to tell harmonious stories, but because that speaks to a successful, integrated, mature, not a diva, you’re wrong, this focus group sucked, blah, blah, blah. Like that’s sort of – that’s what we’ve been saying and I think true or not it’s sort of disharmonious and I feel like the stories of how groups have come together, to me that’s like oh we’re a little more mature as a world now because we can coexist well with everybody. And the qual/quant things used to be cats and dogs, but as you said you guys integrate well. I’ve heard that from lots of groups that I’ve talked to and I’ve seen that. You know clients I work with start to say like oh yeah, we have this group, this group, this group and we can kind of poll lots of – so that’s mixed methods. That’s also, you know – I don’t know sort of mixed disciplines I guess. So it seems like that – to me there’s just a maturity level there with that. Yeah, that’s my big one I’ll leave as the answer for that.
Monal:I agree and I really think that in order to have the most robust insights we do need a good triangulation of different types of data to be able to tell that story. I was also curious, I mean as we talked about UX research has come a long way from you know the mid-90s and where the worldwide web was at that time. I mean just technology has come a long way and UX research has alongside of that. So that said, where do you think UX research needs to develop most given our current landscape?
Steve:These are the questions I should be asking you but I didn’t.
Monal:The tables have turned.
Steve:Yes. Yeah, what’s the sort of opportunity? And this came up as a topic in an earlier episode of this podcast series where I don’t know that we answered it, but we talked about just this idea of research sort of evolving to the point where it disappears. That research right now there are leaders that are trying to develop in a business, but – and you’ve talked about the successful collaborations and so on. Is this an activity or is this a discipline. You know I hear so much about teams trying to – I mean one way to deal with the overwhelming demand of research is that research leaders are empowering teams to go do their own research and so then they have to deal with this question of well what do we do as the researchers vs. what do we kind of let everyone do. I mean that’s the consequence of the evolution of research and we’ve been saying, as part of that evolution, like everyone should be doing it, everyone can do it, right. There’s books that teach people how to do research, like that’s a change and so we – I think in the early days we struggled with do we let go of this or do we hold on to it and we chose to do a bit of both. So how does that play out I think is an interesting question? How should it play out? And research doesn’t exist in a vacuum, so how technology has developed. You know we are in a lot of the way – in a lot of environments that research is being used it’s a close colleague of design, right. I mean you talked about being part of the product design organization, not some other – part of the market research organization. That’s who our sort of, our teammates are. And design is changing and you talked about this already too, right. That companies used to just do technology and sort of experience was secondary. So I think where research goes we’re sort of subject to these larger changes in how – in how experiences are created. So I don’t have an answer to that and I’m not sure it’s a need. I don’t feel like oh we better get this sorted out or we’re dead. I just feel like there’s some interesting, sort of existential questions to reflect on. You know I mean over the course of our careers you and I have seen what kinds of organizations and what kind of roles are there for us. It’s shifted a lot and as you said there weren’t leaders like yourself a few years ago and not in the numbers that there are no. So even just for our own careers, like it’s going to be different in a few years. I don’t know. It’s tough to predict the future, but I think those are the – those are interesting, if abstract areas for me to keep looking at.
Monal:Um-hmm, definitely. I mean I think it’s – I agree with you and I also think it’s really great news that the discipline is growing because there is a lot of value that we can provide. And you know the Lyft research team is also growing so I think it’s a really good sign of, you know how much we’re able to impact the organization in a positive way and I love seeing now, after having come back from Berlin – previously it was like oh we can only work at big companies. And now I feel like every startup has a researcher. Maybe not every startup, but it’s becoming more and more common which is amazing.
Steve:You know the designer as founder sort of thing, it’s like a – it’s a thing that gets recognized and talked about and I feel like maybe we identified with – I feel like I was on some Slack channel or some list and people were saying well when do we get our first researcher as founder. I’m probably misquoting someone that said that to me and I apologize for that and that company may exist, but certainly that idea is nascent whereas designer as founder in the startup world at least is well established. Maybe that’s the, you know sort of next wave for us.
Monal:Um-hmm. Cool. I have one more question for you.
Steve:Alright.
Monal:So I’m really curious, and I know you’ve had a lot of questions about this podcast, but I was curious what you’ve found to be the most striking similarities between all of whom you’ve talked to so far.
Steve:Don’t you love when you go out in the field and the person at the end says like “hey what else have you heard?” Or “am I normal?” They want you to kind of normalize them as part of it. That’s not exactly your question here, but yeah, what do researchers have in common? I think the emphasis on product is really, it’s like a thing that we should be proud of as the field and it goes back again to your list of three things that you talked about. As you kind of researchers needing to do good data collection – I’m paraphrasing you here – make sense of it and then sort of help it live in the organization. I am definitely twisting what you said, but that’s sort of thrust I took away from it. And I don’t know, I carry some scars from my early days of coexisting with people that – where research was their end – they loved – and you know I guess scars, but I’m also definitely guilty of this myself. Like research is fascinating. If you do it well the experience is great and you learn things that have no relevance. And I mean I’m a big fan of not knowing if it has relevance, but you learn things that you don’t know if it’s what you’re going to do with it, but you also learn things that just are great and they’re not applicable, but are part of what you have to do on that journey. And then you make sense of things and you find some really interesting takeaways, or things that are potential takeaways that aren’t necessarily usable by the company. Of course then you find here’s the things that we learned and here’s what we need to do about it and here’s all – but all the way along there’s a lot of fascination, right, like…
Monal:Um-hmm.
Steve:…it’s super engaging and to me very creative and I don’t know. I can be seduced at many points along the way. I mean, I don’t know – I think I’m good at making sure I get to that end point, but there’s a lot of seduction along the way with just stuff that is super inspiring, super thought provoking, that you kind of dig into and start synthesizing and organizing. So the scars are around, you know maybe sort of early days in the field, or my early days, where that’s where people kind of dwelled and someone had to come along and kind of pull them and say like okay you can’t deliver that. You have to show actionability. That sort of discussion was needed. And I mean it makes sense that the people that I speak to, like you who are leaders – I mean in order to be a leader you have to be beyond that. There may still be practitioners and contributors who kind of need to learn more of that, but I feel like if I take the temperature of the practice in terms of how it’s being led it’s about what we do with research.
Monal:Um-hmm.
Steve:And that is a massive change again over when I started in my younger days. And so to see that in common – again, maybe you go well of course Steve look who you’re talking to and what kind of jobs they have and look at what kind of companies they work for. You know again, yeah, you might find people at that skill level, like yourself, and experience level – if we were to like shift back 10 years you would be maybe working in an R&D lab or an advanced products group or something and not shipping stuff. So yeah, I think one of the commonalities is like these are people making products, or helping companies make services, whatever the thing that their company does they’re doing that and obviously that’s – I think that’s really, really good.
Monal:Um-hmm, yeah. One of the things that struck me about what you’re saying is that fascination and I think as researchers like a lot of what we love is having that curiosity and it’s so fulfilling just to do the hands-on work. My role is managerial, but I’m always still – you know I do some hands-on, individual contributor work as well. I just love doing it, right. That’s what makes us passionate about research, we love doing it, but I think in a business one of the dangers is getting too far into that fascination, but then not taking that next step. And maybe that’s where managerials, or managers can help with making sure that the data and the analysis and then these insights and actionable recommendations happen and are affecting product. That we’re not just doing the research and being fascinated for our own curiosity and fulfillment of that, but that we’re actually making sure it’s being put to good use.
Steve:I think that’s really important and it reminds me of just a period in my own career where I was managing my own team in my consultancy and I think like – we always had many different projects for different clients and I want to say maybe for a period of 18 months, which is a long time in sort of consulting years right, I didn’t go in the field at all. I was doing what you’re talking about, right, sort of managing the problem, helping with the synthesis, kind of trying to tie everything back. Sort of a creative lead and it was really cool to sort of – you know you remove one part of the puzzles and the other parts – I guess the other things kind of rise in prominence. So it was real interesting for me to step out of certain tasks and step into other tasks in a larger way. And I actually remember the first time that I went – it just worked out that like oh yeah I was going to do some of these interviews and I almost had a panic attack. Like it had just been so long. And I was not on my own. I just remember like kind of coming up to the door of somebody’s and just about freaking out. And you know the way I work now I’m sort of – I do more of everything, but having had that experience of sort of again playing just part of the role was really, really interesting. It’s good to hear that you – it sounds like you’re in the right spot. You still have your hands in things, but you can lead and kind of manage in that spot.
Monal:Um-hmm, yeah.
Steve:Alright well that’s maybe everything we have time for to talk about today. But this was really interesting. It’s one of these conversations where now I’m curious about a million other things. I’ll have my imaginary next 90 minute conversation with you. But yeah, thank you very much. You’ve really shared a lot today.
Monal:Thank you Steve. This was a lot of fun and yeah, I really enjoyed it.
The post 14. Monal Chokshi of Lyft first appeared on Portigal Consulting.
In this episode I speak with Kate Lawrence, Vice President of User Research at EBSCO Information Services. Our conversation covers where to place user research in the organization, emotions in fieldwork, and empowering others to advocate for information literacy.
The post 13. Kate Lawrence of EBSCO first appeared on Portigal Consulting.
In this episode I speak with Kate Lawrence, Vice President of User Research at EBSCO Information Services. Our conversation covers where to place user research in the organization, emotions in fieldwork, and empowering others to advocate for information literacy.
The ability to write, synthesize, distill and synthesize findings into a report that is digestible and spot on is very challenging and there are people who can do it well and I really strive to find those people for the team. So great writing skills – and part of that writing is storytelling. It’s a combination of communication, analytical skills and research skills. – Kate Lawrence
Show Links Kate Lawrence * Follow Kate on Twitter * EBSCO Information Services * EBSCO Industries * Ipwich, MA * Mark Zuckerbeg’s hoodie * Shonda Rhimes’ Year of Yes* * Qualtrics * UserTesting * Pizza in Japan * Cruella De Vil * Affectiva * Eat This Not That * User Research War Stories * Deirdre Costello * Kristen Arakelian * Flipster
Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes.
TranscriptSteve Portigal: Welcome, Kate, to Dollars to Donuts.
Kate Lawrence:Thank you for having me.
Steve:Why don’t you begin by introducing yourself?
Kate:My name is Kate Lawrence and I’m the Vice President of User Research at EBSCO Information Services. We are the leading provider of research databases and other content to libraries and institutions all around the world and we’re headquartered actually out of Ipswich, Massachusetts – a beautiful town on the North Shore of Massachusetts, north of Boston. And our parent company is actually EBSCO Industries and they’re headquartered out of Birmingham, Alabama. And EBSCO Industries is one of the 200, one of the top 200 privately held companies in the United States and EBSCO is actually an acronym. EBSCO stands for the Elton B. Stephens Company and it’s a family owned company since the 1940s and again it’s one of the largest privately held companies in Alabama and one of the 200 largest in the United States and what’s interesting is one company, many different businesses and so the largest part of EBSCO Industries is EBSCO Information Services where I work in Ipswich, Massachusetts, but what’s interesting is EBSCO Industries has lots of other businesses. For example EBSCO owns the company that produces the most fishing flies of any manufacturer. They have real estate holdings. So it’s quite diverse in that way.
Steve:Does that create a certain culture and not so much EBSCO specific, but companies that have diverse holdings – and I don’t know if you can even answer this, but you know we hear a lot about you know you work at a start-up, you work at this kind of company, but just you work at a very kind of unique organization. It’s part of a very diverse, larger company.
Kate:It is and what makes it fun and unique to me is every day brings something different. So the part of the company where I am focused is the Information Services division. So we’re – we work on products and services that provide essentially information, premium information, to information seekers around the world. And occasionally I will get pulled into another project. So if there’s – we have a – there’s furniture company that makes work stations and so we did this whole research project looking at what are the furniture needs of the new library – you know the new information commons? What is that collaborative furniture that is needed, that’s different from the study carrels of yesterday? So we occasionally get pulled into projects and I’ve actually done a small research project on game cameras for one of our other outdoor product products called PRADCO. So you do get these projects peppered in that keep it very interesting. Absolutely.
Steve:It seems like the first example leverages an area that you’re focused in, that the information work – the information, information user is already focused on the library and people using the library and the work that they’re being done and if I’m guessing the game work is much further outside that.
Kate:It’s true. The game camera work was outside of that. What we do on a day to day basis is our research team focuses on studying the path and the search process for information gathering, information seeking. And like you said libraries are customers and think about libraries and think about library patrons. You have your academic market. You know you have students as young as age 6 and 7 and 8, starting to work on research projects. I mean it might just be the history of the Lego Corporation or finding out about Harriet Tubman, but that’s research. And so we have young students, elementary school students, all the way up to 95, 98, 101 year old public library patrons. So all of us interact with the library in some way and that’s why you see EBSCO products all over the world.
Steve:You know when you describe the young kids doing the Lego Corporation history research project it makes me think that there’s some major shifts that are going to impact the business that you’re in and what your research is going to uncover. Education continues to but what are some of these larger changes that we as a culture are going through that are driving changes in your product and things that you’re looking at in research?
Kate:Changes are because of Google. Google has changed everything about information seeking and information gathering and search. And Google being described by many of our students, they’ve described Google as their oxygen, their mother, their water, their soul mate. And they are completely serious and it makes perfect sense why they would describe Google that way, but Google has been this seismic shift in the way that people conduct searches for information. So when you think about, Steve, when you – when the two of us were in college, when we had a research paper to do we went to the library. We worked on it there and do you remember – everyone remembers the smell of the shelves. You know you went in and you – the library had a certain smell to it. It put you in a whole different zone when you were doing research. Research is different today. It’s a lot of guided self serve because you don’t have to be – because of electronic resources and you can access these materials from home, you can log into your library website from Starbucks, anywhere. You can conduct research on the go and so you’re not necessarily within sight of the librarian. You’re not necessarily even on campus. It’s a different experience and it actually starts in Google.
Steve:So – and you speak about this in a way that makes me think you’ve talked with a lot of people about this over the years.
Kate:Students are an area of tremendous focus for the user research group at EBSCO and as I was saying we have students all over the world. And students by the way is that third grader all the way up through graduate and doctoral students and fellows who are doing intensive research. So you have people who are doing research for a basic introductory class and they’re not particularly invested in the topic – may be in their first semester in college. And then you have your doctoral level researcher. So when we set out to design products and experiences and we think about interface design we have to think about this range of abilities and at EBSCO our customer and our user are not always the same person.
Steve:So the people that are making choices about what we call information products, information systems…
Kate:That’s right.
Steve:…they are choosing something that’s going to be rolled out to students, I guess, and other users as well.
Kate:Students with a very different approach to the usage and very different knowledge of the topic of information seeking and very different level of searcher. Sometimes I equate it to trying to appease the parent through a great experience for the child. So you think about parents buying products for kids and then you think about who are you trying to suit? Who are you trying to serve? And what we’re trying to do is create an experience for research that is delightful for students and then as a result also… also pleases the librarian, but pleases the librarian customer because it’s serving students so well.
Steve:So that’s an interesting area to think about. If as you said Google has really, really changed things – you know I’ve done a little bit in this area so I have a little knowledge which is dangerous – I mean when you describe these two groups, the people providing the service – the facilitators, you kind of used the metaphor of parents – librarians and people in that role. You know my casual observation is that this shift – you know the shift that’s Google and kind of everything that falls under the umbrella that you’ve related, how they’re experiencing it, and this is just natural – how they’re experiencing it as professionals and people with a lot of history with tools as they’ve evolved, and how these students are experiencing it, they’re looking at that shift very differently.
Kate:They do look at the shift differently and it reminds me of when – I remember when Mark Zuckerberg put on that hoodie and started wearing that outfit to work, it was upsetting to people. There was a certain generation and a certain group of people who are really offended that someone who runs a company is going to show up in a hoodie. And I think like – it’s kind of like when parents say “oh the kids today.” There’s a sense that we don’t have to do it the way our parents did it. We don’t have to kind of treat research like the Oldsmobile that my parents had to drive. Like I don’t – you know look Mom, I don’t have to do research the way you did it because I don’t have to be in the library and you had to be in the library. So I think we’re all still adapting and coming to terms with the new rules of research and the new way that students are getting through and studying in college. It’s very different and when you said it sounds like you’ve studied a lot, we do go out and interview users. We do employ ethnography and we sit with students on the floor of the coffee shop in Las Vegas, or right outside their community college. Or we sit with them after swim practice in Texas where they’re in a breezeway in a library trying to do a little research on the go right after swim practice. And so we sit and we watch their organic search process because until you see it, it’s hard to believe that it is the way that it is and it’s piecemeal and it doesn’t necessarily have a solid strategy, like this is how I’m going to go in, this is how I’m going to do it. There’s a lot of trial and error. There’s a lot of trial and error and it absolutely starts in Google because they feel safe there and starting your research in Google by getting this foundation of your topic, it reduces anxiety. And research can be, for some students, a very anxious process.
Steve:I’m thinking sort of the network of – I want to use the word stakeholders in the – in the broadest sense possible. You’ve talked about students. Certainly for younger students there’s the parents. There’s the people that are providing them – let’s just call them librarians, is that the right term? People that are providing them with these tools?
Kate:Yes.
Steve:And those are kind of all outside and then inside your organization there’s you and your folks that are going out and talking to these people. There’s somebody that is taking what you’ve learned and acting on it in some way. What – very generally what does that node look like?
Kate:That is, at EBSCO what we have is we have evidence based product development and we’ve embedded user research into the product management organization for this reason. So what will typically happen is we have achieved in the last – I would say 12 to 18 months – we’ve been able to achieve alignment. And so we understand, for user research, what is coming 12, 18 months from now. So we’re in front of it, we’re doing the research now. And so that was one of my key learnings when I first started the group and I first started doing user research. I did every single project that came my way and we joke about the year of yes. You know Shonda Rhimes just wrote “The Year of Yes” and I said I lived that year. I said yes to everything just to get the demand so that I could get the resources. And what happens when you do that is then you’re doing research on everything, but strategic alignment is critical. And so now we’re very much aligned with what the product development cycle is and we’re able to do the research at the outset instead of what used to happen – and this is typical of many companies. Years ago it was let’s do usability testing before we launch and that is a tough situation for anyone to be in because then you’re this kind of glorified version of QA and you’re just – you don’t have any chance to make an impact if something isn’t testing well.
Steve:Can we pause that thread and can we go back to something? I want to talk more about how this has become adopted and integrated. But can we go back to this earlier point and sort of who’s involved in it to me seems really challenging. So you are sort of uncovering new paradigms. I think there’s – they’re kind of, they’re transgressive paradigms (my word not yours) – but they are changing the way that we think about how people are searching, what outcomes look like for information problems and so on. You’ve established how that works between different parts of the organization internally. And again I’m projecting here, but if I’m in the business of selling products to these librarians and these products – it’s what you said before about trying to please these different audiences. These products have to – they reflect a new model for how people are going to use them, but that new model might be a little – I don’t know, how is that new model being received by the librarians, by the people that are sort of selecting and rolling out the kinds of tools that you have to offer?
Kate:I have been so encouraged by the librarian reaction to the research because one of the talks I give, one of the themes of a talk I give on a regular basis is called Student Researchers – The Reality Show, or the Realities of Student Research – because it’s very different and it’s librarians who come up to me afterwards and say thank you, you’re validating what I’m seeing and now I have greater context. And also many of our findings support the role of the librarian in an active partnership with faculty. Let me give you an example. You are sitting in your Intro to Philosophy class. You’re a freshman at University of Virginia and you’re there and all of a sudden you hear I’m getting a research assignment. And so this kind of fills you with a little bit of anxiety and the professor is giving information about you have to use scholarly sources, things have to be peer reviewed. You’re getting kind of the elements of the rubric. So New York Times, Time Magazine, these are not written for academics. These are not written for an academic audience. They’re not peer reviewed. They’re not considered scholarly peer reviewed materials. So that’s another point of anxiety for students because they have to go to materials they may be not be familiar – as familiar with. Something amazing happens when in addition to the faculty member, your professor assigning this work, there is an amazing phenomenon that occurs when a librarian is brought in to the classroom – okay, taken out of the library. He or she leaves the library and is in active partnership with the professor, talking about this is how to do this, this is how you navigate the library website to find these scholarly databases from EBSCO, etc. It’s the equivalent of when your parents – both your Mom and your Dad – say Katherine, we’d like to speak with you. Something in you is queued up to listen and you’re more apt to learn it and you’re more apt to learn it and absorb it. As opposed to the professor saying well next Tuesday you’re going to meet in the library and the librarian is going to walk you through X or Y. So what happens when we talk about this is some of these findings they empower librarians to advocate for a more active role in this process of teaching students information literacy skills. And that’s exciting to see when they say this is going to help me help my students and they appreciate that. And I’ve been – I’ve been overwhelmed by just how wonderful the librarian community is and how supportive our customers are of our work.
Steve:That’s really fabulous. There’s something too – can you expand a little bit on this notion of – you’re talking about kind of the librarian in the library and then the librarian in the classroom.
Kate:Yes, it’s so interesting.
Steve:There’s some kind of frame shift that’s happening there.
Kate:There is. And so think about – think about the amount of effort it takes you when you’re in the library to get up, leave your laptop unsupervised, walk over to the librarian desk and ask a question. That’s actually harder than any of us might imagine. And so also when you’re in the library, what we’re hearing from students is often they are collaborating or they’re having a meeting and they’re with other people. So to interrupt that and get up and go ask for help is hard. And other obstacle is some students don’t know how to ask. And I have a customer I’ve worked with over in Europe who said that they actually post signs at the reference desk that say – you know how to ask. You know ways to frame your question because that’s an obstacle. And then we’ve had students say to us you know I want to – they actually feel very positively about the librarians. They’ll have Live Chat and they said oh the librarian came to my class. He or she is so nice. But there’s still – are you using a library? Are you using librarian services? Are you going to the reference desk? No, I’m just asking my friend or I’m asking my roommate. And the reason that is happening is – goes back to what I said about a lot of this searching is guided self serve and the guided part is – and the self serve part is it’s 11 o’clock, they’re home, they’re on the couch, they’re not in the library per se. And so who is nearby? That’s your go to.
Steve:Can you talk a little bit about the, I don’t know, mechanics, logistics challenges – you know these kinds of, the people that you’re doing research with and then you described some really interesting contexts that you’re in as well. Are there things that you have had to figure out to make that kind of an established process for your group?
Kate:So working with students is fascinating. We also research physicians. We research nurses because EBSCO has medical products. We research, again, the public library patron which is all of us from the time we’re two, or two months old, to the time when we’re a hundred and something years old. We have relationships with our public library. And we also research corporate users. So we’re – EBSCO’s user research team – in our team we are researching people all the time and I always say that our most important job is matching method to question because everyone gets really excited about big contextual inquiry, ethnography studies, but really that’s not appropriate in every scenario. It’s more of a tool when you’re examining markets. Like let’s study the public library patron, let’s study college students, etc. But you know what I find – one of our greatest challenges is we have to know where to spend our time and so we rely on – we rely on this technique of – that I talked about with the strategic, strategic alignment to understand what’s coming, what are the priorities and how can we do research to best support that. And we have some great product managers and product directors who help – who always come and knock on our door and say I’ve got this thing coming up, I’m going to need research for it. And so we’re – culturally we’re at that point where we’re not having to chase people down anymore. They’re actually coming to us and we have a backlog and we have a wait and we conduct probably close to 80 studies a year and we hope to increase that with every new person we add to the team.
Steve:So, I mean your point that not everything needs to be a contextual research study, what are some other methods that you rely on?
Kate:We have used – we’ve used lots of different methods. So we’ve used traditional surveys. We subscribe to Qualtrics, and we also – usertesting.com has been a great resource for us because it’s allowed us to, you know just do more work in a shorter amount of time and it’s a way to kind of expand the group without adding resources. I will tell you though some of our most fun work has been methods that were new to the group, like for example we did a modified video diary study because we study students and we study students who are under the age of 18 and so we deal with legal issues and consent issues and one of the things that’s challenging about researching 14 year olds, middle schoolers and early high schoolers, is that especially 14/15 year old young men, boys, they don’t necessarily, they’re not the most talkative group in the world. And so I’d read somewhere that Google had put video cameras in envelopes and mailed them off and asked students to – younger kids to track themselves searching, etc. So we basically sent off these video cameras and I think I only spent $50 on each one because we weren’t sure we were going to get them back. So we sent them off in these padded envelopes with some tasks for users to complete and it was great. And it’s a method that we want to use again because we got the quiet 14 year old boys, we got them talking. And they told us lots and lots about how they conduct research and what they’re looking for and they evaluated screens for us. And every single video camera came back. Not a single one was damaged. It was a great method and it was so much more successful than it would have been if we’d had them in a room, asking questions.
Steve:Because they had sort of their own space to respond.
Kate:Yeah and it was this – it’s the selfie generation. And so it was just this kind of extended selfie and it was just extended selfie and it was really precious. It was precious seeing the things – just – they took pride in it. They created a little movie of their life and how they conduct research and we got to be privy to that. I mean the challenge is if you put those 14 year olds in a room, because they’re under 18 we have the parents in there as well. So you’ve got the researcher who’s like a second parent and then another parent, and it just – it’s not an opportunity – it’s not a situation where they’re very talkative in person, but we did manage to get some information out of them through the video cameras.
Steve:You point of the selfie generation reminds me of this great video from a couple of – I want to say there were two Canadian students that were living in Japan. I think they were undergraduate students and they were doing these sort of mini auto ethnographies, for lack of a better term, where they did one video where they ordered pizza and they showed what the interface looked like and you know – I mean Japan is such a great place to talk about how things are different than what your normal reference frames are. So they kind of called them all out and then when the pizza came they sort of showed what the food looked like. They went through sort of, like the whole user journey, to use some jargon – they went through that whole journey in a very delightful, YouTubey kind of way and it made me think about maybe self reporting is – culturally can take on more of a role and you just proved my hypothesis with your camera study.
Kate:Well you know the raw self reporting, that’s what I love, because so often when you’re filling out a survey you’re self report… Our self reporting – I mean this is why we go study users in the wild because people do things differently than they say they will. And I always tease my husband because he’s the only person when you go to the physician for your annual physical – you know the physician will say how many times a week do you exercise, like how many – do you eat all your vegetables ,etc. And he tells the truth and I said no one every tells the truth. No one ever self reports accurately. And they account for that. But he does. And this is why we study people in their environments. This is why when we go out we always come back with new information that’s so fascinating to folks because within the walls of our company in EBSCO, here in Ipswich, you know we make assumptions like everyone does and then you go out and you get this raw data and it’s exciting. So I like the idea of an ethnography app and I like the idea of the raw data capture because I want that. It’s those words and those truths I think from users that they pivot products and they really launch products that are so much better targeted and better suited for the users.
Steve:How do you help these folks feel comfortable or feel like they’re doing the right thing?
Kate:The same way I work to make people successful and then comfortable in an in-person interview. So let me give you an example. Years ago I had a boss who said to me the reason you’re good at usability testing Kate is you serve it up cold. You don’t give any emotion through the process. So if someone is doing something right or someone is doing something wrong they can’t read it on your face. And I was proud of this for about 15 minutes and then I started to realize over the years that that Cruella De Vil kind of cold fish approach to usability testing and user interactions is actually awful. And the reason it’s awful is because you need a genuine connection with people for them to tell you their truths and I have learned – I learned that the hard way as a researcher. And so when you connect with someone in person it’s that warmth. It’s that finding common ground that’s important so you can get – you can get information from that person. They can tell you what’s painful about the research process? Or they can tell you oh I’m trying to book an online cruise and this is what I don’t understand. And then you can actually take those words and take that wisdom and go back and tell the product team, like this is how you can really help these users, these customers. But you can do the same thing if it’s not in person. If you have a self reporting tool – like when we did the video diary study we wrote instructions like we were talking to 7th graders because if you send them a packet that reads like tax filing instructions, you’re not going to get any answers and you’re not going to get an video cameras back. So we put ourselves in the shoes of the user. What is their reading level? What is – how are they likely to talk about this? What terms – I mean they’re not talking about information literacy. They’re talking about finding articles for a paper. So we try very hard to speak their language and I think that’s one of the skills I value, it’s one of the traits I value most in our researchers is the ability to adapt and understand and listen and then adjust accordingly.
Steve:Which you can’t do with the instructions. You have sort of one shot at that.
Kate:You have one shot at the instructions and so you have to write on their level. You have to consider yourself as the user. And you may have to do some sort of pre-survey or pre-work in order to get – in order to get a sense of how they’re going to process the information. For the video diary study our researcher Lin Lin and myself we actually tested the testing instructions. We went through several rounds with people saying, you know do you know what to do here and you have to. So often we’re testing the survey or testing the test instructions to make sure that we’re being clear.
Steve:Right the research – research artifacts are, they’re a designed tool.
Kate:They’re a designed tool.
Steve:And you have to test them just like anything else that has to be tested.
Kate:Yeah, and I think that – I think that we can all get to a point where we feel comfortable. Like okay I’ve done this kind of test before and this was understandable, but you know recent experiences – I’ve had a couple of surveys that were challenging – we got feedback after the fact like this was challenging or – and so now we’re trying to be better about understanding how long the survey should take someone and then adjusting compensation accordingly. Because we had a survey that we thought we – I think we gave a $50 Amazon gift card for, but it ended up taking people almost 40 minutes. And so we realized we have to be better about understanding the amount of effort and how we’re going to compensate for that.
Steve:Can I go back to your Cruella de Vil?
Kate:I know, do you like that?
Steve:I do. So – ‘cuz I think this is an interesting personal style of fieldwork thing.
Kate:Yeah.
Steve:I’d love to get you talk a little more about – you sort of learned what not to do and I wonder if you could characterize what your approach is now in terms of common ground and so on.
Kate:So when we were talking about the usability testing and serving it up cold the whole approach was very kind of generic and you know show no emotion. So people are succeeding or failing. It was very metrics based, like oh did they succeed in this task to add X to their cart, or whatever it was. And so kind of I think as a person you evolve to start to understand that it’s about the holistic experience. It’s not about this one slice of the UI, it’s about this journey of I was in online travel at the time when that happened and it’s about this journey of how do I start to think about what trip I want to book for my honeymoon or what flights I want to book or what vacation package destination I’m thinking about etc. So as you start to think about the whole journey you realize that you’re not just, it’s not about just succeeding or failing. And so the warmer tone – you know you kind of go from Cruella de Vil to I guess more – to more of like the Dr. Oz who’s asking the questions about why and trying to unearth and dig at the whys. And trying to unearth those pain points and trying to look at it from a standpoint of empathy and understanding and not just let’s see how long it takes this user, this participant – how long it takes this person to find the shopping cart button. It’s really about what are your struggles? If you could change any one thing about this process or this product what would it be? You know and when you’re – we always say to users, we always say to students when we study them today, we say the process – how does it make you feel when you get a research paper assignment? How would you complete this sentence – the process of you know finding research for my paper makes me feel – and we ask them to fill in the blank. And through those words and the way they complete the sentence we start to get that picture and I don’t think it’s a coincidence that EBSCO and our product development team and as a company we’re starting to consider emotion when we think about not only the process of buying products as a librarian, but the process of using them as students and other end users. It’s not a coincidence that our research group is all women. I think that we’ve brought that emotional dimension into the discussion in a way that had not been present prior.
Steve:So what I heard is that a way to bring that emotion in is in the questions that you’re asking.
Kate:Yup. In the questions that we’re asking and also we’re going to be exploring later this year emotion analytics. There’s a company right in our backyard here in Massachusetts called Affectiva and they’re out of the MIT labs. It’s an emotion analytics product and it’s something we’re very interested in because for many people conducting research – we have this wonderful researcher on our team, our lead qualitative – our lead researcher. Her name is Deirdre Costello. She’s a librarian and a researcher and she likes to say conducting research for many students is like eating your vegetables. And so you don’t necessarily look for the emotions of they’re jumping up and down or they – like you’re not expecting people to look like they just hit the winning lottery ticket. But you’re looking for certain emotions to register on their face as they’re finding the path forward toward these scholarly resources. And so this is something that we as a team are going to be exploring later this year is how do we – how do we capture and quantify those emotions and what about the emotional state changes with certain design changes? Or changes when you’re looking at different products? We’re going to be exploring that and that is, that’s really new research for our group. We’re excited about it.
Steve:This is a tremendous evolution from this Cruella de Vil era of sort of looking at metrics.
Kate:Metrics.
Steve:Looking at a lot of other kinds of things.
Kate:Well you know that same boss said to me – and actually he’s a really – he’s a wonderful old smart guy. We were all learning at the time. He was the one who said to me I want the findings written up as a white paper, really in white paper format. And so I did that and I would pass people in the hallway and say hey did you read that report? And they’d say yeah, yeah, looked great. And I could tell that they never read it. And so the next time I did a usability study I said to my boss at the time I will write the white paper, but I’m also going to create this parallel version. And I actually based my reports – I did them in PowerPoint but I made them very visual and I based them on the format of that book written by the editor I think of Men’s Health or Men’s Fitness Magazine called “Eat This Not That” because someone, somewhere in my life had that book at their house and I was leafing through it and it was these visuals like look if you go to Panera don’t eat this, eat that. You know this is healthier for you. And it was just very visual. And so in 10 minutes you get a sense of alright if the only option is Jack in the Box this is healthy option at Jack in the Box. And so I did the report – like look users don’t understand this, but they understand that. And I did it in a very visual way, modeled after that book, and all of a sudden we got traction. All of a sudden the people who weren’t reading the reports were calling me into meetings and saying I want more of this, how can I get more, I love this. And so creating results that tell a story in a visual way is very important. Having said that we also in our group now, we do write the white paper and we do speak at conferences and do different things with our findings because we need a variety of different deliverables to satisfy our customers.
Steve:Right. You have different audiences…
Kate:We absolutely do, yeah.
Steve:Can we talk a little about – you made the comment it’s not surprising that your group – did you say it was all women?
Kate:We are. We’re all women and we have a male intern in the summers who is wonderful. So we are certainly welcoming of – we’re welcoming of women and men, but at the moment we are all women, yes.
Steve:I mean what do you think that’s about? ‘Cuz I think if you look at the field of user research in general, and this is just anecdotally for me, it’s you know I think predominantly women is my guess. And someone may be rolling their eyes at that, but that’s my sense of it. I don’t know, what do you think is happening there culturally?
Kate:Half of my group also have librarian degrees and I think librarians you find mostly women. So I do think the qualities I look for in a researcher, you know we are of course gender blind and so I think these qualities can exist in anyone. I think – you know some of the qualities I look for – we want people who are highly adaptable, curious, great communicators is key. And I was mentioning Deirdre previously. Deirdre used to be a writer – she taught writing at Boston College. So the ability to write, synthesize, distill and synthesize findings into a report that is digestible and spot on, that is very challenging and there are people who can do it well and I really strive to find those people for the team. So great writing skills and part of that writing is storytelling. So I’m not sure if you said to someone who do you think has better communication skills, men or women? It’s a combination of communication, analytical skills, research skills. I think it’s just coincidence that we have all women, but boy we have a terrific group. I mean I always say I’m loving this part of my career. I’m loving it because of the chemistry we have in the group and I just – you know sometimes when you’re in a job that’s frustrating you think well maybe someday and I finally found my someday. So it’s nice to know that it’s out there.
Steve:I finally found my someday – seems like that’s a song they should play at somebody’s wedding.
Kate:I know, but I finally found that environment and – but you know part of that is you have to build the culture that respects user research and you have to be – you know my boss is very supportive of our work and if he wasn’t we would go nowhere. I mean you have to be in the right place in the organization working with and for the right people and you have to hire well. You know we actually have an opening right now and we’ve decided that we’re going to just wait for the right person to find us because when you go out and you recruit the person you want isn’t necessarily available at that moment and you may not find that person and then you end up with a pool of six people. You narrow it down to three, and then two and then you end up choosing that person who maybe in a different world you wouldn’t have chosen. So we just have this fantasy that the right person will find us.
Steve:Well and I think – you know your catch phrase is I finally found my someday, but you’re talking about building it. I think you built it.
Kate:I built it and so when you – when I started at EBSCO it was – let’s see it was almost 5 years ago and so when I started at EBSCO user research really hadn’t emerged to be its own entity yet and I remember my husband – I came via the information architect path, up through user experience, and my husband would always say to me why don’t you focus on the research? It’s what you like the best. And at the time research was – there was a lot of usability testing, or at least my version of it and I just said I don’t that I could do that all the time, but he said but you love it and I said you’re right I do. So I took a job at EBSCO and my first job here I was focused on usability, but I was also helping to write requirements. Like I was a requirements analyst for these products and platforms. And I remember at the time – I conducted I think two usability tests and I realized there was this big need – there was a major need for someone to focus exclusively on usability and user research and previously this had fallen to multiple people. It was a dimension of many people’s jobs. And I went into my boss at the time, named Ron, and I said Ron I would like to focus just on the user research piece and he said well I can’t let you do that because I still need you to be able to write requirements if I need you to do that and I said okay I’ll make you a deal – if you let me do more usability and user research I will write requirements whenever you need me. And he said yes, ‘cuz you’re definitely not going to have enough work for just the usability piece. And I said okay, deal. So we had a deal and I never wrote another requirement because the demand, that was my year of yes. I said yes to every project that came my way and then I was in technology at the time and then I moved over to product management and things really took off because then I was embedded in the product development life cycle.
Steve:Is there a point at which doing research sort of morphs into leading research?
Kate:There is and that is how a bill becomes a law or how a girl becomes a woman. You know it’s really – it’s quite a transformation. It is. Like all of a sudden it’s – it’s growth, it’s evolution and what starts to happen – I remember you go from doing testing and doing research to talking about it, to evangelizing for it, to finding it a home within the organization. And that’s where the leadership engine kicks in. And I think one of the reasons I feel well supported at EBSCO in that people value the work of my team and when they promoted me to Vice President that was really exciting for me personally and also for the research in the organization to be given that level of visibility and that was exciting. And it’s, it’s – it means a lot to our customers that we have prioritized this as a function within our company and our organization and so when they buy our products, or they invest in a partnership with us, they understand that we are looking at our users 24/7. We have a relentless and a passionate focus on our users and customers.
Steve:And you described those different milestones in that evolution and was becoming part of the product management organization a significant point along those different stages?
Kate:Absolutely because products – this is where the products – this is where the products live and this is where they’re created and this is where they are evolved. And so when we were located in technology that was fantastic because we were where things were being built. But a lot of the upfront research needed to happen – it needed to happen sooner than we were getting it in technology. And so moving us over to product development was a real culture shift and it was an important one in our growth. And something in user experience, I constantly had that debate of should I be in – at the time at other companies it was should I be in marketing? As a UX person should I be in marketing or should I be in technology? And actually at the time I always opted for technology. And so it was a shift for me to be at EBSCO and say you know what I think we should be in product management and I think that’s the right place for us because my whole career I’d actually fought as a UX person to stay in technology. But at EBSCO it made more sense for us to be in product management. And now, again, we can achieve that alignment a lot better because we’re where those discussions are happening.
Steve:Right. I want to follow-up on something else you said as well. You described these qualities that are just so essential, that you look for in researchers. How are those – how would someone make those qualities visible?
Kate:That is a great question because how do you – how in an hour interview do you say oh well I’m curious, I’m a great communicator, I can distill down findings. I always say the great researchers are like Navy SEALs. They go in, they adapt, they accomplish their mission, they leave without a trace. Because you want to be efficient. You want to go in, you want to get the job done and you want to come out and you want to leave people kind of unruffled and let them go on with their lives. And so I look at communication skills before I even look at research skills. So for example you wouldn’t believe some of the cover letters we get or some of the emails and you know I have very – I actually have – I can’t say I have a low tolerance – I have no tolerance for things like bad writing and bad grammar and so that, because our work is published externally, that’s just a deal breaker for me. There’s a curiosity and a work ethic that we look for. I just wish you could sit in our team meetings and meet some of the researchers and – it’s funny my husband actually calls Deirdre, our lead researcher, he calls her The Oracle because she knows everything. I mean you can sit – you can sit her with a fifth grader to talk about research and she can make a connection and that yet she’s amazing with 60 year old esteemed department heads of you know medical or surgical departments when we’re doing physician research. So she can – she can do it all and my friends – I remember my friends and I, when we were all dating before any of us got married, we had this joke about we’re looking for the guy who’s tent to tuxedo. You know he can go and rough it in nature and then he knows which fork to use at the black tie dinner. And I think as researchers that’s what I appreciate. I appreciate the person who can kind of rough it in the wild and then be completely comfortable at a black tie dinner and everything in between because we research everybody and you have to be able to connect across that continuum.
Steve:Can you talk a little about – we’ve talked about your team and sort of growing your team and a bit about the history. What can you say about kind of the make up of the team? Things that have changed or new ways you’ve found to expand the team?
Kate:We actually just added a user research recruiter and this has changed my life in such a positive way. In fact one of our researchers, Deirdre, said to me have you ever had something amazing come into your life and you didn’t – and it solved a problem that you didn’t even really realize that you had. So we have a new research recruiter whose name is Kristen Arakelian and we brought her on for one purpose and that is to help us curate higher quality samples. And when I say higher quality participant samples I don’t mean okay every person on the panel has to be an Einstein. I mean sometimes we – you know we’re sending surveys to people who aren’t being responsive or we have these dead email addresses or we wanted medical nurses and we ended up with a surgical nurse. We just need to make sure that when we have a study ready to go we have our – we’re building out these participant panels and these participant – uh, these databases that are going to support our work and it has been incredible. And so now as researchers we get to focus on the research and she is doing all of this terrific work building our ground game, making sure she’s connecting. I mean here we are in Boston and we have so many universities and so many hospitals in our backyard and so she’s making these connections and she’s on social media and she is making lists and adding these people to our panels so when we have a research study and it’s ready to go we can actually launch it that day, or we can start it that day. That’s been a life changer. I wish I knew to do this sooner because my solution was always to add more researchers and there’s a lot of stress around recruiting. I used to say recruiting was our largest pain point. Pain point solved. It’s been wonderful.
Steve:What do you think is her secret superpower? Or not-so-secret power that makes that work?
Kate:What makes it work is I didn’t go looking for a user research recruiter. And in the instance of our recruiter Kristen had all the skills that I need someone who is polished. I need someone who is a great communicator and has those writing skills and understands how to kind of reach out to someone she’s never met before. And she has all the qualities that we need and she had no experience doing user research recruitment and I didn’t care because I knew she’d be right for the job. And I knew her prior to EBSCO and so I took her out for lunch and I said I have this opportunity to you and she said well you know do you want to give me a description and I said no because then you’ll think too much about it and I need you to come and start and see how you like it. And she did and she jumped right in and she’s made our – she is curating these samples for us that are yielding tremendous feedback and we don’t have the trial and error of okay we sent this survey to 20 people and only 6 responded and who knows someone who is a teacher in a K-12 school? You know we don’t have to go the friends and family route and we don’t have to bootstrap it anymore. We have a recruiting operation and it’s absolutely changed our lives for the better.
Steve:That’s fabulous. Are there – you mentioned earlier that the things that you do – the work that you do is published in public and you go to – you give talks about it. Are there stories or recent successes or things that you can share with us?
Kate:There are and one of our researchers named Lin Lin is located in Shanghai. She’s based in Shanghai, in China, and she conducted research on Chinese students and how they were similar or different then many of the US based students that we study on a frequent basis. I’m really impressed by her findings because Lin is this researcher – you send her out and she comes back with gold every single time. And it’s impressive because when you research students in China what’s missing from their research experience is Google because of government sanctions. And so what’s interesting is Chinese students have a similar approach in that they get very anxious about research at the outset, upon receiving the assignment. But research for Chinese students is episodic and each of those episodes of research it’s a combination of open web resources. So China doesn’t allow access to Google. They have something called Baidu instead. So there’s open web resources and there’s these episodes of research that happen. So over the three weeks of the assignment there’s these episodes of research and in each of the episodes it’s a mix of open web and scholarly resources which is through the library website and also consultation with the professor and perhaps a librarian is interspersed, is intermixed in there. That’s very different. The researching episodes is very different than what we’ve seen and what we’ve documented in US students. In US based students what we see is research in two microbursts. There’s a flurry of activity when the research assignment is made – is given. And the first microburst is that activity related to sizing and scoping. And so you’re on Google. You are looking for – you’re trying to understand your topic better and that Google search leads you to Wikipedia and there’s a lot of open web searching that’s happening. Then there’s a period of dormancy and we assumed that during that dormancy it’s this procrastination and there’s kind of shame and bad feeling about that because – and dread and oh-no. But no, there’s not. It’s about efficiency and students are – they’re locked and loaded because they did their open web. They understand alright Google and Wikipedia, like I have a grasp of my topic. And then there’s the second microburst and the second microburst of activity is right before it’s due and that’s when you go into the library website and you access the scholarly resources and you find those articles and you find those eBooks and you find those videos that are peer reviewed and scholarly and meet the criteria that your professor has set forth. So if you’re a librarian the reason that that information is valuable to you is because you understand that your opportunity to influence the student research process in the US is at the outset, is during that first microburst of activity. Is getting into the classroom with the professor and you know introducing the research topic and the scholarly resources that are available to them because most likely at the tail end of that, during that second microburst, the US based students are – it’s that flurry of activity related to finishing the project. It’s not about I’m going to go consult with the librarian per se. So we love having an international research team because we have the opportunity to do these cultural comparisons and they’re fascinating.
Steve:It seems like part of – there’s many elements there. It seems like one element is that by looking at what the behavior is for a Chinese student it highlights behaviors in the American use case that you might not have thought to reflect on until you see how it plays out differently.
Kate:It’s true and that’s the kind of research that helps us gain an understanding of students. You know students in China and students in the US, but it also helps us advocate for and help our – advocate for our customers and it empowers our customers to advocate for themselves. So that’s the kind of research that matters because our customers walk away and say great I can now evangelize to have myself as part of the assignment process or in the classroom because I’ve got this research that’s been documented. So we do a fair share of – we write the white paper or we’ll give a talk at library conferences or we’ll – we do a lot of traveling to our customers and talking about our findings and actually collaboratively conducting research with them. One of our favorite activities that we’ve been doing more of is to travel to customer sites and invite in three or four students, or three or four nurses or whatever product we’re looking at, to come in and demonstrate their search process live and we – it’s very interactive. Like people will say well did you know you could do this? Or what about this? And it brings that – again it’s the reality show of how users are interacting with different products and different services and it’s fascinating to watch and usually that exercise garners some great enthusiasm because it opens people’s eyes.
And I will tell you something. I have been very – I have been really pleasantly surprised by the great questions that many students ask around participant privacy and around sharing of information. And you know I think many people think millennials are just posting their whole lives on Facebook and Instagram and they’re very knowledgeable about privacy and I’ve been very impressed. And we’ve had to adapt. We’ve had to make sure that we talk about how this information will be shared. We used to kind of ask casually like can we take a picture of you so as we’re thinking about these results, blah, blah, blah – sometimes we will ask that, but certainly not as much as we did because we’ve had a lot of students say no thanks, or I’m not on Facebook anymore and I – you know when they talk about social media it’s not about this mass sharing. It’s they’re really kind of curating their online profile, their social media presence and they’re not careless about it. They’re very thoughtful and they care about privacy and they read the small print, as they should. I’ve been very impressed with how seriously our participants take that and as a result we are very specific in our releases. And we deal with children so when we talk about what are the qualities you look for – certainly good judgment, ethical research practice is critical. It’s the most important thing at the top of – at the very, very top of our list because without that good judgment I’m putting way too much in your hands that you’re responsible for and if I can’t trust you there’s no job.
Steve:Wow, that’s really good. Can you talk a little about your own background?
Kate:My background is actually – so I studied sociology as an undergrad and I loved, loved, loved studying sociology and I loved the research element of sociology. And I went to graduate school for public health and public health – and epidemiology being part of public health is fascinating because you’re looking at what programs, what interventions have an impact on a population. So unlike medicine which is the study of – you know is curation – you’re about, it’s curative – public health and epidemiology is about prevention. And so you’re looking at behaviors and you’re looking at interventions and you’re tracking those things over time. So the research element is very heavy in that. So I like to say that chapter one of my career was in healthcare and my job understanding end user needs and building products that met end user needs, there was a health reporter for the Boston Globe named Betsy Lehman and she was a cancer patient at Dana Farber in Boston and because of a medical error she actually passed away because she got four times the dose of a medication that she should of and she did not survive that. And so this was heavily covered in the Boston area and my hiring was a part of the staffing up that happened following Betsy and one other person was severely injured as a result of a medication overdose. So what they did, the Partners Healthcare system decided every unit is going on this online order entry. And so I was part of the staffing up that happened following that incident and our job was to understand okay this online order entry, we’re replacing the pen and paper – the clipboard at the end of the bed – it’s different in the labor and delivery unit. It’s different in an emergency department. And so my job was to understand and then – understand the user needs and translate that into system requirements. And it was just an amazing job because we did nursing shifts so I would work 11 pm to 7 in the morning on a labor and delivery unit and see the workflow there and understand how that product needed to be customized. So that was really fantastic work in the healthcare field. And then I went into online travel for the second chapter of my career. And online travel is similar in its complexities because you’re thinking about booking cruises, booking vacations and the complexities involved in that. And so booking engine, it was all about conversion and getting people into the booking engine and helping them shop and compare. I was at TripAdvisor for a little while and primarily doing information architecture and usability work and that was – online travel is fascinating and to this day I think about some of the things I learned about you know why Kayak made some of the decisions they made about their filters and things like that. It’s, it’s – I still refer to some of those points because it’s a complex product and you winnow down these results by using filters and things like that. And then I was in the UX field like I said up through information architecture, into the broader UX and then I – I fell in love with user research and I carved a path – carved a path forward to do just that and it’s been incredibly satisfying.
Steve:Is there something about you as a researcher, whether it’s maybe the product of those experiences or anything else? Like what’s – I think the Cruella de Vil anecdote is one – that’s sort of an evolution. I don’t know, anything else that would characterize – we talked about leadership a bit, but I’m thinking about doing the research. How do you – what’s…
Kate:So doing the – yeah, you know I love research because I love hearing people’s stories and I only – it’s funny I only read non-fiction and so I’m just very curious. I love hearing – I love new experiences and I love hearing people’s stories and I also love distilling that down, finding – you know finding those nuggets. And I love the moment – you talk about this actually in your book, Steve, but you talk about getting from question/answer to question/story. That is such a satisfying moment when it happens in an interaction with a user. And I never get tired of that. And in fact every time – you know we can be working on a large ethnographic study and you think okay you know we’ve got 25 public library patrons or we’ve got 22 physicians we are going to be speaking with and I always walk out of every session – you know you’re drained and you’re exhausted, but it’s that feeling of you just – you just ran a marathon and it’s the greatest feeling in the world because people just shared their stories with you and you get to take those stories – you’re obligated – I mean that’s such a privilege to have someone share that information about how they interact, how they live their life. And then you get to come back and represent that through their words and through their stories to product developers and owners. I just think that’s the best work in the world.
Steve:I love how you describe it and the “it’s hard” but you feel good after.
Kate:And you know it is true and that’s why we joke – you know as a researcher you’re like a Navy SEAL, I mean minus the incredible – minus that torturous training process, but we have different training we go through, but you have to kind of – you need to be able to be dropped into any situation, adapt quickly, find your way, get your answers by making people – you know you have to make them comfortable and then you’ve got to go and you’ve got to go back and you’ve got to deliver what you said you were going to deliver. And so we work hard to find those – we work hard to excavate and to uncover what’s there. And I’ll tell you something, my family and I are in the process of moving now and as I think about it – I always say that moving is an analogy for these design excavations and these changes, these product evolutions, because when you move every single thing in your house you either have to – you have to hold it in your hands and you have to make a decision about it. Does it stay? Does it go? And when we think about changing products and moving – you know evolving platforms, like we have to look at every aspect. So the research is comprehensive. We have to look at every single piece of it. We have to understand users in all aspects of their lives and we have to make decisions about does it stay, does it go, does it change and how?
Steve:So what else should I have asked you about?
Kate:One question that we are asked often is are you guys the user experience group? Are you UX? And we say we’re user research. And I think user experience is a different configuration at every company. And we at EBSCO – we’ve tried a couple of different versions and the version that works now is that user experience, because of our diverse product set – you know we have corporate products, and medical products and school products and we have you know a digital magazine product called Flipster for the public library. All these products are so different and these services are different and so there is no universal user experience because the user is so different. So what we do is user experience is a responsibility that falls to the product managers and product directors within each of those product pods. But user research is central. So user experience is not centralized because of the differences in all of the products. But user research is centralized because I will tell you having done the research 60 year old physicians and 16 year old students they’re searches and their strategies for searching are not as different as you might believe there are. So there are similarities and when we learn about people’s process we can apply – we can extend some of those findings across different markets, I think more than you can when you’re thinking about a product experience.
Steve:So is user experience part of the product management organization?
Kate:User experience falls within product management meaning it’s a responsibility, it’s not a centralized group. It’s decentralized within the product pods. And I think over time, you know these models with evolve, but that’s where we are now because we’ve tried on a couple of occasions to have it a little more central and it just didn’t take off as it should. So I think over time we will continue to revisit that, but right now user experience falls as a responsibility, as a product experience within the product pods.
Steve:I think you know these conversations are – they often cover a key point that you made which is like here’s where it works for us right now and these good things that are evolving, but I like how you’re talking about – you know there’s a responsibility, there’s an action, there’s a task and then there’s a team or a job title or a department and that I think sometimes we conflate the two of them when we talk about who does what, how is it structured? There’s activities that have to take place and then there’s well who does those activities and what do we call them?
Kate:Yeah and there’s a sense of – I think the challenge in centralizing UX is you’re trying to be everything to everyone and that gets challenging and then you’re not as close to the products as you might be. But when we’re – in user research when we conduct research we have the product managers with us. I mean they’re the co – they’re working on that research with us. They’re sitting on the floor of that hotel room in Las Vegas where the student is conducting research and they’re watching it right along with us which actually brings up a great point about user research. When you are doing user research in the wild be safe. Make sure you build in safety measures for your team. I mean we have to meet strangers and so be safe about that and that’s something that we talk about and it’s something we accommodate for. Because we’re going to our users so we don’t go alone. And we make sure that we’re safe about it because I’ll tell you something – even – I had moments where conducting usability tests, you know you’re conducting these sometimes after work and it’s dark and then the usability test staff at the testing center has gone home and you’re in this testing studio and these participants you’ve never met before are coming in. That’s something – I want to write some form of a blog post or an article about that because it’s something that I don’t hear talked about a lot, but safety in conducting user research should be paramount for all of us.
Steve:One of the things that I’ve struggled with and I think I wrote about this in Interviewing Users is the difference between being unsafe and feeling uncomfortable and that – I mean my suspicion is if you don’t kind of check yourself it’s easy to think that one is the other.
Kate:Agreed.
Steve:And not to minimize being unsafe in any way, but there’s feelings of being out of control. There’s feelings of just being out of your comfort zone. If you’re not comfortable being uncomfortable you might not know the difference between being uncomfortable and being unsafe.
Kate:Yeah, and as I said part of research is new experiences, new people, environments that are not your own, not your own native environment and so certainly if you’re someone who’s easily thrown off your game user research might not be for you because you’re put in all sorts of circumstances and people will say the unexpected. There have been – I mean we always talk about this. You know you’ll get asked for a job – I’ve had researchers who’ve been asked for a date. I mean it’s really – I mean the unexpected does happen – does and will happen.
Steve:Yeah, I’ve wondered a lot about the experience of being a woman in user research and the experience of being a man and that if there’s an aspect to that that like men just couldn’t possibly believe. And maybe this is about being a woman in our society also that men couldn’t possibly understand, but – that example of being asked for a date seems like oh yeah, that happens.
Kate:Sure. And you know when – I’ll be honest with you, when we were in Las Vegas doing the interview I had recruited the student I was meeting with off of Craigslist and so I was meeting a stranger that we’d recruited off Craigslist for a research interview when I was out at a library conference in Las Vegas and I asked my colleague Dave to come with us and Dave is very fit and very muscular and I asked because he was part of the project, but I also asked him because he’s strong. And I was – I had recruited someone off Craigslist and that was deal with my husband. He said I’m nervous about this. I said well I’m going to bring this guy Dave and he said okay, well I feel a little better about this. But you have to think about safety and in that situation I wasn’t – I wasn’t protecting against feeling uncomfortable. I was protecting against you know all the crazy stories you’ve heard about Craigslist I guess.
Steve:Is – yeah, is being asked for a date an unsafe experience? I guess it depends on how, but – I’m not even sure how to ask this question, but sort of separate from the Craigslist experience, how does – how do women in the field deal with the set of things that may – the type of content or conversations that might come up with – how do we even frame that?
Kate:I know. It’s a great question and think about it. Think about part of the Cruella de Vil, like serving it up cold, no one’s asking Cruella de Vil on a date. Do you know what I mean? I mean she’s like the ice queen, so who is going to ask this icy researcher who’s not even giving a sense of like am I doing the right thing on this, am I passing this test on the website. So part of being comfortable with people is being comfortable but still having boundaries. Still being the researcher. And part of it is not being alone in a circumstance that would make you uncomfortable, or make a conversation like that too convenient. And you know I think personally I can sense when we might be getting off topic, or we might – or the conversation might be getting too comfortable for the participant and so then I just move it forward. But I think this is an area where some sharing of these thoughts and experiences would help the larger research community to think about these issues and hopefully not have to be in a situation where they’re uncomfortable or unprepared.
Steve:I’m going to pitch hard for the war stories – having people talk about for the last few years. With the aim – I mean there’s many more circumstances that come up than just the one that we’re talking about, but my goal is yours. To hear these stories starts to, you know let people if nothing else be aware that they can – these are the kinds of things that can happen and so, you know, there doesn’t seem to be one size fits all right way to handle it, but to – like you said the Navy SEALs have to adapt.
Kate:You have to adapt and you have to adapt and you have to know your boundaries and you have to be comfortable with uncomfortable conversations. And so I was just having this conversation with my husband the other night about being comfortable with conflict and we talked about that as a trait of good management and I said you know as a researcher you have to be because sometimes you’re putting a product or something in front of a user and it may evoke some emotion, they may be upset about it. They may make that personal against you. So as a researcher it’s a requirement. You have to be comfortable with some level of conflict because not everything is smooth sailing in these research sessions.
Steve:And so conflict could simply be asserting boundaries and it may feel – if you are someone that’s kind of – that doesn’t – that is trying to avoid conflict, drawing a boundary which could be – I liked your example before of just moving things along. That’s asserting a boundary and that may feel like conflict to someone. Conflict is let’s get out of here – I’m going to punch you in the face – it doesn’t have to be dramatic.
Kate:No, and the conflict – right, the conflict could be – the conflict or the confrontation could be as simple as you’re not being responsive to a direction they’re taking the conversation in and you know I think as we move away from methods like the focus group – you know the in-person focus group where you have – that’s not a method that we use often because I think there’s so many challenges with it. So you’re moving to these individual – you know you’re playing Charlie Rose and someone else is sitting across from you and you’re understanding their lives, right. You’re kind of moderating the narrative of your user’s, your participant’s life and their experience. And as we move to these one on one, the deeper exploration, it gets more personal and the stakes are higher and being boundaried is important. It’s important as a researcher. It’s important as a person. It’s important for the participant to understand the parameters of the session.
Steve:And that you as the researcher have permission to continue to reinforce those boundaries.
Kate:As researchers one of our challenges is trying to figure out what should our temperature be and the temperature being about that level of interaction with, empathy with our participants, our users. And you know you don’t want to have yourself dialed all the way down into the freezing zone where you’re disconnected from the user and you’ve got the Cruella de Vil phenomenon happening. But you also don’t want to be so over the top connected and warm that it becomes like a friendship conversation and you have difficulty stepping back and still having those boundaries as the researcher. So this question of temperature, this question of you know how accessible are you, we haven’t solved it yet, but if I had to look at – if I had to look at a temperature scale I’d say it’s in the – the medium to warm is the ideal. The approachable, medium, warm temperature.
Steve:If I can pile on your metaphor a little bit, there’s your internal temperature and there’s your external temperature. And so you know I mean I’ve interviewed people about sad things. I imagine you have as well.
Kate:Yes.
Steve:Talked to people about situations that are sad. There’s a difference to me between how you feel and then what – and then how you do or don’t exhibit that feeling.
Kate:That’s a great point because part of our research has been focused on physicians and nurses and you know because EBSCO has strong presence in the medical market. So when we did a large study of physicians, what we ended up doing was talking to physicians in all different specialties and one particular physician stands out in my mind because she – she saw patients she – she delivered end of life care and her stories were unbelievably touching and several were very sad. And it was hard not to cry because some of these stories were really quite sad and I think what she saw – so inside, inside you’re really feeling the pain of what she’s describing, about families saying goodbye to other family members at the end of life, but to the outward – to the participant, to this physician, this really lovely, just wonderful, compassionate human being , to her I think the researchers, we just looked somber. You know we looked like she had told us something serious, which she had, and we looked somber and we nodded and we were respectful of that. But I don’t think she needed us to cry to know that we had feelings. I think the way we handled it inside, I think we had stronger feelings that we’d led on, but we treated the information soberly and somberly because it was – you know it was sad. It was sad. And that’s part of uncovering people’s truths that is so incredibly rewarding. So if I go into a session like that, and you know I use that approach that was recommended to me years ago, the like serve it up cold, I don’t deserve the privilege of being in that room. I don’t deserve the true honor of speaking with these participants who are helping to evolve our products and you know show us their lives. You don’t deserve the job of a researcher if you’re not truly empathetic to your user.
Steve:Sure. I think what that story illustrates, that seems very important, is that you had the feelings in the experience with the end of life professionals. You had the feelings inside. You acknowledged for yourself that you were having those feelings, but you chose in the context of research how to act on or how to express those feelings, and that you made a different choice that you would make in a social situation – you know anything that’s outside the context of doing research – you chose how to react differently than you would.
Kate:Yes, and that is key because the participant – the person that you’re speaking with needs to feel like the researcher is in control. So if someone starts sobbing uncontrollably all of a sudden the physician might be worried about the researcher and then it creates this shift in the dynamic that is hard to recover from. And so it’s the same as if someone tells you a joke or says something funny. If you laugh so hard that you can’t regain your composure it shifts the dynamics of the interaction. So it’s – that’s where having the boundaries – and it’s not so different than being a teacher or being a parent that you know your kid says something silly, or does something, and you think okay well do I want to reinforce – do I want – how do I need to react in this moment to convey the message that okay we’re going to continue – we’re still going forward or we’re still going to have a nap even though you just said this thing that’s really silly and we could laugh about it for 2 hours. So you have to keep the research – you have to keep the research moving forward, but you have to convey that when someone – you have to convey that you’re listening and you’re processing what they’re saying. So when someone is telling you a very sad story, when you’re somber and when you are listening intently and making eye contact that says I’m here with you, I hear what you’re saying and it’s registering and I understand that it’s serious. I get it.
Steve:If you – I think it’s different with a professional, but if you – I mean this has probably happened to all of us. We’ve been someone who’s had something difficult happen to us and we talk to other people about it and they say oh, I don’t know what to say or something and that we end up in the role of trying to help them.
Kate:Yeah, we end up in the comforter role. And keep – you know something I always say to other researchers is you have to be very clear that the research session is a judgment free zone because one of the challenges we have is when you’re researching and interviewing students sometimes they fear that there’s a right way to do what you’re asking them to describe and they don’t know what it is. So you don’t want people to feel that they have to say well I’m sure there’s a better way to do it, but this is how I do it. You want to be – you want them to feel comfortable and you know show us how you do X, instead of – and you don’t want them to be apologetic and you don’t want them to feel that what they’re showing you isn’t good enough. And often during those – I mentioned those live student demonstrations that we do with some customers that are so enlightening. We start of and we talk about you know your act of conducting research – your act of information seeking is uniquely you. We want to learn about it. There’s no right, there’s no wrong, whatever works for you is what we want you to show us. So we really spend a lot of time on that at the beginning of those sessions because we have a room full of librarians. We have a room full of library staff watching someone conduct research. And so we don’t want them to feel intimidated.
Steve:I had a client that I worked with that said to me that sympathizing is judging. And we think of judging as sort of the right or wrong, but she was saying you know if you say oh that’s awful – I’m not going to do justice to how she put it, but it’s taking a value framework that is yours and putting it on their story.
Kate:Right. You’re handing your world view to someone and saying oh it’s okay. Or you’re making a judgment and there’s no place for that. And so we hear it all. I mean that’s why we talk about you know anything can happen in these research sessions. We hear it all and we hear – you know we hear students talk about how they conduct scholarly research and sometimes the techniques they’re talking about aren’t necessarily scholarly. You know if they’re talking about looking on Wikipedia which isn’t a peer review…which is not a citable source – and so we don’t stop them and say well did you know, because we’re not there to fix anything. We’re not there to teach. This is not a teaching moment. This is understanding and this is actually exploration and one of the things we try to do in our group is we try not to say – when people come to us and say okay we want to validate – we don’t go out to validate. We go out to explore and in that process of exploration and excavation, you know we’re lucky enough to unearth findings, but we do so because we’re listening and not putting our own framework on the participants or the users.
Steve:Well Kate, let’s wrap it up there. It’s been a really excellent experience to talk with you and thanks so much for everything that you shared with us today.
Kate:Steve, thank you and I look forward to seeing you in Boston someday soon.
Steve:I look forward to that too. Thanks.
Kate:Okay, take care.
The post 13. Kate Lawrence of EBSCO first appeared on Portigal Consulting.
In this episode I speak with Kate Lawrence, Vice President of User Research at EBSCO Information Services. Our conversation covers where to place user research in the organization, emotions in fieldwork, and empowering others to advocate for information literacy.
The ability to write, synthesize, distill and synthesize findings into a report that is digestible and spot on is very challenging and there are people who can do it well and I really strive to find those people for the team. So great writing skills – and part of that writing is storytelling. It’s a combination of communication, analytical skills and research skills. – Kate Lawrence
Show Links Kate Lawrence * Follow Kate on Twitter * EBSCO Information Services * EBSCO Industries * Ipwich, MA * Mark Zuckerbeg’s hoodie * Shonda Rhimes’ Year of Yes* * Qualtrics * UserTesting * Pizza in Japan * Cruella De Vil * Affectiva * Eat This Not That * User Research War Stories * Deirdre Costello * Kristen Arakelian * Flipster
Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes.
TranscriptSteve Portigal: Welcome, Kate, to Dollars to Donuts.
Kate Lawrence:Thank you for having me.
Steve:Why don’t you begin by introducing yourself?
Kate:My name is Kate Lawrence and I’m the Vice President of User Research at EBSCO Information Services. We are the leading provider of research databases and other content to libraries and institutions all around the world and we’re headquartered actually out of Ipswich, Massachusetts – a beautiful town on the North Shore of Massachusetts, north of Boston. And our parent company is actually EBSCO Industries and they’re headquartered out of Birmingham, Alabama. And EBSCO Industries is one of the 200, one of the top 200 privately held companies in the United States and EBSCO is actually an acronym. EBSCO stands for the Elton B. Stephens Company and it’s a family owned company since the 1940s and again it’s one of the largest privately held companies in Alabama and one of the 200 largest in the United States and what’s interesting is one company, many different businesses and so the largest part of EBSCO Industries is EBSCO Information Services where I work in Ipswich, Massachusetts, but what’s interesting is EBSCO Industries has lots of other businesses. For example EBSCO owns the company that produces the most fishing flies of any manufacturer. They have real estate holdings. So it’s quite diverse in that way.
Steve:Does that create a certain culture and not so much EBSCO specific, but companies that have diverse holdings – and I don’t know if you can even answer this, but you know we hear a lot about you know you work at a start-up, you work at this kind of company, but just you work at a very kind of unique organization. It’s part of a very diverse, larger company.
Kate:It is and what makes it fun and unique to me is every day brings something different. So the part of the company where I am focused is the Information Services division. So we’re – we work on products and services that provide essentially information, premium information, to information seekers around the world. And occasionally I will get pulled into another project. So if there’s – we have a – there’s furniture company that makes work stations and so we did this whole research project looking at what are the furniture needs of the new library – you know the new information commons? What is that collaborative furniture that is needed, that’s different from the study carrels of yesterday? So we occasionally get pulled into projects and I’ve actually done a small research project on game cameras for one of our other outdoor product products called PRADCO. So you do get these projects peppered in that keep it very interesting. Absolutely.
Steve:It seems like the first example leverages an area that you’re focused in, that the information work – the information, information user is already focused on the library and people using the library and the work that they’re being done and if I’m guessing the game work is much further outside that.
Kate:It’s true. The game camera work was outside of that. What we do on a day to day basis is our research team focuses on studying the path and the search process for information gathering, information seeking. And like you said libraries are customers and think about libraries and think about library patrons. You have your academic market. You know you have students as young as age 6 and 7 and 8, starting to work on research projects. I mean it might just be the history of the Lego Corporation or finding out about Harriet Tubman, but that’s research. And so we have young students, elementary school students, all the way up to 95, 98, 101 year old public library patrons. So all of us interact with the library in some way and that’s why you see EBSCO products all over the world.
Steve:You know when you describe the young kids doing the Lego Corporation history research project it makes me think that there’s some major shifts that are going to impact the business that you’re in and what your research is going to uncover. Education continues to but what are some of these larger changes that we as a culture are going through that are driving changes in your product and things that you’re looking at in research?
Kate:Changes are because of Google. Google has changed everything about information seeking and information gathering and search. And Google being described by many of our students, they’ve described Google as their oxygen, their mother, their water, their soul mate. And they are completely serious and it makes perfect sense why they would describe Google that way, but Google has been this seismic shift in the way that people conduct searches for information. So when you think about, Steve, when you – when the two of us were in college, when we had a research paper to do we went to the library. We worked on it there and do you remember – everyone remembers the smell of the shelves. You know you went in and you – the library had a certain smell to it. It put you in a whole different zone when you were doing research. Research is different today. It’s a lot of guided self serve because you don’t have to be – because of electronic resources and you can access these materials from home, you can log into your library website from Starbucks, anywhere. You can conduct research on the go and so you’re not necessarily within sight of the librarian. You’re not necessarily even on campus. It’s a different experience and it actually starts in Google.
Steve:So – and you speak about this in a way that makes me think you’ve talked with a lot of people about this over the years.
Kate:Students are an area of tremendous focus for the user research group at EBSCO and as I was saying we have students all over the world. And students by the way is that third grader all the way up through graduate and doctoral students and fellows who are doing intensive research. So you have people who are doing research for a basic introductory class and they’re not particularly invested in the topic – may be in their first semester in college. And then you have your doctoral level researcher. So when we set out to design products and experiences and we think about interface design we have to think about this range of abilities and at EBSCO our customer and our user are not always the same person.
Steve:So the people that are making choices about what we call information products, information systems…
Kate:That’s right.
Steve:…they are choosing something that’s going to be rolled out to students, I guess, and other users as well.
Kate:Students with a very different approach to the usage and very different knowledge of the topic of information seeking and very different level of searcher. Sometimes I equate it to trying to appease the parent through a great experience for the child. So you think about parents buying products for kids and then you think about who are you trying to suit? Who are you trying to serve? And what we’re trying to do is create an experience for research that is delightful for students and then as a result also… also pleases the librarian, but pleases the librarian customer because it’s serving students so well.
Steve:So that’s an interesting area to think about. If as you said Google has really, really changed things – you know I’ve done a little bit in this area so I have a little knowledge which is dangerous – I mean when you describe these two groups, the people providing the service – the facilitators, you kind of used the metaphor of parents – librarians and people in that role. You know my casual observation is that this shift – you know the shift that’s Google and kind of everything that falls under the umbrella that you’ve related, how they’re experiencing it, and this is just natural – how they’re experiencing it as professionals and people with a lot of history with tools as they’ve evolved, and how these students are experiencing it, they’re looking at that shift very differently.
Kate:They do look at the shift differently and it reminds me of when – I remember when Mark Zuckerberg put on that hoodie and started wearing that outfit to work, it was upsetting to people. There was a certain generation and a certain group of people who are really offended that someone who runs a company is going to show up in a hoodie. And I think like – it’s kind of like when parents say “oh the kids today.” There’s a sense that we don’t have to do it the way our parents did it. We don’t have to kind of treat research like the Oldsmobile that my parents had to drive. Like I don’t – you know look Mom, I don’t have to do research the way you did it because I don’t have to be in the library and you had to be in the library. So I think we’re all still adapting and coming to terms with the new rules of research and the new way that students are getting through and studying in college. It’s very different and when you said it sounds like you’ve studied a lot, we do go out and interview users. We do employ ethnography and we sit with students on the floor of the coffee shop in Las Vegas, or right outside their community college. Or we sit with them after swim practice in Texas where they’re in a breezeway in a library trying to do a little research on the go right after swim practice. And so we sit and we watch their organic search process because until you see it, it’s hard to believe that it is the way that it is and it’s piecemeal and it doesn’t necessarily have a solid strategy, like this is how I’m going to go in, this is how I’m going to do it. There’s a lot of trial and error. There’s a lot of trial and error and it absolutely starts in Google because they feel safe there and starting your research in Google by getting this foundation of your topic, it reduces anxiety. And research can be, for some students, a very anxious process.
Steve:I’m thinking sort of the network of – I want to use the word stakeholders in the – in the broadest sense possible. You’ve talked about students. Certainly for younger students there’s the parents. There’s the people that are providing them – let’s just call them librarians, is that the right term? People that are providing them with these tools?
Kate:Yes.
Steve:And those are kind of all outside and then inside your organization there’s you and your folks that are going out and talking to these people. There’s somebody that is taking what you’ve learned and acting on it in some way. What – very generally what does that node look like?
Kate:That is, at EBSCO what we have is we have evidence based product development and we’ve embedded user research into the product management organization for this reason. So what will typically happen is we have achieved in the last – I would say 12 to 18 months – we’ve been able to achieve alignment. And so we understand, for user research, what is coming 12, 18 months from now. So we’re in front of it, we’re doing the research now. And so that was one of my key learnings when I first started the group and I first started doing user research. I did every single project that came my way and we joke about the year of yes. You know Shonda Rhimes just wrote “The Year of Yes” and I said I lived that year. I said yes to everything just to get the demand so that I could get the resources. And what happens when you do that is then you’re doing research on everything, but strategic alignment is critical. And so now we’re very much aligned with what the product development cycle is and we’re able to do the research at the outset instead of what used to happen – and this is typical of many companies. Years ago it was let’s do usability testing before we launch and that is a tough situation for anyone to be in because then you’re this kind of glorified version of QA and you’re just – you don’t have any chance to make an impact if something isn’t testing well.
Steve:Can we pause that thread and can we go back to something? I want to talk more about how this has become adopted and integrated. But can we go back to this earlier point and sort of who’s involved in it to me seems really challenging. So you are sort of uncovering new paradigms. I think there’s – they’re kind of, they’re transgressive paradigms (my word not yours) – but they are changing the way that we think about how people are searching, what outcomes look like for information problems and so on. You’ve established how that works between different parts of the organization internally. And again I’m projecting here, but if I’m in the business of selling products to these librarians and these products – it’s what you said before about trying to please these different audiences. These products have to – they reflect a new model for how people are going to use them, but that new model might be a little – I don’t know, how is that new model being received by the librarians, by the people that are sort of selecting and rolling out the kinds of tools that you have to offer?
Kate:I have been so encouraged by the librarian reaction to the research because one of the talks I give, one of the themes of a talk I give on a regular basis is called Student Researchers – The Reality Show, or the Realities of Student Research – because it’s very different and it’s librarians who come up to me afterwards and say thank you, you’re validating what I’m seeing and now I have greater context. And also many of our findings support the role of the librarian in an active partnership with faculty. Let me give you an example. You are sitting in your Intro to Philosophy class. You’re a freshman at University of Virginia and you’re there and all of a sudden you hear I’m getting a research assignment. And so this kind of fills you with a little bit of anxiety and the professor is giving information about you have to use scholarly sources, things have to be peer reviewed. You’re getting kind of the elements of the rubric. So New York Times, Time Magazine, these are not written for academics. These are not written for an academic audience. They’re not peer reviewed. They’re not considered scholarly peer reviewed materials. So that’s another point of anxiety for students because they have to go to materials they may be not be familiar – as familiar with. Something amazing happens when in addition to the faculty member, your professor assigning this work, there is an amazing phenomenon that occurs when a librarian is brought in to the classroom – okay, taken out of the library. He or she leaves the library and is in active partnership with the professor, talking about this is how to do this, this is how you navigate the library website to find these scholarly databases from EBSCO, etc. It’s the equivalent of when your parents – both your Mom and your Dad – say Katherine, we’d like to speak with you. Something in you is queued up to listen and you’re more apt to learn it and you’re more apt to learn it and absorb it. As opposed to the professor saying well next Tuesday you’re going to meet in the library and the librarian is going to walk you through X or Y. So what happens when we talk about this is some of these findings they empower librarians to advocate for a more active role in this process of teaching students information literacy skills. And that’s exciting to see when they say this is going to help me help my students and they appreciate that. And I’ve been – I’ve been overwhelmed by just how wonderful the librarian community is and how supportive our customers are of our work.
Steve:That’s really fabulous. There’s something too – can you expand a little bit on this notion of – you’re talking about kind of the librarian in the library and then the librarian in the classroom.
Kate:Yes, it’s so interesting.
Steve:There’s some kind of frame shift that’s happening there.
Kate:There is. And so think about – think about the amount of effort it takes you when you’re in the library to get up, leave your laptop unsupervised, walk over to the librarian desk and ask a question. That’s actually harder than any of us might imagine. And so also when you’re in the library, what we’re hearing from students is often they are collaborating or they’re having a meeting and they’re with other people. So to interrupt that and get up and go ask for help is hard. And other obstacle is some students don’t know how to ask. And I have a customer I’ve worked with over in Europe who said that they actually post signs at the reference desk that say – you know how to ask. You know ways to frame your question because that’s an obstacle. And then we’ve had students say to us you know I want to – they actually feel very positively about the librarians. They’ll have Live Chat and they said oh the librarian came to my class. He or she is so nice. But there’s still – are you using a library? Are you using librarian services? Are you going to the reference desk? No, I’m just asking my friend or I’m asking my roommate. And the reason that is happening is – goes back to what I said about a lot of this searching is guided self serve and the guided part is – and the self serve part is it’s 11 o’clock, they’re home, they’re on the couch, they’re not in the library per se. And so who is nearby? That’s your go to.
Steve:Can you talk a little bit about the, I don’t know, mechanics, logistics challenges – you know these kinds of, the people that you’re doing research with and then you described some really interesting contexts that you’re in as well. Are there things that you have had to figure out to make that kind of an established process for your group?
Kate:So working with students is fascinating. We also research physicians. We research nurses because EBSCO has medical products. We research, again, the public library patron which is all of us from the time we’re two, or two months old, to the time when we’re a hundred and something years old. We have relationships with our public library. And we also research corporate users. So we’re – EBSCO’s user research team – in our team we are researching people all the time and I always say that our most important job is matching method to question because everyone gets really excited about big contextual inquiry, ethnography studies, but really that’s not appropriate in every scenario. It’s more of a tool when you’re examining markets. Like let’s study the public library patron, let’s study college students, etc. But you know what I find – one of our greatest challenges is we have to know where to spend our time and so we rely on – we rely on this technique of – that I talked about with the strategic, strategic alignment to understand what’s coming, what are the priorities and how can we do research to best support that. And we have some great product managers and product directors who help – who always come and knock on our door and say I’ve got this thing coming up, I’m going to need research for it. And so we’re – culturally we’re at that point where we’re not having to chase people down anymore. They’re actually coming to us and we have a backlog and we have a wait and we conduct probably close to 80 studies a year and we hope to increase that with every new person we add to the team.
Steve:So, I mean your point that not everything needs to be a contextual research study, what are some other methods that you rely on?
Kate:We have used – we’ve used lots of different methods. So we’ve used traditional surveys. We subscribe to Qualtrics, and we also – usertesting.com has been a great resource for us because it’s allowed us to, you know just do more work in a shorter amount of time and it’s a way to kind of expand the group without adding resources. I will tell you though some of our most fun work has been methods that were new to the group, like for example we did a modified video diary study because we study students and we study students who are under the age of 18 and so we deal with legal issues and consent issues and one of the things that’s challenging about researching 14 year olds, middle schoolers and early high schoolers, is that especially 14/15 year old young men, boys, they don’t necessarily, they’re not the most talkative group in the world. And so I’d read somewhere that Google had put video cameras in envelopes and mailed them off and asked students to – younger kids to track themselves searching, etc. So we basically sent off these video cameras and I think I only spent $50 on each one because we weren’t sure we were going to get them back. So we sent them off in these padded envelopes with some tasks for users to complete and it was great. And it’s a method that we want to use again because we got the quiet 14 year old boys, we got them talking. And they told us lots and lots about how they conduct research and what they’re looking for and they evaluated screens for us. And every single video camera came back. Not a single one was damaged. It was a great method and it was so much more successful than it would have been if we’d had them in a room, asking questions.
Steve:Because they had sort of their own space to respond.
Kate:Yeah and it was this – it’s the selfie generation. And so it was just this kind of extended selfie and it was just extended selfie and it was really precious. It was precious seeing the things – just – they took pride in it. They created a little movie of their life and how they conduct research and we got to be privy to that. I mean the challenge is if you put those 14 year olds in a room, because they’re under 18 we have the parents in there as well. So you’ve got the researcher who’s like a second parent and then another parent, and it just – it’s not an opportunity – it’s not a situation where they’re very talkative in person, but we did manage to get some information out of them through the video cameras.
Steve:You point of the selfie generation reminds me of this great video from a couple of – I want to say there were two Canadian students that were living in Japan. I think they were undergraduate students and they were doing these sort of mini auto ethnographies, for lack of a better term, where they did one video where they ordered pizza and they showed what the interface looked like and you know – I mean Japan is such a great place to talk about how things are different than what your normal reference frames are. So they kind of called them all out and then when the pizza came they sort of showed what the food looked like. They went through sort of, like the whole user journey, to use some jargon – they went through that whole journey in a very delightful, YouTubey kind of way and it made me think about maybe self reporting is – culturally can take on more of a role and you just proved my hypothesis with your camera study.
Kate:Well you know the raw self reporting, that’s what I love, because so often when you’re filling out a survey you’re self report… Our self reporting – I mean this is why we go study users in the wild because people do things differently than they say they will. And I always tease my husband because he’s the only person when you go to the physician for your annual physical – you know the physician will say how many times a week do you exercise, like how many – do you eat all your vegetables ,etc. And he tells the truth and I said no one every tells the truth. No one ever self reports accurately. And they account for that. But he does. And this is why we study people in their environments. This is why when we go out we always come back with new information that’s so fascinating to folks because within the walls of our company in EBSCO, here in Ipswich, you know we make assumptions like everyone does and then you go out and you get this raw data and it’s exciting. So I like the idea of an ethnography app and I like the idea of the raw data capture because I want that. It’s those words and those truths I think from users that they pivot products and they really launch products that are so much better targeted and better suited for the users.
Steve:How do you help these folks feel comfortable or feel like they’re doing the right thing?
Kate:The same way I work to make people successful and then comfortable in an in-person interview. So let me give you an example. Years ago I had a boss who said to me the reason you’re good at usability testing Kate is you serve it up cold. You don’t give any emotion through the process. So if someone is doing something right or someone is doing something wrong they can’t read it on your face. And I was proud of this for about 15 minutes and then I started to realize over the years that that Cruella De Vil kind of cold fish approach to usability testing and user interactions is actually awful. And the reason it’s awful is because you need a genuine connection with people for them to tell you their truths and I have learned – I learned that the hard way as a researcher. And so when you connect with someone in person it’s that warmth. It’s that finding common ground that’s important so you can get – you can get information from that person. They can tell you what’s painful about the research process? Or they can tell you oh I’m trying to book an online cruise and this is what I don’t understand. And then you can actually take those words and take that wisdom and go back and tell the product team, like this is how you can really help these users, these customers. But you can do the same thing if it’s not in person. If you have a self reporting tool – like when we did the video diary study we wrote instructions like we were talking to 7th graders because if you send them a packet that reads like tax filing instructions, you’re not going to get any answers and you’re not going to get an video cameras back. So we put ourselves in the shoes of the user. What is their reading level? What is – how are they likely to talk about this? What terms – I mean they’re not talking about information literacy. They’re talking about finding articles for a paper. So we try very hard to speak their language and I think that’s one of the skills I value, it’s one of the traits I value most in our researchers is the ability to adapt and understand and listen and then adjust accordingly.
Steve:Which you can’t do with the instructions. You have sort of one shot at that.
Kate:You have one shot at the instructions and so you have to write on their level. You have to consider yourself as the user. And you may have to do some sort of pre-survey or pre-work in order to get – in order to get a sense of how they’re going to process the information. For the video diary study our researcher Lin Lin and myself we actually tested the testing instructions. We went through several rounds with people saying, you know do you know what to do here and you have to. So often we’re testing the survey or testing the test instructions to make sure that we’re being clear.
Steve:Right the research – research artifacts are, they’re a designed tool.
Kate:They’re a designed tool.
Steve:And you have to test them just like anything else that has to be tested.
Kate:Yeah, and I think that – I think that we can all get to a point where we feel comfortable. Like okay I’ve done this kind of test before and this was understandable, but you know recent experiences – I’ve had a couple of surveys that were challenging – we got feedback after the fact like this was challenging or – and so now we’re trying to be better about understanding how long the survey should take someone and then adjusting compensation accordingly. Because we had a survey that we thought we – I think we gave a $50 Amazon gift card for, but it ended up taking people almost 40 minutes. And so we realized we have to be better about understanding the amount of effort and how we’re going to compensate for that.
Steve:Can I go back to your Cruella de Vil?
Kate:I know, do you like that?
Steve:I do. So – ‘cuz I think this is an interesting personal style of fieldwork thing.
Kate:Yeah.
Steve:I’d love to get you talk a little more about – you sort of learned what not to do and I wonder if you could characterize what your approach is now in terms of common ground and so on.
Kate:So when we were talking about the usability testing and serving it up cold the whole approach was very kind of generic and you know show no emotion. So people are succeeding or failing. It was very metrics based, like oh did they succeed in this task to add X to their cart, or whatever it was. And so kind of I think as a person you evolve to start to understand that it’s about the holistic experience. It’s not about this one slice of the UI, it’s about this journey of I was in online travel at the time when that happened and it’s about this journey of how do I start to think about what trip I want to book for my honeymoon or what flights I want to book or what vacation package destination I’m thinking about etc. So as you start to think about the whole journey you realize that you’re not just, it’s not about just succeeding or failing. And so the warmer tone – you know you kind of go from Cruella de Vil to I guess more – to more of like the Dr. Oz who’s asking the questions about why and trying to unearth and dig at the whys. And trying to unearth those pain points and trying to look at it from a standpoint of empathy and understanding and not just let’s see how long it takes this user, this participant – how long it takes this person to find the shopping cart button. It’s really about what are your struggles? If you could change any one thing about this process or this product what would it be? You know and when you’re – we always say to users, we always say to students when we study them today, we say the process – how does it make you feel when you get a research paper assignment? How would you complete this sentence – the process of you know finding research for my paper makes me feel – and we ask them to fill in the blank. And through those words and the way they complete the sentence we start to get that picture and I don’t think it’s a coincidence that EBSCO and our product development team and as a company we’re starting to consider emotion when we think about not only the process of buying products as a librarian, but the process of using them as students and other end users. It’s not a coincidence that our research group is all women. I think that we’ve brought that emotional dimension into the discussion in a way that had not been present prior.
Steve:So what I heard is that a way to bring that emotion in is in the questions that you’re asking.
Kate:Yup. In the questions that we’re asking and also we’re going to be exploring later this year emotion analytics. There’s a company right in our backyard here in Massachusetts called Affectiva and they’re out of the MIT labs. It’s an emotion analytics product and it’s something we’re very interested in because for many people conducting research – we have this wonderful researcher on our team, our lead qualitative – our lead researcher. Her name is Deirdre Costello. She’s a librarian and a researcher and she likes to say conducting research for many students is like eating your vegetables. And so you don’t necessarily look for the emotions of they’re jumping up and down or they – like you’re not expecting people to look like they just hit the winning lottery ticket. But you’re looking for certain emotions to register on their face as they’re finding the path forward toward these scholarly resources. And so this is something that we as a team are going to be exploring later this year is how do we – how do we capture and quantify those emotions and what about the emotional state changes with certain design changes? Or changes when you’re looking at different products? We’re going to be exploring that and that is, that’s really new research for our group. We’re excited about it.
Steve:This is a tremendous evolution from this Cruella de Vil era of sort of looking at metrics.
Kate:Metrics.
Steve:Looking at a lot of other kinds of things.
Kate:Well you know that same boss said to me – and actually he’s a really – he’s a wonderful old smart guy. We were all learning at the time. He was the one who said to me I want the findings written up as a white paper, really in white paper format. And so I did that and I would pass people in the hallway and say hey did you read that report? And they’d say yeah, yeah, looked great. And I could tell that they never read it. And so the next time I did a usability study I said to my boss at the time I will write the white paper, but I’m also going to create this parallel version. And I actually based my reports – I did them in PowerPoint but I made them very visual and I based them on the format of that book written by the editor I think of Men’s Health or Men’s Fitness Magazine called “Eat This Not That” because someone, somewhere in my life had that book at their house and I was leafing through it and it was these visuals like look if you go to Panera don’t eat this, eat that. You know this is healthier for you. And it was just very visual. And so in 10 minutes you get a sense of alright if the only option is Jack in the Box this is healthy option at Jack in the Box. And so I did the report – like look users don’t understand this, but they understand that. And I did it in a very visual way, modeled after that book, and all of a sudden we got traction. All of a sudden the people who weren’t reading the reports were calling me into meetings and saying I want more of this, how can I get more, I love this. And so creating results that tell a story in a visual way is very important. Having said that we also in our group now, we do write the white paper and we do speak at conferences and do different things with our findings because we need a variety of different deliverables to satisfy our customers.
Steve:Right. You have different audiences…
Kate:We absolutely do, yeah.
Steve:Can we talk a little about – you made the comment it’s not surprising that your group – did you say it was all women?
Kate:We are. We’re all women and we have a male intern in the summers who is wonderful. So we are certainly welcoming of – we’re welcoming of women and men, but at the moment we are all women, yes.
Steve:I mean what do you think that’s about? ‘Cuz I think if you look at the field of user research in general, and this is just anecdotally for me, it’s you know I think predominantly women is my guess. And someone may be rolling their eyes at that, but that’s my sense of it. I don’t know, what do you think is happening there culturally?
Kate:Half of my group also have librarian degrees and I think librarians you find mostly women. So I do think the qualities I look for in a researcher, you know we are of course gender blind and so I think these qualities can exist in anyone. I think – you know some of the qualities I look for – we want people who are highly adaptable, curious, great communicators is key. And I was mentioning Deirdre previously. Deirdre used to be a writer – she taught writing at Boston College. So the ability to write, synthesize, distill and synthesize findings into a report that is digestible and spot on, that is very challenging and there are people who can do it well and I really strive to find those people for the team. So great writing skills and part of that writing is storytelling. So I’m not sure if you said to someone who do you think has better communication skills, men or women? It’s a combination of communication, analytical skills, research skills. I think it’s just coincidence that we have all women, but boy we have a terrific group. I mean I always say I’m loving this part of my career. I’m loving it because of the chemistry we have in the group and I just – you know sometimes when you’re in a job that’s frustrating you think well maybe someday and I finally found my someday. So it’s nice to know that it’s out there.
Steve:I finally found my someday – seems like that’s a song they should play at somebody’s wedding.
Kate:I know, but I finally found that environment and – but you know part of that is you have to build the culture that respects user research and you have to be – you know my boss is very supportive of our work and if he wasn’t we would go nowhere. I mean you have to be in the right place in the organization working with and for the right people and you have to hire well. You know we actually have an opening right now and we’ve decided that we’re going to just wait for the right person to find us because when you go out and you recruit the person you want isn’t necessarily available at that moment and you may not find that person and then you end up with a pool of six people. You narrow it down to three, and then two and then you end up choosing that person who maybe in a different world you wouldn’t have chosen. So we just have this fantasy that the right person will find us.
Steve:Well and I think – you know your catch phrase is I finally found my someday, but you’re talking about building it. I think you built it.
Kate:I built it and so when you – when I started at EBSCO it was – let’s see it was almost 5 years ago and so when I started at EBSCO user research really hadn’t emerged to be its own entity yet and I remember my husband – I came via the information architect path, up through user experience, and my husband would always say to me why don’t you focus on the research? It’s what you like the best. And at the time research was – there was a lot of usability testing, or at least my version of it and I just said I don’t that I could do that all the time, but he said but you love it and I said you’re right I do. So I took a job at EBSCO and my first job here I was focused on usability, but I was also helping to write requirements. Like I was a requirements analyst for these products and platforms. And I remember at the time – I conducted I think two usability tests and I realized there was this big need – there was a major need for someone to focus exclusively on usability and user research and previously this had fallen to multiple people. It was a dimension of many people’s jobs. And I went into my boss at the time, named Ron, and I said Ron I would like to focus just on the user research piece and he said well I can’t let you do that because I still need you to be able to write requirements if I need you to do that and I said okay I’ll make you a deal – if you let me do more usability and user research I will write requirements whenever you need me. And he said yes, ‘cuz you’re definitely not going to have enough work for just the usability piece. And I said okay, deal. So we had a deal and I never wrote another requirement because the demand, that was my year of yes. I said yes to every project that came my way and then I was in technology at the time and then I moved over to product management and things really took off because then I was embedded in the product development life cycle.
Steve:Is there a point at which doing research sort of morphs into leading research?
Kate:There is and that is how a bill becomes a law or how a girl becomes a woman. You know it’s really – it’s quite a transformation. It is. Like all of a sudden it’s – it’s growth, it’s evolution and what starts to happen – I remember you go from doing testing and doing research to talking about it, to evangelizing for it, to finding it a home within the organization. And that’s where the leadership engine kicks in. And I think one of the reasons I feel well supported at EBSCO in that people value the work of my team and when they promoted me to Vice President that was really exciting for me personally and also for the research in the organization to be given that level of visibility and that was exciting. And it’s, it’s – it means a lot to our customers that we have prioritized this as a function within our company and our organization and so when they buy our products, or they invest in a partnership with us, they understand that we are looking at our users 24/7. We have a relentless and a passionate focus on our users and customers.
Steve:And you described those different milestones in that evolution and was becoming part of the product management organization a significant point along those different stages?
Kate:Absolutely because products – this is where the products – this is where the products live and this is where they’re created and this is where they are evolved. And so when we were located in technology that was fantastic because we were where things were being built. But a lot of the upfront research needed to happen – it needed to happen sooner than we were getting it in technology. And so moving us over to product development was a real culture shift and it was an important one in our growth. And something in user experience, I constantly had that debate of should I be in – at the time at other companies it was should I be in marketing? As a UX person should I be in marketing or should I be in technology? And actually at the time I always opted for technology. And so it was a shift for me to be at EBSCO and say you know what I think we should be in product management and I think that’s the right place for us because my whole career I’d actually fought as a UX person to stay in technology. But at EBSCO it made more sense for us to be in product management. And now, again, we can achieve that alignment a lot better because we’re where those discussions are happening.
Steve:Right. I want to follow-up on something else you said as well. You described these qualities that are just so essential, that you look for in researchers. How are those – how would someone make those qualities visible?
Kate:That is a great question because how do you – how in an hour interview do you say oh well I’m curious, I’m a great communicator, I can distill down findings. I always say the great researchers are like Navy SEALs. They go in, they adapt, they accomplish their mission, they leave without a trace. Because you want to be efficient. You want to go in, you want to get the job done and you want to come out and you want to leave people kind of unruffled and let them go on with their lives. And so I look at communication skills before I even look at research skills. So for example you wouldn’t believe some of the cover letters we get or some of the emails and you know I have very – I actually have – I can’t say I have a low tolerance – I have no tolerance for things like bad writing and bad grammar and so that, because our work is published externally, that’s just a deal breaker for me. There’s a curiosity and a work ethic that we look for. I just wish you could sit in our team meetings and meet some of the researchers and – it’s funny my husband actually calls Deirdre, our lead researcher, he calls her The Oracle because she knows everything. I mean you can sit – you can sit her with a fifth grader to talk about research and she can make a connection and that yet she’s amazing with 60 year old esteemed department heads of you know medical or surgical departments when we’re doing physician research. So she can – she can do it all and my friends – I remember my friends and I, when we were all dating before any of us got married, we had this joke about we’re looking for the guy who’s tent to tuxedo. You know he can go and rough it in nature and then he knows which fork to use at the black tie dinner. And I think as researchers that’s what I appreciate. I appreciate the person who can kind of rough it in the wild and then be completely comfortable at a black tie dinner and everything in between because we research everybody and you have to be able to connect across that continuum.
Steve:Can you talk a little about – we’ve talked about your team and sort of growing your team and a bit about the history. What can you say about kind of the make up of the team? Things that have changed or new ways you’ve found to expand the team?
Kate:We actually just added a user research recruiter and this has changed my life in such a positive way. In fact one of our researchers, Deirdre, said to me have you ever had something amazing come into your life and you didn’t – and it solved a problem that you didn’t even really realize that you had. So we have a new research recruiter whose name is Kristen Arakelian and we brought her on for one purpose and that is to help us curate higher quality samples. And when I say higher quality participant samples I don’t mean okay every person on the panel has to be an Einstein. I mean sometimes we – you know we’re sending surveys to people who aren’t being responsive or we have these dead email addresses or we wanted medical nurses and we ended up with a surgical nurse. We just need to make sure that when we have a study ready to go we have our – we’re building out these participant panels and these participant – uh, these databases that are going to support our work and it has been incredible. And so now as researchers we get to focus on the research and she is doing all of this terrific work building our ground game, making sure she’s connecting. I mean here we are in Boston and we have so many universities and so many hospitals in our backyard and so she’s making these connections and she’s on social media and she is making lists and adding these people to our panels so when we have a research study and it’s ready to go we can actually launch it that day, or we can start it that day. That’s been a life changer. I wish I knew to do this sooner because my solution was always to add more researchers and there’s a lot of stress around recruiting. I used to say recruiting was our largest pain point. Pain point solved. It’s been wonderful.
Steve:What do you think is her secret superpower? Or not-so-secret power that makes that work?
Kate:What makes it work is I didn’t go looking for a user research recruiter. And in the instance of our recruiter Kristen had all the skills that I need someone who is polished. I need someone who is a great communicator and has those writing skills and understands how to kind of reach out to someone she’s never met before. And she has all the qualities that we need and she had no experience doing user research recruitment and I didn’t care because I knew she’d be right for the job. And I knew her prior to EBSCO and so I took her out for lunch and I said I have this opportunity to you and she said well you know do you want to give me a description and I said no because then you’ll think too much about it and I need you to come and start and see how you like it. And she did and she jumped right in and she’s made our – she is curating these samples for us that are yielding tremendous feedback and we don’t have the trial and error of okay we sent this survey to 20 people and only 6 responded and who knows someone who is a teacher in a K-12 school? You know we don’t have to go the friends and family route and we don’t have to bootstrap it anymore. We have a recruiting operation and it’s absolutely changed our lives for the better.
Steve:That’s fabulous. Are there – you mentioned earlier that the things that you do – the work that you do is published in public and you go to – you give talks about it. Are there stories or recent successes or things that you can share with us?
Kate:There are and one of our researchers named Lin Lin is located in Shanghai. She’s based in Shanghai, in China, and she conducted research on Chinese students and how they were similar or different then many of the US based students that we study on a frequent basis. I’m really impressed by her findings because Lin is this researcher – you send her out and she comes back with gold every single time. And it’s impressive because when you research students in China what’s missing from their research experience is Google because of government sanctions. And so what’s interesting is Chinese students have a similar approach in that they get very anxious about research at the outset, upon receiving the assignment. But research for Chinese students is episodic and each of those episodes of research it’s a combination of open web resources. So China doesn’t allow access to Google. They have something called Baidu instead. So there’s open web resources and there’s these episodes of research that happen. So over the three weeks of the assignment there’s these episodes of research and in each of the episodes it’s a mix of open web and scholarly resources which is through the library website and also consultation with the professor and perhaps a librarian is interspersed, is intermixed in there. That’s very different. The researching episodes is very different than what we’ve seen and what we’ve documented in US students. In US based students what we see is research in two microbursts. There’s a flurry of activity when the research assignment is made – is given. And the first microburst is that activity related to sizing and scoping. And so you’re on Google. You are looking for – you’re trying to understand your topic better and that Google search leads you to Wikipedia and there’s a lot of open web searching that’s happening. Then there’s a period of dormancy and we assumed that during that dormancy it’s this procrastination and there’s kind of shame and bad feeling about that because – and dread and oh-no. But no, there’s not. It’s about efficiency and students are – they’re locked and loaded because they did their open web. They understand alright Google and Wikipedia, like I have a grasp of my topic. And then there’s the second microburst and the second microburst of activity is right before it’s due and that’s when you go into the library website and you access the scholarly resources and you find those articles and you find those eBooks and you find those videos that are peer reviewed and scholarly and meet the criteria that your professor has set forth. So if you’re a librarian the reason that that information is valuable to you is because you understand that your opportunity to influence the student research process in the US is at the outset, is during that first microburst of activity. Is getting into the classroom with the professor and you know introducing the research topic and the scholarly resources that are available to them because most likely at the tail end of that, during that second microburst, the US based students are – it’s that flurry of activity related to finishing the project. It’s not about I’m going to go consult with the librarian per se. So we love having an international research team because we have the opportunity to do these cultural comparisons and they’re fascinating.
Steve:It seems like part of – there’s many elements there. It seems like one element is that by looking at what the behavior is for a Chinese student it highlights behaviors in the American use case that you might not have thought to reflect on until you see how it plays out differently.
Kate:It’s true and that’s the kind of research that helps us gain an understanding of students. You know students in China and students in the US, but it also helps us advocate for and help our – advocate for our customers and it empowers our customers to advocate for themselves. So that’s the kind of research that matters because our customers walk away and say great I can now evangelize to have myself as part of the assignment process or in the classroom because I’ve got this research that’s been documented. So we do a fair share of – we write the white paper or we’ll give a talk at library conferences or we’ll – we do a lot of traveling to our customers and talking about our findings and actually collaboratively conducting research with them. One of our favorite activities that we’ve been doing more of is to travel to customer sites and invite in three or four students, or three or four nurses or whatever product we’re looking at, to come in and demonstrate their search process live and we – it’s very interactive. Like people will say well did you know you could do this? Or what about this? And it brings that – again it’s the reality show of how users are interacting with different products and different services and it’s fascinating to watch and usually that exercise garners some great enthusiasm because it opens people’s eyes.
And I will tell you something. I have been very – I have been really pleasantly surprised by the great questions that many students ask around participant privacy and around sharing of information. And you know I think many people think millennials are just posting their whole lives on Facebook and Instagram and they’re very knowledgeable about privacy and I’ve been very impressed. And we’ve had to adapt. We’ve had to make sure that we talk about how this information will be shared. We used to kind of ask casually like can we take a picture of you so as we’re thinking about these results, blah, blah, blah – sometimes we will ask that, but certainly not as much as we did because we’ve had a lot of students say no thanks, or I’m not on Facebook anymore and I – you know when they talk about social media it’s not about this mass sharing. It’s they’re really kind of curating their online profile, their social media presence and they’re not careless about it. They’re very thoughtful and they care about privacy and they read the small print, as they should. I’ve been very impressed with how seriously our participants take that and as a result we are very specific in our releases. And we deal with children so when we talk about what are the qualities you look for – certainly good judgment, ethical research practice is critical. It’s the most important thing at the top of – at the very, very top of our list because without that good judgment I’m putting way too much in your hands that you’re responsible for and if I can’t trust you there’s no job.
Steve:Wow, that’s really good. Can you talk a little about your own background?
Kate:My background is actually – so I studied sociology as an undergrad and I loved, loved, loved studying sociology and I loved the research element of sociology. And I went to graduate school for public health and public health – and epidemiology being part of public health is fascinating because you’re looking at what programs, what interventions have an impact on a population. So unlike medicine which is the study of – you know is curation – you’re about, it’s curative – public health and epidemiology is about prevention. And so you’re looking at behaviors and you’re looking at interventions and you’re tracking those things over time. So the research element is very heavy in that. So I like to say that chapter one of my career was in healthcare and my job understanding end user needs and building products that met end user needs, there was a health reporter for the Boston Globe named Betsy Lehman and she was a cancer patient at Dana Farber in Boston and because of a medical error she actually passed away because she got four times the dose of a medication that she should of and she did not survive that. And so this was heavily covered in the Boston area and my hiring was a part of the staffing up that happened following Betsy and one other person was severely injured as a result of a medication overdose. So what they did, the Partners Healthcare system decided every unit is going on this online order entry. And so I was part of the staffing up that happened following that incident and our job was to understand okay this online order entry, we’re replacing the pen and paper – the clipboard at the end of the bed – it’s different in the labor and delivery unit. It’s different in an emergency department. And so my job was to understand and then – understand the user needs and translate that into system requirements. And it was just an amazing job because we did nursing shifts so I would work 11 pm to 7 in the morning on a labor and delivery unit and see the workflow there and understand how that product needed to be customized. So that was really fantastic work in the healthcare field. And then I went into online travel for the second chapter of my career. And online travel is similar in its complexities because you’re thinking about booking cruises, booking vacations and the complexities involved in that. And so booking engine, it was all about conversion and getting people into the booking engine and helping them shop and compare. I was at TripAdvisor for a little while and primarily doing information architecture and usability work and that was – online travel is fascinating and to this day I think about some of the things I learned about you know why Kayak made some of the decisions they made about their filters and things like that. It’s, it’s – I still refer to some of those points because it’s a complex product and you winnow down these results by using filters and things like that. And then I was in the UX field like I said up through information architecture, into the broader UX and then I – I fell in love with user research and I carved a path – carved a path forward to do just that and it’s been incredibly satisfying.
Steve:Is there something about you as a researcher, whether it’s maybe the product of those experiences or anything else? Like what’s – I think the Cruella de Vil anecdote is one – that’s sort of an evolution. I don’t know, anything else that would characterize – we talked about leadership a bit, but I’m thinking about doing the research. How do you – what’s…
Kate:So doing the – yeah, you know I love research because I love hearing people’s stories and I only – it’s funny I only read non-fiction and so I’m just very curious. I love hearing – I love new experiences and I love hearing people’s stories and I also love distilling that down, finding – you know finding those nuggets. And I love the moment – you talk about this actually in your book, Steve, but you talk about getting from question/answer to question/story. That is such a satisfying moment when it happens in an interaction with a user. And I never get tired of that. And in fact every time – you know we can be working on a large ethnographic study and you think okay you know we’ve got 25 public library patrons or we’ve got 22 physicians we are going to be speaking with and I always walk out of every session – you know you’re drained and you’re exhausted, but it’s that feeling of you just – you just ran a marathon and it’s the greatest feeling in the world because people just shared their stories with you and you get to take those stories – you’re obligated – I mean that’s such a privilege to have someone share that information about how they interact, how they live their life. And then you get to come back and represent that through their words and through their stories to product developers and owners. I just think that’s the best work in the world.
Steve:I love how you describe it and the “it’s hard” but you feel good after.
Kate:And you know it is true and that’s why we joke – you know as a researcher you’re like a Navy SEAL, I mean minus the incredible – minus that torturous training process, but we have different training we go through, but you have to kind of – you need to be able to be dropped into any situation, adapt quickly, find your way, get your answers by making people – you know you have to make them comfortable and then you’ve got to go and you’ve got to go back and you’ve got to deliver what you said you were going to deliver. And so we work hard to find those – we work hard to excavate and to uncover what’s there. And I’ll tell you something, my family and I are in the process of moving now and as I think about it – I always say that moving is an analogy for these design excavations and these changes, these product evolutions, because when you move every single thing in your house you either have to – you have to hold it in your hands and you have to make a decision about it. Does it stay? Does it go? And when we think about changing products and moving – you know evolving platforms, like we have to look at every aspect. So the research is comprehensive. We have to look at every single piece of it. We have to understand users in all aspects of their lives and we have to make decisions about does it stay, does it go, does it change and how?
Steve:So what else should I have asked you about?
Kate:One question that we are asked often is are you guys the user experience group? Are you UX? And we say we’re user research. And I think user experience is a different configuration at every company. And we at EBSCO – we’ve tried a couple of different versions and the version that works now is that user experience, because of our diverse product set – you know we have corporate products, and medical products and school products and we have you know a digital magazine product called Flipster for the public library. All these products are so different and these services are different and so there is no universal user experience because the user is so different. So what we do is user experience is a responsibility that falls to the product managers and product directors within each of those product pods. But user research is central. So user experience is not centralized because of the differences in all of the products. But user research is centralized because I will tell you having done the research 60 year old physicians and 16 year old students they’re searches and their strategies for searching are not as different as you might believe there are. So there are similarities and when we learn about people’s process we can apply – we can extend some of those findings across different markets, I think more than you can when you’re thinking about a product experience.
Steve:So is user experience part of the product management organization?
Kate:User experience falls within product management meaning it’s a responsibility, it’s not a centralized group. It’s decentralized within the product pods. And I think over time, you know these models with evolve, but that’s where we are now because we’ve tried on a couple of occasions to have it a little more central and it just didn’t take off as it should. So I think over time we will continue to revisit that, but right now user experience falls as a responsibility, as a product experience within the product pods.
Steve:I think you know these conversations are – they often cover a key point that you made which is like here’s where it works for us right now and these good things that are evolving, but I like how you’re talking about – you know there’s a responsibility, there’s an action, there’s a task and then there’s a team or a job title or a department and that I think sometimes we conflate the two of them when we talk about who does what, how is it structured? There’s activities that have to take place and then there’s well who does those activities and what do we call them?
Kate:Yeah and there’s a sense of – I think the challenge in centralizing UX is you’re trying to be everything to everyone and that gets challenging and then you’re not as close to the products as you might be. But when we’re – in user research when we conduct research we have the product managers with us. I mean they’re the co – they’re working on that research with us. They’re sitting on the floor of that hotel room in Las Vegas where the student is conducting research and they’re watching it right along with us which actually brings up a great point about user research. When you are doing user research in the wild be safe. Make sure you build in safety measures for your team. I mean we have to meet strangers and so be safe about that and that’s something that we talk about and it’s something we accommodate for. Because we’re going to our users so we don’t go alone. And we make sure that we’re safe about it because I’ll tell you something – even – I had moments where conducting usability tests, you know you’re conducting these sometimes after work and it’s dark and then the usability test staff at the testing center has gone home and you’re in this testing studio and these participants you’ve never met before are coming in. That’s something – I want to write some form of a blog post or an article about that because it’s something that I don’t hear talked about a lot, but safety in conducting user research should be paramount for all of us.
Steve:One of the things that I’ve struggled with and I think I wrote about this in Interviewing Users is the difference between being unsafe and feeling uncomfortable and that – I mean my suspicion is if you don’t kind of check yourself it’s easy to think that one is the other.
Kate:Agreed.
Steve:And not to minimize being unsafe in any way, but there’s feelings of being out of control. There’s feelings of just being out of your comfort zone. If you’re not comfortable being uncomfortable you might not know the difference between being uncomfortable and being unsafe.
Kate:Yeah, and as I said part of research is new experiences, new people, environments that are not your own, not your own native environment and so certainly if you’re someone who’s easily thrown off your game user research might not be for you because you’re put in all sorts of circumstances and people will say the unexpected. There have been – I mean we always talk about this. You know you’ll get asked for a job – I’ve had researchers who’ve been asked for a date. I mean it’s really – I mean the unexpected does happen – does and will happen.
Steve:Yeah, I’ve wondered a lot about the experience of being a woman in user research and the experience of being a man and that if there’s an aspect to that that like men just couldn’t possibly believe. And maybe this is about being a woman in our society also that men couldn’t possibly understand, but – that example of being asked for a date seems like oh yeah, that happens.
Kate:Sure. And you know when – I’ll be honest with you, when we were in Las Vegas doing the interview I had recruited the student I was meeting with off of Craigslist and so I was meeting a stranger that we’d recruited off Craigslist for a research interview when I was out at a library conference in Las Vegas and I asked my colleague Dave to come with us and Dave is very fit and very muscular and I asked because he was part of the project, but I also asked him because he’s strong. And I was – I had recruited someone off Craigslist and that was deal with my husband. He said I’m nervous about this. I said well I’m going to bring this guy Dave and he said okay, well I feel a little better about this. But you have to think about safety and in that situation I wasn’t – I wasn’t protecting against feeling uncomfortable. I was protecting against you know all the crazy stories you’ve heard about Craigslist I guess.
Steve:Is – yeah, is being asked for a date an unsafe experience? I guess it depends on how, but – I’m not even sure how to ask this question, but sort of separate from the Craigslist experience, how does – how do women in the field deal with the set of things that may – the type of content or conversations that might come up with – how do we even frame that?
Kate:I know. It’s a great question and think about it. Think about part of the Cruella de Vil, like serving it up cold, no one’s asking Cruella de Vil on a date. Do you know what I mean? I mean she’s like the ice queen, so who is going to ask this icy researcher who’s not even giving a sense of like am I doing the right thing on this, am I passing this test on the website. So part of being comfortable with people is being comfortable but still having boundaries. Still being the researcher. And part of it is not being alone in a circumstance that would make you uncomfortable, or make a conversation like that too convenient. And you know I think personally I can sense when we might be getting off topic, or we might – or the conversation might be getting too comfortable for the participant and so then I just move it forward. But I think this is an area where some sharing of these thoughts and experiences would help the larger research community to think about these issues and hopefully not have to be in a situation where they’re uncomfortable or unprepared.
Steve:I’m going to pitch hard for the war stories – having people talk about for the last few years. With the aim – I mean there’s many more circumstances that come up than just the one that we’re talking about, but my goal is yours. To hear these stories starts to, you know let people if nothing else be aware that they can – these are the kinds of things that can happen and so, you know, there doesn’t seem to be one size fits all right way to handle it, but to – like you said the Navy SEALs have to adapt.
Kate:You have to adapt and you have to adapt and you have to know your boundaries and you have to be comfortable with uncomfortable conversations. And so I was just having this conversation with my husband the other night about being comfortable with conflict and we talked about that as a trait of good management and I said you know as a researcher you have to be because sometimes you’re putting a product or something in front of a user and it may evoke some emotion, they may be upset about it. They may make that personal against you. So as a researcher it’s a requirement. You have to be comfortable with some level of conflict because not everything is smooth sailing in these research sessions.
Steve:And so conflict could simply be asserting boundaries and it may feel – if you are someone that’s kind of – that doesn’t – that is trying to avoid conflict, drawing a boundary which could be – I liked your example before of just moving things along. That’s asserting a boundary and that may feel like conflict to someone. Conflict is let’s get out of here – I’m going to punch you in the face – it doesn’t have to be dramatic.
Kate:No, and the conflict – right, the conflict could be – the conflict or the confrontation could be as simple as you’re not being responsive to a direction they’re taking the conversation in and you know I think as we move away from methods like the focus group – you know the in-person focus group where you have – that’s not a method that we use often because I think there’s so many challenges with it. So you’re moving to these individual – you know you’re playing Charlie Rose and someone else is sitting across from you and you’re understanding their lives, right. You’re kind of moderating the narrative of your user’s, your participant’s life and their experience. And as we move to these one on one, the deeper exploration, it gets more personal and the stakes are higher and being boundaried is important. It’s important as a researcher. It’s important as a person. It’s important for the participant to understand the parameters of the session.
Steve:And that you as the researcher have permission to continue to reinforce those boundaries.
Kate:As researchers one of our challenges is trying to figure out what should our temperature be and the temperature being about that level of interaction with, empathy with our participants, our users. And you know you don’t want to have yourself dialed all the way down into the freezing zone where you’re disconnected from the user and you’ve got the Cruella de Vil phenomenon happening. But you also don’t want to be so over the top connected and warm that it becomes like a friendship conversation and you have difficulty stepping back and still having those boundaries as the researcher. So this question of temperature, this question of you know how accessible are you, we haven’t solved it yet, but if I had to look at – if I had to look at a temperature scale I’d say it’s in the – the medium to warm is the ideal. The approachable, medium, warm temperature.
Steve:If I can pile on your metaphor a little bit, there’s your internal temperature and there’s your external temperature. And so you know I mean I’ve interviewed people about sad things. I imagine you have as well.
Kate:Yes.
Steve:Talked to people about situations that are sad. There’s a difference to me between how you feel and then what – and then how you do or don’t exhibit that feeling.
Kate:That’s a great point because part of our research has been focused on physicians and nurses and you know because EBSCO has strong presence in the medical market. So when we did a large study of physicians, what we ended up doing was talking to physicians in all different specialties and one particular physician stands out in my mind because she – she saw patients she – she delivered end of life care and her stories were unbelievably touching and several were very sad. And it was hard not to cry because some of these stories were really quite sad and I think what she saw – so inside, inside you’re really feeling the pain of what she’s describing, about families saying goodbye to other family members at the end of life, but to the outward – to the participant, to this physician, this really lovely, just wonderful, compassionate human being , to her I think the researchers, we just looked somber. You know we looked like she had told us something serious, which she had, and we looked somber and we nodded and we were respectful of that. But I don’t think she needed us to cry to know that we had feelings. I think the way we handled it inside, I think we had stronger feelings that we’d led on, but we treated the information soberly and somberly because it was – you know it was sad. It was sad. And that’s part of uncovering people’s truths that is so incredibly rewarding. So if I go into a session like that, and you know I use that approach that was recommended to me years ago, the like serve it up cold, I don’t deserve the privilege of being in that room. I don’t deserve the true honor of speaking with these participants who are helping to evolve our products and you know show us their lives. You don’t deserve the job of a researcher if you’re not truly empathetic to your user.
Steve:Sure. I think what that story illustrates, that seems very important, is that you had the feelings in the experience with the end of life professionals. You had the feelings inside. You acknowledged for yourself that you were having those feelings, but you chose in the context of research how to act on or how to express those feelings, and that you made a different choice that you would make in a social situation – you know anything that’s outside the context of doing research – you chose how to react differently than you would.
Kate:Yes, and that is key because the participant – the person that you’re speaking with needs to feel like the researcher is in control. So if someone starts sobbing uncontrollably all of a sudden the physician might be worried about the researcher and then it creates this shift in the dynamic that is hard to recover from. And so it’s the same as if someone tells you a joke or says something funny. If you laugh so hard that you can’t regain your composure it shifts the dynamics of the interaction. So it’s – that’s where having the boundaries – and it’s not so different than being a teacher or being a parent that you know your kid says something silly, or does something, and you think okay well do I want to reinforce – do I want – how do I need to react in this moment to convey the message that okay we’re going to continue – we’re still going forward or we’re still going to have a nap even though you just said this thing that’s really silly and we could laugh about it for 2 hours. So you have to keep the research – you have to keep the research moving forward, but you have to convey that when someone – you have to convey that you’re listening and you’re processing what they’re saying. So when someone is telling you a very sad story, when you’re somber and when you are listening intently and making eye contact that says I’m here with you, I hear what you’re saying and it’s registering and I understand that it’s serious. I get it.
Steve:If you – I think it’s different with a professional, but if you – I mean this has probably happened to all of us. We’ve been someone who’s had something difficult happen to us and we talk to other people about it and they say oh, I don’t know what to say or something and that we end up in the role of trying to help them.
Kate:Yeah, we end up in the comforter role. And keep – you know something I always say to other researchers is you have to be very clear that the research session is a judgment free zone because one of the challenges we have is when you’re researching and interviewing students sometimes they fear that there’s a right way to do what you’re asking them to describe and they don’t know what it is. So you don’t want people to feel that they have to say well I’m sure there’s a better way to do it, but this is how I do it. You want to be – you want them to feel comfortable and you know show us how you do X, instead of – and you don’t want them to be apologetic and you don’t want them to feel that what they’re showing you isn’t good enough. And often during those – I mentioned those live student demonstrations that we do with some customers that are so enlightening. We start of and we talk about you know your act of conducting research – your act of information seeking is uniquely you. We want to learn about it. There’s no right, there’s no wrong, whatever works for you is what we want you to show us. So we really spend a lot of time on that at the beginning of those sessions because we have a room full of librarians. We have a room full of library staff watching someone conduct research. And so we don’t want them to feel intimidated.
Steve:I had a client that I worked with that said to me that sympathizing is judging. And we think of judging as sort of the right or wrong, but she was saying you know if you say oh that’s awful – I’m not going to do justice to how she put it, but it’s taking a value framework that is yours and putting it on their story.
Kate:Right. You’re handing your world view to someone and saying oh it’s okay. Or you’re making a judgment and there’s no place for that. And so we hear it all. I mean that’s why we talk about you know anything can happen in these research sessions. We hear it all and we hear – you know we hear students talk about how they conduct scholarly research and sometimes the techniques they’re talking about aren’t necessarily scholarly. You know if they’re talking about looking on Wikipedia which isn’t a peer review…which is not a citable source – and so we don’t stop them and say well did you know, because we’re not there to fix anything. We’re not there to teach. This is not a teaching moment. This is understanding and this is actually exploration and one of the things we try to do in our group is we try not to say – when people come to us and say okay we want to validate – we don’t go out to validate. We go out to explore and in that process of exploration and excavation, you know we’re lucky enough to unearth findings, but we do so because we’re listening and not putting our own framework on the participants or the users.
Steve:Well Kate, let’s wrap it up there. It’s been a really excellent experience to talk with you and thanks so much for everything that you shared with us today.
Kate:Steve, thank you and I look forward to seeing you in Boston someday soon.
Steve:I look forward to that too. Thanks.
Kate:Okay, take care.
The post 13. Kate Lawrence of EBSCO first appeared on Portigal Consulting.
This episode features Pree Kolari, the Senior Director of Design Strategy and Research at eBay. We talk about the career arc of a researcher, having impact on the product, and breaking down organizational walls.
We can make it a little better in terms of customer experience where they buy with one less click. We can also change the dollar amount eBay makes by .1%. That’s a win and it’s very easy and it is very, very tempting for everybody on the team to go “Yes that’s great, let’s go do it!” But then you ask the why questions and go for deeper insights, not just the surface level insights. You go deeper. You find out that it’s a lot more and you will have small things that you could do on the product that will give you wins two years from now. – Pree Kolari
Show Links* Pree Kolari * Follow Pree on Twitter * eBay * eBay Classifieds * StubHub * eBay Motors * eBay Fashion * local maximum * stack rank * Moto X * The Design Sprint * Kinect * Zune * Metro * Karmann Ghia * A Pattern Language by Christopher Alexander * The Designer is Present * Product Management + User Experience
Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes.
TranscriptSteve Portigal: Thanks, Pree for being a guest on Dollars to Donuts.
Pree Kolari: Absolutely. Looking forward to talking.
Steve: Let’s do the basic intro part of the interview. Tell me something about you and your work and kind of why we’re here to talk.
Pree: Sure. I’m Pree Kolari and I lead the Design Research team at eBay. It’s actually Design Research and Strategy team that I lead at eBay. And we’re a pretty sizable team, about 20 people across the US and Europe. So we have people in San Jose, San Francisco, Portland, Germany and London. So pretty sizable team. We work across all of the products on eBay. So we work within of – we work with the product teams and work on both new products as well as existing products.
Steve: So if I’m not an industry insider and I’m listening to this and I think eBay – eBay has multiple products? Like can you explain you know eBay a little bit for those of us that aren’t in the know.
Pree: Sure, absolutely. So eBay is interesting because you have a couple of different kinds of users. One is the seller and then another one is a buyer. So both of them see different sides of eBay. And they both interact with eBay via desktop, mobile, you know multiple devices as well. So every one of those interaction spaces are product areas from our standpoint. And so that’s one. Then eBay as a company plays in many, many different spaces. One is eBay the marketplace itself. Then there is a bunch of other places which is Classified, StubHub and all of those things. Those are separate teams by itself and generally speaking we don’t necessarily work across as much as we should, but we do. So that’s – does that that give you…
Steve: So where does something like eBay Motors fit into that?
Pree: Absolutely.
Steve: Is that like a StubHub?
Pree: It is – no, no, no eBay Motors is part of our team. Those are all things that are considered verticals. Good point. So within of – the shopping side of things – there are certain areas that have specific verticals and eBay Motors, or Fashion, you know those are the spaces that are a part of our team.
Steve: If you’re an eBay shopper do you experience those things – if you just go in and search do you know when you move between the verticals?
Pree: Ideally if we did our job well from a design perspective, as well as from a research perspective, you shouldn’t notice it. But sometimes we do, you know because it is two different teams that are delivering the same product, right, and so that’s one of the big challenges of doing good research because as a researcher you’re thinking end to end. You’re thinking the customer journey. You’re coming at it from a user standpoint and users don’t see your org structure. They see the product. The moment the users see our org structure we have a problem and that’s exactly what we shouldn’t be doing.
Steve: So it sounds like the experience today for someone, there’s lots of points where you wouldn’t see the structure and some points where you would.
Pree: Absolutely, absolutely. Lots of places where you shouldn’t see it, but some places you see it and that’s kind of why a bunch of us have jobs.
Steve: To try to…
Pree: Try to fix it. Yeah? Yeah, it is about fixing. Some of it is about fixing. Some of it is about like coming up with new things that will make the older behavior that we have started kind of become back – you know take a back seat.
Steve: It makes me think about what – you know how research fits in, which I think we could probably spend a long time talking about – I mean I think sort of the counterexample is always the – you know there’s a question that comes from a certain team and they’re really only interested in the answers that affect their decisions within that.
Pree: That’s right.
Steve: Whereas, as you say, as a researcher you’re looking kind of end to end.
Pree: Yes.
Steve: So the question is you know what – you know how does the work that you do – how do you structure it to touch different teams at different points so that those decision can be made.
Pree: It’s a good question. So one of the things that our team, the whole team does is talks to our counterparts, our colleagues or people who are part of our core teams. Yeah? And the main point there is understanding the goals of what we are trying to achieve. The secondary part there is to train people not to look for local maxima. We have – we have a habit – and this is just a bad habit that’s come over time, of looking for small wins, local maximum. Instead, how can we focus on long term thinking? So there is a little bit of bringing people along that happens. And during that time the framework that we end up using is just a simple one of generative, iterative, evaluative. So here’s the different kinds of spaces that we can look at research. The first part is generative – that’s about generating ideas. That’s about diverging in from an ideation standpoint. Next one is iterative – this is where you’re continuously testing things out to learn quickly and make this product much, much better. And then there is evaluative. Now we’re pretty close to ship. How do we make sure that we ship something really amazing? And so that’s the three step kind of a thing. If you look at a waterfall process, you look at an agile process, pretty much those things fit right in. Yeah.
Steve: Can you say a little more about local maxima? That’s a great term that comes from say like some kind of number theory something.
Pree: It’s a statistics kind of a – it comes from statistics. It’s used a lot in the stock broker side of things, right, and because you’re always – there are people who are there for the short term, they just are day traders that just made money. They want to sell it off and then they want to move on to the next one. So if you look at today’s stock you probably see an up or a down and then the up could be a good up and it could be the maximum up that the stock has ever gone, or you could step back a lot and see that the trend has been going up for years – 25 years or 30 years. And then you suddenly think, okay you know what if I hold on to the stock it’s going to do very well. So now you look into a product or a research side of things. You start to see that we’ve made – we’ve taken a certain path. Let’s say with eBay we’ve taken a certain path with respect to how we make a buyer go through the categories, go through the shopping experience. And we can fix it a little bit. We can make it a little better in terms of customer experience where they feel like hey you know I got to buy with one less click. We can also change the dollar amount by which eBay makes out of this about .1%. Now that’s a win and it’s very easy and it is very, very tempting for everybody on the team, including us researchers, to go yes that’s great, let’s go do it. But then when you ask the why questions and then you ask – you go for deeper insights, not just the surface level insights – you can see patterns, but you have to see the insights. You go deeper. You start to find out that it’s a lot more and you will have small things that you could do on the product that is – that will give you wins two years from now. That’s what I mean by local maxima.
Steve: So I’m thinking about – we won’t get into any specifics, we’re talking about this very generally – I think about things like you might have other kinds of maxima like how often they’ll come back?
Pree: Right.
Steve: But then you might have sort of larger issues like the meaning. Like if you want to change this site – and I’m really just making this up – you want to change this from a transactional partner to something that becomes a support system.
Pree: That’s right.
Steve: Well now that’s about longer – that’s about shifting the overall meaning through a lot of design choices and you have to measure that very differently than did the amount in the cart go up. Is this what we’re talking about?
Pree: You’re absolutely right. It’s not just looking at a financial benefit that you can quickly calculate now. It’s about looking – it’s about taking some amount of risk which are long term financial gains because it will come back, you know, and the team will start to have an intuition of the fact that it will come back. And there is a certain – you know we’re a profit making company, right, and at the end of the day we want to make profit every day, yeah. And so it’s not about just putting – taking all of our eggs and putting it for long term wins. It’s about being very intention and saying hey you know here’s a few things that we’re going to do where it is high risk/high reward – here’s a few things we’re going to do which is absolutely beneficial and we can see the benefits right away.
Steve: You’re describing a strategy.
Pree: Basically.
Steve: Or a road map.
Pree: That’s right.
Steve: Like here’s where we want to go with this.
Pree: I am. I am describing a road map and a strategy.
Steve: And you said right at the top that strategy is in the title of your group and you’re talking about research as a strategic tool…
Pree: Absolutely, right
Steve: …very much.
Pree: Right, right, right. So as in people on the team come from different kinds of backgrounds. There are people who have no research degrees, if you may. There are people with MBA degrees, right, and that’s intention. When you put these kinds of – different kinds of people together there is – there is a radical collaboration if you may. And that kind of collaboration happens because people are coming at it from different mindsets, but they respect each other’s mindsets. Yeah. And when you start seeing that happen other people also join in and I want the engineers, I want the designers, I want the product managers joining in and pitching in and making it their own idea. And that’s how you take it to ship.
Steve: So in this – this maxima example I think is a really great one. We’re sort of avoiding the pitfall of a local maxima I guess maybe is the theme here. What kinds of conversations are you having with people and how are you helping them – I’m assuming you are helping them to see that as a choice or a different path.
Pree: Yeah, good question. So you have – there is – this is where it’s a little bit of an art more than a science. And it’s a little bit of wisdom more than here’s 5 things you can do. What I look at is I think of – you know I don’t do as much – personally don’t do as much research as I used to do out in the field. I do very, very little. But what I end up doing is I end up doing research on – in the corporate side of things if you may. It’s about understanding the organizational psychology. It’s about understanding what are the needs, what are the pain points, what are the motivations of these people who are working inside of an organization. And once you understand that and you map it you can think about like why is this person thinking this way and you can still figure out that there is a motivation that works with what you’re trying to communicate and it becomes very easy then. And so I am always looking for win/win situations. It’s a negotiation at the end of the day too. So it’s a win/ win situation. Let’s figure this out and make it happen. So I don’t think I have a specific answer there, but more of a generic answer.
Steve: You make me think about my own work. I’m someone that works outside the organization. So I come in and the best engagements for me are ones where I’m partnering with somebody who has that insider knowledge and has sort of worked over time to try to find out how to be successful so that I’m not blundering it and saying I know the best way to do it. It always has to be adapted to the context. And there’s often this interesting tension – maybe tension in a good way – when you come in – and you’re newer in this role so you probably have a perspective on this. When you come in you see everything that’s wrong that needs to be changed. And then when you spend time you choose your battles. You figure out your strategy. What I see, because I’m always in the newer side of things, is you also get – I don’t want to say co-opted, it’s too strong a word – but you become part of the system. And so I have these situations where I feel a deep passion, like this is what these people need and people that are my gatekeepers are like yeah there’s maybe not an appetite for that. And it’s not to say that we then fall flat, it’s just – it’s where we put our energy. So I love what – I mean I think it’s – I love what you’re able to do. You have to be embedded. You have to build relationships. You have to kind of go over time. Without being kind of confrontational to you, do you think, have you seen it in yourself, that you’re now part of the system – you can in to sort of drive change, but then you become part of the system. What does that arc look like?
Pree: Very good point. So I’ve done like consulting work like you have for a long time. And then I moved into the corporate side of things – 2008. So been doing this for 8 years. Big change. Big difference. The advantages and good things that corporate or the consulting side comes with is they come with a fresh perspective. And you come – like when you come and work with us you come with a fresh perspective that we never would have thought of. You also come with an expertise. Yeah. So the hierarchy that has already been established within the corporation doesn’t apply to this consultant who has just come in. And so that’s an awesome thing. Yeah. However, these – the guys who come in also do not know the culture of the organization, or do not know the history of the organization. Now within the corporate structure there is researchers inside the corporate who are embedded and they work really, really hard to build the relationships to make sure that they are doing the right thing. And sometimes yes, getting embedded for too long will make them feel like hey you know I’m part of it and I cannot make any changes. But we end up, we end up really working hard not to let that happen and to make that we go back to – like within our team we have a – we build together the team with a set of principles and these are principles that everybody in the team worked together and we just kind of articulated. One biggest principle that we have is about impact and you know we write down like hey what does impact mean? The other one is about transparency. And it’s about transparency with respect to communication. Yeah. And as soon as we end up being extremely transparent we break down walls that otherwise exist within the corporation. Yeah. And it’s part of our jobs to break those walls. Yeah. And we – within my team we are not measured by how many reports you have written or whether you have finished so many projects. You are measured by the impact on the product. Yeah. So it is super important to work with the product manager. It’s super important to work with the business person. It’s super important to work with the designer to make sure that the impact is clear. And that’s when it changes our own behavior a little bit to not conform to the culture in the way you described it. Does that make sense?
Steve: It does. I feel a little skeptical and that’s not a skepticism towards you. It’s just in general sort of having observed this because in order to be successful there’s a certain amount of conforming that has to happen.
Pree: Absolutely.
Steve: It doesn’t mean that – you know what I mean, you have to kind of conform to figure out how to choose the battles.
Pree: Yup.
Steve: And there’s battles that you’re not going to choose now because you’ve found ways to be successful. And think about yourself in 2007 – I’m going to guess, you would come in and look at yourself doing this job and like shine the flashlight in some corner and say like guys you’re denying this. And now you, 2016 you is kind of saying like well no you know we have these principles and we’ve had impact and we have this track record and like we have lots and lots of success and then – you know I’m doing a little puppet show here – 2007 you is kind of saying like no, no, no. Like over here, this thing. That’s the tension that I think is…
Pree: Absolutely. So I agree with you. So if I look back in the 2008 me I would look at the 2016 me and say hey look, you’re not listening to me. You’re not listening to this part of the equation. And that’s the reason why I think – you know it’s learning, right, from a life perspective. Absolutely, so I prioritize certain things to make an impact on? I do. And everybody on my team does. And we do it because we believe in the fact that making impact is more important than making impact on all the 10 things. So let’s imagine there is a study where we saw 10 different things that are very important things to fix, okay – an evaluative study. We might choose to pick only two of the top things and some of it might be confrontational, but we might pick only two of the top things because we know that if we pick those two and go really, really hard it will help us instead of diluting our impact with going with 10 different things. Yeah.
Steve: Um-hmm.
Pree: And that’s the choices we end up making, but the choices are super important because it’s exactly based on impact. Doing the right research means talking about all the ten because those are all observed and those were patterns that were observed. At the end of the day we are not doing research, we’re doing applied research, you know, and we’re working within a product company that ships products. You cannot ship a research report. You can ship a product. So that’s what we’re going after.
Steve: Yeah, that picking the two of the ten I think is really…
Pree: It’s an art. It’s hard.
Steve: …really interesting. And it’s not like you haven’t learned those other 8 or haven’t articulated them. I don’t know do those float around kind of in the ecosystem somewhat?
Pree: They do, right. As in they might come back again and this is where – so I joined the eBay team only less than a year ago actually and our team is quite amazing. Like have been doing amazing work for quite a few years, since ’99. The research team has been there and have been doing like quite, quite a lot of good work. There were challenges in how this work was heard within the organization. There were challenges with respect to resonance on some of the outputs. And so we had to stack rank everything. So this is the engineer from me talking here. I’m all about stack ranking, drawing a cut line and saying hey, here’s the few things we’re going to go after and we’re going to go after really hard.
Steve: Is that a fancier way of prioritizing?
Pree: Basically.
Steve: Was that what you’re saying or is there like a methodology?
Pree: It is prioritizing.
Steve: Okay. That’s a great phrase though.
Pree: Yeah – stack ranking?
Steve: Yeah.
Pree: Yeah, it is. It’s probably engineering talk, but yes. I would rather you know – because at the end of the day we are – again, concentrating on shipping, yeah. And so again I think that those are things that are coming from my background as an engineer. As a person who has like shipped multiple products. As an engineer, as a start-up person. As like – you know – coming back to if we did amazing research and there isn’t a follow through afterwards the amazing research just sat on shelf and nobody benefited from it. Because I know that in my heart, and I know that a lot of people who do this, we’re doing it not to just do research. We do it because we enjoy seeing the output of this which is the experiences that have changed. And that’s what gives us satisfaction and that’s the long term view.
Steve: Even just the idea of sort of the output of – these are significant efforts. A “research project”, whatever it looks like, it’s a significant effort, right?
Pree: Right.
Steve: And that even to get what you learned down to 10 things or 10 big ideas or 10 opportunities that takes a lot of pain. You have to leave a lot of things on the floor and synthesize and organize. And then – I’m just sort of toying with this idea you’re describing that says okay team we’ve kind of got these 10 things, let’s go at them with two. It’s very different than – and I’m sort of thinking about the consultant cycle which is, you know, it’s hey the consultants are back, here’s their thing, it’s a significant effort. It sort of ends the engagement. We have to impart as much as possible because we’re going to leave. So to get it down to 10 is really hard, but it’s like I think about in certain circumstances like what would happen if we said okay there’s two things. Yes we have a lovely Executive Summary that says like here it is on one slide, but just to really shout loud about two things.
Pree: Sure. Actually let me give you an example, okay, which is from my previous work at Motorola. So here’s the interesting thing that happened. So Google had just acquired Motorola and that’s when I joined Motorola as part of the Google, when they acquired, around that time. And one of our – one of the challenges that we got from – during that time it was like a bunch of the Google executives who joined Motorola and it was clear there was a mandate, said let’s reduce the number of phones that we’re making to very few and whatever we are making let’s make something amazing. So we did a bunch of ethnographies where we saw people taking pictures. That’s what we were looking after. We were looking at images and people taking pictures. And one thing we observed over and over again was how hard it was for people to quickly capture an image. And what all things people were doing. And this is where – you know you’ve seen this in the field. Mom holding a kid and trying to take a picture and a selfie – there’s a diaper bag in another hand. It is, it is tough, And how do you allow for that to happen? And we could have written reports and reports about this, but we went in with one phrase. We said, you know, pocket to picture in 4 seconds and one hand. That was our output from a research study that took 2 months if you may.
Steve: And you learned, just to – you learned lots of things…
Pree: We learned lots of things.
Steve: Tons of improvements that could be made…
Pree: Absolutely.
Steve: Lots of opportunities for fixing, creating new.
Pree: Absolutely. And when we know that we want to get from a pocket to picture this much time, okay, and one handed. Now you suddenly, suddenly have, have something that the whole organization can rally around. There is a purpose. It’s a measurable goal. And what happened was with the MotoX we had a gyro – every phone has, you know has, has a sensor in it. We put in a low power sensor as well and people could just twist twice and with one hand take a picture. We even changed the UI. In any big company you could think about like how difficult it is to remove the interface. And we removed the interface to make the whole screen a tough target and you could take a picture. And suddenly the user experience completely changed. And so could we have gone with like hey the pictures that were taken in a dark bar were too dark? Sure. We had a whole bunch of things, right, but we didn’t. We made a choice and we made multiple choices afterwards and so the quality of the picture came up after and we worked very hard to change the quality of the picture and we have specific things about how you can change tones based on the picture, like a skin tone vs. food vs. the sky and make the perception of that picture to be higher quality. And so what we’re talking about is very data driven design. And so when you have data and you’re very specific about how we can use the data to change the design it always works.
Steve: And I think it’s interesting that you research output, if you will, was sort of a vision statement…
Pree: That’s right.
Steve: …you know for a design brief. And that’s different than personas or journey maps. It’s not about the format of the output. It’s not even “Here’s what we learned about people.” You’re kind taking it all the way to say that we did this research and it tells us that this is what the product needs to deliver.
Pree: Yeah. It’s an objective. It’s, yeah it’s an objective.
Steve: It seems like there’s something interesting about how teams are structured, or how these programs kind of run, that lets a research project output a product decision. As opposed to researchers learned this, they talked to designers, they ran a session, the designers decided – you’re sort of putting it all under the same umbrella. We did this activity, the thing that we learned is the product should do “x”.
Pree: Yup.
Steve: So how do you structure – how do you bring the right players together and sort of have the whole endeavor such that that can be the output?
Pree: Everybody who’s on the team is assigned to a product area. So they have a core team that they work with and it works very well in an agile kind of an environment in eBay because otherwise you’re a service organization within – of a big company. We’re not a service organization within a company. We are a product organization. So what that means is let’s imagine a person on the shopping experience. They’re part of the shopping experience team and are embedded within the team and they work with the team everyday on daily stand-ups. Yeah. And they figure out, they have – they maintain a roadmap of what research has to be done. They bring up things within the team and talk about what are hypothesis, what are myths, what are things that we know, what are data, what are insights that we know, that we really need to go after. And so all of those things are super important coming from the researcher’s standpoint.
Then the next step is about participation within the research. So I’m a firm believer that research isn’t done only by researchers. Everybody is a researcher. It’s like everybody is an artist. Like you know we all have this innate ability to observe and make conclusions. That’s part of what we do. Some people just have been trained in this and are probably much better at the structure side of it. Now you have –when you have researchers who are catalysts within the core group, to understand more about the user so that we can make products that are absolutely relevant to the need of the user. The need may be even unknown. You know it might be a blind spot. They might not even know that they need it. Fine. But building that sensitivity for the whole team is part of the researcher’s job and then it is also part of the researcher’s job to collaborate, to get to an output. When I gave you example of phrase that came in the end it is not the researcher who comes up with that kind of a thing. It happens because the designer, the engineer, the PM is all part of that discussion. They own it. They co-own it. And that’s what – that’s when you get successful products.
Steve: That’s a good explanation of sort of how you set up those conditions to make those kinds of big impact. I mean it’s back to your point of impact earlier on. If you can come up with that phrase as your output you’re setting yourself up for impact in a really…
Pree: Absolutely.
Steve: …massive way.
Pree: And that goes a little bit – you know with research we end up – we end up thinking a lot about outputs from only one research report or one research study. That’s I think history. I really believe that it’s about building criteria, putting together principals that kind of collect all the knowledge that we know from multiple studies. And that’s part of the job of an in-house researcher, right, which is different from somebody who is coming at it from a consulting standpoint. And what that does is it helps make good decisions throughout the product development process. We’re making – you know I really believe that the intuitive decision making is bullshit. It’s either data driven decision making or principle based decision making. And so we have to give the ability for people to make the right decisions and that’s – you know it’s getting the data and putting together principles that help us make the decisions. Does that answer your question?
Steve: Yeah. So to try to tie together a couple of things that you’ve mentioned. This idea of everyone is a researcher and then different decision making styles, I’m curious about, you know there’s people that are researchers for a reason. They have kind of an approach. Can you compare and contrast maybe the mindset of someone that has researcher as their title versus. these other people on the team that they’re working with?
Pree: Sure. So I think one of the big things as a researcher, you know we are curious. And the curiosity is what makes a good researcher. The hunger – it’s a sponge. It’s like wanting to learn. And we’re paid to be unbiased. Versus let’s talk about a product manager or let’s talk about a designer. They’re paid to be opinionated. And having opinions and having ideas of where they want to go is what they’re paid for. And now there is a tension. And both people are coming at it from different angles. Both want to understand what the users want, yeah, because we want to finally make a product for our users, but are coming at it from two different angles. And so that tension is what I think is the root of the issue and then like figuring out how to manage that tension – it’s never going to go away. It’s managing it and directing it in the right direction.
Steve: And I think that tension – as you described sort of how that team, your team, came up with – now I can’t get it right – what was the – from pocket to photo in…
Pree: It doesn’t matter.
Steve: …four seconds. Whatever. So that is a product of those different perspectives, right.
Pree: Right.
Steve: So I’m going to just pick on you slightly. I mean you used this phrase which we hear a lot when we talk about research which is like what users want and that’s sort of shorthand, but I think in the examples that you’ve given I think you’re really saying that research is identifying something else. And maybe this is just me being semantically stubborn or something, but no one asked you – and we know this, right – no one said hey I wish I could do this with my phone, right.
Pree: Yeah.
Steve: When you say what people want it implies that the researcher is taking requests, but what you’re describing is something much – it’s that hunger and that curiosity and that – and that synthesis to pull those principles out.
Pree: Absolutely, as in we’ve been doing this kind of work for a long time and we always go for these unarticulated needs. What people say, you know, is very easy to get to. And a lot of times what people say is like they’re lying, you know, and they’re saying what you want to hear. It’s getting to the deeper insights by looking at patterns, looking at behavior, looking at what people are making. And so I go back to Liz Sanders who is a mentor from a long time ago. What people say, what people do and what people make. You start to see that each of these areas give you different kinds of data and triangulating between that data takes a lot of synthesis, takes a lot of analysis. And then you get to the core of it and articulate what it is that our users want. And the point that I think is most important and is lost is it is not two researchers going off for two months, coming back with this magic. This analysis doesn’t necessarily happen in a closed room. The right way, the new way to do it is the analysis happens in an open space, happens with others. It is co-created so that at the end of the day our products are – the objectives for our product is co-owned by everybody building the product.
Steve: So let me ask somewhat of a more tactical question about that. You know you’ve talked about some people – some people are paid to have opinions.
Pree: That’s right.
Steve: And so how do you help people be effective in synthesis when what they’re coming in with is what they’ve been working on, what their vision is, what their intuition is, whatever that is. How do you help them sort of embrace the messiness of the data and be in that synthesis process and – I don’t know. I’ll just stop.
Pree: So some of it – so we’ve been doing a lot of what we call iteration zero sprints. We just started doing a lot of those things. Basically it’s design sprints you know. And Google’s done a good job of writing a little bit about the design sprints. You must have seen some of the design sprints work that Google Ventures has put out. A good amount of work which is really good and – and that’s a very tactical way to get a bunch of people together and not necessarily throwing them right into the middle of swimming pool, but letting them wade in the kiddie pool. Now what happens is everybody gets a sense of what the water is like. How cold the temperature is? What it is like to float a little bit? That kind of a thing. The researcher does most of the heavy lifting, or a bunch of researchers do most of the heavy lifting. Then when we all come together towards the patterns everybody is on board. Otherwise you know this is a challenge that our industry has faced for quite awhile, right, which is trying to explain qualitative vs. quantitative research as well, you know. And we do a good amount of both kinds of research within our team and I see a lot of it where – and I’ve seen this in multiple companies where time it’s like 10 users, even though those 10 users were observed for, each of them observed for a few days maybe. Oh, it’s only 10 people. So do you think we can put a million dollars on this or not? And that question always comes up. And so that’s where getting people really connected with the stories, the real people, is super important. Storytelling is a very important part of design and make good design decision. So it comes back to that part of design that we plug into research.
Steve: So how do you answer that question? Are these stories sufficient to commit an investment? What’s the response?
Pree: It usually ends up being yes. So there’s two things that help quite a bit. If you look at venture capital money going into startups, or you look at like companies investing into new products, there is one which is very good data that’s driving certain directions. Second, conviction. When the team sees certain things and they believe in it there is a certain level of conviction. And you cannot necessarily explain it in words, but you can see it. Us as researchers, what we do is we change that part for the whole team. And that’s the value-add that we bring in. It’s quite amazing.
Steve: I feel like yes and no. I mean yes I know what that feeling is like and I know what that conviction feels like and I know when you just – you know that here’s the opportunity because you’ve been out, you’ve been with these people…
Pree: That’s right.
Steve: …you’ve experienced their stories. You’ve breathed the same air as them.
Pree: How do you then translate it to conviction for everybody else?
Steve: Yeah.
Pree: You know as in like – as a consultant I’ve done immersion rooms. I’ve done videos, video clips and like it’s not enough. And all of those things help. Now what we’re doing a lot more of is we’re actually taking people into the field. As part of our sprints we have users – we have two sets of interactions. We have interactions in the beginning of the sprint with the users. We have interactions at the end of the sprint with users. And through that interactions, first interaction is about understanding the ecosystem, the domain, the landscape. And the next interaction is about evaluation if you may. It’s more concept evaluation. And so everybody has been part of this journey and it’s not just theater. It is actually immersing yourself in the people’s lives.
Steve: I think you’re describing a really nice version of this – we’ve been headed towards sort of everyone being a researcher, research is integrated, everyone participates. And I think there’s a – we keep talking about tensions. There’s an interesting tension there where this thing works best when everybody plays as deeply and richly as possible and yet it’s not – it’s a profession not a practice. What am I trying to say? There’s a better way to say that. You know filling out expense reports is not a job function at a corporation. It’s an activity that everyone that works for that company does. Research is a job. There’s leaders that have that function. I mean I wonder if this is a point and a transition where – because you’re describing something that’s just really fully integrated at its ultimate version. But they need us to facilitate it, to kind of guide it.
Pree: Yeah, and I’m describing an ideal situation. As in it’s called work for a reason. As in it’s not all like easy and simple. We’re learning and we’re making it happen every day. So it is not – it’s not that everything is like done and like flower petals are put in front of us for us to come there. It is definitely a struggle. At times – like and different teams are different. There are day to day stresses and tensions and those are places where again if we believe in where we want to go as a team I think it makes it much easier because again it comes back to our talk about the local maxima as well. You know even within our profession.
Steve: Could we switch gears a little bit?
Pree: Absolutely.
Steve: Talk about your choice to join this group and take on this role. You’re describing the reality and the reality is of course everything is a mix of wonderful and challenging and that’s not – I don’t think that’s unique to your situation, but something was compelling to you about this opportunity and can you talk about what you were looking for in…
Pree: Absolutely –So eBay – so my background is engineering and design. Have worked in design research, innovation consulting for a good while. Then moved into Microsoft, was part of the small team that invented Kinect and also was part of the team that did Zune – all that kind of work, like some of the initial Metro designs. So moved to Motorola afterwards to lead the design research team right when Motorola was changing quite a bit. You know Motorola had gone down in terms of how it was doing and there was only one way to go once you go down that much. It’s like up, or disappear. And my personality overall is more – I guess I’m a betting man. I take risks in spaces – calculated risk of course, but I like that excitement. And so I joined. After a few years some of the work was nicely done already. There was things that were in place. So personally I was looking for other possibilities and this thing came along and I again looked at what was happening. eBay as a company is a 20 year old company, has established itself as a leader within of this marketplaces. Makes a lot of money by doing this. However if it continues to do the exact same thing that it did it probably doesn’t have as much of a growth trajectory. And so how – so overall there is a challenge for the company to reinvent, to think about new possibilities. And for an outsider, for me I was like looking at it and I was thinking hey, you know, that sounds like an exciting opportunity. And a place where somebody like me can make impact. The team was amazingly good. The culture was very good. So I felt like hey you know I could work in this culture. And it felt exciting to move at a time when eBay was just celebrating its 20th anniversary, just split from PayPal as eBay by itself and there is a future that we all can create together. Does that answer your question?
Steve: Yes. So if I’m a researcher and I’m looking at different organizations, because you’ve mentioned a number of different orgs that you’ve kind of come into, you know what are things that you would advise me to look for to try to assess – I mean a good…
Pree: A good fit.
Steve: …a good fit for researchers I think is different than other kinds of professions.
Pree: Yeah.
Steve: You know what do we look for?
Pree: I think – it’s a very good question and I haven’t thought much about it to be honest. You have researchers at various levels. So there is an entry level researcher who’s done a good amount of academic work let’s say, you know, or has gone through some amount of training. And then there is a researcher with like mid – you know two, three years of experience and then there is the researcher with about ten, fifteen years of experience. Or plus, like much more experience. And then there’s also researchers at various kinds of levels. There are researchers who are IC, individual contributor kind of researchers. And then there are managerial kind of researchers. Different places I think offer different things. So there are certain places – like if you look at the Googles and the Facebooks of the world, if you may, there is a lot of opportunities to learn. And you can have a researcher who is just starting out, a younger researcher who is wanting to learn a lot of things. It’s an awesome opportunity. Even companies like – the bigger companies are really good places for you to learn. eBay is one of those. A new person comes in who has no experience and we’ll train them and they get to observe. They get to be part of a pretty big organization and everybody in the organization comes with a different skill and they all are willing to teach. And they’re willing to learn from this new person as well, you know, and so that automatically makes that kind of an environment better. I would say for a mid-level kind of a person it might be valuable to go to a smaller company, a start-uppy kind of a company where you are probably the only person. You can build out the team. You can figure out what – you know build a network, build a network of mentors, build a network of people that have done this before who can become your personal advisors if you may. And then you can go next step. And as you get more senior there’s only a few places that will hire you as well. Because you’re too expensive for – you know it’s just – or it’s where – or you move into different areas. I know of researchers who moved into product management. I know researchers who moved into design. And you move into lateral areas and grow that way. And all of it is learning and it’s something that as researchers we enjoy.
Steve: So maybe a related question, when you look at people that might join whatever team you’ve been on over the years, what are some kinds of things that you see that kind of – that make your antennae go up. Like what are some good, strong signals that say there’s something here with this person?
Pree: Sure. I think it comes, comes down to passion and you can – again, like when you see people really – and I’d say passion translates into several things. It translates into a real hunger for learning. It translates into caring about the product and the user. And you can see the decisions being made that go towards caring about the product and the user. And I’ve noticed that researchers at a certain point they been taught research so much that they sometimes think of research I’d say in a little bit too pure kind of a way and aren’t thinking about it as a tool, a means to an end. And I see some researchers who have gone past to where they have learnt the tools to cause damage. When I say cause damage as in like they start to think about the product. They start to think about…
Steve: The good kind of damage.
Pree: …the good kind of damage.
Steve: It’s back to your impact point, yes.
Pree: Yeah. So that’s what attracts me to like okay saying hey here’s some people that have potential because they care and they care about the product. They care about the user and they’re very passionate about it.
Steve: That’s great. Alright, so just picking up on this notion of passion, we’ve talked, as of course it’s the focus here, we’ve talked about your work and a little bit about your background and kind of how you got to where you are. I’m wondering about other things that you have passions for that reveal something about how you – what we’ve talked about today? Are there any kind of connections or cross pollinations between research, it makes you do these other things, or these other things kind of impact how you think about research?
Pree: Oh yeah, yeah. No, I’m – I have too many interests as in I’m a Dad. I enjoy being a Dad. I am a bicyclist. I bike. I bike commute to work, plus I bike on the weekends anytime I get. I love cooking. That’s kind of a place where I can calm down. Design, generally speaking like design, designed objects. Well-designed objects, well – you know even furniture making, those kinds of things. I’ve been doing this for years and I do a little bit whenever I get time. Life hacking – it’s just more about productivity for me, as well as just like peace of mind, all of those kinds of things where I could probably get more out of less if you may.
Steve: Um-hmm.
Pree: I also recently got back to this. As a kid I grew up in a – I grew up in a small little town back in India and I come from a farming community and I’ve learned – like a lot of the farming side of things very early in life and one of the things was bee-keeping and I keep bees and that’s something again like you know, backyard honey. I enjoy doing it. It’s a – maybe there is a pattern that goes across all of these too. Have – love old air-cooled Volkswagen Karmann Ghia. Again, the engineering side of it, you know fixing things, making it work again. Those kinds of things excite me. All of it, I think the pattern that goes across is it’s all intense stuff and I like the intensity. Intensity kind of calms me down, and that’s why I end up doing it. And you can see some of these things where I bring them back to work. You know if you think about bees and you think about how it works, the bees as a colony is a brain. A single bee by itself will not sting you if you meet the bee somewhere on the road. You know unless you go and try to squeeze it or something like that. It will never sting you. That’s the last thing it wants to do. But as a colony they work together and one of the ways you can get stung is if you try to steal their honey. And anytime they get a notion that you’re going to steal their honey it’s going to sting you. And they go after it together and there is a whole lot of science there. One bee stings, there’s a little bit of smell that it leaves and then every other bee wants to attack that particular thing. And so now again, simple rules and everybody executes. I bring some of those things back to work. I bring principles and principle-based decision making from cooking. Cooking is an interesting thing. I never cook with a recipe. I cannot cook with a recipe. But I cook with principles. There are certain things that go well together and those have been learned over hundreds and thousands of years. Different cultures have learned this many, many ways. I do read books about food. So principles at work for me come from principles in cooking. Cooking is an interesting thing because if you don’t cook with recipes you kind of borrow principles that help you cook well. It might be about like how you put the right amount of garlic and ginger in your food. Or it might be about like how you pair a certain kind of vegetable with a certain kind of meat. And it’s that connection and it sets – it’s a pattern that works well and if you – you know I read a good amount of books about food. I don’t read about recipes, I just read about food. And I love to read about the history of food as well, as in like how food changed over thousands of years. And those kinds of things intrigue me just because of the fact that there’s this knowledge that has been passed on generation to generation – not necessarily written down, but passed on and it’s a set of principles.
Steve: And so now I’m thinking about what you’re describing and my own inept attempts to cook. If you think about how much garlic or how much ginger to put in you can do it, but if you’re a novice, if you’re like me, the only way you know if you’ve done it right is to taste it, at which point it’s too late.
Pree: That’s right.
Steve: Which I think is similar to some of the work things we’re talking about, like how do you create something and how do you evaluate it. You know you sort of invest – you invest time or building effort…
Pree: You do, right, you do. And that’s also like – here’s an interesting thing as in like from a cooking standpoint. Like my Mom was explaining to me when to add salt to vegetables that were cooking. She had no idea about the science of it, but she was right. You know she was talking about like adding the salt just a little bit after it boils. Right when it boils add some salt and add a spoon of water so that it starts to boil again. That’s what she told me. And I started researching – this is the researcher in me. I started researching why are you saying this? What does this have to do? And then I read up about how the salt added at the right time either makes the vegetable mushy or the skin of the vegetable stays intact, the inside gets cooked. And so it’s a science. And some people have learned it by experience and are passing it on to generations via patterns. And that’s why it’s quite amazing. And architecture is another one like you might have heard of Christopher Alexander and the pattern languages. You know that side of architecture always amazes me. It’s interesting because there is patterns to be seen and patterns that have been happening for years. You know we took the pattern language in the software and a lot of the software now is based on this, you know. So it’s quite amazing.
Steve: You made this point early on about what’s an art vs. what’s a science. Maybe we can take some of what you’re describing with cooking and architecture. Can we talk about just field work? I mean we’ve talked a lot about the context in which research gets applied, but if you think about just doing – field work I think is maybe a rich place to look at just for a minute or so. I don’t know – you know can you draw – I’m going to just throw it back at you. You were sort of talking about cooking as a science but I think there’s an art aspect to that as well, architecture. And so how do some of those principles, do those things apply to field work as an activity in either an art way or a science way?
Pree: Oh absolutely, right. So – and so let’s go to one other passion which is parenting. And I struggled with this, which is being there. Being here now. You know being present. And the mobile phone, the computer, all the screens kind of distract me. And I’m sure a lot of people have the same thing where my kids are wanting my attention and I don’t give them as much attention. Now on the field side of things, right, and this comes to bee keeping. It comes with the mechanical work that I will do on my car. You know all of those things, the attention that you have to give is exactly the attention you have to give while you are interviewing somebody. You have to be there. And you have to feel like that’s the only thing in the world for you at this moment. Is it a skill? Is it learned? I don’t know. But that’s something that everybody – you know – and I know you’ve written a whole book about it. It’s crazy.
Steve: That’s great. That’s a wonderful thought. How old are your children?
Pree: 8 and 9 – you have kids too?
Steve: I do not have kids, no. What should we have talked about that we didn’t get to?
Pree: I think that’s an absolutely good question. That’s a question that everybody should be asking at the end of the interview, yeah.
Steve: That’s right, that’s my stock – it’s one of my stock ending questions – everybody’s stock ending question.
Pree: Let’s see. What should we have talked about? I think we covered a lot of things. I’d say I think we might discuss a little bit and I want to ask you your thoughts on this which is where is design research going long term? And I think a lot about is this field a temporary field that was right for a certain time period and will it just disappear? Or is this a field that will morph into new things. What do you think?
Steve: That’s what I was driving at a little bit when I ineptly compared research to expense reporting. I mean I’ve thought for years that there’s a potential for this to be an activity not a role and it’s interesting – I don’t know, it’s very exciting to hear how the activity of research has really, as you’re describing it in these ideal situations, it’s really grown. Like what we mean by research and what it entails and the skills and all that. It’s – there is a point, and you probably remember this from your early days, where, you know buying a video camera was like, that was sort of the step into doing research. Well we bought a video camera. And it was sort of a naïve time, but I love hearing you describe kind of the – I guess the maturity and sophistication and the capability that this set of activities has. But I have just been saying for years like is this just design? Or should it just be design? And I feel like – I don’t know if this is the same trend or an adjacent one. Part of that sort of questing that you and I and people in our industry have been doing, sort of evangelizing, advocating, hoping, wishing, wanting, has been to see research more fully integrated, more fully adopted and I think with that has come a couple of things. There’s sort of a good and bad and as you said there’s an ideal and you’ve laid out the vision for the ideal and you strive every day to achieve that ideal. I also think, and hopefully I’m not just being grumpy about it, but there’s a lot of commoditization of research that’s happened at the same time.
Pree: Oh absolutely.
Steve: Where there’s lots of junior people coming in. They’re not always getting the mentorship that you’re describing. They’re being asked to kind of execute closed end – not explorations. They’re barely doing evaluative. They’re swimming in the local maxima to mix metaphors. And that some people are kind of you know dusting their hands off going well we did it. We hired a researcher, we’ve got research being done. So…
Pree: It’s a checkbox, yeah.
Steve: Yes, yes. I guess that isn’t new. I guess I just see more versions of that where there was a time where research was ignored and now it’s sort of pulled in, kind of kept in the corner and not doing very much. And I feel like oh we wanted everyone to do research and now they are, except they’re not always doing – they’re not always kind of doing it as a leadership, strategic activity the way you guys are and so many of our peers are and so sometimes I wonder like well is this the consequence – is this the downside of us wanting this so much that it has been adopted but not in the way that we had hoped for.
Pree: Yeah. And I think you mentioned something very interesting which is this is an activity that everybody does or it’s just one of those activities. It’s not necessarily a discipline. It’s not a profession. It’s a very interesting thought because I have a sense that you know over the last few years, like it’s been 20-25 years that we have had kind of a new discipline come about which is the product management discipline. You know it used to be more program management before or it used to be more like putting – connecting all the dots, rather than owning the product. And I also wonder out aloud if we somehow connect with the product management discipline somehow or the design discipline even more so. And I don’t have any conclusions. It’s one of those things that kind of keeps me up and I think about like where is this going?
Steve: I’ve said this before on this podcast, it seems like research follows design or follows user experience and that I think some of the things we wring our hands about are things that that profession, that community, was wringing its hands about three years ago. And so the move – I hear UX people say all the time that product management is kind of, is the opportunity for UX to grow. Rosenfeld Media did a virtual conference recently that was like a PM plus UX event. So there’s these signals that these are the conversations that are happening. And so where’s the research side of – ‘cuz sometimes research is sort of like hidden under UX a little bit.
Pree: It is, but if you see some of the latest things that are happening, I feel like research is actually leading design and the reason I feel research is leading design is because of a few things. One is how connected we are with users? And how that gap between a user having a need to a solution that can be shipped is kind of reducing. As that happens you end up having a lot more of this need gathering. A lot more of this co-creation if you may of the end product. That is important. And we’ve always been at the leading edge of this. We’ve been pushing design from design being just inspiration and like let me make this up, to design that is like intentional. And we continue to push this discipline. So I would any day call it as leading the discipline rather than kind of following it. Following it maybe from a time standpoint, maybe. Yes I see where you are going with this, but yeah.
Steve: So that dynamic is interesting. And maybe design is maturing to not be decoration. I mean it’s been maturing that way for a long time and that’s – I’m just reflecting what you’re saying – that starts – as it matures to be sort of creating experiences, creating value, then the hook for design to do that is research which then goes back to your question, which we’re not going to be able to answer, which is should it just go away or will it just go away?
Pree: I don’t know. Like – another interesting development that’s happening with the design side of things is patterns. So over and over again now we are starting to see design patterns that are used multiple places. And so suddenly the value of having to design individual things go down. And it happened in software a few years ago. As in some of the very good software programmers know that the best thing to do is to have a collection of all of these things that they use over and over again and they just pull it in, cut and paste. And most of the code is not written. It’s just cut and pasted together. I’m not reducing the value of a person – of software here. You know I can code and I know it’s a lot of work – patterns and knowing when to pull those patterns together is the skill. So you mature the discipline to something else. And we are very good as a research discipline to figure out those kinds of things, the needs or what patterns to pull together. So I think we’re set for good success. That’s what I’m thinking.
Steve: I love that. Maybe we should leave it on that wonderful high note and thinking about the success we’ll continue to have as a discipline. So thanks very much, Pree. It was a wonderful conversation.
Pree: Absolutely. I enjoyed it. Thank you so much.
Steve: Thank you.
The post 12. Pree Kolari of eBay first appeared on Portigal Consulting.
This episode features Pree Kolari, the Senior Director of Design Strategy and Research at eBay. We talk about the career arc of a researcher, having impact on the product, and breaking down organizational walls.
The post 12. Pree Kolari of eBay first appeared on Portigal Consulting.
This episode features Pree Kolari, the Senior Director of Design Strategy and Research at eBay. We talk about the career arc of a researcher, having impact on the product, and breaking down organizational walls.
We can make it a little better in terms of customer experience where they buy with one less click. We can also change the dollar amount eBay makes by .1%. That’s a win and it’s very easy and it is very, very tempting for everybody on the team to go “Yes that’s great, let’s go do it!” But then you ask the why questions and go for deeper insights, not just the surface level insights. You go deeper. You find out that it’s a lot more and you will have small things that you could do on the product that will give you wins two years from now. – Pree Kolari
Show Links* Pree Kolari * Follow Pree on Twitter * eBay * eBay Classifieds * StubHub * eBay Motors * eBay Fashion * local maximum * stack rank * Moto X * The Design Sprint * Kinect * Zune * Metro * Karmann Ghia * A Pattern Language by Christopher Alexander * The Designer is Present * Product Management + User Experience
Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes.
TranscriptSteve Portigal: Thanks, Pree for being a guest on Dollars to Donuts.
Pree Kolari: Absolutely. Looking forward to talking.
Steve: Let’s do the basic intro part of the interview. Tell me something about you and your work and kind of why we’re here to talk.
Pree: Sure. I’m Pree Kolari and I lead the Design Research team at eBay. It’s actually Design Research and Strategy team that I lead at eBay. And we’re a pretty sizable team, about 20 people across the US and Europe. So we have people in San Jose, San Francisco, Portland, Germany and London. So pretty sizable team. We work across all of the products on eBay. So we work within of – we work with the product teams and work on both new products as well as existing products.
Steve: So if I’m not an industry insider and I’m listening to this and I think eBay – eBay has multiple products? Like can you explain you know eBay a little bit for those of us that aren’t in the know.
Pree: Sure, absolutely. So eBay is interesting because you have a couple of different kinds of users. One is the seller and then another one is a buyer. So both of them see different sides of eBay. And they both interact with eBay via desktop, mobile, you know multiple devices as well. So every one of those interaction spaces are product areas from our standpoint. And so that’s one. Then eBay as a company plays in many, many different spaces. One is eBay the marketplace itself. Then there is a bunch of other places which is Classified, StubHub and all of those things. Those are separate teams by itself and generally speaking we don’t necessarily work across as much as we should, but we do. So that’s – does that that give you…
Steve: So where does something like eBay Motors fit into that?
Pree: Absolutely.
Steve: Is that like a StubHub?
Pree: It is – no, no, no eBay Motors is part of our team. Those are all things that are considered verticals. Good point. So within of – the shopping side of things – there are certain areas that have specific verticals and eBay Motors, or Fashion, you know those are the spaces that are a part of our team.
Steve: If you’re an eBay shopper do you experience those things – if you just go in and search do you know when you move between the verticals?
Pree: Ideally if we did our job well from a design perspective, as well as from a research perspective, you shouldn’t notice it. But sometimes we do, you know because it is two different teams that are delivering the same product, right, and so that’s one of the big challenges of doing good research because as a researcher you’re thinking end to end. You’re thinking the customer journey. You’re coming at it from a user standpoint and users don’t see your org structure. They see the product. The moment the users see our org structure we have a problem and that’s exactly what we shouldn’t be doing.
Steve: So it sounds like the experience today for someone, there’s lots of points where you wouldn’t see the structure and some points where you would.
Pree: Absolutely, absolutely. Lots of places where you shouldn’t see it, but some places you see it and that’s kind of why a bunch of us have jobs.
Steve: To try to…
Pree: Try to fix it. Yeah? Yeah, it is about fixing. Some of it is about fixing. Some of it is about like coming up with new things that will make the older behavior that we have started kind of become back – you know take a back seat.
Steve: It makes me think about what – you know how research fits in, which I think we could probably spend a long time talking about – I mean I think sort of the counterexample is always the – you know there’s a question that comes from a certain team and they’re really only interested in the answers that affect their decisions within that.
Pree: That’s right.
Steve: Whereas, as you say, as a researcher you’re looking kind of end to end.
Pree: Yes.
Steve: So the question is you know what – you know how does the work that you do – how do you structure it to touch different teams at different points so that those decision can be made.
Pree: It’s a good question. So one of the things that our team, the whole team does is talks to our counterparts, our colleagues or people who are part of our core teams. Yeah? And the main point there is understanding the goals of what we are trying to achieve. The secondary part there is to train people not to look for local maxima. We have – we have a habit – and this is just a bad habit that’s come over time, of looking for small wins, local maximum. Instead, how can we focus on long term thinking? So there is a little bit of bringing people along that happens. And during that time the framework that we end up using is just a simple one of generative, iterative, evaluative. So here’s the different kinds of spaces that we can look at research. The first part is generative – that’s about generating ideas. That’s about diverging in from an ideation standpoint. Next one is iterative – this is where you’re continuously testing things out to learn quickly and make this product much, much better. And then there is evaluative. Now we’re pretty close to ship. How do we make sure that we ship something really amazing? And so that’s the three step kind of a thing. If you look at a waterfall process, you look at an agile process, pretty much those things fit right in. Yeah.
Steve: Can you say a little more about local maxima? That’s a great term that comes from say like some kind of number theory something.
Pree: It’s a statistics kind of a – it comes from statistics. It’s used a lot in the stock broker side of things, right, and because you’re always – there are people who are there for the short term, they just are day traders that just made money. They want to sell it off and then they want to move on to the next one. So if you look at today’s stock you probably see an up or a down and then the up could be a good up and it could be the maximum up that the stock has ever gone, or you could step back a lot and see that the trend has been going up for years – 25 years or 30 years. And then you suddenly think, okay you know what if I hold on to the stock it’s going to do very well. So now you look into a product or a research side of things. You start to see that we’ve made – we’ve taken a certain path. Let’s say with eBay we’ve taken a certain path with respect to how we make a buyer go through the categories, go through the shopping experience. And we can fix it a little bit. We can make it a little better in terms of customer experience where they feel like hey you know I got to buy with one less click. We can also change the dollar amount by which eBay makes out of this about .1%. Now that’s a win and it’s very easy and it is very, very tempting for everybody on the team, including us researchers, to go yes that’s great, let’s go do it. But then when you ask the why questions and then you ask – you go for deeper insights, not just the surface level insights – you can see patterns, but you have to see the insights. You go deeper. You start to find out that it’s a lot more and you will have small things that you could do on the product that is – that will give you wins two years from now. That’s what I mean by local maxima.
Steve: So I’m thinking about – we won’t get into any specifics, we’re talking about this very generally – I think about things like you might have other kinds of maxima like how often they’ll come back?
Pree: Right.
Steve: But then you might have sort of larger issues like the meaning. Like if you want to change this site – and I’m really just making this up – you want to change this from a transactional partner to something that becomes a support system.
Pree: That’s right.
Steve: Well now that’s about longer – that’s about shifting the overall meaning through a lot of design choices and you have to measure that very differently than did the amount in the cart go up. Is this what we’re talking about?
Pree: You’re absolutely right. It’s not just looking at a financial benefit that you can quickly calculate now. It’s about looking – it’s about taking some amount of risk which are long term financial gains because it will come back, you know, and the team will start to have an intuition of the fact that it will come back. And there is a certain – you know we’re a profit making company, right, and at the end of the day we want to make profit every day, yeah. And so it’s not about just putting – taking all of our eggs and putting it for long term wins. It’s about being very intention and saying hey you know here’s a few things that we’re going to do where it is high risk/high reward – here’s a few things we’re going to do which is absolutely beneficial and we can see the benefits right away.
Steve: You’re describing a strategy.
Pree: Basically.
Steve: Or a road map.
Pree: That’s right.
Steve: Like here’s where we want to go with this.
Pree: I am. I am describing a road map and a strategy.
Steve: And you said right at the top that strategy is in the title of your group and you’re talking about research as a strategic tool…
Pree: Absolutely, right
Steve: …very much.
Pree: Right, right, right. So as in people on the team come from different kinds of backgrounds. There are people who have no research degrees, if you may. There are people with MBA degrees, right, and that’s intention. When you put these kinds of – different kinds of people together there is – there is a radical collaboration if you may. And that kind of collaboration happens because people are coming at it from different mindsets, but they respect each other’s mindsets. Yeah. And when you start seeing that happen other people also join in and I want the engineers, I want the designers, I want the product managers joining in and pitching in and making it their own idea. And that’s how you take it to ship.
Steve: So in this – this maxima example I think is a really great one. We’re sort of avoiding the pitfall of a local maxima I guess maybe is the theme here. What kinds of conversations are you having with people and how are you helping them – I’m assuming you are helping them to see that as a choice or a different path.
Pree: Yeah, good question. So you have – there is – this is where it’s a little bit of an art more than a science. And it’s a little bit of wisdom more than here’s 5 things you can do. What I look at is I think of – you know I don’t do as much – personally don’t do as much research as I used to do out in the field. I do very, very little. But what I end up doing is I end up doing research on – in the corporate side of things if you may. It’s about understanding the organizational psychology. It’s about understanding what are the needs, what are the pain points, what are the motivations of these people who are working inside of an organization. And once you understand that and you map it you can think about like why is this person thinking this way and you can still figure out that there is a motivation that works with what you’re trying to communicate and it becomes very easy then. And so I am always looking for win/win situations. It’s a negotiation at the end of the day too. So it’s a win/ win situation. Let’s figure this out and make it happen. So I don’t think I have a specific answer there, but more of a generic answer.
Steve: You make me think about my own work. I’m someone that works outside the organization. So I come in and the best engagements for me are ones where I’m partnering with somebody who has that insider knowledge and has sort of worked over time to try to find out how to be successful so that I’m not blundering it and saying I know the best way to do it. It always has to be adapted to the context. And there’s often this interesting tension – maybe tension in a good way – when you come in – and you’re newer in this role so you probably have a perspective on this. When you come in you see everything that’s wrong that needs to be changed. And then when you spend time you choose your battles. You figure out your strategy. What I see, because I’m always in the newer side of things, is you also get – I don’t want to say co-opted, it’s too strong a word – but you become part of the system. And so I have these situations where I feel a deep passion, like this is what these people need and people that are my gatekeepers are like yeah there’s maybe not an appetite for that. And it’s not to say that we then fall flat, it’s just – it’s where we put our energy. So I love what – I mean I think it’s – I love what you’re able to do. You have to be embedded. You have to build relationships. You have to kind of go over time. Without being kind of confrontational to you, do you think, have you seen it in yourself, that you’re now part of the system – you can in to sort of drive change, but then you become part of the system. What does that arc look like?
Pree: Very good point. So I’ve done like consulting work like you have for a long time. And then I moved into the corporate side of things – 2008. So been doing this for 8 years. Big change. Big difference. The advantages and good things that corporate or the consulting side comes with is they come with a fresh perspective. And you come – like when you come and work with us you come with a fresh perspective that we never would have thought of. You also come with an expertise. Yeah. So the hierarchy that has already been established within the corporation doesn’t apply to this consultant who has just come in. And so that’s an awesome thing. Yeah. However, these – the guys who come in also do not know the culture of the organization, or do not know the history of the organization. Now within the corporate structure there is researchers inside the corporate who are embedded and they work really, really hard to build the relationships to make sure that they are doing the right thing. And sometimes yes, getting embedded for too long will make them feel like hey you know I’m part of it and I cannot make any changes. But we end up, we end up really working hard not to let that happen and to make that we go back to – like within our team we have a – we build together the team with a set of principles and these are principles that everybody in the team worked together and we just kind of articulated. One biggest principle that we have is about impact and you know we write down like hey what does impact mean? The other one is about transparency. And it’s about transparency with respect to communication. Yeah. And as soon as we end up being extremely transparent we break down walls that otherwise exist within the corporation. Yeah. And it’s part of our jobs to break those walls. Yeah. And we – within my team we are not measured by how many reports you have written or whether you have finished so many projects. You are measured by the impact on the product. Yeah. So it is super important to work with the product manager. It’s super important to work with the business person. It’s super important to work with the designer to make sure that the impact is clear. And that’s when it changes our own behavior a little bit to not conform to the culture in the way you described it. Does that make sense?
Steve: It does. I feel a little skeptical and that’s not a skepticism towards you. It’s just in general sort of having observed this because in order to be successful there’s a certain amount of conforming that has to happen.
Pree: Absolutely.
Steve: It doesn’t mean that – you know what I mean, you have to kind of conform to figure out how to choose the battles.
Pree: Yup.
Steve: And there’s battles that you’re not going to choose now because you’ve found ways to be successful. And think about yourself in 2007 – I’m going to guess, you would come in and look at yourself doing this job and like shine the flashlight in some corner and say like guys you’re denying this. And now you, 2016 you is kind of saying like well no you know we have these principles and we’ve had impact and we have this track record and like we have lots and lots of success and then – you know I’m doing a little puppet show here – 2007 you is kind of saying like no, no, no. Like over here, this thing. That’s the tension that I think is…
Pree: Absolutely. So I agree with you. So if I look back in the 2008 me I would look at the 2016 me and say hey look, you’re not listening to me. You’re not listening to this part of the equation. And that’s the reason why I think – you know it’s learning, right, from a life perspective. Absolutely, so I prioritize certain things to make an impact on? I do. And everybody on my team does. And we do it because we believe in the fact that making impact is more important than making impact on all the 10 things. So let’s imagine there is a study where we saw 10 different things that are very important things to fix, okay – an evaluative study. We might choose to pick only two of the top things and some of it might be confrontational, but we might pick only two of the top things because we know that if we pick those two and go really, really hard it will help us instead of diluting our impact with going with 10 different things. Yeah.
Steve: Um-hmm.
Pree: And that’s the choices we end up making, but the choices are super important because it’s exactly based on impact. Doing the right research means talking about all the ten because those are all observed and those were patterns that were observed. At the end of the day we are not doing research, we’re doing applied research, you know, and we’re working within a product company that ships products. You cannot ship a research report. You can ship a product. So that’s what we’re going after.
Steve: Yeah, that picking the two of the ten I think is really…
Pree: It’s an art. It’s hard.
Steve: …really interesting. And it’s not like you haven’t learned those other 8 or haven’t articulated them. I don’t know do those float around kind of in the ecosystem somewhat?
Pree: They do, right. As in they might come back again and this is where – so I joined the eBay team only less than a year ago actually and our team is quite amazing. Like have been doing amazing work for quite a few years, since ’99. The research team has been there and have been doing like quite, quite a lot of good work. There were challenges in how this work was heard within the organization. There were challenges with respect to resonance on some of the outputs. And so we had to stack rank everything. So this is the engineer from me talking here. I’m all about stack ranking, drawing a cut line and saying hey, here’s the few things we’re going to go after and we’re going to go after really hard.
Steve: Is that a fancier way of prioritizing?
Pree: Basically.
Steve: Was that what you’re saying or is there like a methodology?
Pree: It is prioritizing.
Steve: Okay. That’s a great phrase though.
Pree: Yeah – stack ranking?
Steve: Yeah.
Pree: Yeah, it is. It’s probably engineering talk, but yes. I would rather you know – because at the end of the day we are – again, concentrating on shipping, yeah. And so again I think that those are things that are coming from my background as an engineer. As a person who has like shipped multiple products. As an engineer, as a start-up person. As like – you know – coming back to if we did amazing research and there isn’t a follow through afterwards the amazing research just sat on shelf and nobody benefited from it. Because I know that in my heart, and I know that a lot of people who do this, we’re doing it not to just do research. We do it because we enjoy seeing the output of this which is the experiences that have changed. And that’s what gives us satisfaction and that’s the long term view.
Steve: Even just the idea of sort of the output of – these are significant efforts. A “research project”, whatever it looks like, it’s a significant effort, right?
Pree: Right.
Steve: And that even to get what you learned down to 10 things or 10 big ideas or 10 opportunities that takes a lot of pain. You have to leave a lot of things on the floor and synthesize and organize. And then – I’m just sort of toying with this idea you’re describing that says okay team we’ve kind of got these 10 things, let’s go at them with two. It’s very different than – and I’m sort of thinking about the consultant cycle which is, you know, it’s hey the consultants are back, here’s their thing, it’s a significant effort. It sort of ends the engagement. We have to impart as much as possible because we’re going to leave. So to get it down to 10 is really hard, but it’s like I think about in certain circumstances like what would happen if we said okay there’s two things. Yes we have a lovely Executive Summary that says like here it is on one slide, but just to really shout loud about two things.
Pree: Sure. Actually let me give you an example, okay, which is from my previous work at Motorola. So here’s the interesting thing that happened. So Google had just acquired Motorola and that’s when I joined Motorola as part of the Google, when they acquired, around that time. And one of our – one of the challenges that we got from – during that time it was like a bunch of the Google executives who joined Motorola and it was clear there was a mandate, said let’s reduce the number of phones that we’re making to very few and whatever we are making let’s make something amazing. So we did a bunch of ethnographies where we saw people taking pictures. That’s what we were looking after. We were looking at images and people taking pictures. And one thing we observed over and over again was how hard it was for people to quickly capture an image. And what all things people were doing. And this is where – you know you’ve seen this in the field. Mom holding a kid and trying to take a picture and a selfie – there’s a diaper bag in another hand. It is, it is tough, And how do you allow for that to happen? And we could have written reports and reports about this, but we went in with one phrase. We said, you know, pocket to picture in 4 seconds and one hand. That was our output from a research study that took 2 months if you may.
Steve: And you learned, just to – you learned lots of things…
Pree: We learned lots of things.
Steve: Tons of improvements that could be made…
Pree: Absolutely.
Steve: Lots of opportunities for fixing, creating new.
Pree: Absolutely. And when we know that we want to get from a pocket to picture this much time, okay, and one handed. Now you suddenly, suddenly have, have something that the whole organization can rally around. There is a purpose. It’s a measurable goal. And what happened was with the MotoX we had a gyro – every phone has, you know has, has a sensor in it. We put in a low power sensor as well and people could just twist twice and with one hand take a picture. We even changed the UI. In any big company you could think about like how difficult it is to remove the interface. And we removed the interface to make the whole screen a tough target and you could take a picture. And suddenly the user experience completely changed. And so could we have gone with like hey the pictures that were taken in a dark bar were too dark? Sure. We had a whole bunch of things, right, but we didn’t. We made a choice and we made multiple choices afterwards and so the quality of the picture came up after and we worked very hard to change the quality of the picture and we have specific things about how you can change tones based on the picture, like a skin tone vs. food vs. the sky and make the perception of that picture to be higher quality. And so what we’re talking about is very data driven design. And so when you have data and you’re very specific about how we can use the data to change the design it always works.
Steve: And I think it’s interesting that you research output, if you will, was sort of a vision statement…
Pree: That’s right.
Steve: …you know for a design brief. And that’s different than personas or journey maps. It’s not about the format of the output. It’s not even “Here’s what we learned about people.” You’re kind taking it all the way to say that we did this research and it tells us that this is what the product needs to deliver.
Pree: Yeah. It’s an objective. It’s, yeah it’s an objective.
Steve: It seems like there’s something interesting about how teams are structured, or how these programs kind of run, that lets a research project output a product decision. As opposed to researchers learned this, they talked to designers, they ran a session, the designers decided – you’re sort of putting it all under the same umbrella. We did this activity, the thing that we learned is the product should do “x”.
Pree: Yup.
Steve: So how do you structure – how do you bring the right players together and sort of have the whole endeavor such that that can be the output?
Pree: Everybody who’s on the team is assigned to a product area. So they have a core team that they work with and it works very well in an agile kind of an environment in eBay because otherwise you’re a service organization within – of a big company. We’re not a service organization within a company. We are a product organization. So what that means is let’s imagine a person on the shopping experience. They’re part of the shopping experience team and are embedded within the team and they work with the team everyday on daily stand-ups. Yeah. And they figure out, they have – they maintain a roadmap of what research has to be done. They bring up things within the team and talk about what are hypothesis, what are myths, what are things that we know, what are data, what are insights that we know, that we really need to go after. And so all of those things are super important coming from the researcher’s standpoint.
Then the next step is about participation within the research. So I’m a firm believer that research isn’t done only by researchers. Everybody is a researcher. It’s like everybody is an artist. Like you know we all have this innate ability to observe and make conclusions. That’s part of what we do. Some people just have been trained in this and are probably much better at the structure side of it. Now you have –when you have researchers who are catalysts within the core group, to understand more about the user so that we can make products that are absolutely relevant to the need of the user. The need may be even unknown. You know it might be a blind spot. They might not even know that they need it. Fine. But building that sensitivity for the whole team is part of the researcher’s job and then it is also part of the researcher’s job to collaborate, to get to an output. When I gave you example of phrase that came in the end it is not the researcher who comes up with that kind of a thing. It happens because the designer, the engineer, the PM is all part of that discussion. They own it. They co-own it. And that’s what – that’s when you get successful products.
Steve: That’s a good explanation of sort of how you set up those conditions to make those kinds of big impact. I mean it’s back to your point of impact earlier on. If you can come up with that phrase as your output you’re setting yourself up for impact in a really…
Pree: Absolutely.
Steve: …massive way.
Pree: And that goes a little bit – you know with research we end up – we end up thinking a lot about outputs from only one research report or one research study. That’s I think history. I really believe that it’s about building criteria, putting together principals that kind of collect all the knowledge that we know from multiple studies. And that’s part of the job of an in-house researcher, right, which is different from somebody who is coming at it from a consulting standpoint. And what that does is it helps make good decisions throughout the product development process. We’re making – you know I really believe that the intuitive decision making is bullshit. It’s either data driven decision making or principle based decision making. And so we have to give the ability for people to make the right decisions and that’s – you know it’s getting the data and putting together principles that help us make the decisions. Does that answer your question?
Steve: Yeah. So to try to tie together a couple of things that you’ve mentioned. This idea of everyone is a researcher and then different decision making styles, I’m curious about, you know there’s people that are researchers for a reason. They have kind of an approach. Can you compare and contrast maybe the mindset of someone that has researcher as their title versus. these other people on the team that they’re working with?
Pree: Sure. So I think one of the big things as a researcher, you know we are curious. And the curiosity is what makes a good researcher. The hunger – it’s a sponge. It’s like wanting to learn. And we’re paid to be unbiased. Versus let’s talk about a product manager or let’s talk about a designer. They’re paid to be opinionated. And having opinions and having ideas of where they want to go is what they’re paid for. And now there is a tension. And both people are coming at it from different angles. Both want to understand what the users want, yeah, because we want to finally make a product for our users, but are coming at it from two different angles. And so that tension is what I think is the root of the issue and then like figuring out how to manage that tension – it’s never going to go away. It’s managing it and directing it in the right direction.
Steve: And I think that tension – as you described sort of how that team, your team, came up with – now I can’t get it right – what was the – from pocket to photo in…
Pree: It doesn’t matter.
Steve: …four seconds. Whatever. So that is a product of those different perspectives, right.
Pree: Right.
Steve: So I’m going to just pick on you slightly. I mean you used this phrase which we hear a lot when we talk about research which is like what users want and that’s sort of shorthand, but I think in the examples that you’ve given I think you’re really saying that research is identifying something else. And maybe this is just me being semantically stubborn or something, but no one asked you – and we know this, right – no one said hey I wish I could do this with my phone, right.
Pree: Yeah.
Steve: When you say what people want it implies that the researcher is taking requests, but what you’re describing is something much – it’s that hunger and that curiosity and that – and that synthesis to pull those principles out.
Pree: Absolutely, as in we’ve been doing this kind of work for a long time and we always go for these unarticulated needs. What people say, you know, is very easy to get to. And a lot of times what people say is like they’re lying, you know, and they’re saying what you want to hear. It’s getting to the deeper insights by looking at patterns, looking at behavior, looking at what people are making. And so I go back to Liz Sanders who is a mentor from a long time ago. What people say, what people do and what people make. You start to see that each of these areas give you different kinds of data and triangulating between that data takes a lot of synthesis, takes a lot of analysis. And then you get to the core of it and articulate what it is that our users want. And the point that I think is most important and is lost is it is not two researchers going off for two months, coming back with this magic. This analysis doesn’t necessarily happen in a closed room. The right way, the new way to do it is the analysis happens in an open space, happens with others. It is co-created so that at the end of the day our products are – the objectives for our product is co-owned by everybody building the product.
Steve: So let me ask somewhat of a more tactical question about that. You know you’ve talked about some people – some people are paid to have opinions.
Pree: That’s right.
Steve: And so how do you help people be effective in synthesis when what they’re coming in with is what they’ve been working on, what their vision is, what their intuition is, whatever that is. How do you help them sort of embrace the messiness of the data and be in that synthesis process and – I don’t know. I’ll just stop.
Pree: So some of it – so we’ve been doing a lot of what we call iteration zero sprints. We just started doing a lot of those things. Basically it’s design sprints you know. And Google’s done a good job of writing a little bit about the design sprints. You must have seen some of the design sprints work that Google Ventures has put out. A good amount of work which is really good and – and that’s a very tactical way to get a bunch of people together and not necessarily throwing them right into the middle of swimming pool, but letting them wade in the kiddie pool. Now what happens is everybody gets a sense of what the water is like. How cold the temperature is? What it is like to float a little bit? That kind of a thing. The researcher does most of the heavy lifting, or a bunch of researchers do most of the heavy lifting. Then when we all come together towards the patterns everybody is on board. Otherwise you know this is a challenge that our industry has faced for quite awhile, right, which is trying to explain qualitative vs. quantitative research as well, you know. And we do a good amount of both kinds of research within our team and I see a lot of it where – and I’ve seen this in multiple companies where time it’s like 10 users, even though those 10 users were observed for, each of them observed for a few days maybe. Oh, it’s only 10 people. So do you think we can put a million dollars on this or not? And that question always comes up. And so that’s where getting people really connected with the stories, the real people, is super important. Storytelling is a very important part of design and make good design decision. So it comes back to that part of design that we plug into research.
Steve: So how do you answer that question? Are these stories sufficient to commit an investment? What’s the response?
Pree: It usually ends up being yes. So there’s two things that help quite a bit. If you look at venture capital money going into startups, or you look at like companies investing into new products, there is one which is very good data that’s driving certain directions. Second, conviction. When the team sees certain things and they believe in it there is a certain level of conviction. And you cannot necessarily explain it in words, but you can see it. Us as researchers, what we do is we change that part for the whole team. And that’s the value-add that we bring in. It’s quite amazing.
Steve: I feel like yes and no. I mean yes I know what that feeling is like and I know what that conviction feels like and I know when you just – you know that here’s the opportunity because you’ve been out, you’ve been with these people…
Pree: That’s right.
Steve: …you’ve experienced their stories. You’ve breathed the same air as them.
Pree: How do you then translate it to conviction for everybody else?
Steve: Yeah.
Pree: You know as in like – as a consultant I’ve done immersion rooms. I’ve done videos, video clips and like it’s not enough. And all of those things help. Now what we’re doing a lot more of is we’re actually taking people into the field. As part of our sprints we have users – we have two sets of interactions. We have interactions in the beginning of the sprint with the users. We have interactions at the end of the sprint with users. And through that interactions, first interaction is about understanding the ecosystem, the domain, the landscape. And the next interaction is about evaluation if you may. It’s more concept evaluation. And so everybody has been part of this journey and it’s not just theater. It is actually immersing yourself in the people’s lives.
Steve: I think you’re describing a really nice version of this – we’ve been headed towards sort of everyone being a researcher, research is integrated, everyone participates. And I think there’s a – we keep talking about tensions. There’s an interesting tension there where this thing works best when everybody plays as deeply and richly as possible and yet it’s not – it’s a profession not a practice. What am I trying to say? There’s a better way to say that. You know filling out expense reports is not a job function at a corporation. It’s an activity that everyone that works for that company does. Research is a job. There’s leaders that have that function. I mean I wonder if this is a point and a transition where – because you’re describing something that’s just really fully integrated at its ultimate version. But they need us to facilitate it, to kind of guide it.
Pree: Yeah, and I’m describing an ideal situation. As in it’s called work for a reason. As in it’s not all like easy and simple. We’re learning and we’re making it happen every day. So it is not – it’s not that everything is like done and like flower petals are put in front of us for us to come there. It is definitely a struggle. At times – like and different teams are different. There are day to day stresses and tensions and those are places where again if we believe in where we want to go as a team I think it makes it much easier because again it comes back to our talk about the local maxima as well. You know even within our profession.
Steve: Could we switch gears a little bit?
Pree: Absolutely.
Steve: Talk about your choice to join this group and take on this role. You’re describing the reality and the reality is of course everything is a mix of wonderful and challenging and that’s not – I don’t think that’s unique to your situation, but something was compelling to you about this opportunity and can you talk about what you were looking for in…
Pree: Absolutely –So eBay – so my background is engineering and design. Have worked in design research, innovation consulting for a good while. Then moved into Microsoft, was part of the small team that invented Kinect and also was part of the team that did Zune – all that kind of work, like some of the initial Metro designs. So moved to Motorola afterwards to lead the design research team right when Motorola was changing quite a bit. You know Motorola had gone down in terms of how it was doing and there was only one way to go once you go down that much. It’s like up, or disappear. And my personality overall is more – I guess I’m a betting man. I take risks in spaces – calculated risk of course, but I like that excitement. And so I joined. After a few years some of the work was nicely done already. There was things that were in place. So personally I was looking for other possibilities and this thing came along and I again looked at what was happening. eBay as a company is a 20 year old company, has established itself as a leader within of this marketplaces. Makes a lot of money by doing this. However if it continues to do the exact same thing that it did it probably doesn’t have as much of a growth trajectory. And so how – so overall there is a challenge for the company to reinvent, to think about new possibilities. And for an outsider, for me I was like looking at it and I was thinking hey, you know, that sounds like an exciting opportunity. And a place where somebody like me can make impact. The team was amazingly good. The culture was very good. So I felt like hey you know I could work in this culture. And it felt exciting to move at a time when eBay was just celebrating its 20th anniversary, just split from PayPal as eBay by itself and there is a future that we all can create together. Does that answer your question?
Steve: Yes. So if I’m a researcher and I’m looking at different organizations, because you’ve mentioned a number of different orgs that you’ve kind of come into, you know what are things that you would advise me to look for to try to assess – I mean a good…
Pree: A good fit.
Steve: …a good fit for researchers I think is different than other kinds of professions.
Pree: Yeah.
Steve: You know what do we look for?
Pree: I think – it’s a very good question and I haven’t thought much about it to be honest. You have researchers at various levels. So there is an entry level researcher who’s done a good amount of academic work let’s say, you know, or has gone through some amount of training. And then there is a researcher with like mid – you know two, three years of experience and then there is the researcher with about ten, fifteen years of experience. Or plus, like much more experience. And then there’s also researchers at various kinds of levels. There are researchers who are IC, individual contributor kind of researchers. And then there are managerial kind of researchers. Different places I think offer different things. So there are certain places – like if you look at the Googles and the Facebooks of the world, if you may, there is a lot of opportunities to learn. And you can have a researcher who is just starting out, a younger researcher who is wanting to learn a lot of things. It’s an awesome opportunity. Even companies like – the bigger companies are really good places for you to learn. eBay is one of those. A new person comes in who has no experience and we’ll train them and they get to observe. They get to be part of a pretty big organization and everybody in the organization comes with a different skill and they all are willing to teach. And they’re willing to learn from this new person as well, you know, and so that automatically makes that kind of an environment better. I would say for a mid-level kind of a person it might be valuable to go to a smaller company, a start-uppy kind of a company where you are probably the only person. You can build out the team. You can figure out what – you know build a network, build a network of mentors, build a network of people that have done this before who can become your personal advisors if you may. And then you can go next step. And as you get more senior there’s only a few places that will hire you as well. Because you’re too expensive for – you know it’s just – or it’s where – or you move into different areas. I know of researchers who moved into product management. I know researchers who moved into design. And you move into lateral areas and grow that way. And all of it is learning and it’s something that as researchers we enjoy.
Steve: So maybe a related question, when you look at people that might join whatever team you’ve been on over the years, what are some kinds of things that you see that kind of – that make your antennae go up. Like what are some good, strong signals that say there’s something here with this person?
Pree: Sure. I think it comes, comes down to passion and you can – again, like when you see people really – and I’d say passion translates into several things. It translates into a real hunger for learning. It translates into caring about the product and the user. And you can see the decisions being made that go towards caring about the product and the user. And I’ve noticed that researchers at a certain point they been taught research so much that they sometimes think of research I’d say in a little bit too pure kind of a way and aren’t thinking about it as a tool, a means to an end. And I see some researchers who have gone past to where they have learnt the tools to cause damage. When I say cause damage as in like they start to think about the product. They start to think about…
Steve: The good kind of damage.
Pree: …the good kind of damage.
Steve: It’s back to your impact point, yes.
Pree: Yeah. So that’s what attracts me to like okay saying hey here’s some people that have potential because they care and they care about the product. They care about the user and they’re very passionate about it.
Steve: That’s great. Alright, so just picking up on this notion of passion, we’ve talked, as of course it’s the focus here, we’ve talked about your work and a little bit about your background and kind of how you got to where you are. I’m wondering about other things that you have passions for that reveal something about how you – what we’ve talked about today? Are there any kind of connections or cross pollinations between research, it makes you do these other things, or these other things kind of impact how you think about research?
Pree: Oh yeah, yeah. No, I’m – I have too many interests as in I’m a Dad. I enjoy being a Dad. I am a bicyclist. I bike. I bike commute to work, plus I bike on the weekends anytime I get. I love cooking. That’s kind of a place where I can calm down. Design, generally speaking like design, designed objects. Well-designed objects, well – you know even furniture making, those kinds of things. I’ve been doing this for years and I do a little bit whenever I get time. Life hacking – it’s just more about productivity for me, as well as just like peace of mind, all of those kinds of things where I could probably get more out of less if you may.
Steve: Um-hmm.
Pree: I also recently got back to this. As a kid I grew up in a – I grew up in a small little town back in India and I come from a farming community and I’ve learned – like a lot of the farming side of things very early in life and one of the things was bee-keeping and I keep bees and that’s something again like you know, backyard honey. I enjoy doing it. It’s a – maybe there is a pattern that goes across all of these too. Have – love old air-cooled Volkswagen Karmann Ghia. Again, the engineering side of it, you know fixing things, making it work again. Those kinds of things excite me. All of it, I think the pattern that goes across is it’s all intense stuff and I like the intensity. Intensity kind of calms me down, and that’s why I end up doing it. And you can see some of these things where I bring them back to work. You know if you think about bees and you think about how it works, the bees as a colony is a brain. A single bee by itself will not sting you if you meet the bee somewhere on the road. You know unless you go and try to squeeze it or something like that. It will never sting you. That’s the last thing it wants to do. But as a colony they work together and one of the ways you can get stung is if you try to steal their honey. And anytime they get a notion that you’re going to steal their honey it’s going to sting you. And they go after it together and there is a whole lot of science there. One bee stings, there’s a little bit of smell that it leaves and then every other bee wants to attack that particular thing. And so now again, simple rules and everybody executes. I bring some of those things back to work. I bring principles and principle-based decision making from cooking. Cooking is an interesting thing. I never cook with a recipe. I cannot cook with a recipe. But I cook with principles. There are certain things that go well together and those have been learned over hundreds and thousands of years. Different cultures have learned this many, many ways. I do read books about food. So principles at work for me come from principles in cooking. Cooking is an interesting thing because if you don’t cook with recipes you kind of borrow principles that help you cook well. It might be about like how you put the right amount of garlic and ginger in your food. Or it might be about like how you pair a certain kind of vegetable with a certain kind of meat. And it’s that connection and it sets – it’s a pattern that works well and if you – you know I read a good amount of books about food. I don’t read about recipes, I just read about food. And I love to read about the history of food as well, as in like how food changed over thousands of years. And those kinds of things intrigue me just because of the fact that there’s this knowledge that has been passed on generation to generation – not necessarily written down, but passed on and it’s a set of principles.
Steve: And so now I’m thinking about what you’re describing and my own inept attempts to cook. If you think about how much garlic or how much ginger to put in you can do it, but if you’re a novice, if you’re like me, the only way you know if you’ve done it right is to taste it, at which point it’s too late.
Pree: That’s right.
Steve: Which I think is similar to some of the work things we’re talking about, like how do you create something and how do you evaluate it. You know you sort of invest – you invest time or building effort…
Pree: You do, right, you do. And that’s also like – here’s an interesting thing as in like from a cooking standpoint. Like my Mom was explaining to me when to add salt to vegetables that were cooking. She had no idea about the science of it, but she was right. You know she was talking about like adding the salt just a little bit after it boils. Right when it boils add some salt and add a spoon of water so that it starts to boil again. That’s what she told me. And I started researching – this is the researcher in me. I started researching why are you saying this? What does this have to do? And then I read up about how the salt added at the right time either makes the vegetable mushy or the skin of the vegetable stays intact, the inside gets cooked. And so it’s a science. And some people have learned it by experience and are passing it on to generations via patterns. And that’s why it’s quite amazing. And architecture is another one like you might have heard of Christopher Alexander and the pattern languages. You know that side of architecture always amazes me. It’s interesting because there is patterns to be seen and patterns that have been happening for years. You know we took the pattern language in the software and a lot of the software now is based on this, you know. So it’s quite amazing.
Steve: You made this point early on about what’s an art vs. what’s a science. Maybe we can take some of what you’re describing with cooking and architecture. Can we talk about just field work? I mean we’ve talked a lot about the context in which research gets applied, but if you think about just doing – field work I think is maybe a rich place to look at just for a minute or so. I don’t know – you know can you draw – I’m going to just throw it back at you. You were sort of talking about cooking as a science but I think there’s an art aspect to that as well, architecture. And so how do some of those principles, do those things apply to field work as an activity in either an art way or a science way?
Pree: Oh absolutely, right. So – and so let’s go to one other passion which is parenting. And I struggled with this, which is being there. Being here now. You know being present. And the mobile phone, the computer, all the screens kind of distract me. And I’m sure a lot of people have the same thing where my kids are wanting my attention and I don’t give them as much attention. Now on the field side of things, right, and this comes to bee keeping. It comes with the mechanical work that I will do on my car. You know all of those things, the attention that you have to give is exactly the attention you have to give while you are interviewing somebody. You have to be there. And you have to feel like that’s the only thing in the world for you at this moment. Is it a skill? Is it learned? I don’t know. But that’s something that everybody – you know – and I know you’ve written a whole book about it. It’s crazy.
Steve: That’s great. That’s a wonderful thought. How old are your children?
Pree: 8 and 9 – you have kids too?
Steve: I do not have kids, no. What should we have talked about that we didn’t get to?
Pree: I think that’s an absolutely good question. That’s a question that everybody should be asking at the end of the interview, yeah.
Steve: That’s right, that’s my stock – it’s one of my stock ending questions – everybody’s stock ending question.
Pree: Let’s see. What should we have talked about? I think we covered a lot of things. I’d say I think we might discuss a little bit and I want to ask you your thoughts on this which is where is design research going long term? And I think a lot about is this field a temporary field that was right for a certain time period and will it just disappear? Or is this a field that will morph into new things. What do you think?
Steve: That’s what I was driving at a little bit when I ineptly compared research to expense reporting. I mean I’ve thought for years that there’s a potential for this to be an activity not a role and it’s interesting – I don’t know, it’s very exciting to hear how the activity of research has really, as you’re describing it in these ideal situations, it’s really grown. Like what we mean by research and what it entails and the skills and all that. It’s – there is a point, and you probably remember this from your early days, where, you know buying a video camera was like, that was sort of the step into doing research. Well we bought a video camera. And it was sort of a naïve time, but I love hearing you describe kind of the – I guess the maturity and sophistication and the capability that this set of activities has. But I have just been saying for years like is this just design? Or should it just be design? And I feel like – I don’t know if this is the same trend or an adjacent one. Part of that sort of questing that you and I and people in our industry have been doing, sort of evangelizing, advocating, hoping, wishing, wanting, has been to see research more fully integrated, more fully adopted and I think with that has come a couple of things. There’s sort of a good and bad and as you said there’s an ideal and you’ve laid out the vision for the ideal and you strive every day to achieve that ideal. I also think, and hopefully I’m not just being grumpy about it, but there’s a lot of commoditization of research that’s happened at the same time.
Pree: Oh absolutely.
Steve: Where there’s lots of junior people coming in. They’re not always getting the mentorship that you’re describing. They’re being asked to kind of execute closed end – not explorations. They’re barely doing evaluative. They’re swimming in the local maxima to mix metaphors. And that some people are kind of you know dusting their hands off going well we did it. We hired a researcher, we’ve got research being done. So…
Pree: It’s a checkbox, yeah.
Steve: Yes, yes. I guess that isn’t new. I guess I just see more versions of that where there was a time where research was ignored and now it’s sort of pulled in, kind of kept in the corner and not doing very much. And I feel like oh we wanted everyone to do research and now they are, except they’re not always doing – they’re not always kind of doing it as a leadership, strategic activity the way you guys are and so many of our peers are and so sometimes I wonder like well is this the consequence – is this the downside of us wanting this so much that it has been adopted but not in the way that we had hoped for.
Pree: Yeah. And I think you mentioned something very interesting which is this is an activity that everybody does or it’s just one of those activities. It’s not necessarily a discipline. It’s not a profession. It’s a very interesting thought because I have a sense that you know over the last few years, like it’s been 20-25 years that we have had kind of a new discipline come about which is the product management discipline. You know it used to be more program management before or it used to be more like putting – connecting all the dots, rather than owning the product. And I also wonder out aloud if we somehow connect with the product management discipline somehow or the design discipline even more so. And I don’t have any conclusions. It’s one of those things that kind of keeps me up and I think about like where is this going?
Steve: I’ve said this before on this podcast, it seems like research follows design or follows user experience and that I think some of the things we wring our hands about are things that that profession, that community, was wringing its hands about three years ago. And so the move – I hear UX people say all the time that product management is kind of, is the opportunity for UX to grow. Rosenfeld Media did a virtual conference recently that was like a PM plus UX event. So there’s these signals that these are the conversations that are happening. And so where’s the research side of – ‘cuz sometimes research is sort of like hidden under UX a little bit.
Pree: It is, but if you see some of the latest things that are happening, I feel like research is actually leading design and the reason I feel research is leading design is because of a few things. One is how connected we are with users? And how that gap between a user having a need to a solution that can be shipped is kind of reducing. As that happens you end up having a lot more of this need gathering. A lot more of this co-creation if you may of the end product. That is important. And we’ve always been at the leading edge of this. We’ve been pushing design from design being just inspiration and like let me make this up, to design that is like intentional. And we continue to push this discipline. So I would any day call it as leading the discipline rather than kind of following it. Following it maybe from a time standpoint, maybe. Yes I see where you are going with this, but yeah.
Steve: So that dynamic is interesting. And maybe design is maturing to not be decoration. I mean it’s been maturing that way for a long time and that’s – I’m just reflecting what you’re saying – that starts – as it matures to be sort of creating experiences, creating value, then the hook for design to do that is research which then goes back to your question, which we’re not going to be able to answer, which is should it just go away or will it just go away?
Pree: I don’t know. Like – another interesting development that’s happening with the design side of things is patterns. So over and over again now we are starting to see design patterns that are used multiple places. And so suddenly the value of having to design individual things go down. And it happened in software a few years ago. As in some of the very good software programmers know that the best thing to do is to have a collection of all of these things that they use over and over again and they just pull it in, cut and paste. And most of the code is not written. It’s just cut and pasted together. I’m not reducing the value of a person – of software here. You know I can code and I know it’s a lot of work – patterns and knowing when to pull those patterns together is the skill. So you mature the discipline to something else. And we are very good as a research discipline to figure out those kinds of things, the needs or what patterns to pull together. So I think we’re set for good success. That’s what I’m thinking.
Steve: I love that. Maybe we should leave it on that wonderful high note and thinking about the success we’ll continue to have as a discipline. So thanks very much, Pree. It was a wonderful conversation.
Pree: Absolutely. I enjoyed it. Thank you so much.
Steve: Thank you.
The post 12. Pree Kolari of eBay first appeared on Portigal Consulting.
This episode features Gabe Trionfi, the Manager of Research at Pinterest. We discuss the evolution of user research, collaboration between disciplines and the journey versus the destination.
The post 11. Gabe Trionfi of Pinterest first appeared on Portigal Consulting.
This episode features Gabe Trionfi, the Manager of Research at Pinterest. We discuss the evolution of user research, collaboration between disciplines and the journey versus the destination.
Great researchers really know the on the ground, in the trenches, skills, tools, things they have to do to move forward with the group, but they also have to play this unique role where they pop out and get the bird’s eye view or the broader context and think about that, but not in a way that distracts the rest of the team. And then they port back down to the blood and the guts of what we’re doing every day in the trenches to figure out how can I take those observations that are abstract and make them useful for the team? So all research conversations for me waver between the two. They’re up in the air, really philosophical, abstract, but then also down on the ground, really practical, really technical. – Gabe Trionfi
Show Links* Gabe Trionfi * Follow Gabe on Twitter * Pinterest * Joe Namath guarantees it * Joe Montana goes to Disneyland * Failure to Lunch * June Jo Lee * Michael Graves and Target * Tiffani Jones Brown * Sherlock Holmes * Evan Sharp * Ben Silbermann * Social graph/interest graph * Altay Sendil * Larkin Brown * Cassandra Rowe * Jolie Martin * Paul Pattishall * Julia Kirkpatrick * Burlington * Country Style Donuts * Tim Horton’s * Millionaire Donut Shoppe
Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes.
TranscriptSteve:Let’s start with the introduction. Tell us who you are, what you do, where we are today.
Gabe:Yeah, so my name is Gabe Trionfi. I’m the Manager of Research at Pinterest. I’m a researcher. It’s what I do. It’s what I want to do. It’s not the journey, it’s the destination for me. I’ve been doing research for about 10 years. I’ve been at Pinterest for about 3 ½ years now and we’re in the office, third floor, new headquarters for Pinterest. And we’re in Alberta, a Canadian province theme named room.
Steve:Very appropriate.
Gabe:Very appropriate.
Steve:We have two Canucks talking research.
Gabe:Absolutely.
Steve:So you made an interesting comment about it’s the destination, not the journey.
Gabe:Yeah.
Steve:So could you explain that?
Gabe:Yeah, I think for me when I think about research it’s my career. It’s what I want to do and I always want to be doing research in some way. I have the opportunity now at Pinterest to manage a team. It’s my first time managing a research team. I’m always going to want to work as a researcher in some way. I’m not looking to become a PM or a head of product. I want to be involved with research. I also think it’s interesting to think about a career path. Like I’m looking for a positive trajectory for my career, but whether it’s being a manager or an individual contributor, that’s like not the most important thing. The most important thing for me is are we working on something that’s interesting? Is there an interesting research question to be have? Is there a product that people are going to be using? That’s really important. Are we shipping stuff that’s making people’s experience better? I want to be a part of that.
Steve:But some of that isn’t research.
Gabe:Absolutely.
Steve:Some of that is the thing that research informs. In the context in which you’re doing research you want to have a good question, but you want someone to make stuff out of it.
Gabe:Yeah, so I think sometimes when we talk about – in design or creative disciplines like writing and research, people are like well you’re a manager now. Are you always going to be a manager? Or does it mean less if you are not the manager? I would much rather be – go forward and have a role where I was a researcher on something interesting that people were using vs. always choosing to be a manager on something that people weren’t using or was potentially not that interesting from a research perspective.
Steve:You mentioned to me before, outside this interview, that you have a history in theater. It’s almost like you want to direct, but you don’t want to give up your acting jobs?
Gabe:Yeah. Yeah. Or it’s all part of acting. It depends on how you think of it. I think it has – there are some really interesting parallels. So I had never managed a team or built a team. At Pinterest I was the first researcher. There was like an opportunity where they said do you want to build a team? I said I want to try that. I didn’t know what it would be like and there’s been a ton of learning. Obviously when you’re first out of the gate you don’t know enough to be great, or even good, but you learn over time. The one thing that was interesting, taking it back to theatre, was I found that building a team reminded me a lot of casting a play because when you’re really putting together a great research team it’s not just skill sets. It’s not just experience, but it’s also understanding how will these people interact with each other when we’re building knowledge together? How will they be perceived as a group, as individuals? It’s like casting a play, like how will this actor relate? Or to this other actor that we’ve already cast? Do they have chemistry? Do they represent the relationship of the characters well by the way they look and what they sound like? And that was really surprising to me. It’s been a really fun part about building the team at Pinterest.
Steve:I want to follow up on that, but I also want to go back. Let’s just go back to a few different things here before we lose them. You opened with this big idea about the journey and the destination so now I’m going to pick it apart until you wished you’d never said it to me.
Gabe:Okay.
Steve:Because I – I mean I feel like you’re describing problems that are sort of journey problems not destination problems if that’s a way to slice it. Like you cast a play, you can be expert in it and you can know all these actors and know what their strengths are and you can do the right kind of – what’s that audition where you see people play off each other. Like a screen test or whatever that is. You can do all those things, but it’s still an emergent thing. You don’t know.
Gabe:Yeah, I think for me I think of research as a destination as quite large. That there’s many things to be learned. I still think of research as early days, particularly in the context of technology and start-ups. And so I think it’s fast and deep and interesting and sometimes I may talk with people who seem like – well this is how I’m getting into tech. So I have a social science degree and a little bit of background in psychology. I’m going to be a researcher for awhile, but really the destination is like to be a product manager or to be a VP or head of – and for me the actual research part is so interesting and there’s so much to learn and do. I definitely feel like it’s the beginning and that’s ironic of course ‘cuz you’ve been doing this for much longer than I have. When I worked at previous places like IDEO, they have people who have been working in research since, you know, the early 60’s in the context of…
Steve:I have not been doing this since the early 60s. Just for clarification..
Gabe:But in the context…
Steve:But the field…
Gabe:Yeah, the field goes back. You can even go back to World War I or World War II cockpits in air planes. The sort of human factor, early ergonomics. That’s sort of the seed of a lot of this stuff. ’63 is when the Royal Design Conference, right, in England – the Royal Academy of Design – they start talking about research for the first time. Engineers start talking about design and research and really what it means. So it’s a discipline that the idea has been germinating for awhile, but I think in practice every day it’s still fairly new, especially when you get outside the context of physical products and service design and you get into like pure software technical. The majority of people I work with here have never worked with a researcher before. And so that’s a challenge because you’re sort of still at the beginning and sometimes you can do really incredible work and people may not know that it’s that good because they actually haven’t worked with anyone before. And that’s sort of humbling and interesting and it’s a challenge at times, but it’s also a huge opportunity. So that’s interesting. I often talk about research being early like – you know the difference between Joe Namath and like Super Bowl 3 I think – who’s like a big time football player. He is the quarterback. He’s sort of glitzy and everyone knows him. He’s the underdog. He says they’re going to win. They win. He’s like celebrated on a bunch of magazine covers. But it’s like in the 60’s, the NFL. So like he goes home and has a Schlitz or something, right. Like he kind of has a regular life. Fast forward like to the 80’s. It’s Joe Montana, right, for the San Francisco 49ers, who wins a bunch of Super Bowls. He then – he’s a big deal, like Joe Namath, but like he goes to Disneyland afterwards and he gets like a lot of money and he’s super wealthy and commercials and whatever it is. The difference between Joe Namath and Joe Montana, we can argue about it as athletes, but the main difference is timing. When were they the quarterbacks of the Super Bowl winning teams and sometimes timing, when you’re at the beginning that’s just a constraint. It’s just part of being at the beginning and I think that research has a long way to go as a discipline and there’s a ton of opportunity, but like if you’re a researcher 10 years from now you’re going to be standing on the shoulders of schlubs like ourselves who had to do it in the early days. It will be way better then because people will understand it and also the market will shift in a way that will require research or something like research to inform product development. Because it will be more competitive. There will be more services and apps within the same space that need to differentiate from each other. So timing, where we are.
Steve:Where are we between Namath and Montana?
Gabe:That’s a good question.
Steve:Is there a visual continuum here and we could get the whiteboard out?
Gabe:Yeah.
Steve:Let’s just challenge that. A recent issue of the New York Times Magazine was all about work and so this article, this article, this article. There was one – it was really about people eating at their desks, but the side bar piece – unfortunately I don’t remember the name of the person, but the side that kind of goes with the article that supports this photo essay says so and so, an ethnographer, is like the first thing. And it was not an article about oh these people are like anthropologists in safari hats going into corporate America. They didn’t go that thing that journalists were doing 3 years ago and 15 years ago which is ooh, look how weird this is. They were really just – it was sort of incidental and characterized the fact that like hey there’s an expert who’s watching people eat lunch at their desks. To me the cultural moment there is like this is a thing that we don’t have to just point to it and go oh-ah, we can use it then to tell another story. So that seems we’re – I don’t know if we’re taking this person to Disneyland, but we’re not having Schlitz.
Gabe:Yeah, so I think this is an important point which is – like in research we often pay attention to context, so my context today is within tech, within start-up land. I think when we look at research trajectories in terms of its development as a discipline it actually tracks differently across different contexts. So I think when you look at physical products, service design, I think that industry what’s happened there, that collection of industries with research is far ahead of where we are within tech.
Steve:You think tech is behind?
Gabe:Well I think tech is tracing the trajectory. To me it looks the same and I think it’s headed in the same direction, but I do think it’s at a different point much like just the industry, the current modern industry, like post-Internet, apps, software design – it’s fairly new itself, right. And so with a lot of new industries, especially industries that have access to a lot of data, right, even the history of psychology itself has had moments like this where people think it’s all about this type of data, this quantitative data, right. That’s how we can – we’ll find the future opportunities within that and then we realize like well actually one type of data may not be enough. We need different types of data. And then we might switch and say like well I’m going to do this other thing, or I’m going to do this other type of data acquisition or method, right, and then really mature fields, they come to usually a realization which is like yeah there’s no one thing that’s going to do it for us. We need multiple things to try and understand complicated phenomena and then within that, especially within the world of business and tech, then you’re still going to have make a risky call because again none of this data can tell you the future. It can paint out trajectories and possibilities, but you still have to take a risk. I want to work at a place where someone’s going to be able to understand the data, triangulate and then make a strategic decision that’s going to lead to a better outcome for the people using it and the people working on it. So I think that when we get over – again, some disciplines, some industries in the beginning there’s things that seem easy that are a lot more complicated than they seem.
The example I always give is like the common thing is qualitative versus quantitative data which of course they both have subjectivity embedded in them. They both have trade-offs. They probably should be used appropriately which is what’s your question, then let’s choose a method that fits the question. But the metaphor I always say is like I would never hire a general contractor to build my house who is like I only hire carpenters who use hammers. There’s no screwdrivers on my site. It makes no sense because what should determine the tool is the rest of the context. So let’s not use a hammer on a screw. Let’s use it for nails and let’s use a screwdriver for screws. And that’s the same thing for me about methods which is it doesn’t matter what the tool is. What matters is the context and what is the appropriate design to get at understanding the question and also the outcome goal. What is our goal? What are we going to do with the answer? Those are the things that are important to me.
Steve:So you could start to sort of map this trajectory of research and tech and look at well what were the predominant tools – if the tools are a proxy for what questions are we asking, we might just have people showing up with hammers. I remember the point at which Microsoft got usability. Like that was going change – and this was I want to say early to mid 90’s and there was a lot of press about, like the physical rooms they were building. I think I went on an interview there and went on a tour there and you realized oh they’re just going to like run, like huge – I mean they’re going to do usability testing at scale. You know the word that we like to say in research or everyone likes to say in tech I guess, about at scale. They were really going to scale the hell out of this and sort of usability their way into something. And there wasn’t a lot of – I mean that was leading edge thinking back then. And I think as you’re describing this trajectory or this changing context it’s about the tools and the tools are the thing that we can see more easily. I don’t know what design questions were being asked of Microsoft, but I can infer based on the fact that this is where they put all their money. So I think you’re sort of describing a long time ago already, you’re saying now we’re like starting – we’re at a point where we need these tools and we need to be asking these questions. And so – I don’t know. Is that fair, the recap?
Gabe:Yeah. And also sometimes the context is like within software and apps the marketplace actually comes to bear on the types of disciplines you need to invest in to differentiate. So if we’re in a marketplace where everybody has a really high quality engineering team, everybody has a really high quality design team, and we start stacking the disciplines up and you’re going to enter in sort of the messaging as UI which is really popular amongst a suite of apps today, right, where you’re just – it’s a concierge service or an on demand service that is using the interface of like a traditional SMS app. There’s a bunch of them out there, right. They all have great designers. They all have great engineers. They have great marketers. They need to understand how to differentiate from the other 5 apps within their space and they’re going to have to invest in something. They could invest in partnerships with other companies. They could invest in celebrities, right – to sort of like build their awareness. They could invest in researchers to try and understand what is most meaningful about this experience that they’re building and what are the opportunities to separate themselves from the other within the space. So some of it is disciplines and methods over time. Some of it’s the market itself becoming more competitive and sort of more compressed in terms of offerings out there. Traditionally design has been used in recent years to separate or differentiate, right. And I think research follows design and if you don’t have a designer it’s really rare that you have a researcher because design is working on the experience on the front end of software that people see and think of when they think of an app or service. If you’re not investing in that you’re probably not investing in like a deeper understanding of your audience’s needs as a way to differentiate. But like the classic examples – like Target hires a famous designer to make like toilet bowl brushes that are really pretty. That’s a way to differentiate from others within their space who are smell – who are selling commodities like that. I think research can – will and is having the same sort of role within tech as more and more apps and start ups start to fill into the same spaces. It’s not about building the infrastructure, say plumbing the Internet with social connectivity. It’s about like okay there are 5 financial service apps now that are redefining the loan space, how are they different from each other and how can we ensure that ours is the most relevant to our audience? Let’s get a researcher in here to help with that.
Steve:I think what I heard you say though is at a strategy level there’s a lot of different things that we could do if we’re trying to differentiate our financial app. So we could try to get Leo and his posse to use that. Or whatever sort of way to generate attention. We could have a controversial CEO that makes outrageous comments about people. Or we could change our pricing model. Or we could – there’s a lot of things that don’t necessarily come from a nuanced insight about the world. It’s just sort of differentiation for its own sake. Or there’s a strategy there and one of those out of – let’s say there’s 20 obvious ones – and I can only come up with 2 right now – I’m just parroting you back. So one of those might be let’s invest in experience which means investing in design, investing in research.
Gabe:Also writing. Writing is a similar discipline to me than research within tech now. It’s newer. Usually it’s working across many different teams within a tech company usually. It’s critical, like what is the voice of your brand, but also of your product itself. And I see a lot of – I work with a lot of writers here. Tiffany Jones Brown started as the lead of the writing team here, has since become a Creative Director. She has a great team of writers that I’ve worked with before. Building a relationship with that team, I feel just a commonality or similarity in terms of where writing is as a discipline. And the – if you invest in writing it says something about your strategy when you have a tech oriented app or software service. It says like hey we want to make sure that people feel differently about our brand or about our product. And you see a big differentiator in lots of tech companies now in terms of how do they talk to their users through their product. And so again it’s just another strategy that you could invest in. And research, and a suite of different research disciplines is part of that. Also data science can be part of that. Business analytics can be part of that. Ways to, you know, bank capital, not just monetary but like understanding. Like you can bank a lot of knowledge as capital to inform your strategies or to help move your product forward.
Steve:If I take a naïve look at what you’re describing I feel like there are some disciplines that are output disciplines and there are some that are input and by that I mean, you know, writing is crafting the words that go on your website. Engineering is the code. Design is the pixels and every designer – everyone is rolling their eyes at that. That’s a really, it’s a very broad brush, but research is an input to decisions. Business analytics is an input to decisions. I don’t know if that’s a fair – if it’s that binary or not.
Gabe:I think about it similarly. So I think about it as a suite of disciplines that are about indirect influence. So I’m a psychologist. I’m not going to build something with regards to the software that’s going to get into the world, but what I’m building is understanding or knowledge, but also sometimes creating experiences that are inspirational. I talk a lot about information and inspiration as part of the research role that is going to indirectly influence that thing that enters into the world. Sometimes there’s a one to one. You can just say like hey that decision there was based on this work over here. Sometimes it’s much more diffuse in that the whole host of teams that are actually contributing to the output – the most important thing is I want to work in an environment where research is a part of that, whether indirectly or directly, and it’s not something that’s on the side or separate. I often talk about like Sherlock Holmes and Dr. Watson. Like I’m very happy to be Watson to someone’s Holmes because I think that’s a real role and those Holmes stories wouldn’t be the same without the indirect influence of Watson, right, sort of guiding, informing, being a foil for Sherlock as he goes ahead and builds the solution to the mystery. And I think indirect influence is really interesting. Really hard to do. It – sometimes it’s easier just to like come in and say we should do this and then people do it, right. But to sort of like change people’s perspective on something they think they know very well, but actually might be very different outside of their own social circle or their own sphere of experience – let’s say on the other side of the planet there’s a different culture, different country, a different type of person who’s using your product and perhaps they just have a different mental model that you didn’t even know existed. Bringing that back, but also not just a shock, but to potentially influence, like oh hey perhaps we should think about this a little bit differently. Or perhaps it’s another way to think about it, that there’s not just one way to think about our product, there’s actual multiples that we should keep in mind. And that’s a good thing because it means the product is relevant to lots of people and complex and so it’s not just a simple like P2P commerce, like I’ve got this thing, do you want to buy it? Yes you bought it. Cool, transaction, safety, all that sort of stuff. It’s more complicated than that in some cases.
Steve:So I think research has suffered from the sort of the naïve Dr. Watson – well not that Watson himself is naïve, but if you take a shallow look at those stories Watson is merely the chronicler. He’s the sidekick. He doesn’t do anything. He and I – you know, you do the close reading that you do and you see what he enables and how he kind of supports and I think that’s the beauty of those stories. You kind of go at them a little bit deeper and you realize there’s a complicated thing going on and that it – the more you invest in learning what those stories are about the more they kind of pay off. So just to bring it back then to research – research has maybe historically been viewed at the way we naively view Watson, that research is just asking people what they want. Sort of chronically things and reporting them back. And research as you’re articulating it, and of course I agree, has the potential and does more often now play more of that influential, maybe Holmes is the one that gets brought to Disneyland. I’m going to crush all metaphors together. Like Holmes is the one that gets carried on the shoulders but the researcher is – ugh, I can’t untangle this here. So I guess I’m trying to ask in a really twisted way here is about you know what are the things that you’ve seen or that you’ve practiced to give you the chance to play Watson, to enable things, as opposed to being relegated to a mere chronicler?
Gabe:So, I think this is an important question. So for me I had one experience. My sort of first design research job was at IDEO and IDEO is sort of at one end of the scale design thinking to an extreme. Huge opportunities to do different sorts of work and to really be part of a deep, embedded process. Then I worked at Facebook which is an engineering company and it has a very strong engineering first perspective. So when I was thinking about my next destination I was thinking about what lies in between? What are the sorts of things that are out there? And there were a few companies and I think at the time the commonalty is they all had founders who were designers. And I think when you have a founder who’s a designer there’s an awareness of what research might be. There’s an awareness of the Watson role and its power in terms of creating new ideas or – and I think you want – I wanted to try and see if I could work at those places. Pinterest was one of those places. So Evan Sharp is a designer. He’s one of the co-founders along with Ben Silbermann. And so Evan knows about research. He’s not an expert in it. That’s important, but he’s open to it. He’s interested in it. And so that’s why they brought me in. I was about – there was probably about 70 employees when I joined, which at the time some folks thought well that’s early for a researcher, but if you understand what research can do it might not be that early and in fact the broader field we see now – I see startups all the time with a handful of people who are bringing researchers in when it’s relevant to what they’re doing. So to tie it back to the metaphor, to me it’s important to be at a place where the Sherlock wants the Watson to be there because a lot of times that may not be the case. And so Watson’s ability to contribute is limited by the partner’s desire for input. I think at Pinterest there’s a big desire for that. It’s a really interesting place to do research. Very open, very focused on the audience that we’re trying to reach. In part it’s probably due to, you know companies have lots of values. Evan will talk about this. I think the one, the value that I think is really unique at Pinterest, and differentiating, is something called “knit” which is how we talk about interdisciplinary collaboration and respect. It’s understanding that there are many people who can contribute to successful outcomes at the table and we should understand each of their disciplines, strengths and weaknesses and we should leverage their strengths. So “knit” is very Pinterest, knitting right, like weaving together disciplines. There are a lot of starts where that sort of multidisciplinary approach is not part of what they do, culturally or strategically, and it’s a huge part of what’s happening at Pinterest, whether it’s Community Ops, Design, Writing, Brand Design, Marketing, Research. There’s a whole bunch of disciplines – Data Sciences, Business Analytics – that are all working together very closely – Project Management, Product Management – to sort of put together what it is Pinterest does. So that context, that environment that’s willing to accept what you have to offer, but also try to like embrace and really use it. That’s important for research and I think that’s a choice you have as a researcher. If you seek those places out you are in part prototyping or designing your own experience. Versus going to a place where you might know from the outset they want a research, they don’t know what researchers do, they’re not certain that they will use the research. And that’s usually pretty clear in the hiring process or early on. And I think that’s not to say that’s a lesser than experience, but it certainly is one you’re choosing as a research practitioner in terms of your daily experience.
Steve:Or you should be aware of that?
Gabe:Yeah.
Steve:And you’d do well for yourself to know that that’s what you’re choosing.
Gabe:Yeah.
Steve:I think I saw an artifact of the knitting culture. In the men’s room there’s some Python script coding advice hung over the urinal and the title for it – I don’t remember what it said, something about like python knitting or something.
Gabe:I think it’s called – officially those posters are calling knitting in the bathroom. And there’s someone – an engineer here is – there’s I think a couple of them working together to sort of pass on tips and tricks for certain software languages by putting those up in the bathroom stalls.
Steve:It’s not the only company where I’ve seen like technical advice in the bathrooms.
Gabe:Yeah.
Steve:But the knitting – I didn’t understand the knitting reference until you kind of explained what that means culturally.
Gabe:Yeah, absolutely and I think it’s – you know again there’s a lot of great – we have other values at Pinterest that are great. But I think that one is really unique. Again this idea of – for me it hearkens back to some of the experiences I had at IDEO. This emphasis on multi-disciplinary approach to design challenges. I think it does lead to different outcomes. My background, I have an undergraduate degree in theater and I care a lot about process and how we’re doing what we’re doing together and so again that part of Pinterest and being a researcher at Pinterest is real exciting.
Steve:So we never resolved this. Again, I was kind of picking at you at the beginning when you talked about journeys and destinations as things that you – and so I think I understand as you explain more that being a researcher is a successful state for you. It’s not a stage to get to something else. But when we kind of break the world up into journeys and destinations I mean you – theatre is about the journey, right. The payoffs are – and you said someone who cares about process is someone that cares about the journey. So I think I got hung up a little bit on how you were using that phrase and I jumped all over you, but I appreciate finally we have – finally I have clarity. So thank you.
Since we haven’t talked about this at all can you explain what Pinterest does?
Gabe:Yeah. So if you listen to Dollars to Donuts you’ve heard that Pinterest is the world’s largest catalog of ideas. The question is like what does that mean? And I think at that – at the heart of that is like an interesting conversation for researchers, but also an interesting characterization of the opportunity and challenge Pinterest has. So Pinterest is very different than a lot of the apps or services or companies it’s often compared to. For a long time the press sort of positioned Pinterest as another type of social network. Pinterest is not a social network. It’s a very personal experience for the people who are using it. It involves things like saving things that you want to then go on and do or plan or make or cook or buy. But a lot of – when you do research with Pinterest you are really deep into it. They see it as this very personal moment. Sometimes people will even – again it has to do with discovery of content, the things you care about. Sometimes it has to do with learning. There’s lots of stories here from Ben our CEO talking to people early on where they would say to him anecdotally like I didn’t know I had a style. Like I didn’t think that was part of who I was, but then I started pinning clothes or things for my home and one day I was looking at my board about my living room – you know best living room, or like my dream living room – and I realized like wait a second, seeing all those things together it all sort of snapped for me. Like I’m – this is actually my style. This is what I’m into aesthetically. There are a lot of moments like that I think for Pinners because it’s very personal.
Steve:A Pinner is someone that’s using Pinterest?
Gabe:Yeah, someone who is using Pinterest is a Pinner. Someone who works at Pinterest is a Pinployee. We’ve got a lot of pin puns going on here, at the Pin Factory, which is what I call it.
Steve:We’re going edit all those out.
Gabe:So back to the point, I think Pinterest is personal. It’s done in a way where it’s in a public space, but it’s not social in the same way many social networks are about broadcast. Like this is my identity. This is what I’ve done. Pinterest is really about – it’s about aspirations. It’s about anticipating. Like I want to do this trip so I’m planning it. And then it’s about participating. Participation on Pinterest is not within the digital space. It’s within the analogue, the real world. The part where you have this pin for this recipe and then you go cook it. You make it, you serve it. Whereas other social networks are often about broadcasting in the moment, like real time, or they’re about broadcasting this happened. So they’re sort of reminiscing or looking back on the event. Where we see Pinterest is used to plan, you know, in the US Thanksgiving dinner. You start seeing people pinning Thanksgiving recipes months in advance. Halloween costumes is a big – DIY Halloween costumes is a big thing in the US on Pinterest and people start looking for those how to do – you know put together their dream costume, the best costume, the cool costume, like 2 months in advance of Halloween. So that Pinterest is a personal experience. It’s about planning to take action in the future. It hopefully supports you taking that action because you have the information you need. It’s very different then a lot of other services. So what does this have to do with the catalog of ideas? I think the important bit is we see in all of our research, market research, qual, quant, that this idea of a place to find ideas is differentiating. It’s useful, it’s fun and people enjoy that. The challenge is when you sort of – you have to contextualize that. From a marketing perspective you have to say you know okay what about a place to find ideas. A place to find ideas is not enough context. So there’s been different ways we’ve approached that. For some period of time we talked about bookmarks. Some period of time we talked about – more recently we were talking about like a physical thing like a catalog where you go and browse and discover lots of ideas. The key thing is the ideas bit. We know that’s important. The question is how do you contextualize that, not just for a US audience, but actually for a global audience and that’s really challenging. Pinterest itself is based on you know a metaphor of like a pin and board – pinboard. The challenge is that that doesn’t actually internationalize. Not every country has a tradition of a pinboard with a pin. And so how to provide context for what this thing is and it’s not – you know it would be easy if it was just like oh it’s a social network for, I don’t know, saving ideas, right. It’s not that though. That’s not really the core. And so how do you contextualize that? That’s the challenge. So as we think about different ways of messaging to audiences, especially audiences who are not using Pinterest today – there’s a ton of research we’ve done. There’s a ton of research to be done as we look for that really killer way to describe a complicated and robustly powerful product to people who don’t know what it’s about. And that’s an opportunity for research, whether you’re a marketer researcher or quantitative researcher. And it’s part of what the team has worked on and will continue to work on with our partners in writing, marketing and then just even overall strategically, like the decision makers of Pinterest. How do they want to – they have to go and speak to the press. This is something they’re saying all the time. They need to feel that it’s accurate, that it’s representative and that it really gets at the – the core or the central sort of essence of what they think the product is and will be in the future.
Steve:So I asked you what the product is and you describe sort of the research and strategy journey of trying to describe it – to answer that question in the way that sort of has the most benefit for Pinterest and current and future Pinners.
Gabe:Yup.
Steve:But I’m going to ask you again, like – I mean as you say people are out there now talking about it. So you kind of said – you took the phrase catalog of ideas and then you unpacked it into all this wonderful stuff which I think tells us what kinds of things you’re thinking about and what your work is meant to address. But just for that person that’s scratching their head right now and going like okay, but what is Pinterest?
Gabe:I gotcha. It is a place to save ideas digitally, to discover ideas related to the things you’re interested in and then to go and do the things that you have saved and discovered. So it’s about saving, discovering and doing.
Steve:So what kinds of digital information are people saving?
Gabe:Well I think we’re limited to digital today, but it could be many, many things in the future. It could be saving many types of objects. Right now we’re within the space of like the Internet. We see the – the most common use cases are we see a lot of folks use Pinterest for recipes. Planning daily meals is a huge part of what pinners. Also like discovering or trying new recipes, especially if you’re into that. It’s often about doing, right. Another popular use case – trips and travel. Like right now you want to go on a trip so you go to various websites and maybe you have a folder on your computer or a piece of paper – you’re writing down like where do I want to go when I’m in Hawaii? Where do I want to stay? You can save all those things to a board on Pinterest which is similar to a folder, but it’s within the bounds of Pinterest itself. You can save those links. You can find them on Pinterest. You can find them off of Pinterest and save it to your Pinterest. So that you can go to one board and have all the ideas of what you want to do on your trip when you go there. And you can access that when you’re actually on your trip, right. We also see a lot of people who use Pinterest for fashion, whether it’s discovering new trends or thinking about inspiration for what they want to wear. In fact there was a story this week in the – oh gosh, I can’t remember the newspaper – and this writer had talked about – she let Pinterest choose her outfit every day. So what she would do is she would take out of her closet an article of clothing she had – let’s say a pencil skirt. She would go to Pinterest and type in pencil skirt and she would look at the most popular street style or fashion images of people wearing pencil skirts. Then she would find the one she would like and then she would mimic it with her available wardrobe. So every day she let Pinterest dress her. Again, that really critical thing there is 1) she was looking for ideas, 2) she was then doing it and what was also interesting about it is that she was using Search. We see Search as being a huge part of a lot of Pinners experience in order to find those ideas. But Search I would say is a little different on Pinterest. The Search in this case, and what we see across many people using Pinterest is it’s not to find one idea that’s like the right idea. We don’t have a ranking algorithm that puts the right idea at the top. When you’re in a discovery system or a discovery mode you can search for outfits with pencil skirts, but there are many possibilities in front of you. You are looking across this set of data or images that’s about discovery and you’re trying to look at not just the best, but there are some that are good and there are some that are even better to make or to learn or to discover something that you want to then go do.
Steve:So you described before this – you know your own professional path a little bit and that you sort of found Pinterest as a company that was in between maybe other places you’d worked at in terms of what was research and how it was understood. But as you’re talking about Pinterest and what’s going on and what you’re interested in exploring, there’s also a unique class of problem that I hear you kind of digging into with joy and I don’t know if that’s the culture or you or what. And that is we don’t know quite exactly what this thing is and the best way to talk about it and it’s changing and people are inventing new things for it and the world is culturally diverse. So the fundamental, like what is Pinterest and how do we describe it and how do we build for it is – like that’s a hard problem. It’s a dynamically shifting problem. I mean did you know that’s what you were getting into?
Gabe:Well I have two answers. So the first one is like – the first one is sort of addressing the like what is it, it’s dynamic – like I think there’s something really interesting about Pinterest which makes it unique, which is again opportunity, but also challenge, which is there’s a utility element to this, right. It helps you to get things done. It helps you to find the things you want to do. But it’s not just a utility. It’s not a task based software. It’s not about lists per se. Although you could visualize pins and boards as lists, right. But it has this other element that I think is related and unexpected which is it has this way of conveying or inspiring people to think about possibilities. To discover things that they didn’t know about and to learn something about themselves. And it does this in a way that – we know from research that people really enjoy the aesthetic presentation of this information. They think it’s beautiful. And so, you know people can get – we interview people and they talk about my Pinterest time, right. They’re on their couch on a Saturday morning. They’ve got TV on in the background, maybe some Netflix. They’ve got a big cup of coffee. They’ve got like their phone or perhaps their iPad, and they’re browsing through Pinterest. And some of it’s about what they want to do. Some of it’s like what might I want to do? This sort of anticipation, aspiration. This sort of beginning of the experience. And Pinterest puts together these two things which are traditionally separate, right – the sort of exploration/discovery, aspiration/inspiration with like utility. And so that’s a tension within the product that’s really interesting. I think it draws people in, but it can make it really challenging because it would be much easier to do one or the other and that’s not where Pinterest is. It wants to do saving and doing, but it also wants to do discovery. And of course the more saving you do the better Pinterest hopefully gets at making recommendations to you about things that you should check out based on what you’ve done. That’s definitely an interesting part of the product and the tensions within the product, or opportunities and challenges.
Secondarily you were talking about did I know that when I was getting in here? I was super interested in discovery systems. I think discovery systems are a really interesting part of the technology sort of landscape at this point in time. As part of discovery systems it’s like moving away from basic search which again is not basic in any sense, but many companies have been able to perfect the “I need this thing – help me find this one thing.” I call that like the needle use case. Right, I need the – help me find the needle in the haystack quickly. That’s impressive and it’s great, but it’s also now become expected. And so when you move into – it used to be called Search and Discovery. When you move into discovery systems that play up the discovery experience first suddenly, yeah you have to be able to help 5 people find the needle which is like perhaps one small moment within a user’s session, but actually the use case of looking through the haystacks is really interesting to people, right. So they’re looking through like haystacks in a field right and they’re like well what’s in this one? What’s this one about? How is this one different than that one? They’re exploring and discovering things that are interesting to them. They spend a lot of time doing that. And then they get to the needle. That’s something that I was already thinking about before I joined Pinterest and I could see Pinterest as having a unique position within that space. The one thing that’s related to that which we don’t talk to users about and it’s a super – I’m not an expert in this, but it’s an intriguing idea – is the notion of the interest graph. Are you familiar with the interest graph as a concept?
Steve:No.
Gabe:Okay, so social graph is sort of like, you can think of that as a subset of the interest graph.
Steve:What’s a social graph?
Gabe:The social graph is basically let’s map out with technology all of the people and their relationships to other people. And what’s interesting is like that’s been a very disruptive technology moment in the world today. But what’s interesting about it is that’s a big task – don’t get me wrong – but it’s finite. There’s a certain number of people on the planet at any point in time. Computers can count them, right, very quickly and probably count them real time if they have the information. The relationships between those people, they’re finite. You can measure them. You can ask people like we think these are people you might know, do you know them? Yes I do. No I don’t. That’s interesting, but again – huge in scale, but finite. The interest graph resides over top of the social graph and it’s this concept or theory that we could map out the relation between ideas, objects and people. And that’s really interesting to me and that’s a place I think Pinterest has a unique place to play in within the context of the other technology companies. What’s interesting about it is that the hallmark of humanity is every day we make new ideas, objects and people. So the interest graph is infinite. And if you had a technology that could map it out, so it could tell you which ideas are related to each other, which objects are related to which ideas, which people are interested in which ideas, right. You can do really cool things that are technically cool, but actually experientially cool. Something that actually would make people’s lives better which is most of the interests I have today are limited by what I’m exposed to. Like I only can know about the things that I know about, but if I had a technology that could tell me about something happening on the other side of the planet – in Germany there’s a new type of music scene happening that I don’t know exists today. I don’t know anyone who knows it, but if the technology knew that I would love it and could tell me about it perhaps I would discover it years before I would naturally. Or perhaps I would discover it when I wouldn’t ever have discovered it on my own. And I think that’s really cool.
What Pinterest does nicely in that is that everybody is building this sort of interest graph rocket ship, this technology, but the fuel actually makes a difference. So on Pinterest today when people put things together in collections or boards they save them – it’s them. It comes from them. They’re saying something semantically about the objects that belong together to them and have meaning to them. They name that board of that collection and that fuels 100% pure interest. It’s really what I care about. Whereas other companies may have fuel that’s somewhat representative of that. It may be representative of a lot of other things. It may have more impurities into it. So I think Pinterest is uniquely positioned to sort of deliver a consumer experience on interest graph that I’m really excited about. I think it’s a really interesting concept and so one of the reasons I came here was discovery system, interest graph, recommendations. Like this is a pretty interesting moment in time for that sort of technology and one that actually has a real human experience on the other side.
Steve:I love that you ended with the human part. Like that was the last thing that you said in this really amazing kind of romp through some big ideas. And I was sort of reflecting as you were talking that, you know – I mean we’re having a conversation today about research. That’s sort of the context and you’re talking about research, I think, I don’t know, in a unique way. I don’t know if you feel that. I’m kind of laying that on top of you. I mean how is this a research conversation? I’m not forcing you to defend it, but let’s just go meta, like why are we talking about this on Dollars to Donuts?
Gabe:Well I will say that for me when I think of research I think of it as a discipline. Like as a blue-collar type discipline. I don’t think about – it’s a set of practices that you can learn and I think those set of practices can be applied to certain businesses or contexts. And so I don’t think of myself as a – I don’t believe in like super empathy powers. I sort of react negatively to that. I do think research is one way, or one discipline that can contribute to product development, design – service design. There are many that can do it. It’s not necessary to do it, but it can be useful. And so it’s a conversation about the context that research can play in, but also just about what are the duties, the set of practices – what are the things that researchers might think about in terms of their contribution and what they bring to the table. Great researchers I think they really know the sort of on the ground, in the trenches, skills, tools, things they have to do to move forward with the group, but they also have to play this unique role where they pop out and get the bird’s eye view or the broader context and thing about that, but not in a way that distracts the rest of the team. And then they sort of port back down to like the blood and the guts of what we’re doing every day in the trenches to figure out how can I take those observations that are abstract and make them useful for the team and actual for the team. So all research conversations for me, like sort of waver between the two, right. They’re up in the air, really philosophical, abstract, but then also like down on the ground, really practical, really technical. And so that’s probably why we’re having a conversation about like how did you choose where to go next and also some of it was like well I wanted to be in a place where they were going to care about the work that I did, but also there was this huge like theoretical thing that I could learn about in the context of their products.
Steve:And moving back and forth between the tactical to blood and the guts and the big picture, philosophical. I don’t know, I’m guessing that – and just sort of watching your face as you talk, that not only do you like those end points, but I feel like, and maybe this is my own thing coming in here, there’s something about – and this I guess is the journey thing coming back in as well, but there’s something about the motion between those two kinds of thinking that I think for me sparks some stuff. That we can talk about the big picture – I mean you can really talk about the big picture, but sort of the energy we get from that vibration, from the well what does this mean, how do we deal with the blood and guts kind of – with the team every day. That – I don’t know. I’m throwing that out there. It’s a very leading question for you Gabe, but…
Gabe:I think there’s certainly like personality differences that attract certain type of people to certain roles, right. And so there’s probably something about that that appeals to folks who are doing research for sure. I also want to make sure to like be clear which is it’s really important to me that as part of that process we’re shipping something. That I can actually see something in the world that I’ve contributed to. Otherwise I would have then stayed in academics. I would have been working at the theoretical level and for me there was something really exciting about work I did at IDEO where I worked on a furniture project and I’ve seen that furniture in the world many, many times. That makes me feel like there was – it’s not just about ideas, it’s actually about building something and doing something. And that’s I think important for me when I hire researchers or when I think about the type of environment I want to be in. It has to be – it can’t just be interesting. It has to be impactful. It has to be action oriented. There has to be something that happens at the end more than an idea. And so that to me is like I like that fluctuation between the tactical and the abstract, but to an end, and that end is to put something into the world that makes it better for people in some way, whatever the “it” is.
Steve:I really like – I agree with you that stuff being interesting is sort of not an end to – it’s really great to be sort of turned on by stuff and like getting new ideas. You know for me trying to put it in a language – I mean what’s the good and bad word now – the A word – actionable. That you know – I don’t know, people go back and forth on whether that’s a good word or a bad word, but I mean I guess in my own career I had – the ship had sailed a long time ago on like measuring my own success by what shipped. And I hear from researchers as well that sometimes it’s frustrating. That – and so I’ve just sort of shifted my own role. Like I’m in the business of helping people see as clearly as possible what decisions they have to make and what actions they have to take in order to ship something, but I can’t – and maybe this is because, you know the difference between being inside and outside of the organization, I can’t measure my own success by what gets shipped or I would just be an alcoholic. But I do measure my own success by – I mean someone said it to me once like oh, what you do is kind of like take someone to the doorway and kind of like open the door and like point them kind of on the way through.
Gabe:I’m probably using like very tacky language when I talk about ship, but I – so to be clear, shipping something can be pretty broad. What I mean is there was a decision made. That there was some sort of result or outcome of the research. And so definitely some of the hardest things I’ve worked on were the shipping moment was convincing someone not to do something. Like that is just as important as convincing someone to do something, in fact it may be harder in some cases because they’ve already decided, no. No, we’re going to do this thing, we’re going to make this product and helping them to understand how that might be a destructive decision for their business. That’s a type of shipping, right. Shipping a decision. So yeah, maybe I’m overly emphasizing the analogy.
Steve:No, I think I’m just one sort of rung before that which is like I think for me I probably ship influence. I empower people to make a decision, but I can’t actually control if they’re going to make that decision. So sometimes you sort of – you learn something, you package it, you convey it and they need to do something. They need to do something. Like the business is just – the world has changed, the business has changed. They need to do something and the other factors are just so large that sometimes doing nothing is the outcome, but you know that you changed somebody’s mind. You gave them – and I’m not trying to undersell what I do. I mean I’m trying to be realistic like I just don’t get to ship decisions as much as I would like to even. I’m going to go drink after this. That’s what you’ve taught me today.
Gabe:Well as long as I’m going with you it’s fine. I think – so one thing you brought up is control. Like I personally have had moments of development where I do not – I try to focus on what’s in my control. I can’t make a team do something, right. They actually have to make the decision. That’s a healthy part of the process and sometimes they will do things that are risky or potentially counter to the recommendations. And there’s a whole host of reasons why that might be. I try to recruit people and hire researchers who are resilient because I do think a huge part about being a great researcher is going out on a limb to say something important, potentially getting knocked down, and getting back up again and continuing to say that. I think really great researchers, in tech especially, they act like it’s a sprint, but they know it’s a marathon and often times at Pinterest we talk a lot about like the research part and delivering that, that’s just the first step. Now the real work is continuing to bring it up with the team. Continuing to make it part of conversations. That takes a long time sometimes. Particularly like if it’s something very meaningful or hard to do. People just don’t, when you deliver like a hey this thing you thought was the case for many years is actually potentially not quite like that. It’s a bit different and you should really think about it differently. Human beings are terrible at that. Like we don’t do that in our personal lives. We don’t just shift our total point of view without some sort of major event, like some sort of like peyote, or some sort of like near death experience. That just doesn’t happen. And so it’s not going to happen at a company or a business and you have to think about that sort of marathon bit, like bringing it up over time. It’s going to take awhile. You have to work at it and you can’t just think you’re going to come in and say no I have the answer, it’s this, and everyone’s going to be like here, here, good job, let’s do it. It’s harder than that. I just think that’s part of the research gig. That’s part of shipping information and inspiration, right. That inspiration bit. It can be very transformative for a designer in a moment, but getting to the long term outcome of that, the result of that usually is isn’t a couple of days. It’s probably a longer period of time.
Steve:So tactically when you are meeting prospective research hires for Pinterest, how do you – that’s – and obviously you look at many things, but how do you look for that?
Gabe:That’s a good question. I think first of all – how do you look for resilience. I think you talk to them a little bit about their career and I ask people to tell stories about the research they’ve done. You ask them to tell stories about, you know, what’s an example of a project that you did that you think showcases your skill set and really shows the best of you. And then you ask them questions about – they talk a lot about doing the research, then you actually have the real conversation of like so what happened next? What did you do? How did you approach communicating that to other people? What were the challenges you faced? And really – because we want to talk about the researchers, researchers, but again I just feel like that’s the smaller part of what we do in a context of an organization. ‘Cuz that’s what we’re good at and we like and we like that and we want to talk about it and we want to make that the big part, but like great – this is like you can build it and they don’t have to come sort of moment. Or even – I heard this great quote awhile ago which was “you don’t have to make a mistake to lose the game” and that’s about context and timing and sometimes like you have a great insight for an organization but the timing is not correct and it’s out of your control and so then what do you do? Do you pack it up and go away? Do you like start making posters and put them up every month? Do you continue to like any time anyone brings up tangentially relation, like oh that’s like what we found in this research from before, are you aware of it? That’s like part of the story of research and try to focus a little bit on that in conversations.
Steve:Is that something you are teaching researchers that you hire? Or does someone have to come in?
Gabe:I don’t think of myself as a teacher. I did not enjoy that part of academic life. I think of myself as a collaborator and I think it’s something we discuss a lot. How are we doing this? What are we – you know deliverables – what’s the nature of deliverables? Trying things – like the one thing I do try to ask the team to do, even when they’re tired of me and of the challenge is like well what’s something you could just prototype? Let’s just try to do something very different. And I think the team itself has built a good sort of muscle around like let’s try it differently to see what happens in terms of communicating and continuing to push those conversations forward. I do think it’s hard at a startup like Pinterest. When I joined I was around – I think there were 3 people in my class who are all still here – or my whatever, my cohort. We were like 70, 71, 72 – pick your number. Now there are 700 people and I think what’s really challenging with research is that you can learn something and communicate it to the entire organization and everyone can hold that knowledge. In the case of a startup like Pinterest where you’re growing – like next week 20 new people are going to start. And those 20 people they don’t know anything about that. And so you have to continue to make sure that people understand – they’re going to come in and say like hey how come we don’t do this and it’s easy just to say like well we tried it and it didn’t work. Or well it’s not going to work. Or, that’s easy. The harder bit is like actually here’s what we’ve learned and here’s why we think that may not be the best way to proceed or move forward and so – that’s a real – like it’s an ongoing challenge of any like company that’s growing quickly, but particularly it hurts researchers because the artifact of what you’ve done is knowledge and knowledge you either have to read it, or communicate it. And so figuring out how to like maybe put it in a space or you create a wiki, but then you’ve still got to get people to go there. It’s a real challenge I think for any researcher at a fast growing company.
Steve:Right. There’s a lot of talk about – you said at scale earlier in this conversation. That you know dealing with insights at scale is a challenge as companies grow and so on. But you’re talking about sort of insights, like insights in the tornado, or insights when the sands are shifting.
Gabe:Yeah.
Steve:You can have a great meeting – you can – you know all the good things we do in research. We talk to our stakeholders. We bring them along in the process. We bring them out to meet with people. We involve them in synthesis. But over that – say that’s an 8 week process, that team has changed and then two weeks later, after you’ve had some really great meeting to make some decisions and have a workshop and do whatever things that we do. Two weeks after that there’s a bunch of new people and two weeks after that there’s a bunch of new people and two weeks after that.
Gabe:Yeah and also the actual, you know the audience. Like perhaps your insights are stale. The audience has changed significantly, right. Or perhaps the technology, right. The technology itself, there’s like a new innovation that you know we have a new team coming in that’s been acquired and they have like a whole new technology they’re introducing that can really change the context. And so again this notion of in a startup land a research has to again believe – act like – know that it’s a marathon because there’s so much changing and you’re going to have to really stay abreast of all of that and check your own assumptions, check your previous recommendations constantly so that you are really providing the best information you can at that time.
Steve:That’s a really good point and I think that’s sort of the researcher arc, right. You come into a situation. You can tell that everybody’s hypotheses are full of crap. And you’re like okay I don’t know what the truth is, but I know that you’re wrong and we’re going to go learn and you come back with a truth, or a set of truths or a set of things that approximate truths and that you advocate for, but that also – then you could risk becoming part of the problem because now you can kind of clench on those things that you discovered through your own sweat and your own effort in the field, but what you’re saying is those things may have a short shelf life given the changing space that a company like Pinterest is involved in and that you have to let go of those things which is sort of counter to the you’ve got to go – the marathon is kind of about persuasion and advocacy and influence and getting out there and giving that same presentation 20 times to different audiences at the same time as also be willing to let go of it.
Gabe:Yeah, and I think – so there are – I think that is a challenge – a real challenge. I think there are some ways to guard against it, or some strategies. So one strategy at Pinterest, I – my vision for the research team was an interdisciplinary team. Trying to put together both qualitative user experience researchers, of which there are different kinds of qualitative user experience researchers. They’re not all the same. A quantitative user experience research group, market research. Altogether. There’s an economy here in terms of like actually getting things done, but there’s actually, like what it does nicely is it brings together different data points to build the knowledge that we’re learning together.
Steve:Is that knitting?
Gabe:That’s knitting within our own research team right. I think a lot of companies you can have, like those groups can get siloed early on and then you have that moment years later where you’re like well we’re working on the same thing, what’s going on? Or we’re competitive with each other. Trying to put those disciplines together and then what allows – it’s like so work that I’ve done in the past that I still think is relevant, well if it’s still relevant we’re probably still working on it and now we work on it with a perspective of multiple methods. So mixed methods. This is a way to sort of take an idea and really, truly understand it deeply because if it’s – we think it’s – we hypothesize something from the qualitative research then we can hand it off to a quantitative researcher who can then quantify that at scale of which you might learn something that is – you think there’s a correlation between a macro question and a whole bunch of smaller questions. Well if you validate that relationship then you can take that macro question and you can put it in all of your say brand equity trackers for market research and all your markets and then you can measure it there and now what you’re doing it you’re building this interlaced qual and quant sort of lineage of knowledge that over time can become very robust and then you can say now that you’ve validated it at scale and you’ve used multiple methods and you’re monitoring it in each market, well now you can say we know this is a real thing, we’re trying to change it. We can continue to measure it over time with a brand equity tracker and market research to see like are we doing better, are we the same or are we doing worse? That’s super exciting and it’s a way to guard against your own bias, like this is true. Maybe when you get to the step of handing off to a quant person – like no this is not true, it’s scale. Or it’s different – it’s often probably true, but it might be different than what you thought it was. So there’s actually three things going on here, not just one. Or you find out when you pass it over to a market researcher who does brand equity or some sort of large survey, they’re like oh this is true for, you know, Western European countries. It’s not true of Pacific Rim countries. And so – again, bringing in more methods, building out – if it’s an important question, building out different levels of understanding can be really robust and guard against your own bias towards – but I said this before and it was true. It’s still true.
Steve:So in terms of this multidisciplinarity and you’ve kind of said this how you sought to build the team, maybe you can describe the team more specifically and where are you at? Where’s the team headed?
Gabe:Yeah. So we have three qualitative user experience researchers. We have Altay Sendil who comes from IDEO. He has an industrial design background. He was at IDEO as a researcher forever. He brings like a really strong perspective on ethnography and really generative work. We have Larkin Brown who comes – she was previously at Google and she’s like – to me like a real digital UX person and she has a lot of areas of focus. She’s also really into – she serves as a stylist within Pinterest and helps with brand styling and also communication to press around fashion. And then we have Cassandra Rowe who’s – she worked at, previously at Intuit, but really did a lot of international research at Netflix. She was part of their early growth on the qual side. So we have an expert in sort of formative, an expert in digital, an expert in international. Then we have one quantitative UX researcher named Jolie Martin and she is really looking at the intersection of attitudes and behavior for Pinterest. So that’s experiments and – AB experiments and surveys, to truly really understand the quantities and quality of experience. And then we have Paul Pattishall who is on the consumer side doing market research for us – things like brand equity trackers, product surveys, NPS, category deep dives. Like what are the most common interests in France right now for people who aren’t using Pinterest so we might approach them with relevant content. And then recently had Julia Kirkpatrick who joined, who’s doing market research for the business side of the company – Partners Products is what we call it. That’s the people, that’s the disciplines. Again, the important bit is that we’re all working together. We’ve hired people who, they work with a team in an embedded type way, but they also know when – when the team has a question that’s outside their skill set to go to a practitioner within the team to get help and support. We also, you know we’re still – so that’s six people now. We’re adding two more people this year. We’re looking for another quantitative user experience researcher because I think one person is not enough. You have to have two to really be a team so you can play off of each other. And a UX or qualitative researcher to focus on our business or Partners Products. So these are things like the web analytics if you own a domain, or advertising on Pinterest which is something that’s a new area for Pinterest where we can do some really innovative things because of the visual display elements of Pinterest. So we’re adding folks. We’re looking for folks, but the important bit is all those disciplines working together – we’re one team, but you could think of it as three separate teams. But because we’re connected there’s a lot of collaboration and interaction amongst the team members which I think is really – it’s an exciting environment for me as a researcher because people know a lot and you learn a lot and you can pick up a lot of tips and tricks along the way.
Steve:So just thinking to wrap up, as you describe the team and you talked before about – I used the word teacher and you kind of shied away from that term. You said you think of yourself as more of a collaborator and given the nature of how that team is structured and kind of what you have created as a collaborative culture and how you’ve inherited kind of – you’re participating in the larger Pinterest culture – I agree with collaborator. And this is just all a preamble to a leading question – the trouble with interviewing somebody that you know a little bit is that there are certain answers you’re trying to draw out of them. So I’m going to try a few different ways. What – you know in terms of your role as a manager, if it’s not teacher, what would the people on the team, what nouns or adjectives would they use to kind of characterize your managerial jim-jam?
Gabe:That’s a good question. I mean this is my first time managing. So I think my jim-jam is under development. What I strive for is one where they feel like supported. That they – I will – if people ask me for feedback I will give them deep feedback on whatever it is, their deliverable they’re creating because I know in this environment you might not get deep feedback from other people and I think if you’re doing work you deserve to get that feedback. So supportive, probably into the – like I care definitely a lot about research and what we’re doing. So I talk a lot about research. At lunch I’m like the person who’s like, answers the question about what they did this weekend and then follows-up with the like hey let’s talk about this project that’s happening. And so I’m – they would probably say I’m pretty focused on research. So supportive, into the research. I definitely want to empower them to make decisions. So we have a model where they’re working with teams and they are the ones that need to develop the roadmap with their teams. They need to make the calls. I’m there as support. I can provide context. I do a lot of like I – like showcase their work and constantly talk about their work because that’s part of my job is to like hey did you know that Cassandra just did this in Germany with that team, the activation team. You should be aware of that work. That’s a huge part of what I do, connecting the dots at an organizational level. So supporting, you know advocating. Nerding out on research.
Steve:What’s your superpower as a researcher or as a manager?
Gabe:I don’t – that’s a good question. I tend to like shy away from superpowers because I think it’s about like training and practice and whatnot. As a manager, well I don’t think I’m there yet. Like I haven’t done this long enough to really know that about myself. And I have a great team. Like I think I have a lot to learn and a huge part about managing probably is like managing through hard times and I probably need some more of those hard times before I really can assess.
Steve:Careful what you wish for!
Gabe:I know! As a researcher, I mean I care a lot about the work that I do. I think that’s like a double edged sword sometimes because sometimes you have to care a little less. But I think I – it’s a good question. I really – the times I’ve thought – I’ve thought to myself well like I’m pretty good at this thing, but then I’m like that’s just an illusion, right. That’s like I don’t have a special power. I just have a set of practices and things that I try to learn about and do. And so I’m hesitant to answer that because maybe there are no superpowers. Maybe it’s just about hard work and just caring about the details.
I can tell you what my weakness is which is I don’t tend to go to conferences enough or get out and socialize enough in the research team. So when someone comes to visit me, to talk about research, or who’s a researcher, that’s huge for me ‘cuz that’s how I try to like understand what’s happening in the world. That’s a moment for me. That’s like my conference. That’s my getting out and interacting with other folks around research and so – ‘cuz I’m pretty heads down. I’m pretty in the trenches, right. And so that’s why I definitely celebrate those moments and I really appreciate when researchers come visit me. I have sometimes students come by who want to talk about their future career or get more information. I tend to like accept more one on one conversations than really approach like group activities, as a way to just get a sense and reflect on what’s going on in the field.
Steve:Is there anything that we didn’t cover in our conversation today? I know we could go and go.
Gabe:We could. I have a lot of things to cover, but I’ll leave it for another time. I do have some questions for you though.
Steve:I don’t even have to ask that!
Gabe:No, I’m like ready to go. So my first question is…
Steve:Oh your first question!
Gabe:My first question is like what’s the biggest surprise – what’s been most surprising to you in terms of the Dollars for Donuts interviews you’ve done? What stands out as surprising that you didn’t expect before you started this?
Steve:There’s probably surprises in terms of just having these conversations and there’s probably some surprises about just having this as a thing that I put out in the world. But I also – it seems I push back on the notion of surprise. I think I wrote about this in my book even where I used the phrase bland curiosity. ‘Cuz as researchers you get asked the what’s the biggest surprise question all the time and I feel like well that supposes that you entered into it with something and then like a bucket of water appeared and splashed you and you went oh…
Gabe:Okay, this is good. Let me reframe then. What has been the most unexpected theme across these interviews?
Steve:I haven’t synthesized them yet. I mean part of – and you know this is sort of the discipline of research like – and you can do it different ways – you can kind of just go and throw yourself into it and then occasionally you’ll be like oh we haven’t seen any themes except blah, blah, blah, blah. Oh that’s a theme. And so I’m just going to keep thwarting you Gabe. I mean sometimes I resist reflecting and just sort of let it be what it’s going to be. So I mean there are things that come up, but also it’s because I ask about them and then – you know how teams form? What people’s career arcs are? These are sort of the topics that I’ve chosen to talk about, so I don’t know that they’re surprising themes. There’s something that we didn’t get into a lot – there sort of a structural issue that’s emerged from the beginning which is – which I remember this being like a design topic from the 90s and probably goes back before then – where do these people sit? Do they sit with their – are they centralized or are they embedded? So it comes up a lot and I think it’s a really interesting sort of managing – what I like is that no one says this is the right way to do it. People have said like this is what we’re experimenting with now and here’s the tradeoff. So I guess that’s a thing that I would never have thought to ask about and that’s also because I don’t have a job inside a company that has a bunch of research people, so it’s outside my own experience. So I’ve learned about that as a topic and it seems to be one that when I talk to people they say like oh yeah, that discussion and this episode was interesting and we’re thinking about how we’re going to do that.
Gabe:Yeah I think that – I mean I have that as a listener all the time. How are they doing it? Why are they doing it that way? Absolutely. Okay, speaking of being a listener, my second question is will there be a Season 3 of Dollars to Donuts? (laughs)
Steve:We’re going to do it in Oculus. We’re under negotiations right now. Podcasting is over. Maybe it should be – maybe they should be something like – something that’s only – they’re going to be 4 seconds long and then they’re going to disappear.
Gabe:I won’t put you on the spot. This is a more serious question. As a young lad growing up in the cold tundra of Canada during the 70s and 80s, who appreciated donuts, there were 2 major chains. Which was your preferred chain – Country Style Donuts or Tim Horton’s?
Steve:Well I’m sorry. We grew up in the same town.
Gabe:We did.
Steve:And I’m older enough than you that I remember Millionaire Donuts. Do you remember Millionaire Donuts?
Gabe:No, I don’t. But it was the donut of choice?
Steve:Oh I don’t know. I mean Tim Horton’s was kind of the place, ‘cuz Tim Horton’s used to make their donuts in the store. Maybe you’re the one that told me this. They don’t do that anymore. I even went for a job interview at Tim Horton’s as the shift – I didn’t get the job – it was like shift work. You had to make the donuts, like 3 shifts a day. They had these big racks in the back where they would – it was an interesting sort of – I mean I didn’t get to do a lot of research, but I saw how those were made, which maybe everyone knows this, but they like have all the sort of naked donuts there and they like kind of pour over like a big vat of chocolate goo and that’s how they get that sort of drippy thing out their – they just pour it over on these racks.
Gabe:So I don’t think we can really reach a higher point then naked donuts as part of our conversation today, so it was like…
Steve:Are you – not only are you taking over the interview, now you’re wrapping up.
Gabe:How can you beat naked donuts?
Steve:Well there’s a visual that no one needs… Alright, I think that is our – I’m going to take your cue…
Gabe:Thank you.
Steve:… and wrap us up. Thanks Gabe, this was great.
Gabe:Awesome. Thank you.
The post 11. Gabe Trionfi of Pinterest first appeared on Portigal Consulting.
This episode features Gabe Trionfi, the Manager of Research at Pinterest. We discuss the evolution of user research, collaboration between disciplines and the journey versus the destination.
Great researchers really know the on the ground, in the trenches, skills, tools, things they have to do to move forward with the group, but they also have to play this unique role where they pop out and get the bird’s eye view or the broader context and think about that, but not in a way that distracts the rest of the team. And then they port back down to the blood and the guts of what we’re doing every day in the trenches to figure out how can I take those observations that are abstract and make them useful for the team? So all research conversations for me waver between the two. They’re up in the air, really philosophical, abstract, but then also down on the ground, really practical, really technical. – Gabe Trionfi
Show Links* Gabe Trionfi * Follow Gabe on Twitter * Pinterest * Joe Namath guarantees it * Joe Montana goes to Disneyland * Failure to Lunch * June Jo Lee * Michael Graves and Target * Tiffani Jones Brown * Sherlock Holmes * Evan Sharp * Ben Silbermann * Social graph/interest graph * Altay Sendil * Larkin Brown * Cassandra Rowe * Jolie Martin * Paul Pattishall * Julia Kirkpatrick * Burlington * Country Style Donuts * Tim Horton’s * Millionaire Donut Shoppe
Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes.
TranscriptSteve:Let’s start with the introduction. Tell us who you are, what you do, where we are today.
Gabe:Yeah, so my name is Gabe Trionfi. I’m the Manager of Research at Pinterest. I’m a researcher. It’s what I do. It’s what I want to do. It’s not the journey, it’s the destination for me. I’ve been doing research for about 10 years. I’ve been at Pinterest for about 3 ½ years now and we’re in the office, third floor, new headquarters for Pinterest. And we’re in Alberta, a Canadian province theme named room.
Steve:Very appropriate.
Gabe:Very appropriate.
Steve:We have two Canucks talking research.
Gabe:Absolutely.
Steve:So you made an interesting comment about it’s the destination, not the journey.
Gabe:Yeah.
Steve:So could you explain that?
Gabe:Yeah, I think for me when I think about research it’s my career. It’s what I want to do and I always want to be doing research in some way. I have the opportunity now at Pinterest to manage a team. It’s my first time managing a research team. I’m always going to want to work as a researcher in some way. I’m not looking to become a PM or a head of product. I want to be involved with research. I also think it’s interesting to think about a career path. Like I’m looking for a positive trajectory for my career, but whether it’s being a manager or an individual contributor, that’s like not the most important thing. The most important thing for me is are we working on something that’s interesting? Is there an interesting research question to be have? Is there a product that people are going to be using? That’s really important. Are we shipping stuff that’s making people’s experience better? I want to be a part of that.
Steve:But some of that isn’t research.
Gabe:Absolutely.
Steve:Some of that is the thing that research informs. In the context in which you’re doing research you want to have a good question, but you want someone to make stuff out of it.
Gabe:Yeah, so I think sometimes when we talk about – in design or creative disciplines like writing and research, people are like well you’re a manager now. Are you always going to be a manager? Or does it mean less if you are not the manager? I would much rather be – go forward and have a role where I was a researcher on something interesting that people were using vs. always choosing to be a manager on something that people weren’t using or was potentially not that interesting from a research perspective.
Steve:You mentioned to me before, outside this interview, that you have a history in theater. It’s almost like you want to direct, but you don’t want to give up your acting jobs?
Gabe:Yeah. Yeah. Or it’s all part of acting. It depends on how you think of it. I think it has – there are some really interesting parallels. So I had never managed a team or built a team. At Pinterest I was the first researcher. There was like an opportunity where they said do you want to build a team? I said I want to try that. I didn’t know what it would be like and there’s been a ton of learning. Obviously when you’re first out of the gate you don’t know enough to be great, or even good, but you learn over time. The one thing that was interesting, taking it back to theatre, was I found that building a team reminded me a lot of casting a play because when you’re really putting together a great research team it’s not just skill sets. It’s not just experience, but it’s also understanding how will these people interact with each other when we’re building knowledge together? How will they be perceived as a group, as individuals? It’s like casting a play, like how will this actor relate? Or to this other actor that we’ve already cast? Do they have chemistry? Do they represent the relationship of the characters well by the way they look and what they sound like? And that was really surprising to me. It’s been a really fun part about building the team at Pinterest.
Steve:I want to follow up on that, but I also want to go back. Let’s just go back to a few different things here before we lose them. You opened with this big idea about the journey and the destination so now I’m going to pick it apart until you wished you’d never said it to me.
Gabe:Okay.
Steve:Because I – I mean I feel like you’re describing problems that are sort of journey problems not destination problems if that’s a way to slice it. Like you cast a play, you can be expert in it and you can know all these actors and know what their strengths are and you can do the right kind of – what’s that audition where you see people play off each other. Like a screen test or whatever that is. You can do all those things, but it’s still an emergent thing. You don’t know.
Gabe:Yeah, I think for me I think of research as a destination as quite large. That there’s many things to be learned. I still think of research as early days, particularly in the context of technology and start-ups. And so I think it’s fast and deep and interesting and sometimes I may talk with people who seem like – well this is how I’m getting into tech. So I have a social science degree and a little bit of background in psychology. I’m going to be a researcher for awhile, but really the destination is like to be a product manager or to be a VP or head of – and for me the actual research part is so interesting and there’s so much to learn and do. I definitely feel like it’s the beginning and that’s ironic of course ‘cuz you’ve been doing this for much longer than I have. When I worked at previous places like IDEO, they have people who have been working in research since, you know, the early 60’s in the context of…
Steve:I have not been doing this since the early 60s. Just for clarification..
Gabe:But in the context…
Steve:But the field…
Gabe:Yeah, the field goes back. You can even go back to World War I or World War II cockpits in air planes. The sort of human factor, early ergonomics. That’s sort of the seed of a lot of this stuff. ’63 is when the Royal Design Conference, right, in England – the Royal Academy of Design – they start talking about research for the first time. Engineers start talking about design and research and really what it means. So it’s a discipline that the idea has been germinating for awhile, but I think in practice every day it’s still fairly new, especially when you get outside the context of physical products and service design and you get into like pure software technical. The majority of people I work with here have never worked with a researcher before. And so that’s a challenge because you’re sort of still at the beginning and sometimes you can do really incredible work and people may not know that it’s that good because they actually haven’t worked with anyone before. And that’s sort of humbling and interesting and it’s a challenge at times, but it’s also a huge opportunity. So that’s interesting. I often talk about research being early like – you know the difference between Joe Namath and like Super Bowl 3 I think – who’s like a big time football player. He is the quarterback. He’s sort of glitzy and everyone knows him. He’s the underdog. He says they’re going to win. They win. He’s like celebrated on a bunch of magazine covers. But it’s like in the 60’s, the NFL. So like he goes home and has a Schlitz or something, right. Like he kind of has a regular life. Fast forward like to the 80’s. It’s Joe Montana, right, for the San Francisco 49ers, who wins a bunch of Super Bowls. He then – he’s a big deal, like Joe Namath, but like he goes to Disneyland afterwards and he gets like a lot of money and he’s super wealthy and commercials and whatever it is. The difference between Joe Namath and Joe Montana, we can argue about it as athletes, but the main difference is timing. When were they the quarterbacks of the Super Bowl winning teams and sometimes timing, when you’re at the beginning that’s just a constraint. It’s just part of being at the beginning and I think that research has a long way to go as a discipline and there’s a ton of opportunity, but like if you’re a researcher 10 years from now you’re going to be standing on the shoulders of schlubs like ourselves who had to do it in the early days. It will be way better then because people will understand it and also the market will shift in a way that will require research or something like research to inform product development. Because it will be more competitive. There will be more services and apps within the same space that need to differentiate from each other. So timing, where we are.
Steve:Where are we between Namath and Montana?
Gabe:That’s a good question.
Steve:Is there a visual continuum here and we could get the whiteboard out?
Gabe:Yeah.
Steve:Let’s just challenge that. A recent issue of the New York Times Magazine was all about work and so this article, this article, this article. There was one – it was really about people eating at their desks, but the side bar piece – unfortunately I don’t remember the name of the person, but the side that kind of goes with the article that supports this photo essay says so and so, an ethnographer, is like the first thing. And it was not an article about oh these people are like anthropologists in safari hats going into corporate America. They didn’t go that thing that journalists were doing 3 years ago and 15 years ago which is ooh, look how weird this is. They were really just – it was sort of incidental and characterized the fact that like hey there’s an expert who’s watching people eat lunch at their desks. To me the cultural moment there is like this is a thing that we don’t have to just point to it and go oh-ah, we can use it then to tell another story. So that seems we’re – I don’t know if we’re taking this person to Disneyland, but we’re not having Schlitz.
Gabe:Yeah, so I think this is an important point which is – like in research we often pay attention to context, so my context today is within tech, within start-up land. I think when we look at research trajectories in terms of its development as a discipline it actually tracks differently across different contexts. So I think when you look at physical products, service design, I think that industry what’s happened there, that collection of industries with research is far ahead of where we are within tech.
Steve:You think tech is behind?
Gabe:Well I think tech is tracing the trajectory. To me it looks the same and I think it’s headed in the same direction, but I do think it’s at a different point much like just the industry, the current modern industry, like post-Internet, apps, software design – it’s fairly new itself, right. And so with a lot of new industries, especially industries that have access to a lot of data, right, even the history of psychology itself has had moments like this where people think it’s all about this type of data, this quantitative data, right. That’s how we can – we’ll find the future opportunities within that and then we realize like well actually one type of data may not be enough. We need different types of data. And then we might switch and say like well I’m going to do this other thing, or I’m going to do this other type of data acquisition or method, right, and then really mature fields, they come to usually a realization which is like yeah there’s no one thing that’s going to do it for us. We need multiple things to try and understand complicated phenomena and then within that, especially within the world of business and tech, then you’re still going to have make a risky call because again none of this data can tell you the future. It can paint out trajectories and possibilities, but you still have to take a risk. I want to work at a place where someone’s going to be able to understand the data, triangulate and then make a strategic decision that’s going to lead to a better outcome for the people using it and the people working on it. So I think that when we get over – again, some disciplines, some industries in the beginning there’s things that seem easy that are a lot more complicated than they seem.
The example I always give is like the common thing is qualitative versus quantitative data which of course they both have subjectivity embedded in them. They both have trade-offs. They probably should be used appropriately which is what’s your question, then let’s choose a method that fits the question. But the metaphor I always say is like I would never hire a general contractor to build my house who is like I only hire carpenters who use hammers. There’s no screwdrivers on my site. It makes no sense because what should determine the tool is the rest of the context. So let’s not use a hammer on a screw. Let’s use it for nails and let’s use a screwdriver for screws. And that’s the same thing for me about methods which is it doesn’t matter what the tool is. What matters is the context and what is the appropriate design to get at understanding the question and also the outcome goal. What is our goal? What are we going to do with the answer? Those are the things that are important to me.
Steve:So you could start to sort of map this trajectory of research and tech and look at well what were the predominant tools – if the tools are a proxy for what questions are we asking, we might just have people showing up with hammers. I remember the point at which Microsoft got usability. Like that was going change – and this was I want to say early to mid 90’s and there was a lot of press about, like the physical rooms they were building. I think I went on an interview there and went on a tour there and you realized oh they’re just going to like run, like huge – I mean they’re going to do usability testing at scale. You know the word that we like to say in research or everyone likes to say in tech I guess, about at scale. They were really going to scale the hell out of this and sort of usability their way into something. And there wasn’t a lot of – I mean that was leading edge thinking back then. And I think as you’re describing this trajectory or this changing context it’s about the tools and the tools are the thing that we can see more easily. I don’t know what design questions were being asked of Microsoft, but I can infer based on the fact that this is where they put all their money. So I think you’re sort of describing a long time ago already, you’re saying now we’re like starting – we’re at a point where we need these tools and we need to be asking these questions. And so – I don’t know. Is that fair, the recap?
Gabe:Yeah. And also sometimes the context is like within software and apps the marketplace actually comes to bear on the types of disciplines you need to invest in to differentiate. So if we’re in a marketplace where everybody has a really high quality engineering team, everybody has a really high quality design team, and we start stacking the disciplines up and you’re going to enter in sort of the messaging as UI which is really popular amongst a suite of apps today, right, where you’re just – it’s a concierge service or an on demand service that is using the interface of like a traditional SMS app. There’s a bunch of them out there, right. They all have great designers. They all have great engineers. They have great marketers. They need to understand how to differentiate from the other 5 apps within their space and they’re going to have to invest in something. They could invest in partnerships with other companies. They could invest in celebrities, right – to sort of like build their awareness. They could invest in researchers to try and understand what is most meaningful about this experience that they’re building and what are the opportunities to separate themselves from the other within the space. So some of it is disciplines and methods over time. Some of it’s the market itself becoming more competitive and sort of more compressed in terms of offerings out there. Traditionally design has been used in recent years to separate or differentiate, right. And I think research follows design and if you don’t have a designer it’s really rare that you have a researcher because design is working on the experience on the front end of software that people see and think of when they think of an app or service. If you’re not investing in that you’re probably not investing in like a deeper understanding of your audience’s needs as a way to differentiate. But like the classic examples – like Target hires a famous designer to make like toilet bowl brushes that are really pretty. That’s a way to differentiate from others within their space who are smell – who are selling commodities like that. I think research can – will and is having the same sort of role within tech as more and more apps and start ups start to fill into the same spaces. It’s not about building the infrastructure, say plumbing the Internet with social connectivity. It’s about like okay there are 5 financial service apps now that are redefining the loan space, how are they different from each other and how can we ensure that ours is the most relevant to our audience? Let’s get a researcher in here to help with that.
Steve:I think what I heard you say though is at a strategy level there’s a lot of different things that we could do if we’re trying to differentiate our financial app. So we could try to get Leo and his posse to use that. Or whatever sort of way to generate attention. We could have a controversial CEO that makes outrageous comments about people. Or we could change our pricing model. Or we could – there’s a lot of things that don’t necessarily come from a nuanced insight about the world. It’s just sort of differentiation for its own sake. Or there’s a strategy there and one of those out of – let’s say there’s 20 obvious ones – and I can only come up with 2 right now – I’m just parroting you back. So one of those might be let’s invest in experience which means investing in design, investing in research.
Gabe:Also writing. Writing is a similar discipline to me than research within tech now. It’s newer. Usually it’s working across many different teams within a tech company usually. It’s critical, like what is the voice of your brand, but also of your product itself. And I see a lot of – I work with a lot of writers here. Tiffany Jones Brown started as the lead of the writing team here, has since become a Creative Director. She has a great team of writers that I’ve worked with before. Building a relationship with that team, I feel just a commonality or similarity in terms of where writing is as a discipline. And the – if you invest in writing it says something about your strategy when you have a tech oriented app or software service. It says like hey we want to make sure that people feel differently about our brand or about our product. And you see a big differentiator in lots of tech companies now in terms of how do they talk to their users through their product. And so again it’s just another strategy that you could invest in. And research, and a suite of different research disciplines is part of that. Also data science can be part of that. Business analytics can be part of that. Ways to, you know, bank capital, not just monetary but like understanding. Like you can bank a lot of knowledge as capital to inform your strategies or to help move your product forward.
Steve:If I take a naïve look at what you’re describing I feel like there are some disciplines that are output disciplines and there are some that are input and by that I mean, you know, writing is crafting the words that go on your website. Engineering is the code. Design is the pixels and every designer – everyone is rolling their eyes at that. That’s a really, it’s a very broad brush, but research is an input to decisions. Business analytics is an input to decisions. I don’t know if that’s a fair – if it’s that binary or not.
Gabe:I think about it similarly. So I think about it as a suite of disciplines that are about indirect influence. So I’m a psychologist. I’m not going to build something with regards to the software that’s going to get into the world, but what I’m building is understanding or knowledge, but also sometimes creating experiences that are inspirational. I talk a lot about information and inspiration as part of the research role that is going to indirectly influence that thing that enters into the world. Sometimes there’s a one to one. You can just say like hey that decision there was based on this work over here. Sometimes it’s much more diffuse in that the whole host of teams that are actually contributing to the output – the most important thing is I want to work in an environment where research is a part of that, whether indirectly or directly, and it’s not something that’s on the side or separate. I often talk about like Sherlock Holmes and Dr. Watson. Like I’m very happy to be Watson to someone’s Holmes because I think that’s a real role and those Holmes stories wouldn’t be the same without the indirect influence of Watson, right, sort of guiding, informing, being a foil for Sherlock as he goes ahead and builds the solution to the mystery. And I think indirect influence is really interesting. Really hard to do. It – sometimes it’s easier just to like come in and say we should do this and then people do it, right. But to sort of like change people’s perspective on something they think they know very well, but actually might be very different outside of their own social circle or their own sphere of experience – let’s say on the other side of the planet there’s a different culture, different country, a different type of person who’s using your product and perhaps they just have a different mental model that you didn’t even know existed. Bringing that back, but also not just a shock, but to potentially influence, like oh hey perhaps we should think about this a little bit differently. Or perhaps it’s another way to think about it, that there’s not just one way to think about our product, there’s actual multiples that we should keep in mind. And that’s a good thing because it means the product is relevant to lots of people and complex and so it’s not just a simple like P2P commerce, like I’ve got this thing, do you want to buy it? Yes you bought it. Cool, transaction, safety, all that sort of stuff. It’s more complicated than that in some cases.
Steve:So I think research has suffered from the sort of the naïve Dr. Watson – well not that Watson himself is naïve, but if you take a shallow look at those stories Watson is merely the chronicler. He’s the sidekick. He doesn’t do anything. He and I – you know, you do the close reading that you do and you see what he enables and how he kind of supports and I think that’s the beauty of those stories. You kind of go at them a little bit deeper and you realize there’s a complicated thing going on and that it – the more you invest in learning what those stories are about the more they kind of pay off. So just to bring it back then to research – research has maybe historically been viewed at the way we naively view Watson, that research is just asking people what they want. Sort of chronically things and reporting them back. And research as you’re articulating it, and of course I agree, has the potential and does more often now play more of that influential, maybe Holmes is the one that gets brought to Disneyland. I’m going to crush all metaphors together. Like Holmes is the one that gets carried on the shoulders but the researcher is – ugh, I can’t untangle this here. So I guess I’m trying to ask in a really twisted way here is about you know what are the things that you’ve seen or that you’ve practiced to give you the chance to play Watson, to enable things, as opposed to being relegated to a mere chronicler?
Gabe:So, I think this is an important question. So for me I had one experience. My sort of first design research job was at IDEO and IDEO is sort of at one end of the scale design thinking to an extreme. Huge opportunities to do different sorts of work and to really be part of a deep, embedded process. Then I worked at Facebook which is an engineering company and it has a very strong engineering first perspective. So when I was thinking about my next destination I was thinking about what lies in between? What are the sorts of things that are out there? And there were a few companies and I think at the time the commonalty is they all had founders who were designers. And I think when you have a founder who’s a designer there’s an awareness of what research might be. There’s an awareness of the Watson role and its power in terms of creating new ideas or – and I think you want – I wanted to try and see if I could work at those places. Pinterest was one of those places. So Evan Sharp is a designer. He’s one of the co-founders along with Ben Silbermann. And so Evan knows about research. He’s not an expert in it. That’s important, but he’s open to it. He’s interested in it. And so that’s why they brought me in. I was about – there was probably about 70 employees when I joined, which at the time some folks thought well that’s early for a researcher, but if you understand what research can do it might not be that early and in fact the broader field we see now – I see startups all the time with a handful of people who are bringing researchers in when it’s relevant to what they’re doing. So to tie it back to the metaphor, to me it’s important to be at a place where the Sherlock wants the Watson to be there because a lot of times that may not be the case. And so Watson’s ability to contribute is limited by the partner’s desire for input. I think at Pinterest there’s a big desire for that. It’s a really interesting place to do research. Very open, very focused on the audience that we’re trying to reach. In part it’s probably due to, you know companies have lots of values. Evan will talk about this. I think the one, the value that I think is really unique at Pinterest, and differentiating, is something called “knit” which is how we talk about interdisciplinary collaboration and respect. It’s understanding that there are many people who can contribute to successful outcomes at the table and we should understand each of their disciplines, strengths and weaknesses and we should leverage their strengths. So “knit” is very Pinterest, knitting right, like weaving together disciplines. There are a lot of starts where that sort of multidisciplinary approach is not part of what they do, culturally or strategically, and it’s a huge part of what’s happening at Pinterest, whether it’s Community Ops, Design, Writing, Brand Design, Marketing, Research. There’s a whole bunch of disciplines – Data Sciences, Business Analytics – that are all working together very closely – Project Management, Product Management – to sort of put together what it is Pinterest does. So that context, that environment that’s willing to accept what you have to offer, but also try to like embrace and really use it. That’s important for research and I think that’s a choice you have as a researcher. If you seek those places out you are in part prototyping or designing your own experience. Versus going to a place where you might know from the outset they want a research, they don’t know what researchers do, they’re not certain that they will use the research. And that’s usually pretty clear in the hiring process or early on. And I think that’s not to say that’s a lesser than experience, but it certainly is one you’re choosing as a research practitioner in terms of your daily experience.
Steve:Or you should be aware of that?
Gabe:Yeah.
Steve:And you’d do well for yourself to know that that’s what you’re choosing.
Gabe:Yeah.
Steve:I think I saw an artifact of the knitting culture. In the men’s room there’s some Python script coding advice hung over the urinal and the title for it – I don’t remember what it said, something about like python knitting or something.
Gabe:I think it’s called – officially those posters are calling knitting in the bathroom. And there’s someone – an engineer here is – there’s I think a couple of them working together to sort of pass on tips and tricks for certain software languages by putting those up in the bathroom stalls.
Steve:It’s not the only company where I’ve seen like technical advice in the bathrooms.
Gabe:Yeah.
Steve:But the knitting – I didn’t understand the knitting reference until you kind of explained what that means culturally.
Gabe:Yeah, absolutely and I think it’s – you know again there’s a lot of great – we have other values at Pinterest that are great. But I think that one is really unique. Again this idea of – for me it hearkens back to some of the experiences I had at IDEO. This emphasis on multi-disciplinary approach to design challenges. I think it does lead to different outcomes. My background, I have an undergraduate degree in theater and I care a lot about process and how we’re doing what we’re doing together and so again that part of Pinterest and being a researcher at Pinterest is real exciting.
Steve:So we never resolved this. Again, I was kind of picking at you at the beginning when you talked about journeys and destinations as things that you – and so I think I understand as you explain more that being a researcher is a successful state for you. It’s not a stage to get to something else. But when we kind of break the world up into journeys and destinations I mean you – theatre is about the journey, right. The payoffs are – and you said someone who cares about process is someone that cares about the journey. So I think I got hung up a little bit on how you were using that phrase and I jumped all over you, but I appreciate finally we have – finally I have clarity. So thank you.
Since we haven’t talked about this at all can you explain what Pinterest does?
Gabe:Yeah. So if you listen to Dollars to Donuts you’ve heard that Pinterest is the world’s largest catalog of ideas. The question is like what does that mean? And I think at that – at the heart of that is like an interesting conversation for researchers, but also an interesting characterization of the opportunity and challenge Pinterest has. So Pinterest is very different than a lot of the apps or services or companies it’s often compared to. For a long time the press sort of positioned Pinterest as another type of social network. Pinterest is not a social network. It’s a very personal experience for the people who are using it. It involves things like saving things that you want to then go on and do or plan or make or cook or buy. But a lot of – when you do research with Pinterest you are really deep into it. They see it as this very personal moment. Sometimes people will even – again it has to do with discovery of content, the things you care about. Sometimes it has to do with learning. There’s lots of stories here from Ben our CEO talking to people early on where they would say to him anecdotally like I didn’t know I had a style. Like I didn’t think that was part of who I was, but then I started pinning clothes or things for my home and one day I was looking at my board about my living room – you know best living room, or like my dream living room – and I realized like wait a second, seeing all those things together it all sort of snapped for me. Like I’m – this is actually my style. This is what I’m into aesthetically. There are a lot of moments like that I think for Pinners because it’s very personal.
Steve:A Pinner is someone that’s using Pinterest?
Gabe:Yeah, someone who is using Pinterest is a Pinner. Someone who works at Pinterest is a Pinployee. We’ve got a lot of pin puns going on here, at the Pin Factory, which is what I call it.
Steve:We’re going edit all those out.
Gabe:So back to the point, I think Pinterest is personal. It’s done in a way where it’s in a public space, but it’s not social in the same way many social networks are about broadcast. Like this is my identity. This is what I’ve done. Pinterest is really about – it’s about aspirations. It’s about anticipating. Like I want to do this trip so I’m planning it. And then it’s about participating. Participation on Pinterest is not within the digital space. It’s within the analogue, the real world. The part where you have this pin for this recipe and then you go cook it. You make it, you serve it. Whereas other social networks are often about broadcasting in the moment, like real time, or they’re about broadcasting this happened. So they’re sort of reminiscing or looking back on the event. Where we see Pinterest is used to plan, you know, in the US Thanksgiving dinner. You start seeing people pinning Thanksgiving recipes months in advance. Halloween costumes is a big – DIY Halloween costumes is a big thing in the US on Pinterest and people start looking for those how to do – you know put together their dream costume, the best costume, the cool costume, like 2 months in advance of Halloween. So that Pinterest is a personal experience. It’s about planning to take action in the future. It hopefully supports you taking that action because you have the information you need. It’s very different then a lot of other services. So what does this have to do with the catalog of ideas? I think the important bit is we see in all of our research, market research, qual, quant, that this idea of a place to find ideas is differentiating. It’s useful, it’s fun and people enjoy that. The challenge is when you sort of – you have to contextualize that. From a marketing perspective you have to say you know okay what about a place to find ideas. A place to find ideas is not enough context. So there’s been different ways we’ve approached that. For some period of time we talked about bookmarks. Some period of time we talked about – more recently we were talking about like a physical thing like a catalog where you go and browse and discover lots of ideas. The key thing is the ideas bit. We know that’s important. The question is how do you contextualize that, not just for a US audience, but actually for a global audience and that’s really challenging. Pinterest itself is based on you know a metaphor of like a pin and board – pinboard. The challenge is that that doesn’t actually internationalize. Not every country has a tradition of a pinboard with a pin. And so how to provide context for what this thing is and it’s not – you know it would be easy if it was just like oh it’s a social network for, I don’t know, saving ideas, right. It’s not that though. That’s not really the core. And so how do you contextualize that? That’s the challenge. So as we think about different ways of messaging to audiences, especially audiences who are not using Pinterest today – there’s a ton of research we’ve done. There’s a ton of research to be done as we look for that really killer way to describe a complicated and robustly powerful product to people who don’t know what it’s about. And that’s an opportunity for research, whether you’re a marketer researcher or quantitative researcher. And it’s part of what the team has worked on and will continue to work on with our partners in writing, marketing and then just even overall strategically, like the decision makers of Pinterest. How do they want to – they have to go and speak to the press. This is something they’re saying all the time. They need to feel that it’s accurate, that it’s representative and that it really gets at the – the core or the central sort of essence of what they think the product is and will be in the future.
Steve:So I asked you what the product is and you describe sort of the research and strategy journey of trying to describe it – to answer that question in the way that sort of has the most benefit for Pinterest and current and future Pinners.
Gabe:Yup.
Steve:But I’m going to ask you again, like – I mean as you say people are out there now talking about it. So you kind of said – you took the phrase catalog of ideas and then you unpacked it into all this wonderful stuff which I think tells us what kinds of things you’re thinking about and what your work is meant to address. But just for that person that’s scratching their head right now and going like okay, but what is Pinterest?
Gabe:I gotcha. It is a place to save ideas digitally, to discover ideas related to the things you’re interested in and then to go and do the things that you have saved and discovered. So it’s about saving, discovering and doing.
Steve:So what kinds of digital information are people saving?
Gabe:Well I think we’re limited to digital today, but it could be many, many things in the future. It could be saving many types of objects. Right now we’re within the space of like the Internet. We see the – the most common use cases are we see a lot of folks use Pinterest for recipes. Planning daily meals is a huge part of what pinners. Also like discovering or trying new recipes, especially if you’re into that. It’s often about doing, right. Another popular use case – trips and travel. Like right now you want to go on a trip so you go to various websites and maybe you have a folder on your computer or a piece of paper – you’re writing down like where do I want to go when I’m in Hawaii? Where do I want to stay? You can save all those things to a board on Pinterest which is similar to a folder, but it’s within the bounds of Pinterest itself. You can save those links. You can find them on Pinterest. You can find them off of Pinterest and save it to your Pinterest. So that you can go to one board and have all the ideas of what you want to do on your trip when you go there. And you can access that when you’re actually on your trip, right. We also see a lot of people who use Pinterest for fashion, whether it’s discovering new trends or thinking about inspiration for what they want to wear. In fact there was a story this week in the – oh gosh, I can’t remember the newspaper – and this writer had talked about – she let Pinterest choose her outfit every day. So what she would do is she would take out of her closet an article of clothing she had – let’s say a pencil skirt. She would go to Pinterest and type in pencil skirt and she would look at the most popular street style or fashion images of people wearing pencil skirts. Then she would find the one she would like and then she would mimic it with her available wardrobe. So every day she let Pinterest dress her. Again, that really critical thing there is 1) she was looking for ideas, 2) she was then doing it and what was also interesting about it is that she was using Search. We see Search as being a huge part of a lot of Pinners experience in order to find those ideas. But Search I would say is a little different on Pinterest. The Search in this case, and what we see across many people using Pinterest is it’s not to find one idea that’s like the right idea. We don’t have a ranking algorithm that puts the right idea at the top. When you’re in a discovery system or a discovery mode you can search for outfits with pencil skirts, but there are many possibilities in front of you. You are looking across this set of data or images that’s about discovery and you’re trying to look at not just the best, but there are some that are good and there are some that are even better to make or to learn or to discover something that you want to then go do.
Steve:So you described before this – you know your own professional path a little bit and that you sort of found Pinterest as a company that was in between maybe other places you’d worked at in terms of what was research and how it was understood. But as you’re talking about Pinterest and what’s going on and what you’re interested in exploring, there’s also a unique class of problem that I hear you kind of digging into with joy and I don’t know if that’s the culture or you or what. And that is we don’t know quite exactly what this thing is and the best way to talk about it and it’s changing and people are inventing new things for it and the world is culturally diverse. So the fundamental, like what is Pinterest and how do we describe it and how do we build for it is – like that’s a hard problem. It’s a dynamically shifting problem. I mean did you know that’s what you were getting into?
Gabe:Well I have two answers. So the first one is like – the first one is sort of addressing the like what is it, it’s dynamic – like I think there’s something really interesting about Pinterest which makes it unique, which is again opportunity, but also challenge, which is there’s a utility element to this, right. It helps you to get things done. It helps you to find the things you want to do. But it’s not just a utility. It’s not a task based software. It’s not about lists per se. Although you could visualize pins and boards as lists, right. But it has this other element that I think is related and unexpected which is it has this way of conveying or inspiring people to think about possibilities. To discover things that they didn’t know about and to learn something about themselves. And it does this in a way that – we know from research that people really enjoy the aesthetic presentation of this information. They think it’s beautiful. And so, you know people can get – we interview people and they talk about my Pinterest time, right. They’re on their couch on a Saturday morning. They’ve got TV on in the background, maybe some Netflix. They’ve got a big cup of coffee. They’ve got like their phone or perhaps their iPad, and they’re browsing through Pinterest. And some of it’s about what they want to do. Some of it’s like what might I want to do? This sort of anticipation, aspiration. This sort of beginning of the experience. And Pinterest puts together these two things which are traditionally separate, right – the sort of exploration/discovery, aspiration/inspiration with like utility. And so that’s a tension within the product that’s really interesting. I think it draws people in, but it can make it really challenging because it would be much easier to do one or the other and that’s not where Pinterest is. It wants to do saving and doing, but it also wants to do discovery. And of course the more saving you do the better Pinterest hopefully gets at making recommendations to you about things that you should check out based on what you’ve done. That’s definitely an interesting part of the product and the tensions within the product, or opportunities and challenges.
Secondarily you were talking about did I know that when I was getting in here? I was super interested in discovery systems. I think discovery systems are a really interesting part of the technology sort of landscape at this point in time. As part of discovery systems it’s like moving away from basic search which again is not basic in any sense, but many companies have been able to perfect the “I need this thing – help me find this one thing.” I call that like the needle use case. Right, I need the – help me find the needle in the haystack quickly. That’s impressive and it’s great, but it’s also now become expected. And so when you move into – it used to be called Search and Discovery. When you move into discovery systems that play up the discovery experience first suddenly, yeah you have to be able to help 5 people find the needle which is like perhaps one small moment within a user’s session, but actually the use case of looking through the haystacks is really interesting to people, right. So they’re looking through like haystacks in a field right and they’re like well what’s in this one? What’s this one about? How is this one different than that one? They’re exploring and discovering things that are interesting to them. They spend a lot of time doing that. And then they get to the needle. That’s something that I was already thinking about before I joined Pinterest and I could see Pinterest as having a unique position within that space. The one thing that’s related to that which we don’t talk to users about and it’s a super – I’m not an expert in this, but it’s an intriguing idea – is the notion of the interest graph. Are you familiar with the interest graph as a concept?
Steve:No.
Gabe:Okay, so social graph is sort of like, you can think of that as a subset of the interest graph.
Steve:What’s a social graph?
Gabe:The social graph is basically let’s map out with technology all of the people and their relationships to other people. And what’s interesting is like that’s been a very disruptive technology moment in the world today. But what’s interesting about it is that’s a big task – don’t get me wrong – but it’s finite. There’s a certain number of people on the planet at any point in time. Computers can count them, right, very quickly and probably count them real time if they have the information. The relationships between those people, they’re finite. You can measure them. You can ask people like we think these are people you might know, do you know them? Yes I do. No I don’t. That’s interesting, but again – huge in scale, but finite. The interest graph resides over top of the social graph and it’s this concept or theory that we could map out the relation between ideas, objects and people. And that’s really interesting to me and that’s a place I think Pinterest has a unique place to play in within the context of the other technology companies. What’s interesting about it is that the hallmark of humanity is every day we make new ideas, objects and people. So the interest graph is infinite. And if you had a technology that could map it out, so it could tell you which ideas are related to each other, which objects are related to which ideas, which people are interested in which ideas, right. You can do really cool things that are technically cool, but actually experientially cool. Something that actually would make people’s lives better which is most of the interests I have today are limited by what I’m exposed to. Like I only can know about the things that I know about, but if I had a technology that could tell me about something happening on the other side of the planet – in Germany there’s a new type of music scene happening that I don’t know exists today. I don’t know anyone who knows it, but if the technology knew that I would love it and could tell me about it perhaps I would discover it years before I would naturally. Or perhaps I would discover it when I wouldn’t ever have discovered it on my own. And I think that’s really cool.
What Pinterest does nicely in that is that everybody is building this sort of interest graph rocket ship, this technology, but the fuel actually makes a difference. So on Pinterest today when people put things together in collections or boards they save them – it’s them. It comes from them. They’re saying something semantically about the objects that belong together to them and have meaning to them. They name that board of that collection and that fuels 100% pure interest. It’s really what I care about. Whereas other companies may have fuel that’s somewhat representative of that. It may be representative of a lot of other things. It may have more impurities into it. So I think Pinterest is uniquely positioned to sort of deliver a consumer experience on interest graph that I’m really excited about. I think it’s a really interesting concept and so one of the reasons I came here was discovery system, interest graph, recommendations. Like this is a pretty interesting moment in time for that sort of technology and one that actually has a real human experience on the other side.
Steve:I love that you ended with the human part. Like that was the last thing that you said in this really amazing kind of romp through some big ideas. And I was sort of reflecting as you were talking that, you know – I mean we’re having a conversation today about research. That’s sort of the context and you’re talking about research, I think, I don’t know, in a unique way. I don’t know if you feel that. I’m kind of laying that on top of you. I mean how is this a research conversation? I’m not forcing you to defend it, but let’s just go meta, like why are we talking about this on Dollars to Donuts?
Gabe:Well I will say that for me when I think of research I think of it as a discipline. Like as a blue-collar type discipline. I don’t think about – it’s a set of practices that you can learn and I think those set of practices can be applied to certain businesses or contexts. And so I don’t think of myself as a – I don’t believe in like super empathy powers. I sort of react negatively to that. I do think research is one way, or one discipline that can contribute to product development, design – service design. There are many that can do it. It’s not necessary to do it, but it can be useful. And so it’s a conversation about the context that research can play in, but also just about what are the duties, the set of practices – what are the things that researchers might think about in terms of their contribution and what they bring to the table. Great researchers I think they really know the sort of on the ground, in the trenches, skills, tools, things they have to do to move forward with the group, but they also have to play this unique role where they pop out and get the bird’s eye view or the broader context and thing about that, but not in a way that distracts the rest of the team. And then they sort of port back down to like the blood and the guts of what we’re doing every day in the trenches to figure out how can I take those observations that are abstract and make them useful for the team and actual for the team. So all research conversations for me, like sort of waver between the two, right. They’re up in the air, really philosophical, abstract, but then also like down on the ground, really practical, really technical. And so that’s probably why we’re having a conversation about like how did you choose where to go next and also some of it was like well I wanted to be in a place where they were going to care about the work that I did, but also there was this huge like theoretical thing that I could learn about in the context of their products.
Steve:And moving back and forth between the tactical to blood and the guts and the big picture, philosophical. I don’t know, I’m guessing that – and just sort of watching your face as you talk, that not only do you like those end points, but I feel like, and maybe this is my own thing coming in here, there’s something about – and this I guess is the journey thing coming back in as well, but there’s something about the motion between those two kinds of thinking that I think for me sparks some stuff. That we can talk about the big picture – I mean you can really talk about the big picture, but sort of the energy we get from that vibration, from the well what does this mean, how do we deal with the blood and guts kind of – with the team every day. That – I don’t know. I’m throwing that out there. It’s a very leading question for you Gabe, but…
Gabe:I think there’s certainly like personality differences that attract certain type of people to certain roles, right. And so there’s probably something about that that appeals to folks who are doing research for sure. I also want to make sure to like be clear which is it’s really important to me that as part of that process we’re shipping something. That I can actually see something in the world that I’ve contributed to. Otherwise I would have then stayed in academics. I would have been working at the theoretical level and for me there was something really exciting about work I did at IDEO where I worked on a furniture project and I’ve seen that furniture in the world many, many times. That makes me feel like there was – it’s not just about ideas, it’s actually about building something and doing something. And that’s I think important for me when I hire researchers or when I think about the type of environment I want to be in. It has to be – it can’t just be interesting. It has to be impactful. It has to be action oriented. There has to be something that happens at the end more than an idea. And so that to me is like I like that fluctuation between the tactical and the abstract, but to an end, and that end is to put something into the world that makes it better for people in some way, whatever the “it” is.
Steve:I really like – I agree with you that stuff being interesting is sort of not an end to – it’s really great to be sort of turned on by stuff and like getting new ideas. You know for me trying to put it in a language – I mean what’s the good and bad word now – the A word – actionable. That you know – I don’t know, people go back and forth on whether that’s a good word or a bad word, but I mean I guess in my own career I had – the ship had sailed a long time ago on like measuring my own success by what shipped. And I hear from researchers as well that sometimes it’s frustrating. That – and so I’ve just sort of shifted my own role. Like I’m in the business of helping people see as clearly as possible what decisions they have to make and what actions they have to take in order to ship something, but I can’t – and maybe this is because, you know the difference between being inside and outside of the organization, I can’t measure my own success by what gets shipped or I would just be an alcoholic. But I do measure my own success by – I mean someone said it to me once like oh, what you do is kind of like take someone to the doorway and kind of like open the door and like point them kind of on the way through.
Gabe:I’m probably using like very tacky language when I talk about ship, but I – so to be clear, shipping something can be pretty broad. What I mean is there was a decision made. That there was some sort of result or outcome of the research. And so definitely some of the hardest things I’ve worked on were the shipping moment was convincing someone not to do something. Like that is just as important as convincing someone to do something, in fact it may be harder in some cases because they’ve already decided, no. No, we’re going to do this thing, we’re going to make this product and helping them to understand how that might be a destructive decision for their business. That’s a type of shipping, right. Shipping a decision. So yeah, maybe I’m overly emphasizing the analogy.
Steve:No, I think I’m just one sort of rung before that which is like I think for me I probably ship influence. I empower people to make a decision, but I can’t actually control if they’re going to make that decision. So sometimes you sort of – you learn something, you package it, you convey it and they need to do something. They need to do something. Like the business is just – the world has changed, the business has changed. They need to do something and the other factors are just so large that sometimes doing nothing is the outcome, but you know that you changed somebody’s mind. You gave them – and I’m not trying to undersell what I do. I mean I’m trying to be realistic like I just don’t get to ship decisions as much as I would like to even. I’m going to go drink after this. That’s what you’ve taught me today.
Gabe:Well as long as I’m going with you it’s fine. I think – so one thing you brought up is control. Like I personally have had moments of development where I do not – I try to focus on what’s in my control. I can’t make a team do something, right. They actually have to make the decision. That’s a healthy part of the process and sometimes they will do things that are risky or potentially counter to the recommendations. And there’s a whole host of reasons why that might be. I try to recruit people and hire researchers who are resilient because I do think a huge part about being a great researcher is going out on a limb to say something important, potentially getting knocked down, and getting back up again and continuing to say that. I think really great researchers, in tech especially, they act like it’s a sprint, but they know it’s a marathon and often times at Pinterest we talk a lot about like the research part and delivering that, that’s just the first step. Now the real work is continuing to bring it up with the team. Continuing to make it part of conversations. That takes a long time sometimes. Particularly like if it’s something very meaningful or hard to do. People just don’t, when you deliver like a hey this thing you thought was the case for many years is actually potentially not quite like that. It’s a bit different and you should really think about it differently. Human beings are terrible at that. Like we don’t do that in our personal lives. We don’t just shift our total point of view without some sort of major event, like some sort of like peyote, or some sort of like near death experience. That just doesn’t happen. And so it’s not going to happen at a company or a business and you have to think about that sort of marathon bit, like bringing it up over time. It’s going to take awhile. You have to work at it and you can’t just think you’re going to come in and say no I have the answer, it’s this, and everyone’s going to be like here, here, good job, let’s do it. It’s harder than that. I just think that’s part of the research gig. That’s part of shipping information and inspiration, right. That inspiration bit. It can be very transformative for a designer in a moment, but getting to the long term outcome of that, the result of that usually is isn’t a couple of days. It’s probably a longer period of time.
Steve:So tactically when you are meeting prospective research hires for Pinterest, how do you – that’s – and obviously you look at many things, but how do you look for that?
Gabe:That’s a good question. I think first of all – how do you look for resilience. I think you talk to them a little bit about their career and I ask people to tell stories about the research they’ve done. You ask them to tell stories about, you know, what’s an example of a project that you did that you think showcases your skill set and really shows the best of you. And then you ask them questions about – they talk a lot about doing the research, then you actually have the real conversation of like so what happened next? What did you do? How did you approach communicating that to other people? What were the challenges you faced? And really – because we want to talk about the researchers, researchers, but again I just feel like that’s the smaller part of what we do in a context of an organization. ‘Cuz that’s what we’re good at and we like and we like that and we want to talk about it and we want to make that the big part, but like great – this is like you can build it and they don’t have to come sort of moment. Or even – I heard this great quote awhile ago which was “you don’t have to make a mistake to lose the game” and that’s about context and timing and sometimes like you have a great insight for an organization but the timing is not correct and it’s out of your control and so then what do you do? Do you pack it up and go away? Do you like start making posters and put them up every month? Do you continue to like any time anyone brings up tangentially relation, like oh that’s like what we found in this research from before, are you aware of it? That’s like part of the story of research and try to focus a little bit on that in conversations.
Steve:Is that something you are teaching researchers that you hire? Or does someone have to come in?
Gabe:I don’t think of myself as a teacher. I did not enjoy that part of academic life. I think of myself as a collaborator and I think it’s something we discuss a lot. How are we doing this? What are we – you know deliverables – what’s the nature of deliverables? Trying things – like the one thing I do try to ask the team to do, even when they’re tired of me and of the challenge is like well what’s something you could just prototype? Let’s just try to do something very different. And I think the team itself has built a good sort of muscle around like let’s try it differently to see what happens in terms of communicating and continuing to push those conversations forward. I do think it’s hard at a startup like Pinterest. When I joined I was around – I think there were 3 people in my class who are all still here – or my whatever, my cohort. We were like 70, 71, 72 – pick your number. Now there are 700 people and I think what’s really challenging with research is that you can learn something and communicate it to the entire organization and everyone can hold that knowledge. In the case of a startup like Pinterest where you’re growing – like next week 20 new people are going to start. And those 20 people they don’t know anything about that. And so you have to continue to make sure that people understand – they’re going to come in and say like hey how come we don’t do this and it’s easy just to say like well we tried it and it didn’t work. Or well it’s not going to work. Or, that’s easy. The harder bit is like actually here’s what we’ve learned and here’s why we think that may not be the best way to proceed or move forward and so – that’s a real – like it’s an ongoing challenge of any like company that’s growing quickly, but particularly it hurts researchers because the artifact of what you’ve done is knowledge and knowledge you either have to read it, or communicate it. And so figuring out how to like maybe put it in a space or you create a wiki, but then you’ve still got to get people to go there. It’s a real challenge I think for any researcher at a fast growing company.
Steve:Right. There’s a lot of talk about – you said at scale earlier in this conversation. That you know dealing with insights at scale is a challenge as companies grow and so on. But you’re talking about sort of insights, like insights in the tornado, or insights when the sands are shifting.
Gabe:Yeah.
Steve:You can have a great meeting – you can – you know all the good things we do in research. We talk to our stakeholders. We bring them along in the process. We bring them out to meet with people. We involve them in synthesis. But over that – say that’s an 8 week process, that team has changed and then two weeks later, after you’ve had some really great meeting to make some decisions and have a workshop and do whatever things that we do. Two weeks after that there’s a bunch of new people and two weeks after that there’s a bunch of new people and two weeks after that.
Gabe:Yeah and also the actual, you know the audience. Like perhaps your insights are stale. The audience has changed significantly, right. Or perhaps the technology, right. The technology itself, there’s like a new innovation that you know we have a new team coming in that’s been acquired and they have like a whole new technology they’re introducing that can really change the context. And so again this notion of in a startup land a research has to again believe – act like – know that it’s a marathon because there’s so much changing and you’re going to have to really stay abreast of all of that and check your own assumptions, check your previous recommendations constantly so that you are really providing the best information you can at that time.
Steve:That’s a really good point and I think that’s sort of the researcher arc, right. You come into a situation. You can tell that everybody’s hypotheses are full of crap. And you’re like okay I don’t know what the truth is, but I know that you’re wrong and we’re going to go learn and you come back with a truth, or a set of truths or a set of things that approximate truths and that you advocate for, but that also – then you could risk becoming part of the problem because now you can kind of clench on those things that you discovered through your own sweat and your own effort in the field, but what you’re saying is those things may have a short shelf life given the changing space that a company like Pinterest is involved in and that you have to let go of those things which is sort of counter to the you’ve got to go – the marathon is kind of about persuasion and advocacy and influence and getting out there and giving that same presentation 20 times to different audiences at the same time as also be willing to let go of it.
Gabe:Yeah, and I think – so there are – I think that is a challenge – a real challenge. I think there are some ways to guard against it, or some strategies. So one strategy at Pinterest, I – my vision for the research team was an interdisciplinary team. Trying to put together both qualitative user experience researchers, of which there are different kinds of qualitative user experience researchers. They’re not all the same. A quantitative user experience research group, market research. Altogether. There’s an economy here in terms of like actually getting things done, but there’s actually, like what it does nicely is it brings together different data points to build the knowledge that we’re learning together.
Steve:Is that knitting?
Gabe:That’s knitting within our own research team right. I think a lot of companies you can have, like those groups can get siloed early on and then you have that moment years later where you’re like well we’re working on the same thing, what’s going on? Or we’re competitive with each other. Trying to put those disciplines together and then what allows – it’s like so work that I’ve done in the past that I still think is relevant, well if it’s still relevant we’re probably still working on it and now we work on it with a perspective of multiple methods. So mixed methods. This is a way to sort of take an idea and really, truly understand it deeply because if it’s – we think it’s – we hypothesize something from the qualitative research then we can hand it off to a quantitative researcher who can then quantify that at scale of which you might learn something that is – you think there’s a correlation between a macro question and a whole bunch of smaller questions. Well if you validate that relationship then you can take that macro question and you can put it in all of your say brand equity trackers for market research and all your markets and then you can measure it there and now what you’re doing it you’re building this interlaced qual and quant sort of lineage of knowledge that over time can become very robust and then you can say now that you’ve validated it at scale and you’ve used multiple methods and you’re monitoring it in each market, well now you can say we know this is a real thing, we’re trying to change it. We can continue to measure it over time with a brand equity tracker and market research to see like are we doing better, are we the same or are we doing worse? That’s super exciting and it’s a way to guard against your own bias, like this is true. Maybe when you get to the step of handing off to a quant person – like no this is not true, it’s scale. Or it’s different – it’s often probably true, but it might be different than what you thought it was. So there’s actually three things going on here, not just one. Or you find out when you pass it over to a market researcher who does brand equity or some sort of large survey, they’re like oh this is true for, you know, Western European countries. It’s not true of Pacific Rim countries. And so – again, bringing in more methods, building out – if it’s an important question, building out different levels of understanding can be really robust and guard against your own bias towards – but I said this before and it was true. It’s still true.
Steve:So in terms of this multidisciplinarity and you’ve kind of said this how you sought to build the team, maybe you can describe the team more specifically and where are you at? Where’s the team headed?
Gabe:Yeah. So we have three qualitative user experience researchers. We have Altay Sendil who comes from IDEO. He has an industrial design background. He was at IDEO as a researcher forever. He brings like a really strong perspective on ethnography and really generative work. We have Larkin Brown who comes – she was previously at Google and she’s like – to me like a real digital UX person and she has a lot of areas of focus. She’s also really into – she serves as a stylist within Pinterest and helps with brand styling and also communication to press around fashion. And then we have Cassandra Rowe who’s – she worked at, previously at Intuit, but really did a lot of international research at Netflix. She was part of their early growth on the qual side. So we have an expert in sort of formative, an expert in digital, an expert in international. Then we have one quantitative UX researcher named Jolie Martin and she is really looking at the intersection of attitudes and behavior for Pinterest. So that’s experiments and – AB experiments and surveys, to truly really understand the quantities and quality of experience. And then we have Paul Pattishall who is on the consumer side doing market research for us – things like brand equity trackers, product surveys, NPS, category deep dives. Like what are the most common interests in France right now for people who aren’t using Pinterest so we might approach them with relevant content. And then recently had Julia Kirkpatrick who joined, who’s doing market research for the business side of the company – Partners Products is what we call it. That’s the people, that’s the disciplines. Again, the important bit is that we’re all working together. We’ve hired people who, they work with a team in an embedded type way, but they also know when – when the team has a question that’s outside their skill set to go to a practitioner within the team to get help and support. We also, you know we’re still – so that’s six people now. We’re adding two more people this year. We’re looking for another quantitative user experience researcher because I think one person is not enough. You have to have two to really be a team so you can play off of each other. And a UX or qualitative researcher to focus on our business or Partners Products. So these are things like the web analytics if you own a domain, or advertising on Pinterest which is something that’s a new area for Pinterest where we can do some really innovative things because of the visual display elements of Pinterest. So we’re adding folks. We’re looking for folks, but the important bit is all those disciplines working together – we’re one team, but you could think of it as three separate teams. But because we’re connected there’s a lot of collaboration and interaction amongst the team members which I think is really – it’s an exciting environment for me as a researcher because people know a lot and you learn a lot and you can pick up a lot of tips and tricks along the way.
Steve:So just thinking to wrap up, as you describe the team and you talked before about – I used the word teacher and you kind of shied away from that term. You said you think of yourself as more of a collaborator and given the nature of how that team is structured and kind of what you have created as a collaborative culture and how you’ve inherited kind of – you’re participating in the larger Pinterest culture – I agree with collaborator. And this is just all a preamble to a leading question – the trouble with interviewing somebody that you know a little bit is that there are certain answers you’re trying to draw out of them. So I’m going to try a few different ways. What – you know in terms of your role as a manager, if it’s not teacher, what would the people on the team, what nouns or adjectives would they use to kind of characterize your managerial jim-jam?
Gabe:That’s a good question. I mean this is my first time managing. So I think my jim-jam is under development. What I strive for is one where they feel like supported. That they – I will – if people ask me for feedback I will give them deep feedback on whatever it is, their deliverable they’re creating because I know in this environment you might not get deep feedback from other people and I think if you’re doing work you deserve to get that feedback. So supportive, probably into the – like I care definitely a lot about research and what we’re doing. So I talk a lot about research. At lunch I’m like the person who’s like, answers the question about what they did this weekend and then follows-up with the like hey let’s talk about this project that’s happening. And so I’m – they would probably say I’m pretty focused on research. So supportive, into the research. I definitely want to empower them to make decisions. So we have a model where they’re working with teams and they are the ones that need to develop the roadmap with their teams. They need to make the calls. I’m there as support. I can provide context. I do a lot of like I – like showcase their work and constantly talk about their work because that’s part of my job is to like hey did you know that Cassandra just did this in Germany with that team, the activation team. You should be aware of that work. That’s a huge part of what I do, connecting the dots at an organizational level. So supporting, you know advocating. Nerding out on research.
Steve:What’s your superpower as a researcher or as a manager?
Gabe:I don’t – that’s a good question. I tend to like shy away from superpowers because I think it’s about like training and practice and whatnot. As a manager, well I don’t think I’m there yet. Like I haven’t done this long enough to really know that about myself. And I have a great team. Like I think I have a lot to learn and a huge part about managing probably is like managing through hard times and I probably need some more of those hard times before I really can assess.
Steve:Careful what you wish for!
Gabe:I know! As a researcher, I mean I care a lot about the work that I do. I think that’s like a double edged sword sometimes because sometimes you have to care a little less. But I think I – it’s a good question. I really – the times I’ve thought – I’ve thought to myself well like I’m pretty good at this thing, but then I’m like that’s just an illusion, right. That’s like I don’t have a special power. I just have a set of practices and things that I try to learn about and do. And so I’m hesitant to answer that because maybe there are no superpowers. Maybe it’s just about hard work and just caring about the details.
I can tell you what my weakness is which is I don’t tend to go to conferences enough or get out and socialize enough in the research team. So when someone comes to visit me, to talk about research, or who’s a researcher, that’s huge for me ‘cuz that’s how I try to like understand what’s happening in the world. That’s a moment for me. That’s like my conference. That’s my getting out and interacting with other folks around research and so – ‘cuz I’m pretty heads down. I’m pretty in the trenches, right. And so that’s why I definitely celebrate those moments and I really appreciate when researchers come visit me. I have sometimes students come by who want to talk about their future career or get more information. I tend to like accept more one on one conversations than really approach like group activities, as a way to just get a sense and reflect on what’s going on in the field.
Steve:Is there anything that we didn’t cover in our conversation today? I know we could go and go.
Gabe:We could. I have a lot of things to cover, but I’ll leave it for another time. I do have some questions for you though.
Steve:I don’t even have to ask that!
Gabe:No, I’m like ready to go. So my first question is…
Steve:Oh your first question!
Gabe:My first question is like what’s the biggest surprise – what’s been most surprising to you in terms of the Dollars for Donuts interviews you’ve done? What stands out as surprising that you didn’t expect before you started this?
Steve:There’s probably surprises in terms of just having these conversations and there’s probably some surprises about just having this as a thing that I put out in the world. But I also – it seems I push back on the notion of surprise. I think I wrote about this in my book even where I used the phrase bland curiosity. ‘Cuz as researchers you get asked the what’s the biggest surprise question all the time and I feel like well that supposes that you entered into it with something and then like a bucket of water appeared and splashed you and you went oh…
Gabe:Okay, this is good. Let me reframe then. What has been the most unexpected theme across these interviews?
Steve:I haven’t synthesized them yet. I mean part of – and you know this is sort of the discipline of research like – and you can do it different ways – you can kind of just go and throw yourself into it and then occasionally you’ll be like oh we haven’t seen any themes except blah, blah, blah, blah. Oh that’s a theme. And so I’m just going to keep thwarting you Gabe. I mean sometimes I resist reflecting and just sort of let it be what it’s going to be. So I mean there are things that come up, but also it’s because I ask about them and then – you know how teams form? What people’s career arcs are? These are sort of the topics that I’ve chosen to talk about, so I don’t know that they’re surprising themes. There’s something that we didn’t get into a lot – there sort of a structural issue that’s emerged from the beginning which is – which I remember this being like a design topic from the 90s and probably goes back before then – where do these people sit? Do they sit with their – are they centralized or are they embedded? So it comes up a lot and I think it’s a really interesting sort of managing – what I like is that no one says this is the right way to do it. People have said like this is what we’re experimenting with now and here’s the tradeoff. So I guess that’s a thing that I would never have thought to ask about and that’s also because I don’t have a job inside a company that has a bunch of research people, so it’s outside my own experience. So I’ve learned about that as a topic and it seems to be one that when I talk to people they say like oh yeah, that discussion and this episode was interesting and we’re thinking about how we’re going to do that.
Gabe:Yeah I think that – I mean I have that as a listener all the time. How are they doing it? Why are they doing it that way? Absolutely. Okay, speaking of being a listener, my second question is will there be a Season 3 of Dollars to Donuts? (laughs)
Steve:We’re going to do it in Oculus. We’re under negotiations right now. Podcasting is over. Maybe it should be – maybe they should be something like – something that’s only – they’re going to be 4 seconds long and then they’re going to disappear.
Gabe:I won’t put you on the spot. This is a more serious question. As a young lad growing up in the cold tundra of Canada during the 70s and 80s, who appreciated donuts, there were 2 major chains. Which was your preferred chain – Country Style Donuts or Tim Horton’s?
Steve:Well I’m sorry. We grew up in the same town.
Gabe:We did.
Steve:And I’m older enough than you that I remember Millionaire Donuts. Do you remember Millionaire Donuts?
Gabe:No, I don’t. But it was the donut of choice?
Steve:Oh I don’t know. I mean Tim Horton’s was kind of the place, ‘cuz Tim Horton’s used to make their donuts in the store. Maybe you’re the one that told me this. They don’t do that anymore. I even went for a job interview at Tim Horton’s as the shift – I didn’t get the job – it was like shift work. You had to make the donuts, like 3 shifts a day. They had these big racks in the back where they would – it was an interesting sort of – I mean I didn’t get to do a lot of research, but I saw how those were made, which maybe everyone knows this, but they like have all the sort of naked donuts there and they like kind of pour over like a big vat of chocolate goo and that’s how they get that sort of drippy thing out their – they just pour it over on these racks.
Gabe:So I don’t think we can really reach a higher point then naked donuts as part of our conversation today, so it was like…
Steve:Are you – not only are you taking over the interview, now you’re wrapping up.
Gabe:How can you beat naked donuts?
Steve:Well there’s a visual that no one needs… Alright, I think that is our – I’m going to take your cue…
Gabe:Thank you.
Steve:… and wrap us up. Thanks Gabe, this was great.
Gabe:Awesome. Thank you.
The post 11. Gabe Trionfi of Pinterest first appeared on Portigal Consulting.
In this episode I chat with Elizabeth Kell, the Senior Director of User Research at Comcast. We talk about the growth of Comcast's user research practice, essential soft skills for research candidates, and putting a human face on the people that use your products.
The post 10. Elizabeth Kell of Comcast first appeared on Portigal Consulting.
In this episode I chat with Elizabeth Kell, the Senior Director of User Research at Comcast. We talk about the growth of Comcast’s user research practice, essential soft skills for research candidates, and putting a human face on the people that use your products.
If you don’t have a natural curiosity then I can’t teach you to be a researcher. It has to be ingrained in you. To be a researcher is to want to understand. I was that kid that was 5 years old, like why, why, why, why, why? And you know I’m still like that as an adult. I need to know how things work. So I look for that [when hiring]. – Elizabeth Kell
Show Links* Elizabeth Kell * Comcast * Comcast Center * Fancast * X1 * MyAccount app * IIT Institute of Design * Steelcase * Livia Labate * Tom Loretan * Alan Wurtzel * Practical Empathy by Indi Young * Hands on a Hardbody * Charlie Herrin * Tipping from questions into stories * Over-the-top * Kano analysis * Prentice-Hall * Information Architects by Richard Saul Wurman * School of Visual Arts * Erik Spiekermann and the Berlin subway map * Joyce Bromberg * World IA Day * Soft Skills Are Hard
Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes.
TranscriptSteve Portigal:Alright Liz, well thank you. Thanks for being part of the podcast.
Liz Kell:Thanks, Steve.
Steve:So I’m going to ask you the question that I always ask everybody at the start of almost every interview about any topic I think it is which is just to give a sort of basic intro to start us off.
Liz:Sure. So my name is Liz Kell. I’m Senior Director of User Research here at Comcast in our headquarters here in Philadelphia. I run a team of 11, a diverse team of user researchers and I’ve been doing this for about 8 years.
Steve:Another question that I often ask was so people in the US will know the Comcast name, other people might not. So – and probably there’s some – I guess you can explain Comcast and we’ll learn something about the company in terms of how you describe it. So with that long preamble, like what is Comcast?
Liz:What is Comcast for the folks who don’t know or don’t have the services at home? Comcast is the largest cable Internet service provider in the United States. We have over I want to say 22 million video subscribers and 23 million high speed Internet subscribers. I may be slightly off on those numbers so hopefully nobody fact checks me. We also provide digital voice service to the home, home security services and yeah, those are our four main business areas right now. And so that’s what we do.
Steve:So within your group’s purview I guess do you get involved in all those different areas of the business?
Liz:We’re involved in all areas of the business and in the layers beneath too, so all the service layers that go into supporting all four of those products and services that we deliver. So we look at the customer service – the whole customer end to end experience and that includes everything from you know the sales and the marketing and going in and observing what’s going on in the call centers, to looking at the installation, riding along with the agents, watching people try to self install kits. Watching people try to get self-help or call into the call center and troubleshoot and things like that. So in addition to evaluating the product we’re also evaluating the services.
Steve:When you say product, for a company like Comcast, what’s a product?
Liz:Product – I think of the product as like the thing.
Steve:Like a physical thing?
Liz:Sort of. Yeah, I mean – I mean it’s a bunch of electrons right, but the television is something that comes to this big screen on your TV and you have the interface that you interact with. You have the set top box that you interact with. You have the remote controls that you interact with. So that’s like kind of thing and we do a lot of evaluating and trying to improve those things. For high speed Internet there’s less to interactive with. That was one of our big insights when we were doing some research on the high speed Internet product. But really how do people interact with that physical gateway which is you modem and router in one. With home security there’s a lot of interesting things to explore around like where do people decide to put the different sensors or what kind of different pieces of home automation are they adopting and why and how are they using them and how are they controlling them, either through mobile or web interfaces. So there’s a lot of like the thing I would call it, like a tangible thing we’re evaluating as well.
Steve:So that’s hardware and software?
Liz:Both, yeah.
Steve:So these questions are naïve ‘cuz I don’t necessarily have the language for them, but so there’s hardware and software and services, but the services I feel like you’re describing sort of are more customer support services. Are there other kinds of services that Comcast delivers that you’re examining or studying, trying to improve, that aren’t about sort of managing my success with the hardware and software I guess?
Liz:Really about self-service and getting help and being able to get the support you need for your things. I keep calling them things now. I got that word from you.
Steve:So Comcast is kind of a hardware and software company and there’s a service layer to enable hardware and software. I’m sort of thinking of someone that’s like more of – a company that’s a services company like has services as a thing that you would consume, but you’d only consume Comcast services in order to get to a point where you’re having the hardware and software, the thing.
Liz:Yeah. I mean people subscribe to our services and they get something for it which I consider – I call product. You know we’re part of a product organization. So people subscribe to our services so it’s really I think about figuring out how to get the help and support you need with the services and then being able to use the products. So we look at both. I don’t know if I’m answering the question or not.
Steve:I think we’re both totally confused now. Let’s see if we can get through this without using the word product or service or thing again. So you started off too by describing the size of your team. Can we go back in time and talk about the history of your role? Sort of the history of user research and insights at Comcast.
Liz:Sure, of course. So I said I’ve been here, I’m coming up on 8 years. I was actually hired to be the first user researcher on staff when we moved into – we have a new headquarters here in Philly which isn’t really new anymore. We moved in almost 8 years ago. It was a month after I started and when we moved in space was actually built out to have an internal usability lab which was something that the company didn’t have before. So looking to bring some of this capability in-house rather than outsourcing as it’s been done in the past. So I started out as a team of one and when I started in 2008 we really – it was supporting a couple of different websites. So Comcast.net which was originally the portal for our customers, high speed Internet customers to get their email and it was similar to like Yahoo or AOL. And then Fancast.com which was really one of the first destinations to go and watch you know full episodes or full movies online. So it was kind of an exciting space to be in. You know the frontier of like watching TV online. You know in some ways it seems like yesterday and some ways it seems like ancient history. But since then you know just every year something new just kept coming into the fold. So that was 2008. In 2009 we were working more on iPhone apps and the iPad came out. Then we started taking over the TV interface and you saw the result of that now. If anyone has our X1 system that’s the latest TV UI that we worked on for many years. We also took on hardware, so looking at the design of the remote controls, set top boxes, all kinds of things like that. Starting then to move into services, looking at the online, the acquisitions, the self service, the MyAccount app. So the scope of what we touch just keeps growing. You know looking at employee tools and some of the software that our agents use out in the field or helping with an engagement to try to evaluate and improve the tools that our techs use out in the field. So it’s gotten to the point where we almost touch like anything that someone needs to interact with in the company which is really exciting. And I’ve grown this from a team of 1 to a team of we’re up to 11 now in 8 years. So I think that shows really great commitment of the company to how important it is to understand your users and design for them.
Steve:So can you go back along that timeline. You’re sort of framing it around the things that Comcast was doing and then the way that you and your team were able to contribute, but something was also happening internally, I’m hypothesizing, that started to say hey this capability, these people that we have can help drive good design or good outcomes for customers. Are there sort of key moments or kind of transitional points that you can think about where wow we’re – yes we can help with that. Yes we’re going to start doing that. How did that come about?
Liz:That’s a great question. Well my background actually was more in field research and anthropology. I did the design research program at IIT Institute of Design. So I was more into field research, worked at Steelcase and that’s what they focused on before I came here. But when I got here you know it was really about we need to start up this usability practice and I think the big wins in the beginning were – there was definitely a hunger and appetite for this and the woman who was leading the IA group at the time, Livia Labate along with Tom Loretan who’s my VP of UX. He’s been my boss for the whole time I’ve been here. They had an appetite for this already. So I came in with two champions ready to support me and it was pretty easy. There was just a hunger and people who wanted testing and they wanted to see customers interacting with their interfaces. It was really easy to just get up and running and show some immediate results and to show some immediate impact on the UI. So with that I think the legend of my team started to grow, first within the UX group with designers coming to me saying hey we want to test this, hey we’re working on this, into you know our close knit neighbors. We were underneath the product organization and we still are. So some of the close knit neighbors in product started coming to us and saying hey I want to learn more about this or I want to learn more about that. And then they would invite someone to a readout and next thing you know someone for the business unit would be like hey can we use your group to learn more about our users in this way. So I think it was kind of a bit of an organic growth.
And thinking about key moments, that’s a great question. You know we’ve had a few of them. I think you know we did a big ethnographic study back in 2011 that got a lot of attention from our GM for the VBU which is the Video Business Unit and then some of our market research partners and then they started spreading the word and having us put our presentation up in front of other audiences within the company. And one of the people I got connected to – this is kind of a cool story – is Alan Wurtzel who’s the President of Research for NBC Universal and has been for a long time. And of course you know we own NBC Universal, we’re all the same family. And I got to build a relationship with him and got to speak at his offsite and present on some of the really cool research we’ve done and some awesome videos of people navigating our interface and talking about how passionate they are about TV and it led to a relationship there. So it’s just been growing organically, but it’s really just about finding your champions and then spreading the word, and letting them spread the word and taking those opportunities every single time it’s presented to you, to just go get in front of more people and share what you do and how exciting it is.
I think one of the keys too is you know when I present I try to bring so much enthusiasm about the subject. To really go in there – I love to be like animated up in the front of the room. I love to show videos and I put a lot of effort and I challenge my team to really be creative with their videos and make sure that they’re really engaging. I think that videos often sometimes have a really awesome impact and help humanize the experience. So I think some of that’s the turning point as well. Just being able to share your work in a really engaging way I think makes a big difference.
Steve:I’m going to try a leading question. There’s a little bit of – I mean a good leader is humble about their accomplishments and I think there’s a certain humility in your answer in that you talked a lot about other people’s interest in your work kind of drove the growth of the demand, but I want to push you a little bit and can you talk about things that were – like work is not just work. There’s work that’s relevant. There’s work that points to opportunities. There’s work that connects. You were able to – in order to drive that kind of increasing demand, it’s not just about being animated. Like it’s all the things that you said, but there’s something, I bet, that’s in the content of what you and your team have been producing and I wonder – again I’m totally leading you, but I wonder if you can, with all due humility, reflect on – what about the content of the work that you’ve done that has driven that organic growth?
Liz:So with all due humility it’s really about I always say you know our teams’ goal is to really bring – the word empathy doesn’t mean like you feel sorry for someone or you know – it’s really about – I think Indi Young just wrote the book about empathy and I was reading it and I was like that’s exactly what it is. It’s like being able to put yourself in someone else’s shoes, but not necessarily understand it because it’s not your world. And I’ve had stakeholders say to me after a readout, or say to members of my team that you know, not necessarily that we learned anything new or that we didn’t know, but you really humanized it. You really illustrated it in a way that was so impactful that we didn’t realize – you know it caused this kind of emotion in our customers. Or you know the problem was – this was the problem or this wasn’t the problem. Like this is really the heart of it. So it’s about being able to illustrate and bring empathy and just humanize the experience for folks who really aren’t out there necessarily interacting with our customers or users every day like we are. I think that’s just – that’s where the passion comes from. I don’t know if I’m answering your question?
Steve:You’re answering it and it’s a wonderful answer. You know I’ve had this experience a lot in years of doing research with – it’s sort of the classic response when you share something that you’ve learned is that we already knew that response. You just kind of flipped it around for me where it’s not – because sometimes that’s sort of confrontational and my response is to get defensive. Well, are you sure you knew it? And I think what you’re sort of talking about – I mean kind of a cultural, organizational value in doing this work is as you said to humanize. And it’s not about oh my God did you know that this thing is going on because that might not – you might not be able to surprise anyone with that, but to bring it back in a way that humanizes. And I think what I’m drawing from what you’re saying too is that’s a powerful way to drive change or to help people take action based on what you rediscover or represent.
Liz:Yeah, exactly. It was kind of an epiphany when someone said that to me you know ‘cuz sometimes we would. We’d be like oh well I don’t know if we’re really like bringing any new insights here. They’re things we already knew, but when someone said you know it’s not about necessarily bringing us anything new – and I don’t want to say that we don’t bring new insights because we bring tons of them, I know we do and we’ve affected a lot of change – but there are some things that just the company knows and they know they need to change, but nobody’s really ever told it in the type of storytelling manner that we bring to the table where like the end to end experience with videos of people, showing them their passion, their joy or their frustration and really talking about it in an eloquent way, or even things like when we go in the homes you know we often try to get multiple family members in the interview because that’s when you can really get the genuine emotion and confusion and frustration out. You know I’ve shown videos of husbands and wives getting into arguments over like how to actually use an interface. That tells a way better story then I could ever tell. Putting together a presentation that shows like for example a matrix of tasks and who succeeded and who failed. It’s a far more powerful story to say like hey this is really confusing and I’m going to show you a video of a husband and wife arguing about it because one understands it and the other one doesn’t.
Steve:This makes me think about documentary films a little bit. There’s documentary films that I like and ones that I sort of almost tolerate. I will bring this back to what you just said. You know there are those documentary films where it’s about a topic and the topic is so amazing that it seems like what the filmmaker has done is sort of open the lens and sort of be there and then kind of chop it up in a way that follows a linear narrative and you’re like I can’t believe these people are trying to win a truck by standing on it – Hands on a Hardbody was a classic documentary, like you just saw people trying to – I don’t know if you know this one. It’s a contest to win a truck and you have to have your hand on it for days and days and days. This amazing human drama emerges from this very simple thing. And not to minimize the art of the filmmaker there, but then there are other documentaries that just do – I don’t know, just wildly creative things and sort of synthesize and pull out of something, some story you never could have ever have found, that like they’ve told that story. And I enjoy them both, but I like to sort of reflect on sort of what’s the hand of the filmmaker. I know we’re not – there is an art to what we do, but we’re not making art. So just to loop it all the way back around to what you’re talking about, it is powerful to bring those stories in and kind of impact people that way. Again, I guess I’m going to lead by offering my point of view. I would hate for someone to listen to this and feel like what the field researcher does is just like point the lens at people and sort of – that we’re just scooping. We’re not digging or we’re not synthesizing. So when you come back with those videos of the husband and wife arguing, like you didn’t just – you weren’t just out shooting video, you were doing something else.
Liz:Obviously – no, we’re doing a lot more, we’re probing in a really meaningful way and I actually – I’ve given talks here at Comcast to groups of product managers. You know last year Charlie Herrin who is our – now our head of Customer Experience and a whole new group, had asked me to set up a training program for all the product managers so that they could come, ride along, or be empowered to do their own listening in the call centers, or ride along with agents, but to really understand like what is the art of what we do and how do you listen to customers. I talk about things like how to build rapport, how you want to go in. You don’t want to necessarily go in with an agenda. You want to ask open ended questions. Don’t be leading. But it’s so important to build rapport because once you build a comfort level with the people who you’re visiting – you’re visiting them in their homes – it’s amazing how they’ll start to open with you and it gets to the point where – I think I ripped this from your book, but I say this anyway, it does have a tipping point from where it’s question and answer to question to story to just you say something and they just start talking. And that’s the point that we really always strive to get to and that takes experience, that takes practice and I think that’s really important. It’s the art of – it’s the art of having a conversation because that’s really what we’re going in there to do. And sometimes you don’t know where a participant is going to take you and that’s okay sometimes. You know you can get some of the most richest or most interesting insights when the participant takes you down the path that’s most interesting to them and then you can bring it back to some of the things that you want to learn about, but I really think that it’s so important to be open minded and let the participant talk to you and show you what’s most important to them.
Steve:There’s an interesting tension – I don’t know that either of us can resolve it, or at least not in this conversation – between that territory with no map – I don’t know, like the exploring to get to what you don’t know you’re going to get to. At the same time the result is things that you mostly knew or knew to some extent, but maybe not with this amount of emotion or this amount of richness.
Liz:I’m thinking of a good example that I can share. When we first got into the home security arena, it was maybe 3 or 4 years ago, we went to go do a field study with some of our early customers and we were just hoping to learn. We were still mostly UX Product. We wanted to see how are they using the system, what are the different ways they’re using the system – you know to improve it or to be inspired for new features or ways to improve the UI of some of the apps or the touchscreen and things like that. And what we found actually was a direct correlation between you know how engaged people were with actually using their home security system and how well their installation and first time use of the system was. And that wasn’t something we were looking for at all. That was something that started to come out naturally as my lead researcher was in the field and she came back with this like amazing story and this amazing end to end process map. It was like well, it was like 6 steps and we went in the home with this expectation of studying what was step 6 and what we learned was X, Y and Z about steps 1 through 5 and how it affects step 6. You know and step 1 is just becoming aware of the system. Step 2 is calling in to order it. Step 3 is getting it installed. You know all these things that happened along the way and that was because the participants wanted to talk about that and we found it necessary to let them talk about that to understand why they were or were not using the system in a certain way. So that’s an example I think of this sort of non direct, or the power of being non directed.
Steve:So that’s an example where using this non directed approach helps you find something that you didn’t know. But you’ve also talked about examples where there’s a great power in the way that you can bring back stories that are not necessarily surprising as you said, but they’re brought back in a way. I’m just wondering, does that approach of – now I’m messing up your words – but sort of a non directed, rapport building. Letting the participant talk about what they want to talk about. Does that also bring back things that you knew, but in that powerful way that’s helping you drive change?
Liz:Yeah. So one of the things that we hear a lot about, we talk a lot about obviously in the world of cable and television is live television. And how attached are people to live TV? I guess in our own little rarefied world we tend to think that people are really moving towards cord cutting and consuming everything on Netflix and over the top HBO and that kind of stuff, yet we’re going in the homes and we’re seeing this incredible attachment still to live television and we’re still seeing that in a lot of the numbers too and a lot of the Nielsen surveys and things like that. So we know it’s true and it seems like there’s this dichotomy and people want to understand it better. So we can go in the homes and you find that people – when you let them speak in a non directed way all of a sudden they start talking about like you know why they’re attached to live TV and it has a lot to do with like a lean back experience vs. maybe a lean forward experience. You know we think of live TV very much as like oh it’s still that classic – you know Alan Wurtzel would call it television the way God intended you to watch it. That’s what he says. Like everyone sits down at 8 o’clock to watch the Cosby Show on Thursday nights. But really the attachment to live TV is so much into sports, into the low commitment, into the serendipity, into I just want to flip around and see what’s on for the next 20 minutes while I decompress, like that kind of thing. And I think we tend to overlook that. So to be able to bring back videos of people showing, like they all have different strategies for browsing the grid, the traditional grid, but how that’s happening, to really illustrate that, to explain like why there’s still an attachment to it, but letting people show us through the context of their own stories and letting us take us down whatever path they want is really interesting and really meaningful.
Steve:I feel like you just hit on what the power is of those humanizing stories. They’re not necessarily surprises, but they can help people reprioritize. We knew that, we weren’t really paying attention to it. By humanizing it you’re helping people to make decisions and understand that something’s really important. That it doesn’t have to be a surprise that blows your hair back. It can be…
Liz:Right. It’s almost like it is really important so we definitely want to still make sure we have a strong way to help people you know browse what’s on live. And it’s not to say we shouldn’t be looking at like alternative ways to do that in the future. We know that live TV is still important. We know this from numbers. We know this anecdotally even though anecdotally there’s also the sense of there’s still cord-cutting, but being able to go in the homes and ask people to just like hey, take me through your day in TV and having them illustrate that sort of brings home this notion that you know some of these behaviors are still happening and we knew they were still happening and here’s a way to illustrate them and help people understand why it’s still so important and so meaningful to the user.
Steve:What are some of the challenges that – maybe it’s less about your team, but just the kind of work – I mean you know user research as a thing. You know in an organization like yours where you’re having these successes and you’re having this kind of impact, you know what’s – how do you characterize the types of challenge that you all are looking to address?
Liz:I mean some of the challenges that we’ve been looking to address lately, at least with my team, is we’ve historically been a qualitative research team you know and qualitative research is intrinsically linked to traditional market research techniques. So there’s a lot of familiarity with many stakeholders outside of UX and especially outside of UX and product with those methods. So really trying to bring in that awareness of how important it is to – you know we’re talking a lot about talking to people, but really the heart and soul of what we do is we come in and we build rapport and then we get them showing us and that’s really what usability is too – you bring people in, but you watch them use your system. When we go in their homes and we try to get them to just recreate their lives and watch them. So really you know trying to illustrate like different approaches to qualitative research and trying to help explain what those different approaches were, that’s been one challenge that I think we’re starting to really successfully illustrate. And then the other challenge too is like I said we’ve always traditionally been a qualitative research group and you know everybody loves numbers, everybody loves analytic surveys and stuff so we’ve started to – I have a quantitative researcher on my team now and we’re starting to bring in some of the quant to compliment the qualitative research we’re doing to emphasize some of the points that we’re making, sort of establish them – some credibility to them at scale.
Steve:And what’s the difference between what the quantitative research on your team does vs. other kinds of quantitative research that Comcast is doing?
Liz:We’re trying to do it from a product development approach. So you know one of the techniques that we just started implementing, or at least we’ve started piloting is Kano analysis which is surveying in a specific way, asking people about features and their expectations, but not do you like it or are you satisfied with it, but do you expect it to be there? If it’s not there are you upset, are you indifferent? There’s a formula that sort of segments things I believe into like three outputs – the must haves, the indifferents and then the delighters. You know we’re always getting – it kind of came because we’re always being asked about what will delight our customers? What new features should we be building to delight people? It’s a great language that really speaks to the company and one of the interesting things we’ve found too is that delighters decay into must haves over time so things – you know for example about the television interface that we would hear were delighters 5 years ago. Anecdotally we kind of see how they’ve decayed into must haves as we started to do some surveying. Like these must haves are things that people just expect to be there as a baseline, almost like a neutral. So anyway, having the – I think having some numbers behind it, but not in a traditional way, but in a way that reframes it, but again still amplifies things that we’re hearing from other types of surveys and other groups in the company I think is really, really powerful.
Steve:It makes me think about car features and how they seem – that industry seems to have sort of figured out – you know things come in at the ultra high end and then you now you can rent a Toyota Corolla at the airport, if they make Corollas or whatever, and it has a backup camera. Whether that’s driven by legislation, whatever, all these features seem to trickle down.
Liz:That’s a great example.
Steve:And is that the same – do you think that’s the same – I don’t know what’s going on in the car industry, if that’s about economics or is that about you know…
Liz:No, but that’s a great example. Like right now backup cameras, it’s a delighter. People who have them love it. But it’s not standard and there are still a lot of people out there driving cars that don’t have them, but I would predict you know in 5 years maybe that’s just going to be a must have and an expectation, especially if they’re putting it in Corollas. So it seems like that’s where you see the decay from delighter to must have when it’s no longer necessarily a luxury or something high end. It’s something that’s expected in any make and model.
Steve:It seems like it’s easy to see that framework in the car industry and probably harder in the things that you work on.
Liz:Right.
Steve:Because there’s just not…
Liz:It’s probably more practical stuff in the things we work on, like the number of tuners that you can use at once to record shows. It used to be like so exciting, I have a DVR. Like oh my God I can watch one show and record the other at the same time. Now people have four tuners and they can record four shows and watch one live at the same – or five tuners. You need 5 tuners to do that. And so things like that have become more than must haves. Same thing with the Internet, like the must haves. The speeds keep going up and how much you’re able to download in a certain amount of time.
Steve:Is the tuner thing about multiple viewers in a household?
Liz:Sometimes, yes. Sometimes. Or sometimes it’s about like a TV passionate – you know there are people who are super attached to TV and like to record a lot of things. They like to stockpile stuff. So there’s a lot of different behaviors out there that drive that.
Steve:Right. I’m doing that thing that people do where like 5 shows at once, that seems like a lot. What’s wrong with that person? But your answer is of course the appropriate, empathetic one. There’s use cases that you support and I’m just doing that thing that people do.
Liz:Exactly.
Steve:And that’s right. Maybe there’s a…
Liz:How many people use five tuners at once? I actually – I’m sure we have analytics on that, but it’s even more the perception.
Steve:Ah, right.
Liz:I think.
Steve:Right. I bet the – you don’t have to answer this, but I bet the incremental cost to add a third tuner and a fourth tuner is probably low and so right now you have more capability. I mean it goes back to what you’re saying about things that are delighters vs. must haves like – and I’ve seen this in studies that I’ve done unrelated to your business area where you know more capability then you think that you could possibly is like that’s very comforting for people. Like, I’m set.
Liz:A great example – a great example within the DVR space maybe is the, like remote DVR. So I remember when I was working on the first app for mobile DVR – oh my gosh I want to say that was like 2009 or so. You know where it was like super exciting to like be able to be like oh my God I’m out and I forget to set the DVR and now I can log in and actually set my DVR. That’s just – you know you can do that now. It’s still a delighter. So I want to say that I don’t know that’s it decayed to a must have, but I would say for – it’s getting close.
Steve:Interesting. Okay, let’s shift gears a little bit. You’ve talked about your team and some of the people on your team. Can you give kind of a 1000-foot view of the kinds of people, the kinds of skills? Like what is – what have you built and kind of who have you looked for and what’s the makeup I guess?
Liz:So like any UX research team we come from a variety of different backgrounds. You know the 31 flavors or whatever you want to call it. So my background is in physics. I’ve got people with backgrounds in information systems, anthropology. You know we’ve got someone from CMU (Carnegie Mellon’s graduate program). I’ve got people from – a PhD at Temple. So it’s really definitely a definite mix. And I think everyone of the team sort of has an area of strength. You know I’ve got someone on the team who’s just this absolutely amazing, almost like anthropologist – go out in the field – you know some of the things we were talking about before, about just being non directed and building rapport. You know that’s a real strength. Someone else on the team who really, really loves doing more of like the usability type evaluations. As I said I have a quantitative researcher on the team so – I don’t want to say – you know everyone is a generalist and we don’t have the luxury of being specialized, but I think everyone on my team has an area of strength I think. And I think it’s really important to have that mix as well. You know we get an open head or we’re looking to hire – among my senior group I have two managers on my team and a principal. You know the four of us will sit down and talk about you know what do we really need? What kind of skill sets do we need? Where do we have gaps or things like that. We’ll look for resumes and people who will fill that gap.
Steve:I have to call an open head as like the best piece of corporate jargon that any of us are going to hear this week. I have an open head.
Liz:I have an open head.
Steve:Well we’re going to send you to the doctor then.
Liz:It just means I have a requisition to hire somebody for a full time job with benefits. One of the things we talk about in user research of course is like using natural language with our customers.
Steve:And do we talk about making fun with our natural language because that’s the principal of interviewing that I just leveraged. So when you’re thinking about managing workload how do you think about – you know with people being generalists, how do you think about you know here’s an internal client with a need, here’s who’s going to work on that? How do you make those decisions?
Liz:That’s a great question. Right now I do have, I mentioned I have like a senior team and I tend to have them embedded in one subject matter area. So I have someone who is really leading most of the television and entertainment research, someone high speed Internet and home and then another one on customer experience. Underneath that I have a pool of researchers. They tend to move around a bit more. I like to try to keep people – you know again it’s kind of like the strengths in terms of methodologies. I also have people who have affinity to certain subject matter. So you know I have someone on my team who’s just a DIY geek about home automation so I try to keep that person working in the home security/home automation space because they just have a lot of subject matter expertise there. So some of it’s like you know if I can and if the demand is there, or if the need is there I’ll try to keep people – you know keep that subject matter expertise like running so that we can leverage that and move faster and build upon you know insights from the past. So that’s kind of how we do it and usually there’s enough work balance between each of the different verticals that it isn’t too much of a – you know people aren’t switching around too much.
Steve:Let’s go back to the open head bit. So now I’m using natural language – I did it straight and you giggled so I was like I’m back in natural language.
Liz:Calling me out on my corporate lingo. Okay.
Steve:So when you’re in an open head situation and you’re looking at resumes what do you look for? What do you see and what do you want to see I guess?
Liz:What do you see and what do you want to see? You know it’s hard with a resume actually. I’m almost always more interested in if someone reaches out to me with a note, you know, or a personal connection. I mean I can look at a resume and sometimes people have, what looks like they check the boxes and then we’ll want to talk to them, but it’s really about evaluating someone over the phone then on that first pass. So the resume you know should have like you know some experience, unless we’re looking for someone really entry level, but even if they’re entry level you know some sort of interest in doing something UX and user researcher related while they were in school. But anyway, yeah, it’s really more about I would say how are you going to demonstrate to me your level of passion for user research and your soft skills. So like for example you know I hired someone a few years ago – I actually got a note from one of our engineers – one of our scrum masters I think she was. And she said I met this woman at a party, she’s getting ready to finish her master’s degree at Temple and she’s really, really interesting and she had a background in mass communications and anthropology. She loved technology and she was doing work in like presence. And she was just a really neat person and I was telling her about your – we have a coop or like internship program and I was telling her about that and she was interested. So and I said put her in touch with me. But to me that’s like the test right, put her in touch with me to see if she actually gets in touch with me. Not only did she get in touch with me, she got in touch with me right away with this incredibly thoughtful email about all the different things she was thinking about and she wasn’t like I want a job or I absolutely must have a career in UX research. She was almost like I’m really interested in technology. I’m really interested in people. Not really sure where this is all going to go and I would just love to learn more about your team and what you do. So I got on the phone with her and I just thought she had amazing aptitude, she had great maturity and so then I brought her in to meet the team and we actually felt like she was qualified to be an entry level researcher. I thought she was way beyond an internship position. But it was really about you know the impression she made with me in terms of the way she communicated with me and communicated beyond like I know how to do these skills. It was more like I’m curious, which I can’t teach. You know I’m personable. I can get along with people. I ask good questions. Those are things that are harder to teach, the aptitude. And she demonstrated that she had that.
Steve:Some of those seem very general to try to reach out to someone inside a company and make them feel like they would want to work with you and some of them seem like there’s elements of like this is what makes for someone who I want to have work with me as a researcher specifically. Can you highlight what are the researcher ingredients and how you’re trying to suss those out?
Liz:Yeah, sure. I mean a big one I just mentioned is curiosity. Like if you don’t have a natural curiosity then I can’t teach you to be a researcher ‘cuz that’s like, it has to be ingrained in you, right. I mean to be a researcher is to want to understand. You know like I was that kid that was 5 years old, like why, why, why, why, why? And you know I’m still like that as an adult. I need to know how things work. So I look for that. Another thing is like pattern recognition. I think a lot of being a researcher is you know when you’re out, especially when you’re doing qualitative work, like really being able to understand. You know you hear this one story over here and you hear this one story over there and how are they related and what’s the common thread between them? I’m trying to think of a good example off the top of my head and of course now I’m blanking. But being able to really recognize patterns I think is important. Being a good listener too. I talked about curiosity, but the other side of curiosity is I love it when I interview somebody and then at the end when I say do you have questions for me, not only do they have questions which demonstrates their curiosity, but they’re not canned questions. They’re questions based on things that I’ve talked about, about my team and about what we do. So demonstrating that you can actively listen and process what you’re hearing and then ask thoughtful questions back I think is really critical. You know I can teach someone you know to go read about heuristics or good read a book, you know how to write tasks, how to run Morae and all these other hard skills, but I can’t necessarily teach someone how to be curious or you know how to be able to recognize patterns. I’m trying to think of another good one that I look for. I mean maybe it’s the soft skills too ‘cuz to be a researcher you know you really need to be able to just interact with people and make them feel comfortable. So I look for people who are good conversationalists, who are just comfortable talking to people they haven’t met before, come in, you know have a certain amount of confidence to them, with still some humility. ‘Cuz that’s what you need when you’re going to be going in and interviewing. Customers – and that’s also what you need when you walk in the room to present to your stakeholders. You know you have to be sensitive to their needs as well and what they need to hear in order to take action on your findings. You know and you may get some – when you’re presenting too you know you get questions at the end and sometimes the questions can be challenging, right. You know, especially if like something that you’re presenting on isn’t you know finding sort of challenges. It’s like one of their core beliefs about what they thought about their product. So how do you handle those questions? How do you react in that kind of a situation where you might be challenged I think is really important to observe as well.
Steve:What’s the soft skill – what’s the soft skill that describes that, like being able to handle these questions?
Liz:Yeah, that’s great. I’m not sure what the word is for it.
Steve:It must be categorized.
Liz:It must be categorized right. You’re going to make me dig back into my IA world in my brain. I think it’s about not being you know – I mean maybe it is humility. You know it’s not being so attached to your idea and your agenda that you need to be able to compromise.
Steve:That’s a great challenge in an interview situation to be able to demonstrate your confidence and your humility together.
Liz:Yeah it is, isn’t it.
Steve:Like that’s a bit of a magic trick. And I’m not saying that people can’t pull it off. It is sort of about who you are as a person. If I had to think about how would I do that I would definitely have to scratch – be scratching my head and want to practice that a little bit. I agree with you those are essentially, but you just – I don’t know it makes me have some empathy for people that are knocking on doors and trying to put all those best feet forward at the same time.
Let’s switch gears again and maybe sort of build on this. You talked a little about – I feel like at the top of the conversation you mentioned some of your other experiences that you had had. You’ve had other careers before this one. You want to talk a little about your background and maybe some of the highlights or milestones for you as you have…
Liz:Sure. How much time do you have? I’m just kidding.
Steve:How much time do you have?
Liz:How much time do we have left? No, I’m just kidding. Yeah, so my degree, my undergraduate degree is actually in physics which I always joked after I was done, the only reason we – my friend and I – would say we majored in physics is because we liked the way it kind of stopped the conversation at a party when someone would ask us what we did, what we majored in. You know I kind of grew up being just you know one of those women that was smart at math and science which is a little bit unusual, especially 20 years ago, and I just assumed I’d major in math and science and then when I was in school I was working in labs and I was like God this is so boring, there’s no way I can do this the rest of my life. I thought I would just get a PhD and go do research and get published in journals and I just thought God how boring is that. So I wasn’t sure what I wanted to do. I went to Penn and I worked in their admissions office and through that I actually just got a job right out of school doing, as an admissions counselor for a local college for a couple of years, which wasn’t as interesting as being an admissions officer for a school like Penn, you know. But I thought it would be cool, but really the lessons I took away from that – you know it was a lot about being independent, being on the road, learning how to sell, how to speak to things, how to present. And I knew that wasn’t going to be my career or anything so I had a friend working for a textbook company, Prentice-Hall, big name, and she was working up at their headquarters and said hey we have an opening for a marketing assistant, you’d be great and it’s in science and math textbooks. So ended up getting a job up there and again I think some of things I learned up there – first of all I worked for two really awesome women who were leaders in the marketing area and you know they taught me not to be afraid to sell myself and to make sure that people knew that I was like doing a good job. It wasn’t enough to get a compliment. It was to make sure you forwarded your compliments on so that people who could help your career knew that you were being acknowledged. You know and I knew – I remember one time they were talking about moving me up to manage a big sales territory in Ann Arbor, Michigan which is you know, in the textbook industry, you want to move up you often need to go manage a sales territory. You need to go out into the field and one of them said to me, she’s like you need to go knock on everyone’s door of these executive editors and stuff and just ask their advice and let them know you’re interested in this position and I like freaked out and I got like all nervous and stuff and they’re like no you have to go do that ‘cuz these are the people who are going to promote your career. So I got like really good advice that way, you know and they were from women which I thought was really important. So anyway I spent five years doing sales marketing and editorial acquisitions in the science and math space in textbooks which is the only job I ever had that was even tangentially related to my degree when I was working on physics textbooks and I was actually a little conversant with the authors. But that was where I sort of made the transition to UX. I always loved – I always loved graphic design and typography and people told me that like I was too smart to do that and I would never make any money and I finally just decided well I didn’t care, I really loved it and we were working on a brand new textbook for biology and the biology editor, I was talking to her about this a little bit and she pulled out this book, it was Richard Saul Wurman’s Information Architects book and it profiled people like Erik Spiekermann and a lot of really just interesting people who thought about how to design things. This was way back in the print age, or signage systems and stuff, but thought about how to design things based on how people were going to actually use them and I was like this is just awesome. And they were hiring somebody as a consultant who was featured in that book to help design like the program for how they were going to design this new textbook. I just thought that was so cool and so I finally just decided you know what I think I want to change careers. And I left Prentice-Hall. I kept doing some freelance work, but I started studying graphic design full time at School of Visual Arts. You talk about key moments. A key moment for me is I was taking a typography class and we were supposed to redesign a spa brochure. It was just one of those handmade things on like – I forget what they were called at the time, but just one of those things where anyone could put together a brochure on their desktop computer and it was pretty ugly. We were supposed to just redesign it with nice type and special attention to typography and I actually rewrote all the content before I redesigned it and I was the only person that did that and I came back in and I remember a couple of the other students were like why did you do so much work? And I was like I can’t design something I don’t understand and that was like almost like the epiphany for me and that’s when I realized I was actually probably more of an information architect then a designer and so that’s how I got into, made the transition into UX and that was – it wasn’t even called UX at the time obviously. This was like around 2000/2001. But it was the right time, right place to sort of jump into that arena and so I moved back to Philadelphia which is where I’m from and through some connections you know got a job at a local agency, basically doing jack of all trades what would be UX now – small agency that built websites for nonprofits and educational clients. So I did everything. I did stakeholder interviews, content strategy, information architecture, you know building site maps. I actually did the visual design. I even did some front end HTML. So that’s where I really learned all my chops and then from there I just got really excited about understanding how people would actually like use the stuff I was designing. Back to like thinking about Erik Spiekermann and the case study of how he thought about redoing the map for the Berlin subway I think it was. Anyway that’s when I went off to Institute of Design for grad school and learned about design research and then got really excited about you know watching users and learning what they do. So from there I just went to Steelcase for a couple of years. Worked for a really awesome woman there named Joyce Bromberg who again was just a great inspiration. She was the one – she’d give me a compliment and I’d be your typical woman like oh it was no big deal and she’d be like take it like a man, say thank you. So she taught me to always just say thank you when someone compliments me. You know you never downplay your successes. It was neat to be out of the digital arena for awhile and then I decided I wanted to move back to Philly and I’ve been here at Comcast ever since. So I think I asked you how much time you had and that was probably the longest answer ever.
Steve:No I thought you like managed to do like a really clipped, swift kind of active version of the whole career arc. That’s great. I think we will keep paying attention to time here. What are some things about you as a leader or a manager that I think might be important for us to understand or maybe things that we wouldn’t see on your LinkedIn profile?
Liz:I mean I don’t know if this would come through in my LinkedIn profile, but as I’ve moved up I never thought I would be a manager, let alone a director or a senior director. When I started here at Comcast it was like hey here’s this lab and we started up and I thought I’d be a team of one, probably forever. No actually, I didn’t. But I think as I’ve moved up and gotten more into the management side of things I’ve really found like a passion for passing it forward. You know as I was talking about in the past you know some of these great mentors I’ve had who you know taught me how to take a compliment or make sure other people knew my successes or you know taught me how to be patient and have a vision for something. I think that’s something I didn’t talk about before. You know sometimes people ask me like how did you grow this team so quickly, because I know that can be a struggle for some folks, you know, in the corporate world, trying to build up a fledgling UX or user research practices. How do you go from a team of one to ten in a short amount of time? You know and it’s really having the patience. It’s like you’re not going to go from 1 to 10 in one year so it’s like celebrate your small successes and have a vision for where you want to go but be patient for it. So I think it’s kind of passing that forward to the folks below me. You know I’m really interested, I read a lot of books on how to motivate, what people are motivated by and trying to understand like a little bit about the psychology of that aspect of our work, and really trying to do for some of the younger folks what those folks did for me and helping them build their confidence, move their own careers forward, you know feel like they can do it and to figure out what you want to do and not be afraid to take risks.
Steve:So as we think about wrapping up I will fall back as I often do in my stock questions because I’m a creature of habit, or I’m lazy or something else. Because they’re great questions. Is there anything that we should have talked about, about you and your work at Comcast or your background that you’d like to make sure that we include in this?
Liz:I mean again, just to sum up, I think one of the things that maybe, that isn’t emphasized enough – you know we talk a lot within UX about like the hard skills and the methods and stuff like that and I know you go around – I think you said you were just at World IA Day and you gave Soft Skills are Hard and I think that that, that’s just so important and I think that’s something I’d love to emphasize to folks that’s so important in your career. Don’t neglect your soft skills. And I think I got lucky in that I had this – you know that period for 5 or 6 years where I was you know working as basically like almost in a sales and marketing capacity where I learned to be comfortable interacting with others. People who – you know learning how to work a room. Learning how to make conversation with people I really didn’t know. Learning how to negotiate. Learning how to present. And then again something I think I talked about is you know never turn down an opportunity. So I’ve had people like present opportunities to me where like I gulp really hard and then I’m really glad I did it you know. And one example is the University of Pennsylvania actually, they contacted me a few years ago for – to come back and speak on a panel at parent’s weekend. If you know anything about Penn, Penn is a very pre-professional school. So you basically go there and if you don’t say I want to be a doctor, a lawyer or an i-banker, or a consultant – it’s like oh my God what’s my child going to do and how are they going to make any money. And so they had noticed that I had this unusual major and this unusual career yet I seemed to be pretty successful. So they invited me to come back and speak on this panel and I remember when the guy called me and I was thinking alright, you know I’ll do it. And then when he told me there would be like 400 people in the audience I almost choked because I had never spoken to a crowd that big before. I think the biggest crowd I ever spoke to is maybe 40 or 50 people and I just like looked out and I was like okay I’ll do it. And it was just so energizing and so much fun and I think you know that was the beginning of like so when Alan Wurtzel invited me to speak at his offsite and they said that would be like 200 people I was like sure. I got a little nervous, but now I’ve gotten to the point where, like I’m super comfortable speaking in front of large groups, to the point where like now I can almost do it on the fly if I already have the content ready and I’m not afraid to be animated or to tell stories, or even be a little self deprecating in front of a large group which helps like humanize yourself and build some rapport with your audience. So don’t ever be afraid to take an opportunity even if it scares you ‘cuz you’ll always learn something for it and the more you do something the more comfortable you become doing it. So I think that that’s just really important.
Steve:But Liz, what if I fail?
Liz:Well you can’t think that, right. I don’t know. You know someone actually asked me that once. I’m so glad you said that because someone asked me. I remember I was working at Prentice-Hall. I can’t remember the context now but they said well what if you fail and I was just like I’m not afraid I’m going to fail. I don’t know, maybe that’s something intrinsic, but I guess don’t be afraid of failure either.
Steve:Well I like – I mean just your framework is revealed when that question sort of just brings you up short, like I don’t know. I’m not thinking about that.
Liz:Right. And it sounds like cocky. We’ve been talking about humility and I’m like I’ve never failed . I mean sure I’ve failed here and there, but I mean it’s not really failure. Maybe it was like a little bit of setback. Oh you know this didn’t go – I didn’t get that job that I wanted to get, but I got the confidence to do even better on the next interview and got another job that turned out to be even better.
Steve:I didn’t hear you say I’ve never failed. I heard you say like…
Liz:I’m not afraid of it.
Steve:…don’t think about it in terms of what if you fail.
Liz:Right.
Steve:You sort of were attacking the question. I didn’t hear you say I don’t fail.
Liz:No.
Steve:Okay. Well those are some lovely sort of parting thoughts. Just to wrap up, wrap up, do you have any questions? No.
Liz:Have other people asked you questions?
Steve:Is that a meta question or an actual question? Have other people asked me questions? Yes, sometimes.
Liz:I don’t know. Steve, is there anything else that maybe, that I could add that would be unique perspective. I listened to some of your other podcasts and there’s always been a lot of talk about the methods and the structure of the team. I think we covered that a bit though. If there’s anything unique…
Steve:I mean you’re going to ask me if there’s anything unique about you? You’re the person that can answer that.
Liz:And I don’t know.
Steve:Okay. It sounds like we’re done then. Alright, thanks very much Liz.
Liz:Thanks, Steve.
The post 10. Elizabeth Kell of Comcast first appeared on Portigal Consulting.
In this episode I chat with Elizabeth Kell, the Senior Director of User Research at Comcast. We talk about the growth of Comcast’s user research practice, essential soft skills for research candidates, and putting a human face on the people that use your products.
If you don’t have a natural curiosity then I can’t teach you to be a researcher. It has to be ingrained in you. To be a researcher is to want to understand. I was that kid that was 5 years old, like why, why, why, why, why? And you know I’m still like that as an adult. I need to know how things work. So I look for that [when hiring]. – Elizabeth Kell
Show Links* Elizabeth Kell * Comcast * Comcast Center * Fancast * X1 * MyAccount app * IIT Institute of Design * Steelcase * Livia Labate * Tom Loretan * Alan Wurtzel * Practical Empathy by Indi Young * Hands on a Hardbody * Charlie Herrin * Tipping from questions into stories * Over-the-top * Kano analysis * Prentice-Hall * Information Architects by Richard Saul Wurman * School of Visual Arts * Erik Spiekermann and the Berlin subway map * Joyce Bromberg * World IA Day * Soft Skills Are Hard
Follow Dollars to Donuts on Twitter (and Stitcher) and take a moment to leave a review on iTunes.
TranscriptSteve Portigal:Alright Liz, well thank you. Thanks for being part of the podcast.
Liz Kell:Thanks, Steve.
Steve:So I’m going to ask you the question that I always ask everybody at the start of almost every interview about any topic I think it is which is just to give a sort of basic intro to start us off.
Liz:Sure. So my name is Liz Kell. I’m Senior Director of User Research here at Comcast in our headquarters here in Philadelphia. I run a team of 11, a diverse team of user researchers and I’ve been doing this for about 8 years.
Steve:Another question that I often ask was so people in the US will know the Comcast name, other people might not. So – and probably there’s some – I guess you can explain Comcast and we’ll learn something about the company in terms of how you describe it. So with that long preamble, like what is Comcast?
Liz:What is Comcast for the folks who don’t know or don’t have the services at home? Comcast is the largest cable Internet service provider in the United States. We have over I want to say 22 million video subscribers and 23 million high speed Internet subscribers. I may be slightly off on those numbers so hopefully nobody fact checks me. We also provide digital voice service to the home, home security services and yeah, those are our four main business areas right now. And so that’s what we do.
Steve:So within your group’s purview I guess do you get involved in all those different areas of the business?
Liz:We’re involved in all areas of the business and in the layers beneath too, so all the service layers that go into supporting all four of those products and services that we deliver. So we look at the customer service – the whole customer end to end experience and that includes everything from you know the sales and the marketing and going in and observing what’s going on in the call centers, to looking at the installation, riding along with the agents, watching people try to self install kits. Watching people try to get self-help or call into the call center and troubleshoot and things like that. So in addition to evaluating the product we’re also evaluating the services.
Steve:When you say product, for a company like Comcast, what’s a product?
Liz:Product – I think of the product as like the thing.
Steve:Like a physical thing?
Liz:Sort of. Yeah, I mean – I mean it’s a bunch of electrons right, but the television is something that comes to this big screen on your TV and you have the interface that you interact with. You have the set top box that you interact with. You have the remote controls that you interact with. So that’s like kind of thing and we do a lot of evaluating and trying to improve those things. For high speed Internet there’s less to interactive with. That was one of our big insights when we were doing some research on the high speed Internet product. But really how do people interact with that physical gateway which is you modem and router in one. With home security there’s a lot of interesting things to explore around like where do people decide to put the different sensors or what kind of different pieces of home automation are they adopting and why and how are they using them and how are they controlling them, either through mobile or web interfaces. So there’s a lot of like the thing I would call it, like a tangible thing we’re evaluating as well.
Steve:So that’s hardware and software?
Liz:Both, yeah.
Steve:So these questions are naïve ‘cuz I don’t necessarily have the language for them, but so there’s hardware and software and services, but the services I feel like you’re describing sort of are more customer support services. Are there other kinds of services that Comcast delivers that you’re examining or studying, trying to improve, that aren’t about sort of managing my success with the hardware and software I guess?
Liz:Really about self-service and getting help and being able to get the support you need for your things. I keep calling them things now. I got that word from you.
Steve:So Comcast is kind of a hardware and software company and there’s a service layer to enable hardware and software. I’m sort of thinking of someone that’s like more of – a company that’s a services company like has services as a thing that you would consume, but you’d only consume Comcast services in order to get to a point where you’re having the hardware and software, the thing.
Liz:Yeah. I mean people subscribe to our services and they get something for it which I consider – I call product. You know we’re part of a product organization. So people subscribe to our services so it’s really I think about figuring out how to get the help and support you need with the services and then being able to use the products. So we look at both. I don’t know if I’m answering the question or not.
Steve:I think we’re both totally confused now. Let’s see if we can get through this without using the word product or service or thing again. So you started off too by describing the size of your team. Can we go back in time and talk about the history of your role? Sort of the history of user research and insights at Comcast.
Liz:Sure, of course. So I said I’ve been here, I’m coming up on 8 years. I was actually hired to be the first user researcher on staff when we moved into – we have a new headquarters here in Philly which isn’t really new anymore. We moved in almost 8 years ago. It was a month after I started and when we moved in space was actually built out to have an internal usability lab which was something that the company didn’t have before. So looking to bring some of this capability in-house rather than outsourcing as it’s been done in the past. So I started out as a team of one and when I started in 2008 we really – it was supporting a couple of different websites. So Comcast.net which was originally the portal for our customers, high speed Internet customers to get their email and it was similar to like Yahoo or AOL. And then Fancast.com which was really one of the first destinations to go and watch you know full episodes or full movies online. So it was kind of an exciting space to be in. You know the frontier of like watching TV online. You know in some ways it seems like yesterday and some ways it seems like ancient history. But since then you know just every year something new just kept coming into the fold. So that was 2008. In 2009 we were working more on iPhone apps and the iPad came out. Then we started taking over the TV interface and you saw the result of that now. If anyone has our X1 system that’s the latest TV UI that we worked on for many years. We also took on hardware, so looking at the design of the remote controls, set top boxes, all kinds of things like that. Starting then to move into services, looking at the online, the acquisitions, the self service, the MyAccount app. So the scope of what we touch just keeps growing. You know looking at employee tools and some of the software that our agents use out in the field or helping with an engagement to try to evaluate and improve the tools that our techs use out in the field. So it’s gotten to the point where we almost touch like anything that someone needs to interact with in the company which is really exciting. And I’ve grown this from a team of 1 to a team of we’re up to 11 now in 8 years. So I think that shows really great commitment of the company to how important it is to understand your users and design for them.
Steve:So can you go back along that timeline. You’re sort of framing it around the things that Comcast was doing and then the way that you and your team were able to contribute, but something was also happening internally, I’m hypothesizing, that started to say hey this capability, these people that we have can help drive good design or good outcomes for customers. Are there sort of key moments or kind of transitional points that you can think about where wow we’re – yes we can help with that. Yes we’re going to start doing that. How did that come about?
Liz:That’s a great question. Well my background actually was more in field research and anthropology. I did the design research program at IIT Institute of Design. So I was more into field research, worked at Steelcase and that’s what they focused on before I came here. But when I got here you know it was really about we need to start up this usability practice and I think the big wins in the beginning were – there was definitely a hunger and appetite for this and the woman who was leading the IA group at the time, Livia Labate along with Tom Loretan who’s my VP of UX. He’s been my boss for the whole time I’ve been here. They had an appetite for this already. So I came in with two champions ready to support me and it was pretty easy. There was just a hunger and people who wanted testing and they wanted to see customers interacting with their interfaces. It was really easy to just get up and running and show some immediate results and to show some immediate impact on the UI. So with that I think the legend of my team started to grow, first within the UX group with designers coming to me saying hey we want to test this, hey we’re working on this, into you know our close knit neighbors. We were underneath the product organization and we still are. So some of the close knit neighbors in product started coming to us and saying hey I want to learn more about this or I want to learn more about that. And then they would invite someone to a readout and next thing you know someone for the business unit would be like hey can we use your group to learn more about our users in this way. So I think it was kind of a bit of an organic growth.
And thinking about key moments, that’s a great question. You know we’ve had a few of them. I think you know we did a big ethnographic study back in 2011 that got a lot of attention from our GM for the VBU which is the Video Business Unit and then some of our market research partners and then they started spreading the word and having us put our presentation up in front of other audiences within the company. And one of the people I got connected to – this is kind of a cool story – is Alan Wurtzel who’s the President of Research for NBC Universal and has been for a long time. And of course you know we own NBC Universal, we’re all the same family. And I got to build a relationship with him and got to speak at his offsite and present on some of the really cool research we’ve done and some awesome videos of people navigating our interface and talking about how passionate they are about TV and it led to a relationship there. So it’s just been growing organically, but it’s really just about finding your champions and then spreading the word, and letting them spread the word and taking those opportunities every single time it’s presented to you, to just go get in front of more people and share what you do and how exciting it is.
I think one of the keys too is you know when I present I try to bring so much enthusiasm about the subject. To really go in there – I love to be like animated up in the front of the room. I love to show videos and I put a lot of effort and I challenge my team to really be creative with their videos and make sure that they’re really engaging. I think that videos often sometimes have a really awesome impact and help humanize the experience. So I think some of that’s the turning point as well. Just being able to share your work in a really engaging way I think makes a big difference.
Steve:I’m going to try a leading question. There’s a little bit of – I mean a good leader is humble about their accomplishments and I think there’s a certain humility in your answer in that you talked a lot about other people’s interest in your work kind of drove the growth of the demand, but I want to push you a little bit and can you talk about things that were – like work is not just work. There’s work that’s relevant. There’s work that points to opportunities. There’s work that connects. You were able to – in order to drive that kind of increasing demand, it’s not just about being animated. Like it’s all the things that you said, but there’s something, I bet, that’s in the content of what you and your team have been producing and I wonder – again I’m totally leading you, but I wonder if you can, with all due humility, reflect on – what about the content of the work that you’ve done that has driven that organic growth?
Liz:So with all due humility it’s really about I always say you know our teams’ goal is to really bring – the word empathy doesn’t mean like you feel sorry for someone or you know – it’s really about – I think Indi Young just wrote the book about empathy and I was reading it and I was like that’s exactly what it is. It’s like being able to put yourself in someone else’s shoes, but not necessarily understand it because it’s not your world. And I’ve had stakeholders say to me after a readout, or say to members of my team that you know, not necessarily that we learned anything new or that we didn’t know, but you really humanized it. You really illustrated it in a way that was so impactful that we didn’t realize – you know it caused this kind of emotion in our customers. Or you know the problem was – this was the problem or this wasn’t the problem. Like this is really the heart of it. So it’s about being able to illustrate and bring empathy and just humanize the experience for folks who really aren’t out there necessarily interacting with our customers or users every day like we are. I think that’s just – that’s where the passion comes from. I don’t know if I’m answering your question?
Steve:You’re answering it and it’s a wonderful answer. You know I’ve had this experience a lot in years of doing research with – it’s sort of the classic response when you share something that you’ve learned is that we already knew that response. You just kind of flipped it around for me where it’s not – because sometimes that’s sort of confrontational and my response is to get defensive. Well, are you sure you knew it? And I think what you’re sort of talking about – I mean kind of a cultural, organizational value in doing this work is as you said to humanize. And it’s not about oh my God did you know that this thing is going on because that might not – you might not be able to surprise anyone with that, but to bring it back in a way that humanizes. And I think what I’m drawing from what you’re saying too is that’s a powerful way to drive change or to help people take action based on what you rediscover or represent.
Liz:Yeah, exactly. It was kind of an epiphany when someone said that to me you know ‘cuz sometimes we would. We’d be like oh well I don’t know if we’re really like bringing any new insights here. They’re things we already knew, but when someone said you know it’s not about necessarily bringing us anything new – and I don’t want to say that we don’t bring new insights because we bring tons of them, I know we do and we’ve affected a lot of change – but there are some things that just the company knows and they know they need to change, but nobody’s really ever told it in the type of storytelling manner that we bring to the table where like the end to end experience with videos of people, showing them their passion, their joy or their frustration and really talking about it in an eloquent way, or even things like when we go in the homes you know we often try to get multiple family members in the interview because that’s when you can really get the genuine emotion and confusion and frustration out. You know I’ve shown videos of husbands and wives getting into arguments over like how to actually use an interface. That tells a way better story then I could ever tell. Putting together a presentation that shows like for example a matrix of tasks and who succeeded and who failed. It’s a far more powerful story to say like hey this is really confusing and I’m going to show you a video of a husband and wife arguing about it because one understands it and the other one doesn’t.
Steve:This makes me think about documentary films a little bit. There’s documentary films that I like and ones that I sort of almost tolerate. I will bring this back to what you just said. You know there are those documentary films where it’s about a topic and the topic is so amazing that it seems like what the filmmaker has done is sort of open the lens and sort of be there and then kind of chop it up in a way that follows a linear narrative and you’re like I can’t believe these people are trying to win a truck by standing on it – Hands on a Hardbody was a classic documentary, like you just saw people trying to – I don’t know if you know this one. It’s a contest to win a truck and you have to have your hand on it for days and days and days. This amazing human drama emerges from this very simple thing. And not to minimize the art of the filmmaker there, but then there are other documentaries that just do – I don’t know, just wildly creative things and sort of synthesize and pull out of something, some story you never could have ever have found, that like they’ve told that story. And I enjoy them both, but I like to sort of reflect on sort of what’s the hand of the filmmaker. I know we’re not – there is an art to what we do, but we’re not making art. So just to loop it all the way back around to what you’re talking about, it is powerful to bring those stories in and kind of impact people that way. Again, I guess I’m going to lead by offering my point of view. I would hate for someone to listen to this and feel like what the field researcher does is just like point the lens at people and sort of – that we’re just scooping. We’re not digging or we’re not synthesizing. So when you come back with those videos of the husband and wife arguing, like you didn’t just – you weren’t just out shooting video, you were doing something else.
Liz:Obviously – no, we’re doing a lot more, we’re probing in a really meaningful way and I actually – I’ve given talks here at Comcast to groups of product managers. You know last year Charlie Herrin who is our – now our head of Customer Experience and a whole new group, had asked me to set up a training program for all the product managers so that they could come, ride along, or be empowered to do their own listening in the call centers, or ride along with agents, but to really understand like what is the art of what we do and how do you listen to customers. I talk about things like how to build rapport, how you want to go in. You don’t want to necessarily go in with an agenda. You want to ask open ended questions. Don’t be leading. But it’s so important to build rapport because once you build a comfort level with the people who you’re visiting – you’re visiting them in their homes – it’s amazing how they’ll start to open with you and it gets to the point where – I think I ripped this from your book, but I say this anyway, it does have a tipping point from where it’s question and answer to question to story to just you say something and they just start talking. And that’s the point that we really always strive to get to and that takes experience, that takes practice and I think that’s really important. It’s the art of – it’s the art of having a conversation because that’s really what we’re going in there to do. And sometimes you don’t know where a participant is going to take you and that’s okay sometimes. You know you can get some of the most richest or most interesting insights when the participant takes you down the path that’s most interesting to them and then you can bring it back to some of the things that you want to learn about, but I really think that it’s so important to be open minded and let the participant talk to you and show you what’s most important to them.
Steve:There’s an interesting tension – I don’t know that either of us can resolve it, or at least not in this conversation – between that territory with no map – I don’t know, like the exploring to get to what you don’t know you’re going to get to. At the same time the result is things that you mostly knew or knew to some extent, but maybe not with this amount of emotion or this amount of richness.
Liz:I’m thinking of a good example that I can share. When we first got into the home security arena, it was maybe 3 or 4 years ago, we went to go do a field study with some of our early customers and we were just hoping to learn. We were still mostly UX Product. We wanted to see how are they using the system, what are the different ways they’re using the system – you know to improve it or to be inspired for new features or ways to improve the UI of some of the apps or the touchscreen and things like that. And what we found actually was a direct correlation between you know how engaged people were with actually using their home security system and how well their installation and first time use of the system was. And that wasn’t something we were looking for at all. That was something that started to come out naturally as my lead researcher was in the field and she came back with this like amazing story and this amazing end to end process map. It was like well, it was like 6 steps and we went in the home with this expectation of studying what was step 6 and what we learned was X, Y and Z about steps 1 through 5 and how it affects step 6. You know and step 1 is just becoming aware of the system. Step 2 is calling in to order it. Step 3 is getting it installed. You know all these things that happened along the way and that was because the participants wanted to talk about that and we found it necessary to let them talk about that to understand why they were or were not using the system in a certain way. So that’s an example I think of this sort of non direct, or the power of being non directed.
Steve:So that’s an example where using this non directed approach helps you find something that you didn’t know. But you’ve also talked about examples where there’s a great power in the way that you can bring back stories that are not necessarily surprising as you said, but they’re brought back in a way. I’m just wondering, does that approach of – now I’m messing up your words – but sort of a non directed, rapport building. Letting the participant talk about what they want to talk about. Does that also bring back things that you knew, but in that powerful way that’s helping you drive change?
Liz:Yeah. So one of the things that we hear a lot about, we talk a lot about obviously in the world of cable and television is live television. And how attached are people to live TV? I guess in our own little rarefied world we tend to think that people are really moving towards cord cutting and consuming everything on Netflix and over the top HBO and that kind of stuff, yet we’re going in the homes and we’re seeing this incredible attachment still to live television and we’re still seeing that in a lot of the numbers too and a lot of the Nielsen surveys and things like that. So we know it’s true and it seems like there’s this dichotomy and people want to understand it better. So we can go in the homes and you find that people – when you let them speak in a non directed way all of a sudden they start talking about like you know why they’re attached to live TV and it has a lot to do with like a lean back experience vs. maybe a lean forward experience. You know we think of live TV very much as like oh it’s still that classic – you know Alan Wurtzel would call it television the way God intended you to watch it. That’s what he says. Like everyone sits down at 8 o’clock to watch the Cosby Show on Thursday nights. But really the attachment to live TV is so much into sports, into the low commitment, into the serendipity, into I just want to flip around and see what’s on for the next 20 minutes while I decompress, like that kind of thing. And I think we tend to overlook that. So to be able to bring back videos of people showing, like they all have different strategies for browsing the grid, the traditional grid, but how that’s happening, to really illustrate that, to explain like why there’s still an attachment to it, but letting people show us through the context of their own stories and letting us take us down whatever path they want is really interesting and really meaningful.
Steve:I feel like you just hit on what the power is of those humanizing stories. They’re not necessarily surprises, but they can help people reprioritize. We knew that, we weren’t really paying attention to it. By humanizing it you’re helping people to make decisions and understand that something’s really important. That it doesn’t have to be a surprise that blows your hair back. It can be…
Liz:Right. It’s almost like it is really important so we definitely want to still make sure we have a strong way to help people you know browse what’s on live. And it’s not to say we shouldn’t be looking at like alternative ways to do that in the future. We know that live TV is still important. We know this from numbers. We know this anecdotally even though anecdotally there’s also the sense of there’s still cord-cutting, but being able to go in the homes and ask people to just like hey, take me through your day in TV and having them illustrate that sort of brings home this notion that you know some of these behaviors are still happening and we knew they were still happening and here’s a way to illustrate them and help people understand why it’s still so important and so meaningful to the user.
Steve:What are some of the challenges that – maybe it’s less about your team, but just the kind of work – I mean you know user research as a thing. You know in an organization like yours where you’re having these successes and you’re having this kind of impact, you know what’s – how do you characterize the types of challenge that you all are looking to address?
Liz:I mean some of the challenges that we’ve been looking to address lately, at least with my team, is we’ve historically been a qualitative research team you know and qualitative research is intrinsically linked to traditional market research techniques. So there’s a lot of familiarity with many stakeholders outside of UX and especially outside of UX and product with those methods. So really trying to bring in that awareness of how important it is to – you know we’re talking a lot about talking to people, but really the heart and soul of what we do is we come in and we build rapport and then we get them showing us and that’s really what usability is too – you bring people in, but you watch them use your system. When we go in their homes and we try to get them to just recreate their lives and watch them. So really you know trying to illustrate like different approaches to qualitative research and trying to help explain what those different approaches were, that’s been one challenge that I think we’re starting to really successfully illustrate. And then the other challenge too is like I said we’ve always traditionally been a qualitative research group and you know everybody loves numbers, everybody loves analytic surveys and stuff so we’ve started to – I have a quantitative researcher on my team now and we’re starting to bring in some of the quant to compliment the qualitative research we’re doing to emphasize some of the points that we’re making, sort of establish them – some credibility to them at scale.
Steve:And what’s the difference between what the quantitative research on your team does vs. other kinds of quantitative research that Comcast is doing?
Liz:We’re trying to do it from a product development approach. So you know one of the techniques that we just started implementing, or at least we’ve started piloting is Kano analysis which is surveying in a specific way, asking people about features and their expectations, but not do you like it or are you satisfied with it, but do you expect it to be there? If it’s not there are you upset, are you indifferent? There’s a formula that sort of segments things I believe into like three outputs – the must haves, the indifferents and then the delighters. You know we’re always getting – it kind of came because we’re always being asked about what will delight our customers? What new features should we be building to delight people? It’s a great language that really speaks to the company and one of the interesting things we’ve found too is that delighters decay into must haves over time so things – you know for example about the television interface that we would hear were delighters 5 years ago. Anecdotally we kind of see how they’ve decayed into must haves as we started to do some surveying. Like these must haves are things that people just expect to be there as a baseline, almost like a neutral. So anyway, having the – I think having some numbers behind it, but not in a traditional way, but in a way that reframes it, but again still amplifies things that we’re hearing from other types of surveys and other groups in the company I think is really, really powerful.
Steve:It makes me think about car features and how they seem – that industry seems to have sort of figured out – you know things come in at the ultra high end and then you now you can rent a Toyota Corolla at the airport, if they make Corollas or whatever, and it has a backup camera. Whether that’s driven by legislation, whatever, all these features seem to trickle down.
Liz:That’s a great example.
Steve:And is that the same – do you think that’s the same – I don’t know what’s going on in the car industry, if that’s about economics or is that about you know…
Liz:No, but that’s a great example. Like right now backup cameras, it’s a delighter. People who have them love it. But it’s not standard and there are still a lot of people out there driving cars that don’t have them, but I would predict you know in 5 years maybe that’s just going to be a must have and an expectation, especially if they’re putting it in Corollas. So it seems like that’s where you see the decay from delighter to must have when it’s no longer necessarily a luxury or something high end. It’s something that’s expected in any make and model.
Steve:It seems like it’s easy to see that framework in the car industry and probably harder in the things that you work on.
Liz:Right.
Steve:Because there’s just not…
Liz:It’s probably more practical stuff in the things we work on, like the number of tuners that you can use at once to record shows. It used to be like so exciting, I have a DVR. Like oh my God I can watch one show and record the other at the same time. Now people have four tuners and they can record four shows and watch one live at the same – or five tuners. You need 5 tuners to do that. And so things like that have become more than must haves. Same thing with the Internet, like the must haves. The speeds keep going up and how much you’re able to download in a certain amount of time.
Steve:Is the tuner thing about multiple viewers in a household?
Liz:Sometimes, yes. Sometimes. Or sometimes it’s about like a TV passionate – you know there are people who are super attached to TV and like to record a lot of things. They like to stockpile stuff. So there’s a lot of different behaviors out there that drive that.
Steve:Right. I’m doing that thing that people do where like 5 shows at once, that seems like a lot. What’s wrong with that person? But your answer is of course the appropriate, empathetic one. There’s use cases that you support and I’m just doing that thing that people do.
Liz:Exactly.
Steve:And that’s right. Maybe there’s a…
Liz:How many people use five tuners at once? I actually – I’m sure we have analytics on that, but it’s even more the perception.
Steve:Ah, right.
Liz:I think.
Steve:Right. I bet the – you don’t have to answer this, but I bet the incremental cost to add a third tuner and a fourth tuner is probably low and so right now you have more capability. I mean it goes back to what you’re saying about things that are delighters vs. must haves like – and I’ve seen this in studies that I’ve done unrelated to your business area where you know more capability then you think that you could possibly is like that’s very comforting for people. Like, I’m set.
Liz:A great example – a great example within the DVR space maybe is the, like remote DVR. So I remember when I was working on the first app for mobile DVR – oh my gosh I want to say that was like 2009 or so. You know where it was like super exciting to like be able to be like oh my God I’m out and I forget to set the DVR and now I can log in and actually set my DVR. That’s just – you know you can do that now. It’s still a delighter. So I want to say that I don’t know that’s it decayed to a must have, but I would say for – it’s getting close.
Steve:Interesting. Okay, let’s shift gears a little bit. You’ve talked about your team and some of the people on your team. Can you give kind of a 1000-foot view of the kinds of people, the kinds of skills? Like what is – what have you built and kind of who have you looked for and what’s the makeup I guess?
Liz:So like any UX research team we come from a variety of different backgrounds. You know the 31 flavors or whatever you want to call it. So my background is in physics. I’ve got people with backgrounds in information systems, anthropology. You know we’ve got someone from CMU (Carnegie Mellon’s graduate program). I’ve got people from – a PhD at Temple. So it’s really definitely a definite mix. And I think everyone of the team sort of has an area of strength. You know I’ve got someone on the team who’s just this absolutely amazing, almost like anthropologist – go out in the field – you know some of the things we were talking about before, about just being non directed and building rapport. You know that’s a real strength. Someone else on the team who really, really loves doing more of like the usability type evaluations. As I said I have a quantitative researcher on the team so – I don’t want to say – you know everyone is a generalist and we don’t have the luxury of being specialized, but I think everyone on my team has an area of strength I think. And I think it’s really important to have that mix as well. You know we get an open head or we’re looking to hire – among my senior group I have two managers on my team and a principal. You know the four of us will sit down and talk about you know what do we really need? What kind of skill sets do we need? Where do we have gaps or things like that. We’ll look for resumes and people who will fill that gap.
Steve:I have to call an open head as like the best piece of corporate jargon that any of us are going to hear this week. I have an open head.
Liz:I have an open head.
Steve:Well we’re going to send you to the doctor then.
Liz:It just means I have a requisition to hire somebody for a full time job with benefits. One of the things we talk about in user research of course is like using natural language with our customers.
Steve:And do we talk about making fun with our natural language because that’s the principal of interviewing that I just leveraged. So when you’re thinking about managing workload how do you think about – you know with people being generalists, how do you think about you know here’s an internal client with a need, here’s who’s going to work on that? How do you make those decisions?
Liz:That’s a great question. Right now I do have, I mentioned I have like a senior team and I tend to have them embedded in one subject matter area. So I have someone who is really leading most of the television and entertainment research, someone high speed Internet and home and then another one on customer experience. Underneath that I have a pool of researchers. They tend to move around a bit more. I like to try to keep people – you know again it’s kind of like the strengths in terms of methodologies. I also have people who have affinity to certain subject matter. So you know I have someone on my team who’s just a DIY geek about home automation so I try to keep that person working in the home security/home automation space because they just have a lot of subject matter expertise there. So some of it’s like you know if I can and if the demand is there, or if the need is there I’ll try to keep people – you know keep that subject matter expertise like running so that we can leverage that and move faster and build upon you know insights from the past. So that’s kind of how we do it and usually there’s enough work balance between each of the different verticals that it isn’t too much of a – you know people aren’t switching around too much.
Steve:Let’s go back to the open head bit. So now I’m using natural language – I did it straight and you giggled so I was like I’m back in natural language.
Liz:Calling me out on my corporate lingo. Okay.
Steve:So when you’re in an open head situation and you’re looking at resumes what do you look for? What do you see and what do you want to see I guess?
Liz:What do you see and what do you want to see? You know it’s hard with a resume actually. I’m almost always more interested in if someone reaches out to me with a note, you know, or a personal connection. I mean I can look at a resume and sometimes people have, what looks like they check the boxes and then we’ll want to talk to them, but it’s really about evaluating someone over the phone then on that first pass. So the resume you know should have like you know some experience, unless we’re looking for someone really entry level, but even if they’re entry level you know some sort of interest in doing something UX and user researcher related while they were in school. But anyway, yeah, it’s really more about I would say how are you going to demonstrate to me your level of passion for user research and your soft skills. So like for example you know I hired someone a few years ago – I actually got a note from one of our engineers – one of our scrum masters I think she was. And she said I met this woman at a party, she’s getting ready to finish her master’s degree at Temple and she’s really, really interesting and she had a background in mass communications and anthropology. She loved technology and she was doing work in like presence. And she was just a really neat person and I was telling her about your – we have a coop or like internship program and I was telling her about that and she was interested. So and I said put her in touch with me. But to me that’s like the test right, put her in touch with me to see if she actually gets in touch with me. Not only did she get in touch with me, she got in touch with me right away with this incredibly thoughtful email about all the different things she was thinking about and she wasn’t like I want a job or I absolutely must have a career in UX research. She was almost like I’m really interested in technology. I’m really interested in people. Not really sure where this is all going to go and I would just love to learn more about your team and what you do. So I got on the phone with her and I just thought she had amazing aptitude, she had great maturity and so then I brought her in to meet the team and we actually felt like she was qualified to be an entry level researcher. I thought she was way beyond an internship position. But it was really about you know the impression she made with me in terms of the way she communicated with me and communicated beyond like I know how to do these skills. It was more like I’m curious, which I can’t teach. You know I’m personable. I can get along with people. I ask good questions. Those are things that are harder to teach, the aptitude. And she demonstrated that she had that.
Steve:Some of those seem very general to try to reach out to someone inside a company and make them feel like they would want to work with you and some of them seem like there’s elements of like this is what makes for someone who I want to have work with me as a researcher specifically. Can you highlight what are the researcher ingredients and how you’re trying to suss those out?
Liz:Yeah, sure. I mean a big one I just mentioned is curiosity. Like if you don’t have a natural curiosity then I can’t teach you to be a researcher ‘cuz that’s like, it has to be ingrained in you, right. I mean to be a researcher is to want to understand. You know like I was that kid that was 5 years old, like why, why, why, why, why? And you know I’m still like that as an adult. I need to know how things work. So I look for that. Another thing is like pattern recognition. I think a lot of being a researcher is you know when you’re out, especially when you’re doing qualitative work, like really being able to understand. You know you hear this one story over here and you hear this one story over there and how are they related and what’s the common thread between them? I’m trying to think of a good example off the top of my head and of course now I’m blanking. But being able to really recognize patterns I think is important. Being a good listener too. I talked about curiosity, but the other side of curiosity is I love it when I interview somebody and then at the end when I say do you have questions for me, not only do they have questions which demonstrates their curiosity, but they’re not canned questions. They’re questions based on things that I’ve talked about, about my team and about what we do. So demonstrating that you can actively listen and process what you’re hearing and then ask thoughtful questions back I think is really critical. You know I can teach someone you know to go read about heuristics or good read a book, you know how to write tasks, how to run Morae and all these other hard skills, but I can’t necessarily teach someone how to be curious or you know how to be able to recognize patterns. I’m trying to think of another good one that I look for. I mean maybe it’s the soft skills too ‘cuz to be a researcher you know you really need to be able to just interact with people and make them feel comfortable. So I look for people who are good conversationalists, who are just comfortable talking to people they haven’t met before, come in, you know have a certain amount of confidence to them, with still some humility. ‘Cuz that’s what you need when you’re going to be going in and interviewing. Customers – and that’s also what you need when you walk in the room to present to your stakeholders. You know you have to be sensitive to their needs as well and what they need to hear in order to take action on your findings. You know and you may get some – when you’re presenting too you know you get questions at the end and sometimes the questions can be challenging, right. You know, especially if like something that you’re presenting on isn’t you know finding sort of challenges. It’s like one of their core beliefs about what they thought about their product. So how do you handle those questions? How do you react in that kind of a situation where you might be challenged I think is really important to observe as well.
Steve:What’s the soft skill – what’s the soft skill that describes that, like being able to handle these questions?
Liz:Yeah, that’s great. I’m not sure what the word is for it.
Steve:It must be categorized.
Liz:It must be categorized right. You’re going to make me dig back into my IA world in my brain. I think it’s about not being you know – I mean maybe it is humility. You know it’s not being so attached to your idea and your agenda that you need to be able to compromise.
Steve:That’s a great challenge in an interview situation to be able to demonstrate your confidence and your humility together.
Liz:Yeah it is, isn’t it.
Steve:Like that’s a bit of a magic trick. And I’m not saying that people can’t pull it off. It is sort of about who you are as a person. If I had to think about how would I do that I would definitely have to scratch – be scratching my head and want to practice that a little bit. I agree with you those are essentially, but you just – I don’t know it makes me have some empathy for people that are knocking on doors and trying to put all those best feet forward at the same time.
Let’s switch gears again and maybe sort of build on this. You talked a little about – I feel like at the top of the conversation you mentioned some of your other experiences that you had had. You’ve had other careers before this one. You want to talk a little about your background and maybe some of the highlights or milestones for you as you have…
Liz:Sure. How much time do you have? I’m just kidding.
Steve:How much time do you have?
Liz:How much time do we have left? No, I’m just kidding. Yeah, so my degree, my undergraduate degree is actually in physics which I always joked after I was done, the only reason we – my friend and I – would say we majored in physics is because we liked the way it kind of stopped the conversation at a party when someone would ask us what we did, what we majored in. You know I kind of grew up being just you know one of those women that was smart at math and science which is a little bit unusual, especially 20 years ago, and I just assumed I’d major in math and science and then when I was in school I was working in labs and I was like God this is so boring, there’s no way I can do this the rest of my life. I thought I would just get a PhD and go do research and get published in journals and I just thought God how boring is that. So I wasn’t sure what I wanted to do. I went to Penn and I worked in their admissions office and through that I actually just got a job right out of school doing, as an admissions counselor for a local college for a couple of years, which wasn’t as interesting as being an admissions officer for a school like Penn, you know. But I thought it would be cool, but really the lessons I took away from that – you know it was a lot about being independent, being on the road, learning how to sell, how to speak to things, how to present. And I knew that wasn’t going to be my career or anything so I had a friend working for a textbook company, Prentice-Hall, big name, and she was working up at their headquarters and said hey we have an opening for a marketing assistant, you’d be great and it’s in science and math textbooks. So ended up getting a job up there and again I think some of things I learned up there – first of all I worked for two really awesome women who were leaders in the marketing area and you know they taught me not to be afraid to sell myself and to make sure that people knew that I was like doing a good job. It wasn’t enough to get a compliment. It was to make sure you forwarded your compliments on so that people who could help your career knew that you were being acknowledged. You know and I knew – I remember one time they were talking about moving me up to manage a big sales territory in Ann Arbor, Michigan which is you know, in the textbook industry, you want to move up you often need to go manage a sales territory. You need to go out into the field and one of them said to me, she’s like you need to go knock on everyone’s door of these executive editors and stuff and just ask their advice and let them know you’re interested in this position and I like freaked out and I got like all nervous and stuff and they’re like no you have to go do that ‘cuz these are the people who are going to promote your career. So I got like really good advice that way, you know and they were from women which I thought was really important. So anyway I spent five years doing sales marketing and editorial acquisitions in the science and math space in textbooks which is the only job I ever had that was even tangentially related to my degree when I was working on physics textbooks and I was actually a little conversant with the authors. But that was where I sort of made the transition to UX. I always loved – I always loved graphic design and typography and people told me that like I was too smart to do that and I would never make any money and I finally just decided well I didn’t care, I really loved it and we were working on a brand new textbook for biology and the biology editor, I was talking to her about this a little bit and she pulled out this book, it was Richard Saul Wurman’s Information Architects book and it profiled people like Erik Spiekermann and a lot of really just interesting people who thought about how to design things. This was way back in the print age, or signage systems and stuff, but thought about how to design things based on how people were going to actually use them and I was like this is just awesome. And they were hiring somebody as a consultant who was featured in that book to help design like the program for how they were going to design this new textbook. I just thought that was so cool and so I finally just decided you know what I think I want to change careers. And I left Prentice-Hall. I kept doing some freelance work, but I started studying graphic design full time at School of Visual Arts. You talk about key moments. A key moment for me is I was taking a typography class and we were supposed to redesign a spa brochure. It was just one of those handmade things on like – I forget what they were called at the time, but just one of those things where anyone could put together a brochure on their desktop computer and it was pretty ugly. We were supposed to just redesign it with nice type and special attention to typography and I actually rewrote all the content before I redesigned it and I was the only person that did that and I came back in and I remember a couple of the other students were like why did you do so much work? And I was like I can’t design something I don’t understand and that was like almost like the epiphany for me and that’s when I realized I was actually probably more of an information architect then a designer and so that’s how I got into, made the transition into UX and that was – it wasn’t even called UX at the time obviously. This was like around 2000/2001. But it was the right time, right place to sort of jump into that arena and so I moved back to Philadelphia which is where I’m from and through some connections you know got a job at a local agency, basically doing jack of all trades what would be UX now – small agency that built websites for nonprofits and educational clients. So I did everything. I did stakeholder interviews, content strategy, information architecture, you know building site maps. I actually did the visual design. I even did some front end HTML. So that’s where I really learned all my chops and then from there I just got really excited about understanding how people would actually like use the stuff I was designing. Back to like thinking about Erik Spiekermann and the case study of how he thought about redoing the map for the Berlin subway I think it was. Anyway that’s when I went off to Institute of Design for grad school and learned about design research and then got really excited about you know watching users and learning what they do. So from there I just went to Steelcase for a couple of years. Worked for a really awesome woman there named Joyce Bromberg who again was just a great inspiration. She was the one – she’d give me a compliment and I’d be your typical woman like oh it was no big deal and she’d be like take it like a man, say thank you. So she taught me to always just say thank you when someone compliments me. You know you never downplay your successes. It was neat to be out of the digital arena for awhile and then I decided I wanted to move back to Philly and I’ve been here at Comcast ever since. So I think I asked you how much time you had and that was probably the longest answer ever.
Steve:No I thought you like managed to do like a really clipped, swift kind of active version of the whole career arc. That’s great. I think we will keep paying attention to time here. What are some things about you as a leader or a manager that I think might be important for us to understand or maybe things that we wouldn’t see on your LinkedIn profile?
Liz:I mean I don’t know if this would come through in my LinkedIn profile, but as I’ve moved up I never thought I would be a manager, let alone a director or a senior director. When I started here at Comcast it was like hey here’s this lab and we started up and I thought I’d be a team of one, probably forever. No actually, I didn’t. But I think as I’ve moved up and gotten more into the management side of things I’ve really found like a passion for passing it forward. You know as I was talking about in the past you know some of these great mentors I’ve had who you know taught me how to take a compliment or make sure other people knew my successes or you know taught me how to be patient and have a vision for something. I think that’s something I didn’t talk about before. You know sometimes people ask me like how did you grow this team so quickly, because I know that can be a struggle for some folks, you know, in the corporate world, trying to build up a fledgling UX or user research practices. How do you go from a team of one to ten in a short amount of time? You know and it’s really having the patience. It’s like you’re not going to go from 1 to 10 in one year so it’s like celebrate your small successes and have a vision for where you want to go but be patient for it. So I think it’s kind of passing that forward to the folks below me. You know I’m really interested, I read a lot of books on how to motivate, what people are motivated by and trying to understand like a little bit about the psychology of that aspect of our work, and really trying to do for some of the younger folks what those folks did for me and helping them build their confidence, move their own careers forward, you know feel like they can do it and to figure out what you want to do and not be afraid to take risks.
Steve:So as we think about wrapping up I will fall back as I often do in my stock questions because I’m a creature of habit, or I’m lazy or something else. Because they’re great questions. Is there anything that we should have talked about, about you and your work at Comcast or your background that you’d like to make sure that we include in this?
Liz:I mean again, just to sum up, I think one of the things that maybe, that isn’t emphasized enough – you know we talk a lot within UX about like the hard skills and the methods and stuff like that and I know you go around – I think you said you were just at World IA Day and you gave Soft Skills are Hard and I think that that, that’s just so important and I think that’s something I’d love to emphasize to folks that’s so important in your career. Don’t neglect your soft skills. And I think I got lucky in that I had this – you know that period for 5 or 6 years where I was you know working as basically like almost in a sales and marketing capacity where I learned to be comfortable interacting with others. People who – you know learning how to work a room. Learning how to make conversation with people I really didn’t know. Learning how to negotiate. Learning how to present. And then again something I think I talked about is you know never turn down an opportunity. So I’ve had people like present opportunities to me where like I gulp really hard and then I’m really glad I did it you know. And one example is the University of Pennsylvania actually, they contacted me a few years ago for – to come back and speak on a panel at parent’s weekend. If you know anything about Penn, Penn is a very pre-professional school. So you basically go there and if you don’t say I want to be a doctor, a lawyer or an i-banker, or a consultant – it’s like oh my God what’s my child going to do and how are they going to make any money. And so they had noticed that I had this unusual major and this unusual career yet I seemed to be pretty successful. So they invited me to come back and speak on this panel and I remember when the guy called me and I was thinking alright, you know I’ll do it. And then when he told me there would be like 400 people in the audience I almost choked because I had never spoken to a crowd that big before. I think the biggest crowd I ever spoke to is maybe 40 or 50 people and I just like looked out and I was like okay I’ll do it. And it was just so energizing and so much fun and I think you know that was the beginning of like so when Alan Wurtzel invited me to speak at his offsite and they said that would be like 200 people I was like sure. I got a little nervous, but now I’ve gotten to the point where, like I’m super comfortable speaking in front of large groups, to the point where like now I can almost do it on the fly if I already have the content ready and I’m not afraid to be animated or to tell stories, or even be a little self deprecating in front of a large group which helps like humanize yourself and build some rapport with your audience. So don’t ever be afraid to take an opportunity even if it scares you ‘cuz you’ll always learn something for it and the more you do something the more comfortable you become doing it. So I think that that’s just really important.
Steve:But Liz, what if I fail?
Liz:Well you can’t think that, right. I don’t know. You know someone actually asked me that once. I’m so glad you said that because someone asked me. I remember I was working at Prentice-Hall. I can’t remember the context now but they said well what if you fail and I was just like I’m not afraid I’m going to fail. I don’t know, maybe that’s something intrinsic, but I guess don’t be afraid of failure either.
Steve:Well I like – I mean just your framework is revealed when that question sort of just brings you up short, like I don’t know. I’m not thinking about that.
Liz:Right. And it sounds like cocky. We’ve been talking about humility and I’m like I’ve never failed . I mean sure I’ve failed here and there, but I mean it’s not really failure. Maybe it was like a little bit of setback. Oh you know this didn’t go – I didn’t get that job that I wanted to get, but I got the confidence to do even better on the next interview and got another job that turned out to be even better.
Steve:I didn’t hear you say I’ve never failed. I heard you say like…
Liz:I’m not afraid of it.
Steve:…don’t think about it in terms of what if you fail.
Liz:Right.
Steve:You sort of were attacking the question. I didn’t hear you say I don’t fail.
Liz:No.
Steve:Okay. Well those are some lovely sort of parting thoughts. Just to wrap up, wrap up, do you have any questions? No.
Liz:Have other people asked you questions?
Steve:Is that a meta question or an actual question? Have other people asked me questions? Yes, sometimes.
Liz:I don’t know. Steve, is there anything else that maybe, that I could add that would be unique perspective. I listened to some of your other podcasts and there’s always been a lot of talk about the methods and the structure of the team. I think we covered that a bit though. If there’s anything unique…
Steve:I mean you’re going to ask me if there’s anything unique about you? You’re the person that can answer that.
Liz:And I don’t know.
Steve:Okay. It sounds like we’re done then. Alright, thanks very much Liz.
Liz:Thanks, Steve.
The post 10. Elizabeth Kell of Comcast first appeared on Portigal Consulting.
Today I chat with Kavita Appachu, the Senior Manager of User Experience Research at Kelley Blue Book. She describes the different roles she's had in different organizations, moving from design to research, and explains the change effort underway at Kelley Blue Book.
The post 9. Kavita Appachu of Kelley Blue Book first appeared on Portigal Consulting.
In today's episode I speak with Aviva Rosenstein, the Senior Manager of User Experience Research at DocuSign. We explore how to make all types of research actionable, the benefit of doing your own recruiting, and the evolution from building a usability lab to having an in-house research capability.
The post 8. Aviva Rosenstein of DocuSign first appeared on Portigal Consulting.
We kick off the second season with Judd Antin, the Director of Experience Research at Airbnb. Judd and I speak about their model for embedding talented generalists with product teams, skill-sharing among researchers, and just what exactly makes research "sexy."
The post 7. Judd Antin of Airbnb first appeared on Portigal Consulting.
Today's guest is Carol Rossi. She's the Senior Director of UX Research at Edmunds.com. In our conversation, we discuss her small-but-mighty team, Edmund.com's collaborative workplace culture, and the personal driver of "doing good."
The post 6. Carol Rossi of Edmunds.com first appeared on Portigal Consulting.
Today's guest is Kerry McAleer-Forte, the Director of User Experience Research for Sears Holdings. We discuss how researchers need to think like storytellers, getting at the underlying need behind a research request, and the risk of using research to make recommendations.
The post 5. Kerry McAleer-Forte of Sears Holdings first appeared on Portigal Consulting.
My guest today is Nancy Frishberg, the manager of user research at Financial Engines. We discuss recruiting participants in an enterprise setting (where users are customers of your customers), finding the generative in the evaluative and how to think about collaborative workspace as entirely separate from reporting structure.
The post 4. Nancy Frishberg of Financial Engines first appeared on Portigal Consulting.
Today’s guest is Frances Karandy, a senior manager within the Customer Experience Group at Citrix. We discuss doing product-focused research in a company with a large number of products, what to look for when hiring researchers, and how to select projects that not only support the business but also help team members to develop.
The post 3. Frances Karandy of Citrix first appeared on Portigal Consulting.
Today’s guest is Alex Wright, who is the director of research at Etsy. We discuss the partnership between qualitative and quantitative research at Etsy and how his background in journalism helps him with the storytelling aspects of managing the research function.
The post 2. Alex Wright of Etsy first appeared on Portigal Consulting.
Welcome to the debut episode of Dollars to Donuts. Today's guest is Gregg Bernstein, who manages customer research at MailChimp. We discuss how MailChimp uses research to uncover new product opportunities, how the right research artifacts can best provide value to different internal audiences and how humility is an essential soft skill for successful researchers.
The post 1. Gregg Bernstein of MailChimp first appeared on Portigal Consulting.