Improving Team Health

Austin Bagley

OK, welcome to the 4th episode of Transformation. Here with us we have Doctor Carol Lee, Senior Researcher on the Developer Success Lab and a Clinical Psychologist. This is your first time joining us. Transformation is a multi episode miniseries for engineering leaders undergoing transformation, which is pretty much all of us. We've had incredible time interviewing leaders like David Farah, Kat Hicks, and Charlie Mode.

 

Austin Bagley

If you haven't seen their episodes yet, I highly encourage you to go back and watch those because we'll be building off of those for our conversation today with Carol. One of the big themes last time with Charlie, director of engineering at Salesloft, was the art of rolling out Flow in a culturally positive way. Today we're going to go and talk a little bit about more about what that means with Carol. As a Senior Research Scientist at Developer Success Lab at Pluralsight Flow, Carol leverages her expertise in mental health and thoughtful measurement to study how developers cope and thrive through stressful circumstances. Carol has over a decade of experience leading academic and industry research in clinical health, measurement and human behavior.

 

Austin Bagley

Carol serves as a research Fellow at the Integrated Behavioral Health Research Institute and as a Clinical Science advisor for Bravely Mental Health. She holds a PhD in Clinical Psychology from UMass Boston. We're so excited to have you here once again. Welcome, Carol.

 

Carol Lee

Thanks for having me, Austin. Hi.

 

Austin Bagley

OK, so let's set some context. You are a clinical psychologist, but you work for a lab that is funded by a technology company. What is that all about?

 

Carol Lee

Yes. So yes, I am a clinical psychologist. What that really means is that I'm formally trained to both give therapy and conduct research, but I've almost kind of been more focused on the research side. My expertise is in stress and anxiety, which developers I'm sure can really relate to, and but specifically looking at what factors kind of mitigate stress and anxiety as well as what factors kind of maintain or exacerbate it, right. I actually first dipped my toes and detect when I was asked to consult on some research kind of looking at these factors of anxiety and mental health.

 

Carol Lee

And it's kind of interesting. This kind of just got me thinking about, OK, well, who are the people building the tech, right? Who are these people behind this mental health tech? And so obviously, like a very good researcher, I started looking into who these people are, which kind of led to a few realizations. So first and foremost, it's this idea that, hey, the world that we know it, including our healthcare systems, actually run on code.

 

Carol Lee

Which I know everybody watching this right now is like, yeah, Carol, we know. But you know, as a psychologist, that just wasn't something I'd really thought about. So that was kind of an interesting moment for me personally. The second thing is that, you know, the people writing this code are often under a ton of pressure. They're also really misunderstood, which just so you know, maintains that are and exacerbates high rates of stress, anxiety, depression, etcetera.

 

Carol Lee

And then finally. It was this note that there is almost no research on how to mitigate developers's stress and anxiety. So even though we rely on folks's work on a daily basis, or healthcare systems rely on developers on a daily basis, we do not study how to make sure that these developers are doing well.

 

Austin Bagley

And so for me, it's really a matter of using my own expertise in clinical psychology to really generate evidence of how to support folks supporting us every day, which is, yeah, perfect intro for this because it's exactly what we're going to be talking about. If you could give us like a maybe a sneak peek behind the curtain a little bit about what do you do on a Developer Success Lab?

 

Carol Lee

Yeah. So I leave the quantitative arm of our research, but more specifically, I integrate my research in things like clinical theory, clinical frameworks, methods, assessment measures, interventions, right. All of that to conduct original research on software developers with the aim of doing two major things. First, I really want to create awareness on what software developers are experiencing and then second I want to generate evidence of how we can actually help software developers in an evidence based way.

 

Austin Bagley

So which is like leads us right into what we're going to be talking about. I love this, this idea of evidence based, you know methods in terms of we're not just like coming up with us in the back, you know, you know back of our mind. This is actually things you're researching and studying and proving, which is amazing. So we had Kat also on the developer's successor at Lab with us on Transformation a month ago, and she showed us this chart. And so I want to kind of like bring it back up because it's a very kind of good kind of starting point for what we're going to be talking about.

 

Austin Bagley

Like, can you tell us a little bit about what's going on for those who haven't seen this yet on this chart?

 

Carol Lee

Yeah. So this is a really great chart. So in this chart we see that 88% of managers say that a critical part of their job. Is to make developers as technical work and effort visible, which is a lot, right? 88%.

 

Carol Lee

What's interesting though is that when we survey a sample of over 1200 developers, only 24% of them said that their managers actually had that visibility right. So we're looking at 88% versus 24%, which is a pretty big difference, right.

 

Austin Bagley

And so, like, why does that matter? Like, as you guys are thinking about this as researchers?

 

Carol Lee

Yeah, it's interesting because, you know, despite managers saying, hey, this is a really important piece of my job is to make folks as good work visible, the data itself actually shows that we're failing to make that happen. And that's actually a pretty big problem, right? Because in our research, we see that when folks feel like their work is seen, recognized, and valued, that is, it's visible. This actually directly increases their sense of thriving. So in other words, when folks feel like their work is visible, they experience greater agency where they feel like they actually have a voice and how their work is measured.

 

Carol Lee

They also experience learning cultures where they feel like the entire learning process is celebrated right, including the process of making mistakes. They also experience greater support and belonging where they feel like they're accepted for who they are and supported to know as developers. And then finally, they also experience greater self efficacy and motivation where they're pretty sure that they can manage any problems that come up.

 

Austin Bagley

So we've kind of laid the groundwork here like visibility matters. I think I think everyone who's listening on this kind of like, you know, naturally understands that it's kind of cool to see the actual objective data on it. But like, hey, like this makes sense. I think the next kind of key step for us is like, OK, that's great. What do we do about it?

 

Carol Lee

I think there's often this question of, OK, well, how do we make things disable, right. So I actually want to answer this first by giving a little reflection on something I see as a clinical psychologist in this field. So often times we in software world talk about needing to fix or improve team health, as we say, right? And this visibility is a part of this team health. So I think what happens is that often times folks will try to, you know, improve team health but throw in whatever strategy they can.

 

Carol Lee

You're just kind of like, let's just throw all the good things at it and great, we've done the thing right. And you'll see this kind of like desperate cadence to this where folks get more and more reactive and then they start throwing out more and more strategies and they're not even really sure if they're working and they're just kind of like falling down this hole, right. So in the clinical world, we do have a term for this. We call this taking the kitchen sink approach where you were throwing everything but the kitchen sink, OK. We explicitly teach people to not do this right because this is bad clinical care.

 

Carol Lee

This is actually a really inefficient and ineffective way of addressing a problem. And oftentimes you see that the biggest reason why this happens is that the person has failed to adequately assess the problem itself. They're not adequately keeping in mind or tracking what is happening to begin with. You can't come up with a solution if you don't even know what the problem is, right? And so kind of similarly in that vein, one really high impact way to increase the visibility is to use metrics, but in a healthy way, right.

 

Carol Lee

And by healthy, I mean that you are tracking performance in a team driven way and you're really using metrics that fit for your team's contest.

 

Austin Bagley

When you say like healthy metrics and ones that are specific for you, like what have you seen in the wild? Like, what are some examples of things that have have risen to like, the top of hey, this is actually working for us.

 

Carol Lee

Yeah, I think it really again depends on what the team is looking for. Sometimes teams are looking to increase collaboration on their teams, and in those cases, maybe you want to start looking at things like how often or how or how quickly folks are reviewing pull requests, right? Other times, folks might be saying that their teams just don't get enough coding time, they're bogged down by meetings all the time.

 

Austin Bagley

So in that case maybe you want to look at the number of coding dates or teams are able to take on, which is great. It's a great couple examples. But again going back to the kitchen sink, it's not 25 different things, not only if in the sphere of kind of like you know, developer health and and team culture, but then even within metrics we don't want to throw the kitchen sink, you know, once we get on that subject too. So it's like one or two, like how do we actually like, you know, to your point, fit for purpose, like what are the right things that matter to this team at this point?

 

Carol Lee

Yeah, I kind of describe it as like. It would be like if you went to the doctor and they were like, I'm going to take every test possible, You're also going to do an MRI, you're going to do ACT scan, you're going to get a sonogram and you're like, why? Right. It doesn't make sense to do every single thing at once. It makes sense to figure out what the problem is first or what you're trying to change and then move from there.

 

Austin Bagley

Yeah, I love that. Like one of the things I think from your study that was really impactful for me that surprised me at least was you went and said hey you know and and you can correct correct me where I kind of I'm wrong here. But like essentially like 87% developers said I agree that healthy metrics are are a positive impact on the way that I work and you can correct me if I'm where I maybe I'm I'm erring on the actual binacular there, but like that surprised me. So First off, what was the actual like metric And like, let's talk a little more about that.

 

Carol Lee

Yeah. So the idea was that 87% of her sample agreed that healthy metrics use is a great way to improve visibility. Anytime we talk about the statistic, folks are like, well, I don't know, that's too much. That's so surprising, right? And I think that it's because folks often have some fear around metrics use.

 

Carol Lee

But actually as a clinical psychologist, this makes perfect sense to me. I saw this number and I was just not surprised at all. Like it anyway. So just to kind of explain that rationale a little bit, you know there is just so much research in the behavioral sciences and the clinical sciences and healthcare showing that tracking helps and tracking helps us for several reasons. So first of all, tracking helps give us concrete evidence of our own work.

 

Carol Lee

It turns out that humans are just really, really, really bad at knowing what we've done each day. So for example, you might have had this experience where? You sit down at the end of the day and someone's like, how was your day? And you're like, I'm really tired. Like, what'd you do?

 

Carol Lee

And you're like, I don't know. I oh, like, I have no idea what I did today. All I know is that I'm really tired, right? This is a pretty common experience. We just really suck at knowing what we've done every day.

 

Carol Lee

And So what happens is that we only don't know what we've done each day. We have a tendency to devalue or minimize our work. So we might say things like. I guess I just didn't do a lot. I don't know.

 

Carol Lee

I'm just tired for no reason. But in reality, and maybe you were doing a ton of complex work all day, You had a ton of focused time. You were thinking really hard. Of course, you're tired, right? And so having that kind of data is really helpful because it keeps us from devaluing or minimizing our work.

 

Carol Lee

We now have concrete evidence saying, hey, this is what you did today, this is what you did this week, this month, this quarter, no longer. You're tired. So what's really helpful is that. Now you can take that data saying after three words of complex work, I hit burnout, right? You can do that as a point of intervention for future.

 

Carol Lee

So maybe next time, after two weeks of complex work, you can say, hey, you know, last time when I hit week three, things got pretty intense and I get burned out. I don't want that to happen because we all know it's really hard to come back from burnout. So since we're only at Week 2, let's go ahead and take on some less complex work. Give my brain a great right? And that kind of helps things stay more sustainable.

 

Austin Bagley

I love that's a great example of taking data because we're always thinking about like looking back, looking back, looking back. And you have to think about, hey, here's this thing that's happened, you know, once or twice or or maybe more times. Let's use that to go and change how we look at work going forward. And that's what really makes the impact, I think. So not only is it visibility, but it's actually like actually improving the way that you're doing work, which is beyond just the benefits of visibility, which is great.

 

Austin Bagley

I love that.

 

Carol Lee

It actually also improves through thriving as well, because now you're creating agency because now you feel like you have a say in what your work day looks like, right? So it kind of has this bonus effect of also improving thriving.

 

Austin Bagley

Yeah, OK, so we're getting to a whole other component of that thriving framework. So it's like spilling over and in goodness, which is, which is great. So developers a like don't think managers have the right visibility, but also agree that healthy metrics can help. Like where are we at? What is the state of the world right now when it comes to this?

 

Carol Lee

Yeah, so our data shows that actually only 14% of developers so that they were using metrics in a healthy way. I personally call this an oh Shit statistic because it kind of makes you go, oh shit, there's a problem here, right? So only 14%, which is not a lot at all. So while we dig a little bit deeper, we also see that only 27% of developers actually say they're using the right metrics for them, right. Really pointing to this idea that there is this misalignment on how developers think that they want to be measured or think they should be measured versus how they're actually being measured.

 

Austin Bagley

So now that we know this, like we're not quite there, only 14%, you know, of, developers agreed that we were using it in a healthy way. How is it that we do? Like how do we start? I'm an energy manager. I'm managing leader.

 

Austin Bagley

What do I do first?

 

Carol Lee

Yeah. So you can actually use this acronym that I made that is TRACE. It stands for the fact that healthy metrics are transparent, it rewards growth, gives folks agency, is consistent, and then finally explores context.

 

Austin Bagley

I think I love this because I think we've talked about almost every single one of these things kind of like throughout our conversations and transformation as because we're and we're now like kind of giving it like a kind of a cool way to actually remember all those things, which is great. Let's dive into that. First one is transparency. Tell us some more about this one.

 

Carol Lee

I actually want to start this one with a fun statistic. So we saw that 26% of our developers said that they weren't sure if their teams were even using, Which is just kind of wild, right? At a minimum, your developer should know if you're using metrics. So that's kind of one big step towards transparency. They should also know when metrics are being used and where they're being used.

 

Carol Lee

Right. Don't keep this a secret. Don't pretend that you're not using metrics, but actually be using them. You know, don't keep it a secret about when metrics matter versus when they don't. But you want to make sure that you're nice and transparent with your developers on exactly how, when, why, and which metrics are being used.

 

Austin Bagley

I love that. And the second one is growth. Tell me more about growth.

 

Carol Lee

Yeah. So for this one, we actually have a pretty cool quote from an IC developer during one of our quality of reviews. And the quote says I don't review P Rs fast enough. And like I know that's my problem and he their managers like, yeah, I see that you're stuck in this area. Are there any blockers here?

 

Carol Lee

Right. Is there any way that we can improve on this? And so they created an initial goal of it. Try to do at least one PR review within 24 hours. So what I really love about this quote is that you see the manager is using data to identify, right?

 

Carol Lee

They're identifying the fact that this person is not very fast at reviewing PRS. From this they create a learning goal. The goal being it's trying to do at least one PR review within 24 hours. What's really awesome about this goal is that the goal is specific and realistic. Because it's realistic, it's actually achievable, right?

 

Carol Lee

You've probably made the mistake of having goals like. Be better at this or I'm going to be faster. And it kind of begs this question of like what does better mean? What does faster mean? How do you know if you got better?

 

Carol Lee

How do you know if you got faster? Like, you don't want to operate on vibes here, right? Because vibes are pretty biased. You want to operate on data. And so here we see that the goal is nice and specific.

 

Carol Lee

They're saying, OK, we know that you have gotten better at this because you've done one PR review within 24 hours, right? That's really nice and measurable. What's also great is that the goal celebrates growth and change over time. You see here in this bolded section at the bottom, it's an initial goal. They're not saying, yeah, you just need to do one, like within 24 hours and boom, best developer in the world.

 

Carol Lee

Yeah, like that's all it takes, turns out, right? Like they're saying, hey, this is the initial goal. We're going to build up to it. We're going to move towards it slowly because nobody can become the fastest reviewer overnight, right? We're going to kind of build this in slowly.

 

Carol Lee

Which is also just a nice reminder that change is slow and nonlinear. So this person might not get 2 within 24 hours this week, but then you know what, next time they might do none within 24 hours. And that's OK, right? Change goes up and down. It's slow.

 

Carol Lee

It's a process. And they're understanding that in this kind of quote here.

 

Austin Bagley

No, I love that because I when I first read Nonlinear, I just thought you were talking about like an exponential curve and I was like that sounds like really like ambitious. So nice to know that it's like it's up and down and that's like generally we're speaking we're, we're looking at the overall trend not the onesies and twosies you know as we get you know throughout our day and our weeks.

 

Carol Lee

Yes, absolutely.

 

Austin Bagley

So next one, So we've done transparency growth. Next one is agency right?

 

Carol Lee

So we have this other quote here from a different IC who says sometimes they the manager had a suggestion for a metric that I might work on and sometimes I agree with that suggestion. And other times I say, well, these are the reasons why I think it'd be difficult to focus on that metric right now. How about this metric instead? And then we have a negotiation with them about it, right? So you see here that the manager avoids shadow measurement.

 

Carol Lee

They instead allow the developer to have a voice and how they're being measured, right? They're not keeping it a secret. They're letting the developer know this is what I'm going to be measuring or looking at right now. Do you agree with this? Right.

 

Carol Lee

The developer then can kind of give their opinion about this and use the metric to advocate for themselves, right? I think what's really important here is that they're asking developers for their opinions. Oftentimes what happens with managers is that they feel like, you know, if they don't have all the answers right now, if they don't, you know. If they're not able to somehow read their developers as minds, that somehow makes them a bad manager, right? And that's not true.

 

Carol Lee

Asking for opinions doesn't make you a bad manager. It actually makes you a really, really good manager who's giving your developers agency. And I think that's just a really big misconception that happens.

 

Austin Bagley

Yeah, I think that's something that you have to kind of soak in a little bit, like that's OK. It's OK not to have all the answers. It's OK to adjust. It's OK to be flexible. That doesn't mean that you're something that is constantly waffling.

 

Austin Bagley

It's it's it's actually good to continue to receive inputs you know, in perpetuity.

 

Carol Lee

I think also just to kind of add to that, right, it's not just OK to not know the answers, it's that you shouldn't know all the answers. You shouldn't be able to. I mean, there would be something very wrong if you could just read your developers as minds all the time that that's something wrong there, right? Like there's a weird boundary being crossed. So like you, you actually want to be able to ask people for input and get their opinions.

 

Austin Bagley

And I think that's what's really cool here is, is coming back to the comment you made kind of earlier in this talk is it's not just, you know, agency around healthy metrics per SE, but it's agency around what do we do with this information? What do we do now that we have this information? Like how are we going to change? What should we work on as a team? And that's coming from the ground up versus mandates, which goes back to what David was saying in our first episode, which I love this.

 

Carol Lee

Yes, absolutely.

 

Austin Bagley

And I think actually speaking another thing that kind of Dave was talking about, one of his big themes was it was all about having stable like long running teams. That was kind of like getting after this like theme of consistency which you actually have also in this framework which is really cool. So let's dive into that as well, which I think is really important for managers to kind of like, understand and like how do I actually incorporate this in the way I work with my teams week by week?

 

Carol Lee

Yeah, so the C here stands for be consistent. And by consistent we name being consistent across time teams and individuals on when you're using metrics, whether you're using metrics, how much metrics matter and how you're applying metrics, right. And so we see here in this graph right here that the majority of folks report using metrics during kind of these planning phases, things like spread planning stand ups, but less so during kind of few more reflective or evaluated phases like monthly or quarterly business reviews, right. And that's actually a mistake. You want to be using metrics pretty consistently throughout the life cycle so that you actually know how things have panned out over time, right?

 

Carol Lee

If you're only looking at metrics to plan for things, but you're never following up to see how those things ended up, I mean, now you have an incomplete picture. Like I said earlier, you also want to be using them pretty consistently across teams. So you don't want to say hey I? I really like Joe over here. And so I'm just talking to look at Joe's metrics.

 

Carol Lee

I'm just going to believe whatever Joe is telling me because we've got good vibes, but I don't know much about this other person. So I'm going to look at their metrics, right. We know that our vibes, our sense of what's happening by ourselves, that they're pretty biased people. And that's not, you know, a criticism on any manager. That's just human nature.

 

Carol Lee

We're pretty bi. We're pretty biased, OK? And so you wanna be using metrics pretty consistently across people, regardless of what you're picking up on. Vibes.

 

Austin Bagley

Lies, actually, I love that point specifically on vibes, cuz that's actually something that having worked on the flow product for five years now, that's something we pick up pretty consistently like. We found that generally speaking, like not always true, but often times you're gonna have a number of quiet engineers and they are typically really like impactful on the team. They're tributing a lot. They're pushing everything forward. But the problem is they don't speak up.

 

Austin Bagley

And so sometimes we look at the people who are speaking up, oh, I know what they're doing because they're taught like they, they, you know, are active in the conversation, in a meeting or you kind of know what they're doing. And that's not a fair way, It's a very biased way actually of of looking at a team because you're basing it on vibes, which is not reality often.

 

Carol Lee

Yes, we want to be looking at reality or facts, not vibes, folks.

 

Austin Bagley

As much as I love vibes. So yeah, vibes.

 

Carol Lee

Are great. You know, Vibes are great for friendships, relationships, whatever, right? But when it comes to measuring success, you really want to be relying on data.

 

Austin Bagley

OK, so last up is Explore again, another theme that has been pretty central to our conversation so far.

 

Carol Lee

The E in phrase stands for explore context, right? I really love this quote here from an IC who says. A good manager is going to be hands on, but also context. For example, your metrics might look really bad, but your manager is going to know that it's because you've been researching something for a week and 1/2. That doesn't involve committing or reviewing code, right?

 

Carol Lee

We all know this context matters, and so you know, I really want to highlight that metrics are the start of a conversation, not the end of a conversation. They're not the only conversation. Use metrics as a starting point, right? And this is actually true of all data. So, for example, you know, I gave a lot of health examples because of my background, so apologies.

 

Carol Lee

But when you go to the doctor's office, your doctor doesn't just, like, take your vitals and say, hey, you know, your blood pressure is really high. Oh, well. And then, like, that's the end of it, right? You have a yeah, they're not just, like, sucks sia or have a conversation with you about it. They ask you Well, what do you think is going on?

 

Carol Lee

Maybe you're feeling really anxious that day and you're like, you know things are going well, but I'm just feeling really anxious right now and so my blood pressure is elevated. Or maybe you're like, I feel fine. So maybe there is something going on with my blood pressure, right. Context really matters. And so similarly, you know with with software metrics, don't just say, hey, I'm noticing something is different.

 

Carol Lee

Oh well, right. Or don't say, hey, something's different. I hate this by. Right. Use this as an opportunity to say, hey, something's different.

 

Carol Lee

What do you think is going on here? Why do you think this is different? What do you think we could do about it moving forward? Is it something worth paying attention to? Or, you know, is this just kind of a fluke?

 

Carol Lee

Have a conversation about it because again, you know, context really matters.

 

Austin Bagley

So oftentimes we get in the frame of thinking about, you know, the engineer to manage your relationship. Let's light it up a little bit. Let's think of like I am AVP of engineering. I have multiple multiple teams reporting to me and I see something that's off the data like what would you recommend for me as as you know someone who may not have all that context because I'm not working with a team day by day do like start to you know, explore what's going on.

 

Carol Lee

I mean, it sounds silly and easy because it is easy, but that doesn't mean it's not effective. But I would recommend you reach out to that team and say, hey, what's going on. I mean, truly, it's that simple. The person whose data that is knows best about what's going on in their context.

 

Austin Bagley

And often times instead of, you know, throwing yourself into a tailspin, trying to interpret data that you have no context on or trying to read people's minds, right, go ahead and ask the person what they think is going on, which I love that and I love that for two reasons. One is it goes back to your agency. You know kind of point of like it's it's we're working on this together. But also what I found is that it builds a really cool muscle memory of like we start to look at insights or to look at kind of like this change, this changed and it becomes easier and easier to like quickly understand this is what's happening. This is what we're this is where we're going.

 

Austin Bagley

This is A, something that is fine or B is something that we need to go and you know work on. And as we make that part of the way we work like day in, day out, just become so like so second nature that like it no longer takes all this effort to understand, which is like so great, we start to see teams start to grok the data this way. So I love this kind of last thing about yeah, really blow this context because it gets easier over time, Which is really.

 

Carol Lee

Cool. Yeah, absolutely. So just to kind of recap this, right? Because I know we've talked about a lot of things. We might as well just like go over it one last time.

 

Carol Lee

Healthy metrics follow the acronym of TRACE. That is, they're transparent about metrics use rewards. Growth over time gives hooks agency by giving developers a voice. It's consistent in how metrics are applied and then finally explores context via the metrics.

 

Austin Bagley

Perfect. I love this. This has been a great conversation. I hope this, you know these five points can really help you understand how we actually implement healthy metrics within our team. It's not just throwing the kitchen sink as you brought up at the very beginning of this and we're so excited to have future conversations.

 

Austin Bagley

So we've now had four episodes transformation. So thankful for each and everyone of you for joining us. We've got two more coming we're super excited about both talking about how we use metrics to optimize our workflow as well as advocate for our teams. You know two things as you'll see are very connected to things we've already talked about. So with that, so grateful for the time and we will see you in two weeks.