028 - A conversation with Chief Cloud Economist Corey Quinn

Corey Quinn of the Duckbill Group chats with Jeremy about the role of a Cloud Economist, AWS optimization and how you can improve your understanding of AWS.

• The Duckbill Group
• Corey's Twitter account
• Screaming in the Cloud podcast

If you enjoy this episode, please consider leaving a review on Apple Podcasts or wherever you listen.

Please send any questions or comments to podcast@pluralsight.com.

00:00:06.5 Jeremy Morgan

Hello, welcome to All Hands on Tech where today's leaders talk tomorrow's technology. I'm Jeremy Morgan. 

Corey Quinn is one of the founders of the Duckbill Group. The Duckbill Group acts as a guide to navigating the murky waters of AWS. They help with reducing costs and even predicting future costs with your AWS infrastructure. Now, this requires an intimate knowledge of AWS and how to optimize it as well as some good old-fashioned business sense. Now, I follow Corey online and he's a very interesting character. I want to introduce you to him and shine some light on this fascinating work in our rapidly changing world. 

So, let's welcome Corey Quinn. So, how are you doing today Corey?

00:00:44.1 Corey Quinn

Oh, can't complain, given that there's a global pandemic on. Other than that, fair to middling. Yourself?

00:00:50.4 Jeremy Morgan

Other than that? Yeah. I'm doing great. So, tell us a little bit about what you do and what's a Chief Cloud Economist?

00:00:58.5 Corey Quinn

Oh geez, so, at a high-level, I go into companies and I help them with their horrifying AWS bills. The easy way for the people to imagine that is that I make the bill smaller, which I do, but that's sort of a by-product of what I really do, which is helping companies understand and predict what those bills are going to be. At least, that's what it was like until a few weeks ago when, suddenly, we see this global event happening and suddenly people start caring about cutting the bill rather than understanding and predicting the bill a lot more than they used to. Turns out that when, suddenly, everyone's belt-tightening, actually saving money is once again in vogue.

00:01:38.6 Jeremy Morgan

Oh, that makes perfect sense. Are you seeing a big shift in what you're doing with your company just in the last week or so?

00:01:45.8 Corey Quinn

I would say everyone’s seeing a bit of a big shift in that everyone is starting to figure out how to work in a way that we never had to work before. Where suddenly everyone is remote. Where we have much lower bandwidth conversations than we do in person and suddenly everything is murky and uncertain. It used to be, a few short months ago, that money was almost free. If you needed to raise additional investment, interest rates were low. It was very easy to go about getting additional funding. Now, with layoffs happening across the board, mass unemployment, we're seeing a very different dynamic play out and it's still very early to identify specific trends but we're definitely seeing a lot of renewed interest.

00:02:30.5 Jeremy Morgan

Did you work remotely before or have you switched to remote?

00:02:35.2 Corey Quinn

Oh, we've been entirely distributed. My business partner lives up in Portland, Oregon and I live in San Francisco and we have staff scattered across all kinds of places. So, we're about 10 people now and I don't believe that any two of us... So well, that's not fair. I don't believe that any more than three of us live in the same city.

00:02:52.9 Jeremy Morgan

Okay, so could you tell us a little bit about the Duckbill Group and what that is?

00:02:57.9 Corey Quinn

Sure. It started off once upon a time as a bit of a lark. I was independent and everything I did, as a company, was sort of a thin wrapper around my crappy excuse for a personality where I have a newsletter, Last Week in AWS, that gathers the news from Amazon's Cloud ecosystem and then gently and lovingly makes fun of it and it was more or less a vehicle by which I could do consulting work. I wound up merging with a long-time friend of mine a little over a year ago now, Mike Julian, he and I are now the co-owners of the company and we've been on a bit of a hiring tear in recent months. But we go into companies in a pure advisory sense and help them understand, reduce, and predict their AWS bills. We also have a Media Group that does my newsletter and two additional podcasts: Screaming in the Cloud and the AWS Morning Brief. Both of which are, well, sarcastic doesn't seem to quite go far enough so we tend to call them snarky.

00:04:00.4 Jeremy Morgan

Yeah, definitely. I followed you on Twitter for a while and that's one of the things I particularly enjoy, is the amount of snark.

00:04:09.8 Corey Quinn

Everyone loves it until they suddenly find it aimed at them.

00:04:13.0 Jeremy Morgan

Yeah, exactly, and one of the things that I kind of admire is like “OK, this person's in a business and he's making fun of everybody pretty much equally”.

00:04:24.2 Corey Quinn

That can often be what is perceived, but it's not technically true. If you take a look at how I view the rules of snark, from how I handle things, I don't call out individuals in almost any case because it doesn't go well. It's the rules. Always punch up, not down. I make fun of large publicly traded companies and that's generally fair game. I don't make fun of small scrappy start-ups. That's why I own twitterforpets.com. So, I have a fake start-up that I can make fun of without actually crapping on people's hard work. There's more nuance to it than a lot of people would expect.

00:05:00.9 Jeremy Morgan

Yeah. So, how did you get started in the cloud, being involved in the cloud? So, before creating this company?

00:05:07.1 Corey Quinn

I spent almost 15 years as a grumpy Unix systems administrator originally, because it's not like there's a second kind of Unix administrator out there, and in time I saw the world changing. Cloud became much more of a clear thing than it had been previously and it was very obviously going to be this transformative force. 

I could either sit there, shake my fist and complain, continue writing Pearl on Solaris or I could evolve. And I sort of stumbled my way through a variety of jobs called SRE, DevOps, Production Engineer, Systems Engineer. It really depends on what company for the flavor du jour but, after I left my last full-time job, it was all right. 

I dabbled with consulting for a while, on and off for years, usually for other companies, and I figured if I was going to try consulting one more time, what would I do differently to make it more viable? And the few things I came up with were, one: I would pick a problem that I would absolutely never be woken up about in the middle of the night, because on-call had basically burned me out. It was going to be an expensive business problem and I could never charge by the hour because as soon as you start doing that you are artificially capping how much revenue you're going to be able to bring in. There's only so many hours in the day and what people don't expect when they start down the path of running their own business. I spent probably 70 percent of my time in those first couple of years on client development, which means there wasn't a lot of time left in the day for coding.

00:06:39.1 Jeremy Morgan

So, what do you think prepared you for being so good at what you're doing now? So, you mentioned being a Unix administrator. That one I could imagine certain things, like scaling and provisioning and things like that, were probably second nature.

00:06:52.3 Corey Quinn

Well, you would think that, wouldn't you? But, surprisingly, back in those early days it was less of a problem because it was more capacity planning because we need to scale up. Great. I'm ordering some servers and they'll be here in six short weeks. Yeah, that is some very slow auto-scaling behaviour. Not quite as slow as Amazon some weeks, but, you know, close.

00:07:12.9 Jeremy Morgan

So, did you have any crazy job before you were doing anything in tech?

00:07:17.1 Corey Quinn

Oh, I did a few things here and there. It's a good question. Most people don't tend to think to ask that. I spent time as a corporate recruiter, which was bizarre and ridiculous. I was in sales for a while selling tape drives into the AS/400 market. Many moons ago I did some moonlighting as IT help desk support in Windows environments. A little bit of Windows admin work and, back in school, I put myself through school, in the summers, as a telemarketer selling credit cards. The disturbing part is, I was good at it.

00:07:48.4 Jeremy Morgan

Did you actually enjoy it?

00:07:49.7 Corey Quinn

Oh, absolutely not. Bothering people at dinner was super obnoxious. But you figure out a way to tell a story and articulate benefits and then get out of people's way and you get very good at accepting rejection.

00:08:02.0 Jeremy Morgan

Yeah, I could imagine. So, do you think storytelling... Do you think that skill comes into what you do now to where, you know, you can present a bunch of numbers, you can present a bunch of graphs, you could do things like that? Do you find it easier if you can kind of create a story around, you know, here's where your company's at, here's where you're going?

00:08:19.7 Corey Quinn

Everything comes down to storytelling. It doesn't matter what your job is. There are two kinds of salespeople. People who know that they're selling something and people who don't and everyone's in sales whether they want to be or not. It comes down to selling a vision. Selling an idea. Selling a course of action. You want to build things a certain way. Well, you have to sell that idea to people who you work with. Telling business stories is something that a lot of people tend to forget is important. It's the same reason that you see the tired old trope of "Well, I don't see the point of having a consultant come in. They're just going to say what I've been saying for years. Why are we just going to throw money at them?". Well, in some cases, because they know how to position that story with framing that resonates with the business. And something else that tended to surprise me early on is that people tend to value advice about as much as they've paid for that advice and I've never had more challenging work than when I did some volunteer work in the AWS building space for a few large non-profits. When people aren't paying for advice, they're not going to recognize it. So, it comes down to charging commensurate to the value you deliver. That's something I think a lot of people in business have not intuitively grasped and it's not easy. I certainly didn't grasp it at all when I started this. It's one of those rapid education moments.

00:09:40.9 Jeremy Morgan

Yeah, and that makes a lot of sense. I think with software it's much in the same way. I think the quality of software when... because I spent years as a software engineer, an independent and in corporate. I found that the same thing. The more expensive the software was, the more organizations tend to trust it and value it and it didn't seem to kind of tie to the quality as much as just the cost. You know, somebody would come in and say “Well, it's going to cost you a million dollars to build this” and then the perceived value there was "This must be amazing software. We paid a million dollars for it".

00:10:15.5 Corey Quinn

Exactly. People tend to make a buying decision or whether they're going to continue down a path. They go based upon signal. There are some services out there that you can't get any pricing information out of without filling out a form and waiting for a salesperson to call you. That tells you A: it's going to be expensive and 2: that even to get a quick question answer to kick the tires on it and I've got to go through a whole sales process. Not interested, personally, with the general case. 

Conversely, if I wind up seeing something that is "Oh, there are two tiers, three tiers. $10, $20 and $30 a month" then, for a lot of things, I'm not even going to begin to consider using that. Because, if I'm looking to build something at significant scale, it's very clear that I would be one of their largest customers for some things that I would do and you don't ever want to be the biggest customer of a particular company because, at that point, you wind up with some really weird and strange side effects. So, pricing is a big signal that I think people don't tend to realize that they're sending and even when we say that "Oh, that stuff doesn't apply to me", yes, it does. We all make decisions based upon perceived value.

00:11:27.2 Jeremy Morgan

Yeah. Absolutely. What does the current landscape look like for your company and for what you're doing? What are people talking about right now, every day?

00:11:36.5 Corey Quinn

Across the board, industry-wide, there have been a lot of changes in a very short period of time, due to the cultural pandemic that we're experiencing and living through. These are historic times but, stepping back from that a second, the overall trend of people going towards microservices, because they read that going with microservices was a good idea on Hacker News or maybe some thought leader on a stage said it somewhere, and not really understanding the complexity and trade-offs you're making by going down that path has been fascinating. 

Something we'll notice is that if someone goes all-in on rolling their own Kubernetes and deploying that on top of a cloud provider. Well, great. You have a hundred applications running in those Kubernetes clusters, but your cloud provider sees a single application called Kubernetes. So, it's very challenging to do any form of allocation of which application's the expensive one. It's just this one really weird monolith from the cloud provider perspective. So, allocating "What's the misbehaving app that's doing a lot of data transfer?", there are no tools that solve for those problems today. You've got to do an awful lot of manual spelunking. So that's been a subject of some concern in our space. 

We also, in the more macro sense away from the economic side, there's been the ongoing discussion about cloud providers and is multi-cloud a good idea? Do you pick one provider and go all in? Everyone has opinions on these but it's easier to sound off on what you believe other people should be doing than it is to understand that the answer to those questions is generally nuanced and complex. It comes down, in many respects, to the microcosm of our entire industry function in which we generally tend to provide marriage counselling between finance and engineering.

00:13:17.3 Jeremy Morgan

So, what role does tech skills play in all of this? So, I'm imagining what you do on a day-to-day basis and I imagine you have to know a ton about AWS, for instance, and that is such a huge landscape. We could spend the rest of this podcast naming off AWS services.

00:13:34.0 Corey Quinn

I could easily start naming ones that don't exist and who in the world could call me out on that? Even AWS employees can't keep up.

00:13:40.0 Jeremy Morgan

Exactly. How important are tech skills in keeping up with some of the services in AWS, for instance, or the people in the group?

00:13:50.0 Corey Quinn

That has been one of the ongoing challenges for us. What's strange is that because we tend to focus, by and large, on understanding and reducing large bills it's less important than you might expect. As much as we want to talk about all the 200 services, not an exaggeration, that AWS offers there are really only five to ten that are the significant cause drivers that we see across the board. Sure, you'll see something esoteric and odd from time to time, but it's EC2, S3, RDS and a few others that are basically the big fundamental building blocks of how these things work. 

Sure, it's nuanced and it's challenging and there's a lack of a learning curve but these are patterns and they tend to repeat. The first time you spend two weeks on something trying to tease out an answer it's maddening and it takes forever and requires an incredibly deep technical dive. The second time you see it, it takes about seven seconds to understand what's going on and you look like a wizard from the future.

00:14:45.1 Jeremy Morgan

Definitely a good way to approach it. Do you do any kind of personal projects or anything with AWS to learn new features of a new service?

00:14:53.9 Corey Quinn

Constantly. I spin things up on an ongoing basis. The way that I learn is that I tend to see what they say about a new service that they've just launched and then I start looking for things that they've gotten wrong or things that are not well documented. The way I've discovered those things is by building something myself. That's easier to do with some services, like a new Lambda function runtime, than it is other services like Ground Station, a service that talks to satellites in orbit. It turns out it's really, really expensive to launch a satellite on short notice.

00:15:27.2 Jeremy Morgan

So, what would you recommend to somebody who, say, had their sights set on your group and they said, "This is something that's great and awesome and I want to be a Cloud Economist”? What would you suggest would be a good way for them to start?

00:15:39.4 Corey Quinn

First, I don't think I've ever met anyone who said, "I want to be a Cloud Economist". It's something that happens to you rather than something you aspire to do, in my experience. But it comes down as well to finding things that identify with how people learn best. Some folks are terrific at learning through reading or through guided, classroom instruction style or through video. I'm terrible at those things. Things go in one ear and out the other for me. So, the way I learn best is by building something myself. If that resonates with someone, I encourage them to pursue that. Find a relatively short project that you can start playing with a service you don't fully understand and see what happens. 

I mean, that's how I learned things like DynamoDB. I had to build something to solve a particular problem I was facing and, a week later, I now know more than I ever wanted to about Dynamo just because, now that I've actually worked with it, I have a much keener understanding than reading any number of books on the subject.

00:16:35.6 Jeremy Morgan

Yeah, that's pretty cool. What are some common misconceptions that people might have about what you do?

00:16:41.6 Corey Quinn

People tend to assume from the outside that all I'm going to do is come in and do a bunch of analysis with some scripts and dump the output into a binder and throw it over the wall. If that is what I did, I would not be doing this in a consulting capacity. I would have built a software as a service product. But the trouble is that there's no API for business insight. This stuff is all contextually bound, and people think "Oh, you're competing with all of those CLASP... those cloud platform management tools out there". No, I'm not. Most of my customers are using at least one of them and, from that perspective, what we're doing is not tied to any particular tool or any particular API call that we wind up making. I mean sure, there's some of that that goes into it to inform our analysis, but it really is about understanding what the customer pain is and what the customer need is. The first question I always ask on a sales call when someone says, "Our AWS bill's too high" is "Great, why do you care?".

00:17:41.5 Jeremy Morgan

Well, yeah, it's a valid question.

00:17:43.1 Corey Quinn

Which sounds ridiculous and naive and why would you be asking me that question? And, sometimes, the answers are illuminating: 

"Well, because my company is spending eighty thousand dollars a month on this cloud environment, but it could be spending forty thousand a month". 

"Cool. How many Engineers are working on that thing?". 

"50". 

"Cool. So they're embezzling more in office supplies every month than they're spending on cloud services. Good, good. Let's take it a step beyond that. What did your boss say when you talked to them?".

"Oh, they said it wasn't important. Build this feature". 

"OK. What are you actually building?". 

"Oh, it's an experiment and, if it works, we're going to capture a giant opportunity and if not, we'll shut that thing down in six months". 

Yeah, the reason no one cares is because it's not an expensive problem. You can always cost-optimize things better than they are but you don't need to do that on day one. If you set out today trying to build something for the least possible amount of money, you're almost certain to fail. Build something. See if it's possible. See if it works and then optimize later and in time you'll see cost creeping up again and then it's time for another optimization pass. It's not something that you're going to solve by getting everyone building these things to understand costing in a nuanced way and I'd argue you shouldn't.

00:18:57.8 Jeremy Morgan

Yeah, and that actually brings me to another question I was going to ask you. Do you come into a company and... a set amount of time and say "OK, now that we fixed everything and we've set you on a good path, it was great working with you. See you later" or is it more of an ongoing relationship where there is a constant need to come back in and optimize things?

00:19:18.6 Corey Quinn

Both. We'd have some very long-term retainer customers and we have some that are effectively after a quick hit of "We need to cut the bill immediately. We don't particularly care how and then get out of here. We don't want to see you anymore". Most tend to be somewhere in the middle of those two extremes. It really comes down to customer pain and customer need. Sometimes it's a very short period time frame, such as negotiating a new contract. Other times, it's a much longer-term story as far as ongoing allocation and continuing to provide insight to the larger business side of the world from engineering acting as a bridge between them. It really tends to depend.

00:19:58.5 Jeremy Morgan

Okay. What does the future look like where AWS, in your opinion, is? Do you the see their services getting bigger and more complex and spreading out?

00:20:07.8 Corey Quinn

Well, I don't see them go in the other direction. I mean product strategy is effectively the word "Yes". There's very little that I would say they wouldn't get into and they are famous for not turning things off as opposed to Google, where turning things off is a core competency. 

The problem that leads to is that anything that they build is, more or less, going to be around forever. So, while that does cause sprawl and complexity, and then some, I also would have no reservations about building a business on top of any ridiculous service that they happen to launch on the AWS side because I know it's going to be around longer term.

00:20:45.5 Jeremy Morgan

Yeah, so I guess that is an optimistic way of looking at it. Most of the people that I've asked about AWS Services, that's usually what they say is if you sit and stare at the console long enough, you'll see new ones pop up as you're there.

00:20:58.7 Corey Quinn

Yeah, and the trick to remember is that every service is for someone, but no service is for everyone. There's always going to be things that wind up evolving, and it's easy to look at that and get overwhelmed but, if you're eyeing getting started with the world of cloud, there are maybe five to ten services that you need to understand and that's really about it to land your first job in the space. The rest of it you learn as you go. 

The problem is, is one of those services for example is EC2, their run a virtual instance as a system product, where you can just spin up a bunch of virtual computers. Terrific. It's incredibly complicated. It's incredibly nuanced and deep and by the time you finally learn enough about that to be dangerous you're "Oh my God. There's 190 other services that they all must be this bad". They're not. That is probably the most complicated service, or pretty close to it, and it's one of the first things people encounter. 

One of the others is IAM, their security permissions model. Great. That's also stupidly complex and you wind up hitting a lot of the complexity upfront which, in turn, means that you tend to believe that the entire space is like this. It's really not.

00:22:09.4 Jeremy Morgan

Yeah, I spent a lot of time with IAM in a lot of frustration with it and I actually, very recently, experienced exactly what you're talking about. So, I worked a lot with IAM and S3 and things like that. A lot of these things were really complicated. So, I needed to spin up just a really small web server to host my website for a little while until I figured out what I was going to do with it. So, I was expecting Lightsail. I'd heard about it and I'm like: "I'll try out Lightsail. I'm sure this is going to be a complicated mess so I'm going to set aside a few hours to do this" and I was pleasantly surprised that, within 15 minutes, I had this Unix server, little FreeBSD server up and running, grab my SSH keys, jump in and was able to do everything I needed to do. I was also expecting "OK, here's this big complicated permissions structure" and I was expecting a lot more and was pleasantly surprised by Lightsail in particular.

00:23:00.5 Corey Quinn

Yes, and networking stuff is... the VPC stuff is almost incomprehensible the first few times you see it and that's something you have to smack into to get an EC2 instance up and running. It feels like Lightsail is very much a reimagining of it. EC3, if you will. It's a glimpse of a future we could have had but didn't.

00:23:17.8 Jeremy Morgan

Yeah, and I know that’s EC2 running in the background so it kind of seems like it was probably an interface more than anything. Like Lightsail's just an interface.

00:23:27.4 Corey Quinn

Well, we saw this before back when I first started at this. I think I want to say 2010 or so. RightScale was a company, before they disappeared into oblivion, that all they did was they wrapped EC2 and provided a window interface, a dashboard, that a human being could understand because, back then, you had to pick which kernel you were going to run, there was a whole RAM disk style, there was an initRD that you had to wind up worrying about. 

It was incredibly complicated and nuanced if you've never done it before, and their entire business value was they added a bit of fee on top and then you just got the click to receive instance. Now the problem with building a product like that is that AWS Services don't generally tend to get worse over time, they get better. And as it became more and more straightforward to do these things without a third-party, the value they added became less and less. 

We see this with a lot of people building tools around the AWS space, and the cloud space in general, where this is a hard problem so we're going to make it better. Well, if your business can be eviscerated by a few feature releases from the cloud provider, you don't have a particularly unassailable position and you need to be aware of that and have contingency plans and a path towards evolving. 

With everything that I do, the rounding up of AWS' news, the snark of making fun of them, the cost optimization stuff, I've had conversations with people about AWS along all three of those axis and given them various blueprints for how to put different aspects of what I do out of business. And I'm hoping that they'll do it and I really, at some point, think that they need to for at least two of those. I don't think they're ever going to get really good at making fun of themselves to the level that I make fun of them. But then again, they, with a straight face, named a service Systems Manager Session Manager. So, maybe I'm wrong on that. Maybe they're going to be better at self-mockery than I am at mocking them.

00:25:26.4 Jeremy Morgan

Did they really do that?

00:25:28.0 Corey Quinn

Oh, yes. AWS Systems Manager Session Manager. It's a terrific service with a stupid name. What it functionally does is, it winds up popping up a web console where you are now effectively staring at a terminal inside of an EC2 instance. The thing that people have always wanted. It doesn't use SSH. It winds up hooking under the hood to an agent running on these things. It's a great service. It's free. It provides a way for people to shut off SSH across the board to their systems, but still have a back door in in case something breaks and they really need to fix it. It's audited. It uses IAM permissions and you can have every command that gets typed in logged to CloudTrail or S3. So, the service is awesome. The name is ridiculous.

00:26:16.0 Jeremy Morgan

Yeah, actually that sounds like a pretty cool service.

00:26:20.5 Corey Quinn

It's one of those many services and sub-services that people don't know tend to exist. But, that's the sort of thing that, just by mentioning that off the top of my head in conversations with this, someone listening to this is going to sit bolt upright from that description and say "Wait a minute" and learn that this exists and, suddenly, it potentially changes their workflow. And that's the entire point. 

Everyone has those moments. People think "Oh, I must know everything about every AWS Services, every available service out there". Hell no. I have no idea how some of these things work under the hood and the way that I learn these things is I, and this is the trick that I think most people lose sight of, I thrive on being the dumbest person in the conversation. Where I am thrilled to ask the dumb idiot question that everyone knows the answer to because one of two things happens. 

One: it turns out that I'm not asking a dumb question and other people have the same question so now "Oh good. It's not just me" and then we can start working to find an answer together and start yelling at the right people.

Or, it is a dumb question. There's this stupid easy right answer. Someone says it. I didn't know it worked that way. It corrects something I didn't know and now I have that. I've never regretted asking the question or saying "I don't know". 

In fact, when I was hiring, one of the biggest indicators I looked for when interviewing for senior people was how willing they were to admit they didn't know something. Everyone says that giving a wrong answer in an interview is a terrible sin. They're right, to some extent, but the right answer is "I don't know but if I had to guess" and then speculate wildly. If you get it right, it shows your instincts are spot on. If you get it wrong, no one's going to hold it against you and you're showing how your mind works. But if you are authoritatively wrong about something, spoiler, in a job interview scenario, if someone's asking you a technical question, they probably know the right answer. You're not going to be able to bluff.

00:28:22.1 Jeremy Morgan

Yeah, absolutely. That's always great advice to give for people who are doing technical interviews. I've written a lot about that recently and even over the years spending time as a hiring manager.

00:28:33.7 Corey Quinn

One of my favorite talks that I gave was on salary negotiation about a year ago with Sonia Gupta. She and I gave a team talk at DevOps Day, Charlotte and that was a lot of fun. I love talking about salary negotiation. About how to handle technical interviews. About how to handle the job interview process because I got really good at it because one of my primary skills as an employee was getting fired a lot. So, I got a lot of practice and now it's great. All of that was super helpful and now useless because I'm not going to work anywhere because I'm basically unemployable. So, running my own company is out there. I did take a lot of that into consideration when designing how we handle compensation. How we handle the process of interviewing people for various roles here and it turns out you can get surprisingly far by treating people how you wanted to be treated. As a manager, I found that I get surprisingly far, as well, by looking at some of the managers that I've had in the past and then doing the exact opposite of what they did. And you wouldn't think that would be as effective a management strategy as you would want it to be but it works disturbingly well.

00:29:36.0 Jeremy Morgan

Yeah, absolutely. I could definitely see that. What do you think of whiteboard interviews for developers?

00:29:44.0 Corey Quinn

Well, certainly better than water board interviews. The problem that I see with whiteboard interviews is that it's generally a skill that people tend to trot out only during job interviews. It is not representative of how most people solve problems. When I write code, I don't know about you, the first thing I do is I go to my preferred editor which is called Stack Overflow. Then I use the copy and paste functionality and there we go, because I am indeed a Full Stack Overflow developer. 

The problem is, is that now draw this thing out on a whiteboard. First, that's not a coding environment anyone is comfortable in. There is no syntactic sugar and you feel dumb. Two, for the person interviewing you it's any random Thursday but, for you, your entire career hangs in the balance and you're now performing in front of someone. It is incredibly stressful and even outages, where you're desperately trying to fix them, don't have that same kind or level of stress. 

Also, surprise, not everyone is terrific at interviewing skills and presenting like that and that's not a skill you generally tend to use in your day-to-day work. So, I have a whole host of problems with that. I think that people tend to get job interviews wrong across the board. One of the things I like to ask is, instead of me finding something you're weak on so I can just kick the living crap out of you over it, that doesn't serve anyone, and that's trying to hire for absence of weakness and I hate the model. Instead, what are you best at? What if I could ask you a deep technical question about something? What area would you want that to be? What can I do that lets you shine at the thing that you're best at? And, when you give people a chance to do that, they really will surprise you.

00:31:27.3 Jeremy Morgan

Yeah, that's definitely an interesting approach.

00:31:29.5 Corey Quinn

I'm also very conflicted on take-home interviews because it tends to bias for people who have a bunch of spare time to spend working on your problem. The thing that I absolutely despise has been take-home interviews that are very clearly something that they want to throw into production. It's like effectively taking projects to Upwork except it's free. You just have 15 different interview candidates build different components of it and then stitch it all together. I hate that model.

00:31:54.9 Jeremy Morgan

Yeah. I've never actually seen that. So, as a manager I have given take-home interviews, but I never did it for them to build a feature. But is that something that you've actually seen happen? I mean, I've heard of it happening.

00:32:08.6 Corey Quinn

I've never worked at a company that did it because I tend to at least pick companies that are not outright unethical, at least in that way. But as a candidate, I'll never forget one where they had this random Java app sitting on GitHub and they wanted me to build a fully working CI/CD pipeline and monitoring system for this inside of an AWS account that they spun up, and do the whole thing so it can be run with one script, from the ground up and there are a few other requirements in there. 

And I remember this because I started working on it. It took me two full days and at the end of it there were still fundamental problems with how the application was built. It was very clear that no one had actually solved this problem. Or, I was just far worse than everyone else in the space. I finally got it working and then declined to submit it because at that point I had such a bad taste in my mouth that it was not worth pursuing.

00:33:04.5 Jeremy Morgan

Yeah. Absolutely. I could definitely understand that.

00:33:08.4 Corey Quinn

This is why I love toy problem. A toy problem I loved to ask people was "Are you familiar with TinyURL, a URL shortener?". And what I like about that is the answer is either yes or, if the answer is no, you can explain in 30 seconds what that is to someone. It takes a long URL and spits out a unique short URL. There we go. 

And then I want people to design that however they want, put that on a whiteboard, let’s have a conversation what the architecture looks like. Whiteboards are great for architecture discussions, lousy for writing code. Cool. Now it goes by, six months later. It's slow. Now what? Where is it getting slow? Find the bottleneck. Cool. Now I want it to be multi-site. Now I want it to be active across three sites. If I kill one of these sites, what happens? What technologies do you build this on? 

And there are no wrong answers there. It lets me see how people think and it lets me see how people respond to being challenged on some of their architectural decisions, respectfully, of course, and it lets me see how things work. This leads to great conversations and in some cases some spectacularly wrong answers. 

One of the best bad answers I ever got was "I would never build that thing because that's a stupid thing to do and I don't do stupid things". So, I go back and forth with this person three or four times and it's a toy problem, work with me. There's just: "I don't know I guess I'd Google it". 

"Thank you. I have no further questions". 

Yeah, cause nothing says bad interview like short interview. Interview horror stories. And again, because it feels a little sad to be dunking on people who are bad at interviewing just because, otherwise, it becomes this really unfortunate story of going too far towards punching down. Not everyone's good interviewing and that's kind of the point. Now, you want people to be able to collaborate with you and be at work in a professional environment. But not everyone needs to be an incredibly polished presenter in front of groups in order to excel at jobs that do not require that skill. 

If you are good at that and you're terrific at doing it, great, by all means find ways to showcase that but that's why it comes down to being so important to meet candidates where they are.

00:35:16.1 Jeremy Morgan

Yeah, and I think the Agile movement and having to do demos quite a bit has helped engineers, in particular, with that. I've worked at organizations where we had to produce something every two weeks to demo to stakeholders. And so, that made us better at demoing things and better at talking about things. So, of course the next job interview you go to and there's several people sitting around and your like "Oh, I'm going to demo something that I've built like I've done every two weeks for the last five years” or whatever. So, I think that has kind of helped because, at least in my experience, engineers aren't generally the type of people that are very outgoing and very focused on presentation skills and things like that, but I think possibly the Agile movement might have helped in that a little bit.

00:35:57.7 Corey Quinn

That is not always true though. The challenge is the ones who are, tend to wind up finding themselves pulled in other directions pretty quickly. I think that's where DevRel came out of.

00:36:05.6 Jeremy Morgan

Yeah, absolutely. So, do you have any projects you're working on right now that you could tell us about?

00:36:11.4 Corey Quinn

Nothing that is hugely relevant to most folks. There's the entire newsletter that I send out every week, has an entirely custom production system. The last time I did the numbers it had 27 different Lambda functions behind four API Gateways to DynamoDB tables and a few other things. But that's mostly a technology testbed but also solves a real business problem. So, that's been fun. I'm debating how much of that I want to open source down the road. It's been terrific for building conference talks and, if that's something that people ever wind up doing again, that's going to give me fertile material to continue having conversations for years. But, right now, as far as public work goes, not as much.

00:36:48.8 Jeremy Morgan

Right. So, you have a couple of podcasts you mentioned. Did you want to talk about those?

00:36:53.5 Corey Quinn

Sure. Screaming in the Cloud is a non-snarky, serious interview-style show similar to this one, where I have a different guest every week and I have a discussion about what it is they're working on, how it works. Guests have ranged from an intern at Facebook all the way up to the head of Azure and most folks in between. It's non-specific to any particular technology or approach and the topic is business in the cloud. The other one is the AWS Morning Brief which is slowly turning into a five-day-a-week morning show where it is incredibly sarcastic, cynical and snarky. And mostly, at the moment, is monologue style but that may be changing in the near future just because I want an opportunity to be cynical and sarcastic but an interview show with different guests every week is not the venue for that if people don't know what they're getting into.

00:37:41.3 Jeremy Morgan

Yeah, absolutely. Those are some of my favorite things but yeah, in an interview, might be tough. If you were going to name an AWS service, what would you name it and why?

00:37:51.7 Corey Quinn

Oh, that's an interesting one. See, it's way easier for me to go ahead and make fun of existing names than it is to name things. I have proposed services that they build to various teams there and given them stupid names for them, but it would all come down to what the service did. It really depends. There's the cynical answer. There's the snarky answer. My rules for naming are: it has to be something you can Google. They have a new service called HTTP APIs. Yeah, good luck Googling that term. Other things are great, like AWS Fargate. You can Google that super easily, but you have no idea what that is from the name. The right answer is usually somewhere between the two. But I might hire a new Systems Manager Marketing Manager.

00:38:34.4 Jeremy Morgan

Awesome. Well, thank you very much for talking with me today.

00:38:38.5 Corey Quinn

No, thank you for having me on it is always a pleasure.

00:38:47.6 Jeremy Morgan

So, that concludes our interview with Corey Quinn. You can find him online: @QuinnyPig on Twitter, at lastweekinaws.com or check out his podcast Screaming in the Cloud. 

Thank you for listening to All Hands on Tech. If you like it, please rate us. You can see episode transcripts and more info at pluralsight.com/podcast.