Imagining a Smarter Pluralsight
What does it mean to be a software engineer on a machine learning team? I couldn’t answer this question when I joined Dev-asauraus-rex, the first dedicated machine learning team at Pluralsight. I immediately hit the books as soon as I knew what our team’s problem space was going to be. I started watching Pluralsight courses on machine learning, refreshing my knowledge of calculus and statistics, and listening to podcasts. As I was drinking from the firehose of knowledge, I kept uncovering even more questions. I had undoubtedly taken the red pill and followed the white rabbit down its hole.
On my quest to figure out what my newfound responsibilities as a software engineer specializing in machine learning meant, I learned a lot. One of my earliest stumbling points is that there are quite a few ambiguous terms that have entirely different meanings in machine learning. Three months ago if you asked me what feature engineering was, I would have given you an answer somewhere along the lines of “It’s adding functionality to an application.” What I have since learned is that feature engineering is the process of gathering and cleaning data so that a machine learning model can make sense of it. Another term that has made me look twice is testing. To a developer testing usually refers to unit testing, end-to-end testing, or integration testing. To a machine learning engineer, it can refer to the process of pushing data into a model to see if it can make the right assumption about the data or pitting models against each other to find the best model. There are more confusing terms in the crosswalk between machine learning and software development than I have time to share. The best piece of advice I can give to someone on a similar journey is to make no assumptions.
After learning the basics of applied machine learning, things started to coalesce. I was able to start to see the light at the end of the tunnel. I had learned that machine learning engineers and software engineers aren’t that different. The software engineer is usually better versed in architecture, software best practices, automation, and optimization of an application. A machine learning engineer has more domain knowledge surrounding applied machine learning, data modeling, statistics, and optimizing. Both should be familiar with the basics of computer science, web development, and database operations. The most exciting thing is that we are a genuinely cross-functional team. We are regularly practicing pair programming and mobbing, attempting always to have a software engineer working with the machine learning engineers or vice-versa. A huge part of the day to day work we do has been shepherding the data scientists and machine learning engineers through our engineering best practices and architecture. Most of the code we write revolves around feature engineering.
Working on a machine learning team is an incredible experience. Every day on my commute to work I can’t stop myself from thinking about how I’m helping to build a smarter Pluralsight that not only teaches its users but learns from them. Most importantly we are trying to reduce the time it takes to learn a subject or a given task, and we believe that machine learning is a viable path to that goal.