Growing a Data Science Team
A nebulous term more often defined as a noun than a verb, we believe data science consists of transforming your business into not merely a data-driven company with data science in a sidecar but a model-driven company with data science in the driver seat. At The General® we define data science as building predictive algorithms that make real-time decisions, and integrating them into our product to drive change throughout the company and dominate the non-standard auto insurance market.
Our team was created in mid-2016 and we have certainly had our share of growing pains, but we are ecstatic for the current direction we are taking and what the future holds. We have great support from our senior leadership, which allows us to stay singularly focused on data science and to avoid the many distractions of ad hoc business analytics and “nice-to-knows” of so many data-driven questions. (As much as possible, we try to let those questions stay within the analytics teams of specific functional areas.) How we have arrived to this point has not been by chance: we maintain a selective hiring process, have been intentional in what projects we take on, and have a clear vision.
While the majority of current data science literature (online and otherwise) is full of wonderful advice for aspiring data scientists and for companies looking to start data science teams, it generally lacks in practical advice for the data science manager or director looking to grow an existing team. Achieving success and advancing data science in the workplace – where there are often competing interests and a lack of resources – is a serious challenge. This blog series will focus more on growing an existing team, but we will explore the initial process as well. Split into three sections, we will focus on hiring a data scientist, how we operate as a team, and where we are going in the future.
Of the many great sources of advice on how to start a data science team, most follow the same structure of an analytically minded person emerging, starting small, and showing value. This person is usually someone who loves tinkering and exploring data available to them with tools they are already comfortable with; possibly starting with Excel or some BI tool. She enjoys data analytics, learns a more powerful tool such as Python, and begins by completing small tasks that involve automating repetitive tasks others cannot. Thus she shows value and starts to get buy-in and influence those around her. (These small wins often involve rudimentary rules-based approaches designed to quickly stand up some type of decision-making algorithm. As an organization evolves to integrating data science to their existing process, rules-based approaches should be validated and phased out where possible.)
No doubt in your organization there are many talented individuals with decades of industry experience who rely on gut instinct as a final barometer. They have been successful in their careers up to this point, but are often resistant to change. Successful young teams will need excellent communication skills to overcome this final hurdle by explaining data science concepts with analogies, and show business leaders how data science can save them time or help improve an aspect of their role. Do not take the outcome of these initial meetings personally. Data science is built on relationships and trust and with time you will learn to effectively communicate technical material to a non-technical audience. Laying out the opportunity cost of not implementing some change can also be effective. To as much extent as possible, always tie a task or project you are working on to a specific, measureable KPI.
This young data science team will naturally grow to having made their first hires. (Jesse Spencer-Smith has given wonderful talks about what kind of data scientist a company should hire given what stage company they are currently, and we have been heavily influenced by his work.) You now have a sandbox in AWS set up, you know what tools you are using and have a process in place to explore new ones, and you know what type of candidate you are looking for. The next step is often the hardest: securing funding. Convincing the powers that be needs to be carefully planned and executed, especially if it is an off-cycle budget period. (Hint: it will always be an off-cycle budget period.) To this regard we align our projects in three areas of optimization: Product, Process, and Revenue. This allows us to show where we are adding value, and where we could be adding value with additional resources. By tying every project to a specific and measureable business metric in the planning process, there are no (or minimal) disagreements about the amount of value we are adding. (In the next post in this series, we will show how we prioritize our work and what our intake process looks like.)
With our funding secured, we are now ready to start the recruiting process. Looking internally first, especially for companies in an industry with specific knowledge, can cut down on the opportunity cost of taking six months to source an external candidate and another six months to teach them the business. (Not to mention saving cost of recruiting fees and sign-on bonuses.) In her Wall Street Journal bestseller Multipliers, Liz Wiseman details the idea of “The Talent Magnet”:
“In their quest to assemble the finest talent, Talent Magnets are blind to organizational boundaries. They see multiple forms of intelligence everywhere. Talent Magnets live in a world without walls and without hierarchical or lateral restrictions. Instead, they see talent networks.”
We have sourced two internal hires by keeping an eye out for those expressing an interest in this field and demonstrating promise; we invest our time in them and grow them. We also have close partnerships with area cultivators of talent, including the Nashville Software School, Austin Peay State University, Belmont University, and local high schools. These partnerships helped us find Taylor Perkins, an incredible engineer with a knack for data science; and an awesome intern this summer, James Bertrand (who we recently converted to a full-time employee).
Our hiring process starts by writing a detailed job description. We know exactly what type of candidate we are looking for and tailor each job description specifically to our open position. If a company or hiring manager is not willing to spend a significant amount of time creating the job description and recruiting, the position may not be needed. Even worse, failed interviews and frustrated candidates, hiring managers, and recruiters will certainly follow. The field of data science is getting deeper and wider by the day, and if you asked ten people what data science entails you would get 15 different answers. Jake VanderPlas, the director of open software at the University of Washington’s eScience institute, describes data science as “an interdisciplinary subject” that:
“…comprises three distinct and overlapping areas: the skills of a statistician who knows how to model and summarize datasets (which are growing ever larger); the skills of a computer scientist who can design and use algorithms to efficiently store, process, and visualize this data; and the domain expertise—what we might think of as ‘classical’ training in a subject—necessary both to formulate the right questions and to put their answers in context.”
Simply put, we look for candidates who know how to model, code, and communicate with business owners. In our job descriptions we have few requirements, mostly consisting of R or Python and SQL. After that it is about where the candidate can add value we do not currently have on the team. Perhaps we are looking for a specific skillset like natural language processing, machine learning, or Bayesian statistics. We do expect each of our data scientists to possess a certain degree of engineering skill (more on that later).
AN ACT, IN FOUR PARTS
We work closely with our Talent Acquisition team throughout the hiring process. This team helps us source candidates and conducts an initial resume screen for us. Because we have spent the time to sit with them and walk them through resumes of candidates we want to speak with or come back to, they are able to quickly identify top talent. Educational background and industry experience are considered, along with personal projects and how a candidate has added value to their organization. Unique projects candidates have worked on are also given weight: if all a candidate has worked on is the Titanic data set, or analyzing tweets for sentiment analysis, we will likely pass.
The next step is a short phone screen. We ask only a handful of questions that get at cultural fit and how they approach problem solving. Cultural fit goes beyond the classic “airport test” – the heart of the matter is how they work with others. It is helpful to ask candidates how they were supported by teammates in a recent project, or how they overcame a roadblock. We hire collaborative, curious, and passionate people who enjoy being a part of something bigger than themselves. We ensure our TA partners present a diverse candidate pool and are actively working with our HR partners to establish The General’s first H1B sponsorship. An additional question in the phone screen poses a problem to the candidate as if they were a consultant using data science to solve a business problem. Because the question is open-ended, we can drill down on specific topics depending how they approach the problem and where they take the conversation. If a candidate jumps right into the analysis before asking any additional questions to understand the business problem, they may not be the right fit for us. After this conversation we have a good feel for who we want to pass on to the next round.
The last interview round before an onsite involves a coding challenge. While we do not expect our data scientists to be “full-stack”, we do expect them to have a certain engineering ability. Our coding challenge is not designed to see how well they can visualize or clean data, or if they split their data into test and train groups, or how well they can optimize a model. Evaluation of these skill sets will come later in the interview process. We are only gauging their ability to code. The challenge involves training a bivariate model, serializing a model object, and writing a predict function taking as input raw JSON and outputting a probability and ID for each record. The challenge is tool and language agnostic (insert corny “if Perl is your jam” joke here), and we expect it to take a Data Scientist II about two hours. Each of these three steps weed out about half the candidate pool, and now we are ready to invite a candidate for an onsite interview.
Everything in an onsite interview should mimic what the real day to day of the job will be. We do not code by whiteboarding, your boss does not pop quiz you at the water cooler, and googling is an undervalued skill. We do start with a code review – even if the code is correct, we like to understand why the candidate approached the problem the way they did. Candidates then meet with members of the Data Science team and key business partners within the organization. The final portion of the onsite interview is the most important: the presentation. To this point, the most promising candidates all have the technical chops to cut it. What separates good from great candidates is a presentation of a take home project we ask them to complete mimicking a business problem they would face at The General. (Yes, this is asking a lot of candidates – but data science salaries are not small sums and we have yet to be turned down.) We know actions speak louder than words; and to show candidates we value their time and effort on the project we offer a $100 gift card to all candidates who participate in the onsite interview. The presentation allows candidates to really show off their creativity: be it a modeling technique; specific feature engineering; or how they incorporate domain knowledge into their approach.
The presentation of the take-home project provides a key opportunity to learn how the candidate approaches an actual problem the business is facing, and more importantly how they are able to communicate their findings to real business owners. Since we instruct candidates to prepare their presentation for a business audience, we evaluate their ability to explain technical concepts to a non-technical audience. We expect them to know to stay away from talking about log loss and AIC and instead find a way to translate their findings using analogies and storytelling to relate to the business. (Time is reserved at the end of the business presentation to drill down on technical questions.) We recommend designing a task that uses your own data and a question you have answered previously, to give candidates an example of their day-to-day work in the future. If your team or organization has worked on a data-driven project to answer a particular business question, a good starting point would be to convert this into a take-home task. Ideally the solution (or potential solution if not already complete) touched multiple functional areas – invite those business leaders to the presentation to ask engaging, challenging questions. This is the first time candidates have a chance to ask questions directly of the business owners, and it is not without precedent for these questions to be a litmus test for the final thumbs-up-thumbs-down decision.
The interview process for hiring machine learning engineers differs slightly. The coding challenge is more extensive and expands on a candidate’s knowledge of various frameworks. Since machine learning engineers interact more with IT and DevOps resources than business owners, they do not present a project and instead spend time during their onsite interview with leaders from those areas.
FAIL AND LEARN
We have continually refined our hiring process as we grow and learn what works and what does not. We are not afraid to make mistakes provided we learn from them. In determining what skill set we should look for in our next candidate, we use radar charts to identify core skills important to the team. We match potential candidates’ talents with our current talents; looking for a healthy mix of overlap while expanding the team’s abilities with an eye to the future. A successful candidate will be proficient at interacting with business owners, have an aptitude for understanding data, and the technical abilities to access it, visualize it, model it, and can work with our machine learning engineers to implement their code into production.