Day in the Life: Machine Learning Engineer

At The General®, Machine Learning Engineers work on a variety of different projects ranging from model retraining and monitoring, to API development for model deployment. We also devote much of our time to research and development since most projects do not have a clear best practice and toolset. With that, each day on the job can be quite different from the last. This post will use an entire week’s worth of tasks as an example of a “Day in the Life” of a Machine Learning Engineer at The General.

 Below is an Eisenhower Matrix that categorizes the different activities a Machine Learning Engineer at The General may complete in a given week:

I recently heard about the Eisenhower Matrix from Kyle Nakatsuji, co-founder and CEO of insurance startup Clearcover, while attending the Venture Fellows program produced by the American Family Ventures team. The matrix provides four categories (Do first, Schedule, Delegate, and Don’t do) which fall into quadrants based on urgency and importance. Each quadrant lists different tasks that I aim to complete for a given week.

Screen Shot 2019-07-03 at 11.49.50 AM.png

The “Do first” quadrant lists all of the urgent and important tasks that require completion within a few days or at most by the end of the week. Notice the “validate prediction stability for retrained Survival Analysis model” task is before the “deploy retrained Survival Analysis model into production” task in the “Do first” quadrant. Machine Learning Engineers must test and validate the performance of new or retrained models before releasing to production. Metrics created to validate predictions are used as KPIs for ongoing model performance monitoring, and these can differ greatly across projects.  Since individual models have specific requirements, we work closely with the Data Scientists who built the model and the business owners to implement a unique solution.  Of course, there are unforeseen urgent and important tasks that can overtake planned activities, such as a failed batch prediction job or API performance degradation.  It is important to be flexible and stay on your toes to be successful in this role.


Screen Shot 2019-07-03 at 11.50.26 AM.png

The “Delegate” quadrant lists all urgent, but less important tasks for a given week. Tasks listed in this quadrant focus on submitting requests to different groups within Data and Analytics or Information Technology. Tasks can range from a simple request for information about a database table, to complex Amazon Web Services network configuration required for model deployment. Machine Learning Engineers must work closely with DevOps to ensure cloud deployments are completed correctly and securely. Occasionally, tasks from the Delegate quadrant can move into the “Do first” or “Schedule” quadrant depending on the priorities of different groups.


Screen Shot 2019-07-03 at 11.50.53 AM.png

The “Schedule” quadrant lists the less urgent, but still important tasks for a given week. These “Scheduled” tasks focus on research and development or documentation activities and are usually part of a larger solution. For example, the “research Falcon web API framework for Python…” task is something all Machine Learning Engineers on the team are currently researching for a proof of concept we are working to launch. It is important to schedule time each week to learn about new functionality and technology in the field of Data Science due to the pace of change within the practice. Many CEOs, Presidents, and thought leaders are on record of how important continual learning is for lifelong success, and it is no different for us.  The ability to quickly learn and apply new concepts is important for Machine Learning Engineers and Data Scientists at The General. A “Schedule” quadrant task can eventually be a skill necessary for a future “Do first” task.


Screen Shot 2019-07-03 at 11.51.19 AM.png

The “Don’t do” quadrant lists neither urgent nor important tasks, but can also list certain habits or activities a Machine Learning Engineer should eliminate. The example matrix has “delay project completion seeking perfection…” which serves as a reminder to avoid trying to create a perfect solution for every project. We have saying: to not let perfect be the enemy of done; and it is important to be reminded of that daily.  For example, in a perfect world, the retraining of a model is automated with a Python script but the work to create the Python script should not delay the positive business impact a retrained model will provide. I find that activities that do not fit into the scope of a current project can eventually end up in the “Schedule” quadrant. On the other hand, Machine Learning Engineers must have the instincts to know what is essential so a deployed solution does not crash and create an unplanned “Do first” quadrant task. Over time and through experience we can hone these instincts.

 

Machine Learning Engineers at The General wear many hats and have the opportunity to work on unique, high business impact projects.  The most important aspect of the role is to deliver functional and reliable deployments with low technical debt so that the Data Science team can continue to productionalize models that make our customers’ lives easier.

Jack Pitts