Page 2
2 Table of Contents Contents Why MLOps Matters? (Foreword) 3 People of Machine Learning 5 How Is Machine Learning Different from Traditional Software? 8 The MLOps Workflow 10 What Is the Point of MLOps? 10 Risk in Machine Learning 11 Time to Market 15 The MLOps Workflow Enforces Best Practices 18 How to Quantify Success in an MLOps Project? 20 Speak the Same Language 22 Use Every Project as an Opportunity to Educate Your Organization about Machine Learning 22 Define Clear Shared Objective and Metrics 22 Real-World Example - The Story of Two Companies 25 Company 1 25 Company 2 26 Learnings26 The MLOps Toolchain 28 Model and Data Exploration 28 Metrics and Model Optimization 31 Productionalization - End-to-End Pipelines 35 Productionalization - Feature Stores 39 Testing 43 Deployment and Inference 46 Conclusion 50 About the Authors 52
Page 3
Why MLOps Matters? 3 Why MLOps Matters? (Foreword) Traditionally, machine learning has been approached from a perspective of individual scientific experiments which are predominantly carried out in isolation by data scientists. However, as machine learning models become part of real-world solutions and critical to business, we will have to shift our perspective, not to depreciate scientific principles but to make them more easily accessible, reproducible and collaborative. In May 2020, we surveyed 330 data scientists, machine learning engineers and managers in a broad range of companies to ask what they were focused on for the next 3 months and what major obstacles they were faced with. Although 20% of respondents said they are still focusing more on the experimentation and learning phase, half of the respondents said that they were focused on developing models for production use and over 40% said they would be deploying models for production. Source: State of ML 2020, Valohai 330 respondents 50% 40% 30% 20% 10% 0% Collect more Figure out how Make data more Learn machine Prove potential Develop models Deploy models Optimize Automate Monitor data data can be accessible learning of machine for production to production models retraining model(s) in used technologies learning use model(s) production
Page 4
4 Why MLOps Matters? Automating retraining of models and monitoring models in production were still not relevant for most respondents which speaks to machine learning in production being relatively nascent for most. However, the steadily rising interest in MLOps based on search volume and news articles speak to these aspects becoming more and more relevant. Production models raise new challenges, not just for data scientists but the extended team of engineers, product managers and compliance officers, which will need to be solved collaboratively. In most real-world applications, the underlying data changes constantly and thus models need to be retrained or whole pipelines even need to be rebuilt to tackle feature drift. Business and regulatory requirements can also change rapidly requiring a more frequent release cycle. This is where MLOps comes in to combine operational know-how with machine learning and data science knowledge. In machine learning, 2020 is the year of production models and we predict that 2021 will be the year of MLOps. We hope that you enjoy our eBook on MLOps and get new, fresh ideas for your use case.
Page 5
People of Machine Learning 5 People of Machine Learning The size and scope of real-world machine learning projects has surely surprised most if not all of us. What seems like a straightforward task of gathering some data, training a model and then using it for profit ends up becoming a deep rabbit hole that spans from business and operations to IT. A single project covers topics such as data storage, data security and access control, resource management, high availability, integrations to existing business applications, testing, retraining and so much more. Many machine learning projects end up being some of the biggest multidisciplinary and cross-organizational development efforts that the companies have ever faced. To understand the multidisciplinary nature of ML projects, we start by looking at the roles involved in them. These roles come from our unique collective vantage point of seeing around 500 different organizations from startups to Fortune 500 over the last 4 years. While the list of roles below is not entirely exhaustive, it gives an overview of the people involved. Remember that these roles are not necessarily one per person but rather a single person can - and in smaller organizations often has to - cover multiple roles. For example, many data scientists also handle ML Engineering tasks, and sometimes DevOps and IT are synonymous. The rule of thumb is the larger the organization, the more specialized and siloed individuals are which requires all the more collaboration. Data Scientist Data scientists are tasked with finding data-driven solutions to business problems. Data scientists tend to be highly educated with a master’s degree or higher in mathematics and statistics, computer science or engineering. Their strengths lie in data wrangling, statistical analysis, data visualization and applying machine learning. Generally, they are deployed first to analyze existing data and to find
Page 6
6 People of Machine Learning patterns that could be used to solve underlying business problems. For example, they might be looking at user data to find meaningful user segments and building models that can classify those users in segments in order to differentiate the end-user experience and drive more engagement. While the primary purpose of data science is to explore data and build models, often a large portion of time is spent manipulating data to suit the use case. Data Engineer Data engineers are responsible for the infrastructure that ensures data is stored and available for data scientists. They build the systems that ingest raw data from various sources and store it in a centralized data lake. Data engineers often also build other data warehouses that derive data from the data lake but transform it so that it is more useful for the end-user; for example, image data might be resized and reformatted so that data scientists can focus on data analysis rather than data manipulation. Machine Learning Engineer Machine learning engineer is a rather nascent role that is becoming increasingly relevant as companies push to get real-world value from machine learning. While data scientists explore and prove a theoretical concept, ML engineers take those concepts and build them into a production-scale system. As the title implies, ML engineers understand the theoretical side of machine learning but are more involved with paradigms found in engineering such as continuous integration (CI) and continuous delivery (CD). They tackle automating and scaling model training, testing models before deployment and deploying and monitoring models in production. Often they will rewrite or restructure code written by data scientists; for example, turning notebooks into scripts. DevOps Engineer DevOps engineers combine the domains of software engineering and cloud infrastructure. In traditional software projects, DevOps
Page 7
People of Machine Learning 7 handles automation around testing and releasing new code, and often also manages testing and production environments. In machine learning projects, DevOps can be the connection between the ML model and the end-user application; for example, releasing a new recommendation engine to a web application. Scalability, stability and responsiveness of the end-user application are often at the forefront of their mind and they’ll be involved in ensuring machine learning models don’t degrade these elements. IT IT operations can be found in most larger organizations. They are generally tasked with resource management, access control and information security. While often seen as red tape, IT operations and the processes involved can help a project run more smoothly, whether it’s through requesting that computing resources be provisioned or ensuring that outside vendors are approved. Business Owner There isn’t a specific title for a business owner in a machine learning environment but it’s a key role to ensure success. Business owners have the deepest understanding of the end-user and use case, and guide the team to producing ML systems that are not just accurate and well-engineered but also valuable. They’ll help define what kind of predictions are useful and help understanding of where risks lie from an operational and regulatory perspective. Manager The management role in a team working with machine learning tends to be a little unclear as these teams tend to be relatively new. The key aspects to consider are that machine learning projects are deceptively complex and multidisciplinary. All the required resources to execute a project are often not in a single team and a manager must secure these from different parts of the organization. Another difficult task for a manager is to understand and communicate the ROI of the team’s work as costs and benefits are likely not directly linked with the team.
Page 8
8 How Is Machine Learning Different from Traditional Software? How Is Machine Learning Different from Traditional Software? We already established that the cross-domain nature of ML causes a new level of discipline in organizations but that’s only part of the picture. There is an additional dimension that differentiates ML and classical software — data. It might sound self explanatory and simple but the addition of data, usually big data, brings entirely new challenges to the development process. While software can generally be developed locally with almost instant feedback on how a code change affects the end-result; in machine learning, to see the effects of a code change requires retraining a model. When working with models that are trained with large datasets, this poses huge infrastructure challenges as either you will be waiting for a long, long time or will need to use remote computing. Just imagine the difference between an application developer just saving a code change and pressing refresh in their browser, and a data scientist saving a similar code change and then spinning up a cluster of GPUs, deploying code, transferring data and training a model before seeing any results. However, it’s not only how the introduction of data changes how code works in the development process but also that data is another aspect that can change. In classical software development, a version of the code produces a version of the software, while in machine learning, a version of the code and a version of the data together produce a version of the ML model.
Page 9
How Is Machine Learning Different from Traditional Software? 9 Data as a new variable in the development process increases complexity drastically, as to reproduce results you need to be able to reproduce the data that you used. While you might not be producing every possible variant of the model, the components must be versioned to ensure reproducibility. The matter of versioning both data and code becomes increasingly complex when you start thinking about whole machine learning systems which have steps for data preparation, augmentation and generation.
Page 10
10 The MLOps Workflow The MLOps Workflow What Is the Point of MLOps? You might have downloaded this book because you intuitively know that MLOps is essential. This intuition arguably comes from the pursuit of doing things the right way and feeling the pain of constant technical struggle in one’s daily work. However, intuition alone can be hard to argue with when talking to teammates looking for approval for an investment – of time or money. This chapter solidifies the idea of MLOps to help understand why it is worth focusing on and how to justify this. The chapter will also help assess what areas to focus on, and the following chapters will dive deeper into concrete ways to measure and show the ROI. When we look at how the current literature describes a model lifecycle, we often see a picture like this: Design Model Operations Development However, in reality, most organizations today still operate in a cycle where hand-overs between model development and operations are – simply put – messy and manual. As fun as development in a vacuum is (we all have done it), the reality is that only a model running in production can bring value. Models have zero ROI until they can be used. Therefore, time to market should be the number one metric to look at and optimize for any ML project.
Page 11
The MLOps Workflow 11 Time to Market 10% Model Development Design Operations 90% Glue Coding Complex & unstructured handover The goal of MLOps is to reduce technical friction to get the model from an idea into production in the shortest possible time to market with as little risk as possible. Let’s break down the statement. It contains two parts, reducing the time to market and reducing the risk involved. These goals help data scientists and business stakeholders to speak the same language and frame MLOps as a business-driven necessity. Albeit MLOps will help automate manual steps and improve the quality of code, but they are not the underlying goals the whole organization will rally around. Next, we will talk about risk and time to market. The area of risk assessment is a more straightforward concept, so let’s start there. Risk in Machine Learning The goal of MLOps is to reduce technical friction to get the model from an idea into production in the shortest possible time to market with as little risk as possible. We have identified three primary types of risks the MLOps workflow tackles: 1. Loss of knowledge 2. Failures in production 3. Regulatory and ethical
Page 12
12 The MLOps Workflow There is plenty of industry, implementation, and even company- specific detail involved in each of these main topics. However, the most significant risks that machine learning in production introduces fall under one of these categories. Keep in mind that we are skipping basic IT risk that is already well documented and covered elsewhere but is not exclusive to machine learning in production; for example, data security. It’s clear that no- one should be able to steal data from your database, but that is true for all databases – not just ones used for machine learning. Loss of Knowledge No! Only I know how to train our production models! Bus factor The bus factor is a common term in software engineering describing the risk of a key contributor disappearing unexpectedly from a project – because they get hit by a bus. While it is a well-documented problem for classical software development, machine learning magnifies the bus factor significantly. In ML, in addition to code that is often somewhat self-documenting, there is data. A single data set can contain vast amounts of hidden information about how it’s been collected and then how features are extracted. There may be many versions of notebooks or scripts that produce a single data set. Loss of knowledge can happen in many different ways. Most obviously by a data scientist leaving the company, and most larger teams have already faced this. But it can also occur without personnel changes. When you work on multiple projects over a more extended period, you
Page 13
The MLOps Workflow 13 tend to forget details about your own work too. It can be challenging to return to a six-month-old project and retrain a model without breaking things. Many readers may have existing models running that they dread getting back to for fear of breaking something. For traditional software development, reproducibility is mostly taken care of by proven version control tools like Git. To achieve proper version control and reproducibility for ML, you will need a bit more on top of that. Input Execution Output Training code Used hardware Model Parameters Environment Logs Dataset Experiment cost Results Statistics Version control for machine learning experiments In ML, a model is a combination of code, data, parameters, and the training environment. To pick up where someone else (or yourself six months ago) left off with a model, you’ll need much more than just a Git repository. You need to know what data the code is using and how that data came to be. There are often multiple scripts bringing together different data sources and doing feature aggregation to build the dataset you need. Putting together the puzzle of several scripts and data sources someone else made leaving behind zero documentation can often be more complicated than just starting from scratch. To tackle this, we recommend a version control system that can track your whole pipeline from raw data to the model, including your code, environment, configurations, and parameters. Failures in Production The next category of risk is failures in production. Everyone who has worked in software development knows that shipping broken code
Page 14
14 The MLOps Workflow to production is a prevalent failure point for systems. Teams spend extensive amounts of time building CI/CD pipelines so that developers can be confident that their latest update does not break anything in production. The same thing applies to ML. From basic syntax errors to changes in the data stream or unexpected model performance. In a perfect world, not only does your MLOps tooling support various methods of testing your code and data throughout the pipeline, but you’ve also adopted a practice to start codifying these tests, starting from the design phase of an ML project. We will cover different parts of the ML workflow and related testing in later chapters of the book, but here are some general pointers to look out for. 1. Ensure that the data used to train a model looks like you expected it to. There may be changes upstream, for example, how the data is collected or stored. 2. Ensure that the model works not only in training but in a real-world environment. Overfitting is a real risk that should be addressed by your pipeline and process. 3. Ensure that the infrastructure works consistently. Your ML pipeline should produce identical results if the inputs stay the same, and you should always be able to roll back. Regulatory and Ethical Machine learning algorithms come under severe regulatory and ethical scrutiny due to not being explicitly defined by a person. When realized, consequences can be financial but maybe even more prevalent is the loss of trust and reputation. Therefore a data scientist might be looking at frequent governance audits, making many people nervous. Enforcing MLOps best
Page 15
The MLOps Workflow 15 practices makes reproducibility a priority, and introduces version control for every component of a machine learning model. Think of this as bookkeeping, and when books are in order, there is nothing to worry about. Most importantly, if a prediction a production model has served comes into question, it can always be traced back to how the model was trained. Most MLOps tooling will introduce version control to your workflow, but that’s only half the battle in minimizing regulatory risk. The second half is ensuring that production models are unbiased, and that is generally much trickier as tooling can only provide a framework, but subject matter expertise is required to enforce the right rules. A prejudiced model likely won’t fail from a technical perspective, but it might from a regulatory perspective. Therefore building on the technical tests by including tests that combat bias and other ethical concerns need to be considered in an ML pipeline These concerns vary wildly depending on the application; for example, a healthcare application will have very different ethical concerns to a financial application. While MLOps can’t fix model biases, it comes with the notion that these concerns should be addressed by codifying tests for them into a machine learning pipeline. Checkpoints and safeguards for how the data used for training should look and what the expected predictions should be will make governance audits much less intimidating. All of the production models will be produced in compliance with stable and reliable rules. Time to Market It’s hard to beat an ad-hoc workflow on speed to market when first developing a machine learning model. For many data scientists and engineers, that is the reason they are hesitant to adopt rigor into their process. However, machine learning in production is rarely about a single version of a single model getting into production but rather a long-term commitment to building machine learning capabilities that
Page 16
16 The MLOps Workflow are continuously evolving. In most cases, you have a lot of dimensionality in model building. First of all, models rarely work over long periods and tend to require retraining somewhat regularly. Secondly, there are often several other dimensions to think about. It is common for models to be, for instance, retrained per customer based on their specific data or by geography, etc. The same model architecture can be running in production as multiple versions of multiple datasets are being re-trained periodically over time. This quickly results in exponential retraining, and ad-hoc workflows will quickly result in chaos, customer dissatisfaction, and poor results. To get a truly scalable business model around ML usually requires that you step back and think of your product in terms of the pipeline instead of a single instance of a trained model. Here’s where the most successful companies in ML are today. Model Customer 1 Customer 2 Data Source 1 Data Source 3 Data Source 1 Data Source 3 Time 1 Time 2 Time 3 Data Source 2 Time 1 Time 2 Time 3 Time 1 Time 2 Time 3 Data Source 2 Time 1 Time 2 Time 3 Time 1 Time 2 Time 3 Time 1 Time 2 Time 3 18 Trained Models
Page 17
The MLOps Workflow 17 Broadly speaking, there are two ways that MLOps helps accelerate the time to market of machine learning: 1. It creates a shared language among the extended team. 2. It automates manual tasks. Shared Language We have seen a drastic uptake in software development delivery speed and rate over the years. Modern tooling and shared work methods (CI/CD, version control, microservices) have enabled companies to scale their throughput in software development exponentially. A shared language enables new team members to hit the ground running when you need to accelerate your development. There is very little to suggest that the same requirements for exponential delivery growth would not hold for machine learning too. Secondly, as discussed earlier, MLOps revolves around turning the process of creating a model into a machine learning pipeline. To achieve this, teams have to codify and componentize their work. While requiring extra effort compared to a single notebook and some manual actions, splitting work into components allows work to scale in a completely different way. This is very similar to what micro services have accomplished in larger development projects. A single data scientist doesn’t need to create the whole model, but rather a team can work on separate pieces such as data preprocessing, model training, and testing. As the complexity of model creation increases, so do the benefits of working around a shared pipeline. Also, a pipeline consisting of various code pieces is inherently self- documenting – to some extent – and can be more easily handed to the next person than manual actions. Automation DevOps has allowed software developers to move from a monthly
Page 18
18 The MLOps Workflow or quarterly release cycle to a daily or weekly cycle. Despite the complexities we’ve explained above with machine learning containing more variables than just code, we should still ensure optimal progress. Building a CI/CD pipeline for machine learning is more challenging than for traditional software, but the same benefits apply. Automation of collecting data, training, and evaluating models allows the most scarce resources, i.e. data scientists, to focus their efforts on further development. These pipelines are also not throw-away as you can re- use parts when developing a model for a different purpose. Again the importance varies by application as the benefit of a quicker release cycle is not the same across different use cases. However, in the most extreme cases, your model needs to be updated all the time to keep up with the pace at which the underlying data is changing. This is one reason why some trailblazing companies such as Facebook and Uber had built their own MLOps infrastructure before the term was even coined. Time to Market Model Design Development Operations Version Control Pipelines Testing Automation The MLOps Workflow Enforces Best Practices MLOps ties model development and operations together into a continuous loop through a set of best practices. These best practices can be enforced in many ways, most frequently through a shared platform. To quickly summarize, we identified four best practices that will help you reduce technical friction to get the model from an idea
Page 19
The MLOps Workflow 19 into production in the shortest possible time to market, with as little risk as possible. • Version control everything, including models, code, data, parameters, and environment. Enable anyone to trace how a model was produced. • Componentize the steps of the model creation process and build them into a pipeline. A single notebook is not a pipeline. • Codify testing. With checkpoints and safeguards in place, there is a standard that models have to adhere to. • Automate work to increase how much time can be spent on future development.
Page 20
20 How to Quantify Success in an MLOps Project? How to Quantify Success in an MLOps Project? Quantifying success is not just about the success of the business outcome. In this stage of MLOps maturity, it’s also about the success of the process, including multi-disciplinary collaboration and documentation of learning. “Only a model that is running in production can bring value.” As we move from projects that experiment with machine learning to projects that are aimed to bring real business value in production systems, we grow the complexity of the project. With it, the uncertainty of success increases. Here are three steps to get started: 1. Define your objective and clear, concrete, and measurable metrics with all the relevant stakeholders • Ensure everyone understands the “why” behind the project and the expected benefits it will bring, and educate everyone on the basics of how the ML project will be evaluated. Everyone should know the basics of how a model’s accuracy can be measured and what is over-/under-fitting. • Document and share how the work is currently done and benchmarked within the team and what it will take to replace the current way of working. • Uncover assumptions that different team members have by hosting a session to discover pre-conceptions about the problem and come up with possible solutions. Assumptions could be related to, for example, what data sources should be used, how the data should be processed, or how the model should be published. Try to uncover these assumptions and scope out small experiments to test the hypotheses out, before taking them as facts.
Showing the first 20 pages. Download the document to see it all.
Comments (0)
No comments yet. Be the first!