 Hey, well thank you for the super warm introduction. I'm super excited to be here today to talk through what I personally think is one of the most important product areas in Weights and Biases for enterprise teams working on model development. But I'm biased because I'm one of the product managers here at Weights & Biases, and I'm primarily focused on a bunch of different areas, but primarily artifacts, model registry, and some of the features that we will be talking through around automations. Before coming to Weights & Biases, I was a product manager at Scale.ai and did work around synthetic data. So I'm excited to kind of take my learnings from, you know, managing complex models and data sets and walking through some of the stuff that we've been doing here at Weights and Biases to make it easier to manage models in production. Cool. So just to set the stage on where the model registry becomes relevant within a machine learning team's workflow. So you can see here that on the left hand side of the screen, we have kind of our experimentation. This is where folks like data scientists, machine learning engineers are super involved in this workflow of creating models, evaluating them, packaging them. And then on the right-hand side is kind of the second part that's more focused on deploying models to production, releasing them, and monitoring them. So to walk quickly through what a practitioner's workflow might look like. And we'll go more into detail a bit later, but you can imagine you have your data scientist or machine learning engineer that's developing these models in the experimentation process. They're producing hundreds of training and evaluation runs to try to improve a model's performance. And then eventually we get to the kind of part around like deployment, where a new model that comes from all of those hundreds of training runs is eventually released. And in the middle, we can see the staging step. So very similar to the world of DevOps, staging is the step in between where a practitioner says, hey, I have a new candidate for release, and we need the model registry to house these staged models and manage all of the workflows associated with candidate models. So again, left-hand side, very data science, research heavy, and then the right is kind of all the work that's being done to keep the product running and operational. And the goal of the model registry is to be this kind of centralized point as models pass from left to right. So one thing I wanted to spend some time talking about is, you know, are we inventing the wheel here or where have we seen this in other workflows? And I think the best example is how do we solve this problem in software now? So I think we can really think of, you know, the model registry and you'll notice similarities in terminology here, but it's kind of like a container registry, like Docker Hub. So a developer is publishing an image to this container registry, and anybody can download and consume this image, obviously, if they have the appropriate permissions to be doing so. And there's a few other pieces of similarity. So, you know, versioning and tagging of these images, each container image is versioned and kind of has its own tag. This piece is crucial for kind of how you maintain differentiation between versions of an application or a service. And then also kind of how do you ensure that specific versions can be deployed or rolled back if needed. Image distribution, so kind of enabling the distribution of container images across many different environments. And the same theme that we see around collaboration, so teams can now share and collaborate on a standardized container image. The other two pieces to mention, which we'll dive more into how they're relevant to model registry, but we see patterns around using these container registries for access control and integration with CI-CD. The other example that I have here on the bottom right again is a different type of registry that we're familiar with from the world of software. So similar to Python package registries like PyPy, So similar to Python package registries like PyPy, we have this public registry of Python libraries, and this facilitates the distribution of Python packages. So developers can easily share their libraries and kind of post them to this registry. And then anyone in the community, in the broader Python community, can kind of use them and install them. So across these two examples and the patterns we saw around publishing and using in model registry, what do all these things have in common? And then what do these software solutions enable? So they enable a way for developers to build and package code and publish usable versions of their code to a central repository so that others on the consumer side, so that might be a dev team, it might be an ops team, can go ahead and consume these models directly. Or, sorry, in this case code, but in the world of model registry, consume models directly. Nice. So I'm going to spend some time talking through, you know, you're an enterprise team, you have a bunch of machine learning engineers, and you know, you're putting models into production. Why should you care about this tool called a model registry and what are kind of the big, the most important highlights and benefits of implementing one into your team's model CICD workflow. So the first piece is abstracting out the messiness of experimentation. So you can see here that on the left-hand side, this is kind of a screenshot of our Artifacts product, which really at a project level is storing all of the different model checkpoints that are being created. So you have a bunch of different runs, and even within those runs, you know, you might have hundreds of different checkpoints and you need a way to kind of post the checkpoints that are candidates for production so that whoever needs to consume it, so like the MLOps engineer or the team lead that's trying to see the latest and greatest model or the product manager isn't sifting through hundreds of checkpoints and trying to figure out, hey, which was the one that you wanted me to put into production? The second piece is on governance and control. So being able to answer questions for models that are in production, who promoted this model? When did they promote it? What was the exact version of the dataset used? What kind of post-processing happened to this dataset before model training? So a production model is going to be served to users, and the implications of a model that's deployed without proper approvals are a lot greater. There's a lot more at stake than publishing to a conference papers. So one example that I wanted to pull in from one of the customers that we're working with closely, they're a large enterprise customer and they're putting large generative AI models into production and they have kind of requirements from their legal and compliance teams and need to be able to prove that for the models that are putting into production, each model that's being deployed can be traced back to prove that it was trained on a data set that was licensed or public and that the company has permissions to be training these generative models on a specific data set. And that's where the story of lineage becomes really important that we saw in this slide here. This is kind of one snapshot, but having anyone from the compliance or security or audit team being able to say, hey, this is the model in production. I want to be able to work backwards myself and see that the data set you used to train it on is licensed so that we don't get in trouble with legal implications downstream. And then this is kind of a nice comic that I've been referencing a lot when talking to customers, but you'd be shocked at how many customers we talk to regularly that when they first start using the product, one of the biggest questions that they have a tough time answering is, I have a model in production, I don't know which exact snapshot of data was used. So yeah, that's on the piece of kind of governance and control. And then the last story is around accelerating model rollout. And this becomes relevant, if we think of that kind of graphic that we saw in the beginning around we have experimentation on the left-hand side and we have production on the right-hand side. Typically within large enterprise teams, there's different cohorts or different types of folks that are responsible for doing the experimentation and then the folks deploying these models to production. And the personas, we'll dig into a little bit more of that in the next slide, but what we see there is this critical handoff point that without a model registry is super slow, frustrating, and error-prone. So again, we see kind of these scary questions like, or comments, hey, like, I think you deployed, you know, the wrong model. I sent the wrong file. Oh, well, I didn't know this new model was even ready. Oh, I kind of deployed this model because I thought that it was ready to be deployed. And that's what, you know, Person XYZ told me. So the more time practitioners are spending doing this manual process of handing off to the MLOps team, the less time they're spending on their work on model improvement. And then the same story goes for folks on the MLOps side. So they're spending kind of a bunch of time trying to find the right model and then rolling it back in case a mistake was made and the wrong model was deployed. And in the end of the day, you know, the longer it takes end users to use the current best model, this is kind of dollars lost for the team because there's a new model that's ready to be used. Let's make it as fast as possible to roll this out to the surface. And just to talk a little bit about these personas that I've been throwing in. So I imagine that there's folks who are joining this course who are familiar with the other pieces of the Weights and Biases ecosystem. And one thing to note about registry is that it actually introduces new personas into the puzzle. As I mentioned, it's this handoff point between ML practitioners, so the folks regularly using experiment tracking, creating their runs, logging models, and the ops team that's deploying the model. And so some of the new personas, there's, there's that example of the MLOps team. They're responsible for this handoff and they need to be able to answer what is the best model and, you know, am I ready to be able to deploy it? Then there's the compliance legal security stakeholders that I mentioned in the customer example there. To them, lineage is really important. They want this assurance that the entire history of the model and the ingredients and components, whether that's metadata, which libraries were used, what pre-processing happened, can all be traced. And then lastly, you know, the folks, ML, an exec, a lead who wants this view of all of the models a team is working on. You know, experimentation is super messy, and that piece is more of an implementation detail. What people really need to see is, you know, what are the published versions? And there's a few folks that I also put in here, including, you know, product manager. So this is, you know, the space that I'm in. And as PMs, we're currently, or we're constantly looking for, hey, where is the documentation for, you know, what the models the team is working on? What is up-to-date performance? What are the expected output? So all of this kind of documentation and model card work also needs to be there for this persona. And, you know, before I'll hand it back to Hamil to demo, you know, the product and these APIs live, I first wanted to mentally prepare you as to how easy this really is to do. And that I'm talking about all these kind of like features and important aspects of enterprise model management. You might be thinking this kind of sounds like a mess to set this all up. And kind of my commitment to you is that you can actually do this all log and link your model in one line of code. Three, if you include, you know, installing weights and biases and starting your experiment. you know, installing weights and biases and starting your experiment. And with that being said, I'm going to hand it back to Hamelin and super excited to walk through the product itself, some of the APIs, and transition into the second part of the course where we'll talk about automating workflows for kind of streamlined model management and connecting this model management piece to your CI-CD pipelines.
What is Model Registry?
Dive in with Noa, Product Manager at Weights & Biases, to learn what Model Registry is!

The definition we will be using in this course is: model registry is a repository of a team's trained models where ML Practitioners publish candidates for production to be consumed by downstream teams and stakeholders.