 Welcome to Unified and Open Governance for Data and AI. My name is Zeeshan Papa. I'm a product manager here at Databricks, and I focus on Unity Catalog, data, and product governance. My name is Fabien Tukor from MasterCard, and I'm not here to talk to you guys about data governance. I'm here to talk to you guys about what it enables, namely getting access to data, finding data for the right use case, and using it for innovation. Thank you, Fabian. Fabian will be beginning his slides a little bit after I initially talk about Unity Catalog. So let's start with what is governance and why is it important. We've talked to lots of customers over the years and we believe that governance is critical to scale your data and your AI initiatives. In the age of generative AI and the age of large language models, the governance for things like data and AI has actually become even more paramount because we're now utilizing and things like data, ML ML models and we're seeing explosive growth in the sector but the foundational principles of governance things like accountability compliance quality transparency etc these things were all traditionally extremely important and essential for data management they now extend to artificial intelligence and AI and this is becoming a really top-level topic at most companies for example Forrester predicts that AI governance is now the next big board-level topic at most organizations. Now everybody sort of understands the importance of this, but governance of the data state is actually practically very difficult. Let's see how. In most organizations, you have a bunch of different data living in a bunch of different places. You have data in different data sources like data lakes and data warehouses and databases and catalogs, possibly multiple catalogs. You have all these different formats, things like iceberg and Delta Lake and Parquet. You also have fragmentation when it comes to things like the actual types of assets, so things like notebooks and files, machine learning models, dashboards, tables, views, et cetera. You have all these different diverse data and AI assets and they're across a variety of different heterogeneous tools. What you end up with is a bunch of different silos, silos across formats, silos across assets, silos across sources, et cetera. And this causes a whole bunch of challenges from a governance standpoint. For starters, you suffer generally from fragmented governance for data and AI. And this makes it hard to manage things such as access policies or auditing, monitoring, observability, lineage across all the individual silos that you might have in your organization. There's also a lack of open connectivity which makes it extremely challenging and difficult for consumers to be able to access and share data and AI across different tools, engines, and platforms in order to be able to collaborate and interoperate in a broad way. Finally, there's a lack of built-in intelligence and that really means the fancy way of saying that if I think about trying to discover data and I want to understand and get a domain-centric view of my data, it requires having and being able to connect information from all these different silos and putting them in one place. What really ends up happening is you end up having a few key technical experts that are sort of the tribal knowledge holders within your organization. And so without that intelligence inside your platform, the platforms tend to fall short of connecting things like business concepts with underlying data and models. And this event inevitably creates that bottleneck where these experts are needed. And these governance challenges have significant impact in terms of business value. So fragmented governance often leads to increased complexity in terms of compliance, security, and quality risks, as well as increased operational inefficiency. So businesses tend to struggle to maintain a unified view of their data and AI estates. Lack of open connectivity often leads to vendor lock-in, and this limits an organization's ability to switch vendors or integrate things like new technologies as the needs and the ecosystem sort of evolves. Poor inoperability also further hinders collaboration and the scaling of data and AI initiatives. This leads again to higher costs as the organization has to now leverage different tools or multiple tools, replicate things back and forth, creating more silos and more of the same problem. Furthermore, the lack of built-in intelligence actually limits your ability to have broad data and AI democratization across your organization and effective use of these platforms as well. So what this means is for the folks that are non-technical users, they're not able to find and understand the various data and data and AI assets that are in your organization. Ultimately, this slows down things like innovation and hampers the ability to make really effective and timely business decisions, which is, as you can understand, limiting your organization's ability to fully leverage the data and the AI that you have for an actual competitive advantage. Our point of view here at Databricks is that if you create an ideal governance solution from the ground up, that it needs three things. Unified governance across data and AI assets, and open formats so that organizations have complete visibility, monitoring, security, and compliance across all their data sets. Open connectivity so that teams can connect to any data source and can all their data sets. Open connectivity so that teams can connect to any data source and can access their data and AI assets from any tool, any engine, or any platform. Built-in intelligence that connects the dots across all these different diverse data and AI assets regardless of format, understanding things like semantics, understanding usage, understanding lineage, and how they are all connected in order to be able to provide that domain knowledge and those insights to all users across any type of workload and autonomously perform things such as platform optimizations, which also leverage that information. So there's a whole bunch of really cool things that we think about when we think about governance. It's not just access controls on data. It's the ability to find data. It's the ability to find data. It's the ability to manage data effectively across your enterprise. And it's the ability to use that data across a variety of different tools in and outside of the Databricks ecosystem. We took a pioneering approach here at Databricks with Unity Catalog by releasing the industry's only unified and open governance solution for all data and AI. This solution works across any open data format and any data and AI asset and provides comprehensive governance capabilities, including access management, lineage, discovery, observability, auditing, monitoring, et cetera, in a single system. But Unity is more than just a catalog. It is an enabler of data intelligence for your platform. Whereas traditional catalogs are just limited to things like access management, tabular data, metadata exposability, Unity is in fact open to any external data source and can be accessed from any external tool or engine. Now, let's take a quick look at what Unity Catalog actually provides in terms of these three foundational pillars that we talked about, starting with unified governance. First, you drive operational efficiencies with a single catalog to manage all of your organization's data and AI assets, such as files, tables, machine learning models, AI tools, business metrics, et cetera. This works across any open format, so for example, Delta Lake, Iceberg, Parquet, and more. You can enhance compliance with a simple singular interface to be able to manage access policies and auditing across your data and AI. And you can also improve your security posture with fine-grained access controls and attribute-based policies for data and AI to meet your unique security needs. With Unity Catalog, you also reduce the operational overhead of understanding data flow and dependencies with real-time lineage for data and AI. Ultimately, this helps you improve compliance by classifying and tagging sensitive data and AI assets, for example, such as PII, PCI, PHI, etc. We actually come out of the box with things like quality monitoring alert capabilities that help you ensure that consumers ultimately have trust in the quality of your data. This is of course a very important part of data management and data governance as a whole, is being able to signal to other consumers of your data that these in fact are trusted data sources. All these capabilities help you build this unified and compliant view of your data estate. One of the features that we talked about earlier this year at Summit is Unity Catalog Metrics. And this is coming out hopefully within the next quarter or two, don't quote me on that. But it gives us the ability to discover, govern, and query business metrics. So what this really means is building a semantic layer effectively above your data and being able to inform unity catalog through the creation and the management of these metric objects what the correct version of data is in your organization and this then obviously informs the consumer experience so when you go into you know genie or any workspace in in databricks you have the ability to quickly find and understand the correct organizational view of data this is accessible from Databricks, you have the ability to quickly find and understand the correct organizational view of data. This is accessible from Databricks as well as external tools, and it also works with third-party metric providers as well, so your cubes and at scales, et cetera. Switching into open connectivity. Unity Catalog empowers your users with open source access, with open access and collaboration in your organization while actually avoiding vendor lock-in. First, it provides rich federation capabilities that allow you to actually connect to any database. So for example, Postgres, Azure SQL, Oracle, Salesforce, Teradata, or data warehouses like Snowflake, GCP BigQuery, Azure Synapses, other catalogs such as Glue, Hive Metastore, Iceberg catalogs, et cetera. You can keep the data where it is without actually having to copy it over. And you can still get full governance capabilities that we've discussed earlier. So for example, things like lineage or tagging, fine-grained access controls, auditing, search and discovery all in one place. Now you can access and join that data from different sources and run queries with performance optimizations. Now this actually helps mitigate the cost overheads you face with other solutions where you typically have to replicate or migrate data from one system to another in order to kind of get past the proprietary storage and system A and bring it into Databricks, etc. so you can work with it in an open format. This helps you get past all of that without having to worry about it. Secondly, Unity Catalog also provides open APIs. So we've actually just released new features in this arena as well. So teams can actually read and write data that's managed by Unity Catalog from any client or engine or platform across Delta and iceberg ecosystems. So this gives you the flexibility to choose the right tool for the right job and the tool of your choice, more importantly, with your data, and provide that broader interoperability across the heterogeneous platforms that might actually exist in your organization. Third, Unity Catalog enables you to be able to share data and AI assets across clouds, regions, and platforms, while still keeping the data in your own storage and without actually needing to copy it across clouds, regions, or wherever you're sending it to. This ultimately reduces cost because copying data is expensive, right? As in other solutions, you typically have to copy data from one location to another, out of a proprietary storage format into an open format in order to be able to make it accessible to customers, partners, vendors, et cetera. And your consumers, in many cases, also have to be on the same platform. So again, we view those as sort of a lock-in type system and that's something we're 100% trying to avoid building here at Databricks. Finally, Unity Catalog is open source. So if you're building a future proof governance model that mitigates the whole catalog lock-in concern that many organizations tend to have, building on top of Unity Catalog makes a whole lot of sense. But data intelligence isn't just limited to the data and workloads that you run directly on Databricks. You can actually unify your governance and your discovery by federating all of your data platforms with Unity Catalog. So also with a Delta sharing protocol, you can securely share that data with any system that supports Delta Lake without ETL, which is what I just mentioned a minute ago. So talking about built-in intelligence, Unity comes with an integrated AI that helps your teams improve productivity and build a competitive advantage. First, it actually provides AI-powered tagging and AI-generated comments at the column level and at the table level, or the asset level, by understanding the data and leveraging how that data is connected. This helps data owners with things like, for example, efficient documentation. And it also ultimately helps your consumers build a better understanding of data and drive clarity. Second, all of this AI-powered documentation is combined with our lineage information, which powers a context-aware search that helps users search more efficiently and be more productive. Users get out-of-the-box data insights, such as popularity, usage, frequently used queries, et cetera, to get started faster. Third, Unity Catalog's metadata, documentation, lineage, usage insights, and so on, also power the intelligent assistant that's inside the product that understands the unique context of your data and helps your users ask questions, generate, explain, and fix code using completely natural language. This provides, of course, a massive productivity boost for any user of the platform and helps a moderately SQL literate or a minorly SQL literate user suddenly become incredibly proficient in terms of being able to access data. Finally, Unity Catalog also provides AI-powered autonomous performance optimizations to keep costs low and query performance high. And we've seen this in talking to customers. We've gone and interviewed hundreds of customers. Our customers have been experiencing significant productivity gains with Unity Catalog. On average, they actually report a 57% improvement in overall productivity and a 49% faster delivery of their use cases. These productivity gains span across all governance capabilities, which actually showcases the value that Unity Catalog brings to streamlining data and AI workflows. It's pretty exciting to see what we're able to do for the organizations that use our product. There's about 10,000 customers that use Unity Catalog today across a variety of different industries, and they're all benefiting from this vision that we have for unified governance. Unity Catalog also works with your existing data catalogs and governance solutions, as I've mentioned. So you can continue to leverage whatever existing investments that you have and build future-proof governance models without necessarily having to incur expensive migration costs. All right. I think we're going to flip to a demo now real quick. So if you will bear with me, I'm going to hit the magic button. Hopefully this works. Voila, it works. That's amazing. All right. So what you're looking at here is a Databricks workspace. So I'm going to quickly go into full screen mode here. All right. So this is a Databricks workspace. If I look over here on the left-hand side, I've got a little icon here that says Catalog. If I click that, I can now open up the Catalog Explorer. This is effectively the main browsing interface of Unity Catalog. Now, I've got I can now open up the Catalog Explorer. This is effectively the main browsing interface of Unity Catalog. Now, I've got a whole lot of data in this particular workspace. But I did load up and connect a Postgres database. So I did that by actually going into the Connections tab here, clicking there. And there's a Postgres database that I configured. Don't look at my passwords or my URLs, but you get the picture. And what I did was I was able to use that particular connection that I defined to create a catalog inside Unity that I could actually run in query. So I'm going to quickly refresh this and connect my warehouse. Select the right compute. OK. Select my warehouse. Select the right compute. So this is my... Bear with me for one second. This particular warehouse is a little slow. Let's try this one. All right. There's try this one. All right, there we go. OK, so I went ahead and loaded a few different tables in here and I generated some data, specifically just so we could talk about it. And I went ahead and generated a customer sample. Now, kind of quickly looking at this, I have no idea what it is because I just connected this random database into Databricks. And as you can see here, it pops up an AI suggested description that I can accept. I can also do the same thing by clicking into a column here. So if I say, for example, couldn't guess what the address column was, I could actually generate a comment for it, right? And that will tell me what the address column is. And it's actually surprisingly really very good. We use it internally all the time to tag our own data sets. We're really happy with the way that our models give out really meaningful information. Once all that information is in there, I can start searching for things in the platform. So as a consumer, being able to search for that information is super useful. So just searching for financial fraud analytics brings up all the transaction tables, accounts, customer tables, and so on and so forth. So clicking back into that customer's table, if I wanted to go and browse the sample data, as a consumer of this, as I mentioned earlier, I can ask natural language queries of my data. So let's pull up our genie here. I click the Ask Genie button right there. And let's say, how many customers do we have? Let's give it a second to spin. All right, we've got 12 customers. We're not doing that great as a bank. We should probably get a few more customers, right? But yeah, you get the idea, right? The point is that a person who doesn't necessarily have the ability, like a non-technical user or a user who's perhaps only literate in a small amount of SQL, can quickly open up our platform, discover things, regardless of where that data lives, whether it's in a Delta Lake, Parquet, Iceberg, in a Postgres database, as I'm showcasing here, and quickly figure out and make use and sense of that data, all of being able to take advantage of the access control, monitoring, auditing capabilities that I mentioned earlier. So it's a pretty cool product. I don't want to spend much more time on it, as we have the lovely Fabian here to come and tell you about MasterCard's journey. So I'm going to switch back over to him. And thank you. Awesome. Thank you. Thank you, sir. Round of applause for Zeeshan. And I know it's not Q&A yet, but I wanted to ask you, any Cyber Monday deals on Databricks? I would be buying some if there was, yes. But unfortunately, there's not. So would I. So before we start, I wanted to ask you to remember when you started at the company that you're currently working at. Do you remember what it felt like to be brand new at that company? And you didn't know anyone, probably. You didn't know your company's products. You certainly didn't know the company's data. Now imagine if you had started and your role was to innovate with data. Maybe you're an analyst. Maybe you're a data scientist over there. Maybe you're a product manager down there. And your role is to use the data to solve problems. How do you find the right data? Are you confident that you could find the right data, the right quality? How do you gain access to the right data and in a compliant way, in a responsible way? And then once you've done that, you're not done. You still need to figure out, how do you use it? Can you do AI on it? Can you train an LM? So all of these things are what my team thinks about to enable for MasterCard and our data users. First, let me just give you a quick reminder of who MasterCard is. I'm sure you guys know, but there are some myth that we try to dispel. Number one is we don't issue cards. Only banks do that. That means we don't set your AP don't issue cards. Only banks do that. And that means we don't set your APRs. Banks do that. But what we really are is a technology company. We are a network company that connects you and your cards, your phones, your watches, whatever you want to pay with. We connect that with the merchants you want to do commerce we connect that with the merchants you want to do commerce with and with the banks to ensure commerce that's what we do so we're not really a financial services company we're a technology company the data and technology company and when we think about picking a partner, picking a technology partner, the first thing that we think about is that they must scale. Because everything we do at MasterCard involves, well, a lot of scale. 220 countries, 3.5 billion cards and credentials, 150 million merchants, 200 billion card transactions, I think this year, and those are big numbers. By the way, 200 billion transactions is a little over 6,000 per second. So still hard to figure out, what does that mean. So if we think that we're 200 people in this room and we clap at three claps per second, then you guys would have to clap for 10 seconds to reach 6,000. So maybe we can do that at the end when we're done. You guys can clap for 10 seconds. And that's a long time to clap, by the way. So now you might be thinking, well, why is MasterCard so interested in innovating with data, in using AI? Like, why does it matter? Well, it matters because we use it to prevent fraud. And do you guys worry about fraud? We just went through Black Friday. I'm sure a lot of you have done some online shopping. Do you worry about money leaving your account without your permission? We don't. We don't worry about that. Why? Because, well, we're not liable for this yeah if we don't authorize the transaction then we'll get reimbursed that's called actually zero liability protection it's a benefit from using your MasterCard but the other reason is because we really work hard to prevent fraud and to stop it before it spreads. And if we didn't do that, then it would cost a lot of money to banks and merchants, and they would eventually pass it down to you guys. So obviously nobody wants that. So it's really important that we stay at the forefront of data and AI so we can prevent fraud and we do that by using a large diverse and complex data set so we can understand if the patterns that are happening on our network are normal or if they're out of whack and we need to stop something. Now when I mentioned scale for example our data warehouses 15 petabytes of data so it's quite significant and we have been at this for decades. I recently found an article in the Wall Street Journal and I think the headline was MasterCard stops three different ATM attacks using machine learning. This article was from 2016. So we've been at it for a while and we need to keep working at it and stay ahead of fraud story, you could say. So when I think about data governance and when I think about the impact our team can have, when I think about the impact our partnership with Databricks can have, that's what's at stake. We need to be able to continue to innovate rapidly, prevent fraud. But in the same breath, what we also need to be able to do is use data responsibly. And as a privacy-minded European, sorry, that was mean. I'm sure you guys are privacy-minded too in the US. I was really proud when in 2020 we announced our data and responsibility principles and at the heart of it it means that the data we have is not our data it's your data right as individuals you own your data and you control it, you should benefit from the use of it, for example with reduced fraud, and we have to protect it and we have to use it responsibly. So again, that's what we try to do, enable rapid cutting edge innovation with data and AI, but do it responsibly so how do we do that so again like I said at the beginning we try and think about helping people find data for use case helping people gaining access to it helping people get started on how to use it and how to use AI. Right? Sounds simple. So data discovery, think about what that means. And think about the scenario where you just started at your company. And you're a data analyst or a data scientist. And you have to find data for use case. Well, looking for data is not unlike looking for a hotel in Las Vegas when you want to attend the conference. At first, you think, how hard can this be? And then soon enough, you're browsing through hundreds of options. You give up, and you just ask your colleagues where they're staying. A lot of that is happening with data as well. You look for data and in the end, it's just way easier to just ask somebody who knows. But that's not very efficient and also you don't really reuse the knowledge across the company. So at MasterCard, when I think about helping people find the right data, I think about what information I should give them. I should give them descriptions, definitions, okay, that's a start, but also quality metrics. Because you want to know, is this a good hotel that I'm staying at, or is this a one-star review or kind of a hotel? You want to show them lineage. Where is the data coming from? You want to get them to understand the data restrictions associated with data. So at MasterCard, a lot of this information we have it is just in different systems. And what's pretty exciting with Databricks is that it's all under one roof. And a lot of it is automated. So for example, lineage is calculated automatically. And if you've tried to do lineage in your firm on-prem that's very hard to do you have to talk to a lot of people and they don't want to talk to you about lineage so the fact that it happens automatically is is really cool so let's imagine you found data for use case now Now you just have to, you know, gain access to it. Again, how hard can this be? And here the analogy can be, well, it's trying to go past security at the airport. How hard can this be? Well, it kind of depends on what the rules are. The problem for us is that the rules around how to use data, how to use AI, are still being defined, right? You have all of these countries trying to think about what's right for them. They don't all think about these problems in the same way. And so you end up with a patchwork. And for us at MasterCloud, because we're global, well, it's hard to navigate. So we spend a lot of time thinking about it and designing for it. And because we don't want to get you jammed up with long security lines, where you have to get your data business case reviewed by privacy, but there's like 50 other business case right ahead of you. We have to think about creating TSA pre-check for your data, right? So you get automation and you don't have to wait. Now if we imagine, so sorry, the cool thing about Databricks is that we can automate these rules in the Unity catalog and the other cool thing is that it's a platform where the usage of data and the governance of data and the access control around data, everything happens in that same platform. Lineage happens in that same platform. Whereas for us, for the longest of time, our catalogs sat as a separate platform from the data warehouse, from the tools that people were using to use data. So with Databricks, that's really helping us also because we can see the transparency in how people are using data, and we can follow the thread. And I can tell you that our privacy attorneys are really excited about that. They're really excited about being able to ask, how is this person using data? I gave them access two months ago. What have they done since? We could pull that thread. It's easy to do. It's auditable. OK, last but not least. So now you have access to data. And all you have to do is solve problems with it. Well, good luck because it's not always easy to figure out what query am I supposed to write, how am I supposed to join these two data sets. Yeah, some of our tables have 500 fields. Okay, so again documentation, but it's hard to produce. People don't want to do it. I don't blame them. So what we try and think about when we think about data governance is we think about showing aggregating information in an automated way. For example, if you can see all of the queries that have been placed on a given data set you're interested in, that helps you. That gets you started. So you can see that in Databricks. You can see the queries that have been placed. And that gives you a head start. The demo that Zeeshan showed, that was pretty cool. I really like this. The genie, the AIBI where you just talk in English to the data, gives you information back. That's huge. That's huge. We have over 1,000 data scientists at MasterCard that are essentially playing that role between the sales team that talk to customers and the data. But what we found too is when we tried using it, well, it doesn't really work right away. You ask questions about your data, and you don't really get the right answers. I'm sure you guys have played with Judge GPT and other tools like that. So you can imagine what happens. But then what we also learn is that if you catalog your data, if you provide the queries that have been placed against this data set, then it figures it out a lot better. The last thing I'll talk to you about is that's very exciting too, is the fact that you don't have a separate catalog for machine learning. It's the same catalog. So you have your data catalog, your machine learning catalog. It's one catalog. I was listening to Ali, the CEO of Databricks, explaining that they had started building two different catalogs. And then they realized, well, it's actually the same functionality. So they merged both. And I think it's brilliant. Because when you have to comply with AI regulations, you have to explain how AI arrived at a decision. Much easier if you can see the thread from the AI model to the data, that lineage. So those are some of the things we're pretty excited about with Databricks. And although we're at the beginning of our journey, already I can offer some lessons that we've learned. OK, prioritize, it sounds like, of course, why would you not prioritize? So you have to prioritize based on business value, based on executive support. And the unintended consequence is that when you do that, well, it becomes a lot easier to ask people to do data management and data cataloging, these tasks that they really don't like doing. It's kind of like cleaning your garage. Nobody wants to do data management and data cataloging, these tasks that they really don't like doing. It's kind of like cleaning your garage. Nobody wants to do it. Everybody loves it when it's done, right? Same thing with data cataloging. So prioritize because it'll help you. Make friends because it takes a village. We work really closely with the teams that are moving data to cloud, with the teams that the privacy team, with the data users, with security. All of these teams must work together if we want to enable the kind of user experience that I talked about, where it's easier to find data, it's easier to gain access to data, it's easier to use data. And's easier to gain access to data, it's easier to use data. And last but not least, learn by doing. And what's really cool about Databricks Unity is that I saw the demo that Zeeshan gave. It wasn't you giving the demo, but I thought, well, this is really cool. We should try it. I talked to somebody on my team who had access. Within 30 minutes, she was trying it on a dataset. So it's actually much better to learn by doing and to see what doesn't work, to see what works. And yeah, that's where we are. So thank you very much. And I look forward to hearing about your journeys..
AWS re:Invent 2024 - Unified and open governance for data and AI (AIM256)
Fragmented governance of data and AI can stifle innovation. This session highlights how Databricks Unity Catalog offers unified discovery and governance for tables, files, ML models, and AI tools while supporting any open table format like Apache Iceberg and Delta. It simplifies access management, facilitates data and AI quality monitoring, and provides column-level lineage tracking. Learn how Unity Catalog empowers you to own your data, connect to any data source, and access it from any tool or compute engine through open APIs—eliminating vendor lock-in. Discover how Mastercard effectively uses Unity Catalog to govern their data and AI assets securely. This presentation is brought to you by Databricks, an AWS Partner.

Learn more:
AWS re:Invent: https://go.aws/reinvent.
More AWS events: https://go.aws/3kss9CP 

Subscribe:
More AWS videos: http://bit.ly/2O3zS75
More AWS events videos: http://bit.ly/316g9t4

About AWS:
Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

#AWSreInvent  #AWSreInvent2024