Consequences of an Insightful Algorithm
Zona, Carina C.
We have ethical responsibilities when coding. We’re able to extract remarkably precise intuitions about an individual. But do we have a right to know what they didn’t consent to share, even when they willingly shared the data that leads us there? A major retailer’s data-driven marketing accidentially revealed to a teen’s family that she was pregnant. Eek. What are our obligations to people who did not expect themselves to be so intimately known without sharing directly? How do we mitigate against unintended outcomes? For instance, an activity tracker carelessly revealed users’ sexual activity data to search engines. A social network’s algorithm accidentally triggered painful memories for grieving families who’d recently experienced death of their child and other loved ones. We design software for humans. Balancing human needs and business specs can be tough. It’s crucial that we learn how to build in systematic empathy. In this talk, we’ll delve into specific examples of uncritical programming, and painful results from using insightful data in ways that were benignly intended. You’ll learn ways we can integrate practices for examining how our code might harm individuals. We’ll look at how to flip the paradigm, netting consequences that can be better for everyone.
 We didn't do a mic level check. It does work. Yay. Okay. All right. So, yeah, this talk is meant to be a toolkit for empathetic coding. And we're going to be delving into some pretty specific examples of uncritical programming. So the first thing I want to do, because we're going to be talking about painful results of doing things in ways that are perfectly benignly intended and then sides-white people is I'm not going to do that to you. So start with a content warning here. We're going to be delving into examples that include a bunch of things, including grief, PTSD, depression, miscarriage, infertility, racial profiling, concentration camp surveillance, sexual history consent, and assault. And while we won't be dwelling on those, that is topics that are going to come up. All right. So algorithms impose consequences on people all the time. We're able to extract such remarkably precise insights about any individual. But we haven't really asked ourselves enough, do we have a right to know what they didn't consent to share with us, even when they willingly share other data that ends up leading us there? And we have to be asking ourselves more about how do we mitigate against unintended consequences of that stuff. So it's helpful to step back a little and start with asking just a really basic question. What is an algorithm? And generically speaking, it's just a step by step set of operations for predictively arriving at a conclusion. And predictively is a really operative word in that. So obviously when we talk about algorithms, usually we meet in the computer science sense and the mathematics sense. Patterns of instructions articulated in code or in formulas. But you can also think of algorithms as being part of everyday life all the time. So just patterns of instructions articulated in other ways, such as for instance, a recipe or a lovely shawl. You know, if your worst code nightmare looks like this, you're doing well. Okay, so deep learning is a particular new, actually new old style of algorithms. So the technology goes back to, in theory at least, the 1950s, but there's been some recent breakthroughs since 2012 and 2013 that have really reshaped the landscape of what's possible with machine learning. It's really just the new hotness because of this. Over simplified, you can think of deep learning as algorithms for fast, trainable, artificial neural networks. It's a technology that, you know, like I said, it's been around in academia, for instance, for a long time, but really mostly at theoretical scale. Breakthroughs in parallelization, in availability of processors at massive scale, these things are making it much more viable to actually take this now into production use. So because of that, deep learning has become realistically able to extract insights out of vast, big data in ways that we've never really even been able to imagine about before. Deep learning is a particular approach to building and training artificial neural networks. You could think of them as decision making black boxes. So what does that mean? Okay, so we have some inputs. You're just giving an array of numbers, representing words, concepts, objects in some sense, and then executing it by running a series of functions that you write and executing against the array. And the outputs you get are the machine learning predicting properties that it thinks will be useful in the future for drawing intuitions about similar data sets. So you give it this training data set, it says, hmm, I notice some patterns here. This is probably how you can figure out similar results next time, and then you can throw much larger data sets at it, and it's able to come up with similar predictions. So what would that look like, for instance? Okay, so there's a few problems here. Black box. We'll get back to that in a second. So because deep learning relies on an artificial neural networks automated discovery of patterns within that training data set, it gets to apply those discoveries to draw intuitions about the future inputs. However, that means that every flaw or assumption in the training data set or in those original functions that you've written are going to have unrecognized influence on the algorithms and on the outcomes that they generate. So that's something that we're going to revisit, but let's look at a really just basic practical example of deep learning. RIO, it's an artificial neural network that teaches itself how to play Super Mario World. It starts with absolutely no clue whatsoever. There's a great YouTube video of this, by the way, in action. It spends 24 hours simply manipulating numbers and seeing what happens when it changes, like, pixel locations, for instance, and discovers that after 24 hours by golly, it can play the whole game. Right. I told you it was amazing. So it's learning. It is not being given rules. It's not being given information. It's exploring its world. So can you imagine on the data sets that we have access to what we could do with that? Learning to identify patterns and then using patterns to draw insights. This is something we try to do all the time. So speaking of games. Oops. Ruin my punchline here. There we go. Would you like to play a nice game? Global thermo-nuclear war. I love that word. We're going to play a different game, and we'll call it bingo. It is the screwiest game of bingo that you will ever play. There are definitely odd angles, but we're going to be able to cover this board real well. So let's take a look at some of the ways that we have serious pitfalls to deal with. We need to appreciate how many of those exist, why they present new challenges, why we have to be able to consider and deal with them in the very near future. So let's go ahead and play some rounds using some case studies as examples. Target. Many people have heard of this one, right? They had figured out that, hey, you can get people to radically change their buying habits if you catch them in the second trimester of pregnancy, something that specific. But how do you find out that you have a customer who is in their second month of pregnancy? I'm not second month, second trimester. That's certainly not something they ask on the visitor cards. So they found that there were a few data points that, in fact, would reliably predict this. And so they started sending out ad circulars, and those ad circulars were very focused on things that you need in your second trimester. And as you are planning your little nest, one day a man comes in, he's actually furious. Why did my teenage daughter get this? This is outrageous. Are you trying to encourage her to have sex? And then supposedly he came back a day or two later and said, wow, I'm so sorry. I didn't know it was going out of my household. My daughter is, in fact, pregnant. So we have this really useful predictability. Target took a lesson from that conversation. The lesson they took was we really should disguise the intent of these ads better. So now what you get is you get diaper ad, but you also get lawn mower ad and auto oil ad. So you don't know that, in fact, only one of these ads is the one that's being actually targeted at you. So here's a lesson about mining data is let's be honest about what we're doing. This was not actually the way to solve this problem that we knew too much. There are lots of different ways we could deal with it. Shutterfly. This one is somewhat less known, but in somewhat similar territory. They sent out an ad or an email saying congratulations, new parent. Trying to pick out cards to send us thank you notes. Unfortunately, they did not really predict well. Thanks, Shutterfly, for the congratulations on my new bundle of joy. I'm horribly infertile, but hey, I'm adopting a cat. I lost a baby in November who would have been due this week. It was like hitting a wall all over again. Shutterfly responded, the intent of the email was to target customers who've recently had a baby. Yeah, I get your intent. False positives are a problem. We can't treat them as edge cases. What is the impact of a false positive? What is the impact of a false negative? They didn't really apologize for having targeted wrong, just for you having received something. Mark Zuckerberg of all people has a little something to say about this. A couple of months ago, he announced that he's going to get father soon. What he also wrote about are the three miscarriages that they experienced before that. He said, you feel so hopeful when you learn you're going to have a child. You start imagining who they'll become and dreaming of hopes for their future. You start making plans and then they're gone. It's a lonely experience. I can't imagine that the Mark Zuckerberg of 10 years ago would have understood this. I think of all the VCs who focus on hiring the Mark Zuckerbergs of 10 years ago who believe that people who have had not nearly enough experiences of hard things in life are the best people to develop apps for people who have. This is the Mark Zuckerberg who's far more interesting to me. He's got a much better insight on all the possible ways that we can screw up or at least one of them. Facebook year interview. For years they have been doing this, but each year they've kind of tweaked how just servicing some posts from the past year that they feel were kind of the highlights reel of your year. Problem is that not everything that had a lot of likes in the past year is something that necessarily was positive for you. Not everything that people commented on was something you want to remember. Sometimes the things that were back then exactly that exciting and memorable and wonderful have changed over the course of a year. We have changes in our relationships, in our jobs, in our location, in our feelings. Having the belief that we can predict how people are feeling now about something that happened in the past is so absurdly arrogant. Eric Raymond coined the term inadvertent algorithmic cruelty. He defines it as the result of code that works in the overwhelming majority of cases, but doesn't take into account other use cases. The reason that he gets to coin this is because he's one of the people that happened to. His daughter died and year in review brought that back and kept bringing it back over and over. There's no obvious way to stop it. This is an algorithm too. This was a predictable result, right? Someone didn't predict that not everyone wants to have this come forth unwillingly. His recommendation is that we need to increase awareness and consideration of failure modes, the edge cases, worst case scenarios. I'm trying to do that today. What we can do is I hope accomplish a little something, a little bit moving forward that conversation. With that in mind, my first recommendation here for avoiding this particular specific pitfall along with a few others is be humble. We cannot intuit the inner state emotions and private subjectivity of our users. Not yet anyway. Which brings us to issues of consent, especially when de-anonymizing private data. Fitbit. You think of it as the tracker of all sorts of activities, right? One of the things that it used to track is your sex life. It doesn't anymore. There's a reason for that. It was in your profile. The profiles were public. The profiles were pretty interesting too. I can't decide whether to be really happy or really sad for this person. When you think about how people are feeling about the Ashley Madison exposure, okay. Some people would want to brag and be all competitive. For other people, this is a horrible violation of their privacy and their expectation of privacy. This was something crucially what's matter here is default. You need to be thinking through what is default private, what is default public, and what are the consequences of making a poor default decision. Hilariously, how they dealt with this deep technical problem is a change to robots.txt. I'm sure you all use your Fitbits. Yes. This is why they no longer do sex tracking because they really weren't willing to put in the effort to do it well. You know what? I actually think that that is a fair decision to make. If we're not prepared to handle data really thoughtfully and well, especially if it's something that can harm them, then yeah, opt out of doing it all together. That's a reasonable product decision. Uber had GodView. If you hadn't heard of this, in many ways it's similar to anyone else's admin tools. We always need stuff for monitoring, analytics, blah, blah, blah, making sure that the service is doing what you expect. Uber just happened to be called GodView. Well, it didn't happen to me. It was called GodView. The problem here was not that they had some sort of monitoring system, but the way they used it, and the way they used it was for funsies. They used it to show potential investors, for instance, at dinner parties, hey, look, let's track all of the guests coming to the party. They don't need to know. Some of the people were not okay with that. They used it for things like an executive was impatient that a reporter was on the way and taking a little too long, so he tracked her in order to meet her at the lobby and let her know and share her data with her. These are not necessary uses of data. We need to be thinking about how much access is actually made to private data and does it really need to be used in the way that we're using it. We make tools and then we expand too much on how they're used. Uber was certainly a case of that. I think it's a little bit telling that they do really evil things like this in their code. This is actual GodView code. Look at that, autoplay true. If you remember OKCupid's blog, they don't really do it anymore, but the research group at the dating site, OKCupid, they used to blog about things that they were learning from the aggregate data trends and they would just focus on sharing some sort of little insight into simple ways that an OKCupid user could use their dating site better, right? User used to blog about their data too, crucially different a few things. There's was not about improving customers' experience of the service. There's was about invading people's privacy in order to pass judgment on their sex lives. This is not a predictable consequence of signing up for a transport service. You don't expect that you're going to have someone being drawing conclusions about your private life. Again, it's so easy to look at something like Ashley Madison say, OK, well those people have some sort of plausible deniability of I didn't do what you said I did. Uber is saying we know for sure. We think our data definitely predicts that you were a slut last night. This is again a lot of opportunity for consequences for the user. Did they expect it? No. What do consent like Uber? Architect for real consent. Architect for informed consent. So what does that mean? Informed consent is permission freely granted where no is a totally consequence free alternative and it's the default value. And it's giving informed appreciation and understanding ahead of time of the facts, implications, and consequences involved in giving a yes. And if that sounds overly burdensome, come on, look at the TOS. We can set aside a little bit of information to genuinely say like, yo, when you sign up, we also do this really fun stuff where we look at your data and we say interesting things about it, you OK with that? This doesn't have to be tremendously hard, but we do have to make some sort of effort to give people better disclosure of what they're getting into. Google AdWords is another really interesting example. There was Harvard Research into basically it took two sets of names, one that's highly correlated with people who are black, one that is highly correlated with people who are white, and then found real names of working academics who had those first names and some sort of last name. So if we say that LaToya is a name highly correlated with black women, then we say, OK, so LaToya Adams is a real academic and we do some search center. And what they found were that when you search for names of white people versus black people, you have ads far more likely to show up that imply that the black person has an arrest record. Now think about what that means because AdWords algorithm is focused on predicting behavior and that's it. It doesn't care about content. It just has a template to fill in and select which one. Its job is just to figure out what is going to make us click, what based on what it knows about us and what it knows about others' behavior with those same ads. So what is this reflecting back on us? Deep learning is also really useful for doing photo analysis. We're already seeing some uses. Facial recognition, facial recognition, this actually was not using deep learning. But we're familiar with some of the interesting trends and possibilities that you can see, right? It's getting a lot better. iPhoto is more helpful now. But we're also seeing that even with far more sophisticated implementations that are using deep learning, this is from just two months ago. This is not, in fact, a jungle gym. This is Auschwitz. This is not an animal. This is Flickr doing auto tagging as well. This is Google doing auto tagging. To their credit, they acted on it really quickly once told, not to their credit. This was a month after Flickr had had that happen with black people being apes. So learn, pay attention to the news. They should have been acting on that a month sooner. And they did get it taken care of within 24 hours. But that says that 29 days earlier, this could have never happened. So why does something like this happen at all? And the answer goes back to the 1950s when Kodak was first developing ways to have really calibrated photos. So in order to make sure that photos were consistently rendered with nice, accurate colors, they created what are called Shirley cards. And these were cards used to color balance the printer. And you might notice something about them. Shirley cards were of white women. Film stock was optimized to accurately reproduce details in white-skinned people. The digital sensors of today are still trying to mimic the results of film processing. If we saw something radically different, we'd be complaining that the sensors suck. The problem is that that means that we have decade after decade of legacy data and current data that is all really poor data if it's of dark-skinned people. We think we have objective, vast data sets, but the data set is polluted with noise. I'm going to have to skip a few because we're running low on time, so I'm going to jump ahead a bit. So rationales for an algorithm can only be seen from inside of that black box that I talked about. And so that's great. Let's look at the rationale. That's a picture of the inside of the black box. It's in there, I swear. The thing about making decisions inside that black box is there's no accountability. There's no way to check our work, and there's no recourse for people when they tell us we've gotten it wrong. So there are some things that we can do to take some lessons from that. And there actually are professional ethicists, including ones in our own profession. I sure hadn't heard about them, and probably you haven't either. I adapted a few rules from various sources, including the Association for Computing Machinery. So let's take a look at a few of those. One, avoid harm to others. This might sound a lot like, say, medical ethics rules. How do we do that? Well, consider decisions' potential impact on others while we're doing development, while we're making decisions. Project the likelihood of consequences. Take to human well-being, minimize negative consequences to others. It's not enough to just say, ah, this is a problem. Minimize, minimize. What is the least invasive solution? We need to be really honest and trustworthy, even more so, because we are going to screw up at some point. We're going to need to be able to sincerely apologize and be believed. We need to have engendered enough trust that we can keep moving forward after we've made a fail. We need to provide others with full disclosure of the limitations of what we do and call attention to signs of risk. We also need to actively counter bias and inequality, because code is made by people. It's not objective. It reflects our tunnel vision. It replicates our flaws. Algorithms always have underlying assumptions about meaning, about accuracy, about the world in which they're generated, about how code should assign meaning to data. Indicating assumptions influence outcomes and consequences being generated always. So we need to do things like audit outcomes. You remember that adage, trust, but verify? No. I tell it to apply here. That's insufficient, because photo of black box. Instead, what we have to do is not trust, but audit constantly. This has to be part of our practices of checking what's coming out. Is it what it should be? Is it what's expected? Is it unbiased? So we're in this arms race right now, because Google, Microsoft, Apple, Facebook, they're all deeply investing in these technologies, and they're doubling down, and they're already rolling out implementations that are fascinating, but they're moving forward very quickly. And the problem with that is that they are both getting more precise in their correctness and more damaging in their wrongness. So we need to be really thoughtful about how we move forward with this to make sure that we are focused on being more precise in our correctness and using that correct information very carefully. We have to insist on getting this right because we want to be empathetic. So this is what we need to do. We have to have decision making authority in the hands of highly diverse teams who can anticipate diverse ways to screw up. And finally, we have to commit to transparency. We have to be able to apply our own expertise to say no to things that are going to harm users at all. We have to be able to say no over and over again to pitfalls like these as much as is necessary because we understand the consequences. Refuse to play along with it. Thank you. Thank you.
 We didn't do a mic level check.  It does work.  Yay.  Okay.  All right.  So, yeah, this talk is meant to be a toolkit for empathetic coding.  And we're going to be delving into some pretty specific examples of uncritical programming.  So the first thing I want to do, because we're going to be talking about painful results  of doing things in ways that are perfectly benignly intended and then sides-white people  is I'm not going to do that to you.  So start with a content warning here.  We're going to be delving into examples that include a bunch of things, including grief,  PTSD, depression, miscarriage, infertility, racial profiling, concentration camp surveillance,  sexual history consent, and assault.  And while we won't be dwelling on those, that is topics that are going to come up.  All right.  So algorithms impose consequences on people all the time.  We're able to extract such remarkably precise insights about any individual.  But we haven't really asked ourselves enough, do we have a right to know what they didn't  consent to share with us, even when they willingly share other data that ends up leading us there?  And we have to be asking ourselves more about how do we mitigate against unintended consequences  of that stuff.  So it's helpful to step back a little and start with asking just a really basic question.  What is an algorithm?  And generically speaking, it's just a step by step set of operations for predictively  arriving at a conclusion.  And predictively is a really operative word in that.  So obviously when we talk about algorithms, usually we meet in the computer science sense  and the mathematics sense.  Patterns of instructions articulated in code or in formulas.  But you can also think of algorithms as being part of everyday life all the time.  So just patterns of instructions articulated in other ways, such as for instance, a recipe  or a lovely shawl.  You know, if your worst code nightmare looks like this, you're doing well.  Okay, so deep learning is a particular new, actually new old style of algorithms.  So the technology goes back to, in theory at least, the 1950s, but there's been some  recent breakthroughs since 2012 and 2013 that have really reshaped the landscape of what's  possible with machine learning.  It's really just the new hotness because of this.  Over simplified, you can think of deep learning as algorithms for fast, trainable, artificial  neural networks.  It's a technology that, you know, like I said, it's been around in academia, for instance,  for a long time, but really mostly at theoretical scale.  Breakthroughs in parallelization, in availability of processors at massive scale, these things  are making it much more viable to actually take this now into production use.  So because of that, deep learning has become realistically able to extract insights out  of vast, big data in ways that we've never really even been able to imagine about before.  Deep learning is a particular approach to building and training artificial neural networks.  You could think of them as decision making black boxes.  So what does that mean?  Okay, so we have some inputs.  You're just giving an array of numbers, representing words, concepts, objects in some sense, and  then executing it by running a series of functions that you write and executing against the array.  And the outputs you get are the machine learning predicting properties that it thinks will  be useful in the future for drawing intuitions about similar data sets.  So you give it this training data set, it says, hmm, I notice some patterns here.  This is probably how you can figure out similar results next time, and then you can throw  much larger data sets at it, and it's able to come up with similar predictions.  So what would that look like, for instance?  Okay, so there's a few problems here.  Black box.  We'll get back to that in a second.  So because deep learning relies on an artificial neural networks automated discovery of patterns  within that training data set, it gets to apply those discoveries to draw intuitions  about the future inputs.  However, that means that every flaw or assumption in the training data set or in those original  functions that you've written are going to have unrecognized influence on the algorithms  and on the outcomes that they generate.  So that's something that we're going to revisit, but let's look at a really just basic practical  example of deep learning.  RIO, it's an artificial neural network that teaches itself how to play Super Mario World.  It starts with absolutely no clue whatsoever.  There's a great YouTube video of this, by the way, in action.  It spends 24 hours simply manipulating numbers and seeing what happens when it changes, like,  pixel locations, for instance, and discovers that after 24 hours by golly, it can play  the whole game.  Right.  I told you it was amazing.  So it's learning.  It is not being given rules.  It's not being given information.  It's exploring its world.  So can you imagine on the data sets that we have access to what we could do with that?  Learning to identify patterns and then using patterns to draw insights.  This is something we try to do all the time.  So speaking of games.  Oops.  Ruin my punchline here.  There we go.  Would you like to play a nice game?  Global thermo-nuclear war.  I love that word.  We're going to play a different game, and we'll call it bingo.  It is the screwiest game of bingo that you will ever play.  There are definitely odd angles, but we're going to be able to cover this board real  well.  So let's take a look at some of the ways that we have serious pitfalls to deal with.  We need to appreciate how many of those exist, why they present new challenges, why we have  to be able to consider and deal with them in the very near future.  So let's go ahead and play some rounds using some case studies as examples.  Target.  Many people have heard of this one, right?  They had figured out that, hey, you can get people to radically change their buying habits  if you catch them in the second trimester of pregnancy, something that specific.  But how do you find out that you have a customer who is in their second month of pregnancy?  I'm not second month, second trimester.  That's certainly not something they ask on the visitor cards.  So they found that there were a few data points that, in fact, would reliably predict this.  And so they started sending out ad circulars, and those ad circulars were very focused on  things that you need in your second trimester.  And as you are planning your little nest, one day a man comes in, he's actually furious.  Why did my teenage daughter get this?  This is outrageous.  Are you trying to encourage her to have sex?  And then supposedly he came back a day or two later and said, wow, I'm so sorry.  I didn't know it was going out of my household.  My daughter is, in fact, pregnant.  So we have this really useful predictability.  Target took a lesson from that conversation.  The lesson they took was we really should disguise the intent of these ads better.  So now what you get is you get diaper ad, but you also get lawn mower ad and auto oil  ad.  So you don't know that, in fact, only one of these ads is the one that's being actually  targeted at you.  So here's a lesson about mining data is let's be honest about what we're doing.  This was not actually the way to solve this problem that we knew too much.  There are lots of different ways we could deal with it.  Shutterfly.  This one is somewhat less known, but in somewhat similar territory.  They sent out an ad or an email saying congratulations, new parent.  Trying to pick out cards to send us thank you notes.  Unfortunately, they did not really predict well.  Thanks, Shutterfly, for the congratulations on my new bundle of joy.  I'm horribly infertile, but hey, I'm adopting a cat.  I lost a baby in November who would have been due this week.  It was like hitting a wall all over again.  Shutterfly responded, the intent of the email was to target customers who've recently had  a baby.  Yeah, I get your intent.  False positives are a problem.  We can't treat them as edge cases.  What is the impact of a false positive?  What is the impact of a false negative?  They didn't really apologize for having targeted wrong, just for you having received something.  Mark Zuckerberg of all people has a little something to say about this.  A couple of months ago, he announced that he's going to get father soon.  What he also wrote about are the three miscarriages that they experienced before that.  He said, you feel so hopeful when you learn you're going to have a child.  You start imagining who they'll become and dreaming of hopes for their future.  You start making plans and then they're gone.  It's a lonely experience.  I can't imagine that the Mark Zuckerberg of 10 years ago would have understood this.  I think of all the VCs who focus on hiring the Mark Zuckerbergs of 10 years ago who believe  that people who have had not nearly enough experiences of hard things in life are the  best people to develop apps for people who have.  This is the Mark Zuckerberg who's far more interesting to me.  He's got a much better insight on all the possible ways that we can screw up or at least  one of them.  Facebook year interview.  For years they have been doing this, but each year they've kind of tweaked how just servicing  some posts from the past year that they feel were kind of the highlights reel of your year.  Problem is that not everything that had a lot of likes in the past year is something that  necessarily was positive for you.  Not everything that people commented on was something you want to remember.  Sometimes the things that were back then exactly that exciting and memorable and wonderful  have changed over the course of a year.  We have changes in our relationships, in our jobs, in our location, in our feelings.  Having the belief that we can predict how people are feeling now about something that  happened in the past is so absurdly arrogant.  Eric Raymond coined the term inadvertent algorithmic cruelty.  He defines it as the result of code that works in the overwhelming majority of cases,  but doesn't take into account other use cases.  The reason that he gets to coin this is because he's one of the people that happened to.  His daughter died and year in review brought that back and kept bringing it back over and  over.  There's no obvious way to stop it.  This is an algorithm too.  This was a predictable result, right?  Someone didn't predict that not everyone wants to have this come forth unwillingly.  His recommendation is that we need to increase awareness and consideration of failure modes,  the edge cases, worst case scenarios.  I'm trying to do that today.  What we can do is I hope accomplish a little something, a little bit moving forward that  conversation.  With that in mind, my first recommendation here for avoiding this particular specific  pitfall along with a few others is be humble.  We cannot intuit the inner state emotions and private subjectivity of our users.  Not yet anyway.  Which brings us to issues of consent, especially when de-anonymizing private data.  Fitbit.  You think of it as the tracker of all sorts of activities, right?  One of the things that it used to track is your sex life.  It doesn't anymore.  There's a reason for that.  It was in your profile.  The profiles were public.  The profiles were pretty interesting too.  I can't decide whether to be really happy or really sad for this person.  When you think about how people are feeling about the Ashley Madison exposure, okay.  Some people would want to brag and be all competitive.  For other people, this is a horrible violation of their privacy and their expectation of  privacy.  This was something crucially what's matter here is default.  You need to be thinking through what is default private, what is default public, and what  are the consequences of making a poor default decision.  Hilariously, how they dealt with this deep technical problem is a change to robots.txt.  I'm sure you all use your Fitbits.  Yes.  This is why they no longer do sex tracking because they really weren't willing to put  in the effort to do it well.  You know what?  I actually think that that is a fair decision to make.  If we're not prepared to handle data really thoughtfully and well, especially if it's  something that can harm them, then yeah, opt out of doing it all together.  That's a reasonable product decision.  Uber had GodView.  If you hadn't heard of this, in many ways it's similar to anyone else's admin tools.  We always need stuff for monitoring, analytics, blah, blah, blah, making sure that the service  is doing what you expect.  Uber just happened to be called GodView.  Well, it didn't happen to me.  It was called GodView.  The problem here was not that they had some sort of monitoring system, but the way they  used it, and the way they used it was for funsies.  They used it to show potential investors, for instance, at dinner parties, hey, look,  let's track all of the guests coming to the party.  They don't need to know.  Some of the people were not okay with that.  They used it for things like an executive was impatient that a reporter was on the way  and taking a little too long, so he tracked her in order to meet her at the lobby and  let her know and share her data with her.  These are not necessary uses of data.  We need to be thinking about how much access is actually made to private data and does  it really need to be used in the way that we're using it.  We make tools and then we expand too much on how they're used.  Uber was certainly a case of that.  I think it's a little bit telling that they do really evil things like this in their code.  This is actual GodView code.  Look at that, autoplay true.  If you remember OKCupid's blog, they don't really do it anymore, but the research group  at the dating site, OKCupid, they used to blog about things that they were learning  from the aggregate data trends and they would just focus on sharing some sort of little  insight into simple ways that an OKCupid user could use their dating site better, right?  User used to blog about their data too, crucially different a few things.  There's was not about improving customers' experience of the service.  There's was about invading people's privacy in order to pass judgment on their sex lives.  This is not a predictable consequence of signing up for a transport service.  You don't expect that you're going to have someone being drawing conclusions about your  private life.  Again, it's so easy to look at something like Ashley Madison say, OK, well those people  have some sort of plausible deniability of I didn't do what you said I did.  Uber is saying we know for sure.  We think our data definitely predicts that you were a slut last night.  This is again a lot of opportunity for consequences for the user.  Did they expect it?  No.  What do consent like Uber?  Architect for real consent.  Architect for informed consent.  So what does that mean?  Informed consent is permission freely granted where no is a totally consequence free alternative  and it's the default value.  And it's giving informed appreciation and understanding ahead of time of the facts, implications,  and consequences involved in giving a yes.  And if that sounds overly burdensome, come on, look at the TOS.  We can set aside a little bit of information to genuinely say like, yo, when you sign up,  we also do this really fun stuff where we look at your data and we say interesting things  about it, you OK with that?  This doesn't have to be tremendously hard, but we do have to make some sort of effort  to give people better disclosure of what they're getting into.  Google AdWords is another really interesting example.  There was Harvard Research into basically it took two sets of names, one that's highly  correlated with people who are black, one that is highly correlated with people who  are white, and then found real names of working academics who had those first names and some  sort of last name.  So if we say that LaToya is a name highly correlated with black women, then we say,  OK, so LaToya Adams is a real academic and we do some search center.  And what they found were that when you search for names of white people versus black people,  you have ads far more likely to show up that imply that the black person has an arrest record.  Now think about what that means because AdWords algorithm is focused on predicting behavior  and that's it.  It doesn't care about content.  It just has a template to fill in and select which one.  Its job is just to figure out what is going to make us click, what based on what it knows  about us and what it knows about others' behavior with those same ads.  So what is this reflecting back on us?  Deep learning is also really useful for doing photo analysis.  We're already seeing some uses.  Facial recognition, facial recognition, this actually was not using deep learning.  But we're familiar with some of the interesting trends and possibilities that you can see,  right?  It's getting a lot better.  iPhoto is more helpful now.  But we're also seeing that even with far more sophisticated implementations that are using  deep learning, this is from just two months ago.  This is not, in fact, a jungle gym.  This is Auschwitz.  This is not an animal.  This is Flickr doing auto tagging as well.  This is Google doing auto tagging.  To their credit, they acted on it really quickly once told, not to their credit.  This was a month after Flickr had had that happen with black people being apes.  So learn, pay attention to the news.  They should have been acting on that a month sooner.  And they did get it taken care of within 24 hours.  But that says that 29 days earlier, this could have never happened.  So why does something like this happen at all?  And the answer goes back to the 1950s when Kodak was first developing ways to have really  calibrated photos.  So in order to make sure that photos were consistently rendered with nice, accurate colors, they  created what are called Shirley cards.  And these were cards used to color balance the printer.  And you might notice something about them.  Shirley cards were of white women.  Film stock was optimized to accurately reproduce details in white-skinned people.  The digital sensors of today are still trying to mimic the results of film processing.  If we saw something radically different, we'd be complaining that the sensors suck.  The problem is that that means that we have decade after decade of legacy data and current  data that is all really poor data if it's of dark-skinned people.  We think we have objective, vast data sets, but the data set is polluted with noise.  I'm going to have to skip a few because we're running low on time, so I'm going to jump  ahead a bit.  So rationales for an algorithm can only be seen from inside of that black box that I  talked about.  And so that's great.  Let's look at the rationale.  That's a picture of the inside of the black box.  It's in there, I swear.  The thing about making decisions inside that black box is there's no accountability.  There's no way to check our work, and there's no recourse for people when they tell us  we've gotten it wrong.  So there are some things that we can do to take some lessons from that.  And there actually are professional ethicists, including ones in our own profession.  I sure hadn't heard about them, and probably you haven't either.  I adapted a few rules from various sources, including the Association for Computing Machinery.  So let's take a look at a few of those.  One, avoid harm to others.  This might sound a lot like, say, medical ethics rules.  How do we do that?  Well, consider decisions' potential impact on others while we're doing development,  while we're making decisions.  Project the likelihood of consequences.  Take to human well-being, minimize negative consequences to others.  It's not enough to just say, ah, this is a problem.  Minimize, minimize.  What is the least invasive solution?  We need to be really honest and trustworthy, even more so, because we are going to screw  up at some point.  We're going to need to be able to sincerely apologize and be believed.  We need to have engendered enough trust that we can keep moving forward after we've made  a fail.  We need to provide others with full disclosure of the limitations of what we do and call  attention to signs of risk.  We also need to actively counter bias and inequality, because code is made by people.  It's not objective.  It reflects our tunnel vision.  It replicates our flaws.  Algorithms always have underlying assumptions about meaning, about accuracy, about the world  in which they're generated, about how code should assign meaning to data.  Indicating assumptions influence outcomes and consequences being generated always.  So we need to do things like audit outcomes.  You remember that adage, trust, but verify?  No.  I tell it to apply here.  That's insufficient, because photo of black box.  Instead, what we have to do is not trust, but audit constantly.  This has to be part of our practices of checking what's coming out.  Is it what it should be?  Is it what's expected?  Is it unbiased?  So we're in this arms race right now, because Google, Microsoft, Apple, Facebook, they're  all deeply investing in these technologies, and they're doubling down, and they're already  rolling out implementations that are fascinating, but they're moving forward very quickly.  And the problem with that is that they are both getting more precise in their correctness  and more damaging in their wrongness.  So we need to be really thoughtful about how we move forward with this to make sure that  we are focused on being more precise in our correctness and using that correct information  very carefully.  We have to insist on getting this right because we want to be empathetic.  So this is what we need to do.  We have to have decision making authority in the hands of highly diverse teams who can  anticipate diverse ways to screw up.  And finally, we have to commit to transparency.  We have to be able to apply our own expertise to say no to things that are going to harm  users at all.  We have to be able to say no over and over again to pitfalls like these as much as is  necessary because we understand the consequences.  Refuse to play along with it.  Thank you.  Thank you.
DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_0_431.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_432_2963.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_2964_3023.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_3024_3323.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_3324_12371.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_12372_16859.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_16860_19607.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_19608_22631.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_22632_26279.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_26280_26531.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_26532_28631.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_28632_29423.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_29424_30503.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_30504_31931.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_31932_33311.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_33312_34067.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_34068_34679.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_34680_34775.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_34776_35039.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_35040_35807.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_35808_36083.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_36084_36467.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_36468_36743.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_36744_36887.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_36888_37235.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_37236_38003.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_38004_38327.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_38328_38915.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_38916_39095.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_39096_39371.png DjangoCon_US_2015_-_Consequences_of_an_Insightful_Algorithm_by_Carina_C__Zona_5e-mZnYAih8_39372_39372.png
We didn't do a mic level check. It does work. Yay. Okay. All right. So, yeah, this talk is meant to be a toolkit for empathetic coding. And we're going to be delving into some pretty specific examples of uncritical programming. So the first thing I want to do, because we're going to be talking about painful results of doing things in ways that are perfectly benignly intended and then sides-white people is I'm not going to do that to you. So start with a content warning here. We're going to be delving into examples that include a bunch of things, including grief, PTSD, depression, miscarriage, infertility, racial profiling, concentration camp surveillance, sexual history consent, and assault. And while we won't be dwelling on those, that is topics that are going to come up. All right. So algorithms impose consequences on people all the time. We're able to extract such remarkably precise insights about any individual. But we haven't really asked ourselves enough, do we have a right to know what they didn't consent to share with us, even when they willingly share other data that ends up leading us there? And we have to be asking ourselves more about how do we mitigate against unintended consequences of that stuff. So it's helpful to step back a little and start with asking just a really basic question. What is an algorithm? And generically speaking, it's just a step by step set of operations for predictively arriving at a conclusion. And predictively is a really operative word in that. So obviously when we talk about algorithms, usually we meet in the computer science sense and the mathematics sense. Patterns of instructions articulated in code or in formulas. But you can also think of algorithms as being part of everyday life all the time. So just patterns of instructions articulated in other ways, such as for instance, a recipe or a lovely shawl. You know, if your worst code nightmare looks like this, you're doing well. Okay, so deep learning is a particular new, actually new old style of algorithms. So the technology goes back to, in theory at least, the 1950s, but there's been some recent breakthroughs since 2012 and 2013 that have really reshaped the landscape of what's possible with machine learning. It's really just the new hotness because of this. Over simplified, you can think of deep learning as algorithms for fast, trainable, artificial neural networks. It's a technology that, you know, like I said, it's been around in academia, for instance, for a long time, but really mostly at theoretical scale. Breakthroughs in parallelization, in availability of processors at massive scale, these things are making it much more viable to actually take this now into production use. So because of that, deep learning has become realistically able to extract insights out of vast, big data in ways that we've never really even been able to imagine about before. Deep learning is a particular approach to building and training artificial neural networks. You could think of them as decision making black boxes. So what does that mean? Okay, so we have some inputs. You're just giving an array of numbers, representing words, concepts, objects in some sense, and then executing it by running a series of functions that you write and executing against the array. And the outputs you get are the machine learning predicting properties that it thinks will be useful in the future for drawing intuitions about similar data sets. So you give it this training data set, it says, hmm, I notice some patterns here. This is probably how you can figure out similar results next time, and then you can throw much larger data sets at it, and it's able to come up with similar predictions. So what would that look like, for instance? Okay, so there's a few problems here. Black box. We'll get back to that in a second. So because deep learning relies on an artificial neural networks automated discovery of patterns within that training data set, it gets to apply those discoveries to draw intuitions about the future inputs. However, that means that every flaw or assumption in the training data set or in those original functions that you've written are going to have unrecognized influence on the algorithms and on the outcomes that they generate. So that's something that we're going to revisit, but let's look at a really just basic practical example of deep learning. RIO, it's an artificial neural network that teaches itself how to play Super Mario World. It starts with absolutely no clue whatsoever. There's a great YouTube video of this, by the way, in action. It spends 24 hours simply manipulating numbers and seeing what happens when it changes, like, pixel locations, for instance, and discovers that after 24 hours by golly, it can play the whole game. Right. I told you it was amazing. So it's learning. It is not being given rules. It's not being given information. It's exploring its world. So can you imagine on the data sets that we have access to what we could do with that? Learning to identify patterns and then using patterns to draw insights. This is something we try to do all the time. So speaking of games. Oops. Ruin my punchline here. There we go. Would you like to play a nice game? Global thermo-nuclear war. I love that word. We're going to play a different game, and we'll call it bingo. It is the screwiest game of bingo that you will ever play. There are definitely odd angles, but we're going to be able to cover this board real well. So let's take a look at some of the ways that we have serious pitfalls to deal with. We need to appreciate how many of those exist, why they present new challenges, why we have to be able to consider and deal with them in the very near future. So let's go ahead and play some rounds using some case studies as examples. Target. Many people have heard of this one, right? They had figured out that, hey, you can get people to radically change their buying habits if you catch them in the second trimester of pregnancy, something that specific. But how do you find out that you have a customer who is in their second month of pregnancy? I'm not second month, second trimester. That's certainly not something they ask on the visitor cards. So they found that there were a few data points that, in fact, would reliably predict this. And so they started sending out ad circulars, and those ad circulars were very focused on things that you need in your second trimester. And as you are planning your little nest, one day a man comes in, he's actually furious. Why did my teenage daughter get this? This is outrageous. Are you trying to encourage her to have sex? And then supposedly he came back a day or two later and said, wow, I'm so sorry. I didn't know it was going out of my household. My daughter is, in fact, pregnant. So we have this really useful predictability. Target took a lesson from that conversation. The lesson they took was we really should disguise the intent of these ads better. So now what you get is you get diaper ad, but you also get lawn mower ad and auto oil ad. So you don't know that, in fact, only one of these ads is the one that's being actually targeted at you. So here's a lesson about mining data is let's be honest about what we're doing. This was not actually the way to solve this problem that we knew too much. There are lots of different ways we could deal with it. Shutterfly. This one is somewhat less known, but in somewhat similar territory. They sent out an ad or an email saying congratulations, new parent. Trying to pick out cards to send us thank you notes. Unfortunately, they did not really predict well. Thanks, Shutterfly, for the congratulations on my new bundle of joy. I'm horribly infertile, but hey, I'm adopting a cat. I lost a baby in November who would have been due this week. It was like hitting a wall all over again. Shutterfly responded, the intent of the email was to target customers who've recently had a baby. Yeah, I get your intent. False positives are a problem. We can't treat them as edge cases. What is the impact of a false positive? What is the impact of a false negative? They didn't really apologize for having targeted wrong, just for you having received something. Mark Zuckerberg of all people has a little something to say about this. A couple of months ago, he announced that he's going to get father soon. What he also wrote about are the three miscarriages that they experienced before that. He said, you feel so hopeful when you learn you're going to have a child. You start imagining who they'll become and dreaming of hopes for their future. You start making plans and then they're gone. It's a lonely experience. I can't imagine that the Mark Zuckerberg of 10 years ago would have understood this. I think of all the VCs who focus on hiring the Mark Zuckerbergs of 10 years ago who believe that people who have had not nearly enough experiences of hard things in life are the best people to develop apps for people who have. This is the Mark Zuckerberg who's far more interesting to me. He's got a much better insight on all the possible ways that we can screw up or at least one of them. Facebook year interview. For years they have been doing this, but each year they've kind of tweaked how just servicing some posts from the past year that they feel were kind of the highlights reel of your year. Problem is that not everything that had a lot of likes in the past year is something that necessarily was positive for you. Not everything that people commented on was something you want to remember. Sometimes the things that were back then exactly that exciting and memorable and wonderful have changed over the course of a year. We have changes in our relationships, in our jobs, in our location, in our feelings. Having the belief that we can predict how people are feeling now about something that happened in the past is so absurdly arrogant. Eric Raymond coined the term inadvertent algorithmic cruelty. He defines it as the result of code that works in the overwhelming majority of cases, but doesn't take into account other use cases. The reason that he gets to coin this is because he's one of the people that happened to. His daughter died and year in review brought that back and kept bringing it back over and over. There's no obvious way to stop it. This is an algorithm too. This was a predictable result, right? Someone didn't predict that not everyone wants to have this come forth unwillingly. His recommendation is that we need to increase awareness and consideration of failure modes, the edge cases, worst case scenarios. I'm trying to do that today. What we can do is I hope accomplish a little something, a little bit moving forward that conversation. With that in mind, my first recommendation here for avoiding this particular specific pitfall along with a few others is be humble. We cannot intuit the inner state emotions and private subjectivity of our users. Not yet anyway. Which brings us to issues of consent, especially when de-anonymizing private data. Fitbit. You think of it as the tracker of all sorts of activities, right? One of the things that it used to track is your sex life. It doesn't anymore. There's a reason for that. It was in your profile. The profiles were public. The profiles were pretty interesting too. I can't decide whether to be really happy or really sad for this person. When you think about how people are feeling about the Ashley Madison exposure, okay. Some people would want to brag and be all competitive. For other people, this is a horrible violation of their privacy and their expectation of privacy. This was something crucially what's matter here is default. You need to be thinking through what is default private, what is default public, and what are the consequences of making a poor default decision. Hilariously, how they dealt with this deep technical problem is a change to robots.txt. I'm sure you all use your Fitbits. Yes. This is why they no longer do sex tracking because they really weren't willing to put in the effort to do it well. You know what? I actually think that that is a fair decision to make. If we're not prepared to handle data really thoughtfully and well, especially if it's something that can harm them, then yeah, opt out of doing it all together. That's a reasonable product decision. Uber had GodView. If you hadn't heard of this, in many ways it's similar to anyone else's admin tools. We always need stuff for monitoring, analytics, blah, blah, blah, making sure that the service is doing what you expect. Uber just happened to be called GodView. Well, it didn't happen to me. It was called GodView. The problem here was not that they had some sort of monitoring system, but the way they used it, and the way they used it was for funsies. They used it to show potential investors, for instance, at dinner parties, hey, look, let's track all of the guests coming to the party. They don't need to know. Some of the people were not okay with that. They used it for things like an executive was impatient that a reporter was on the way and taking a little too long, so he tracked her in order to meet her at the lobby and let her know and share her data with her. These are not necessary uses of data. We need to be thinking about how much access is actually made to private data and does it really need to be used in the way that we're using it. We make tools and then we expand too much on how they're used. Uber was certainly a case of that. I think it's a little bit telling that they do really evil things like this in their code. This is actual GodView code. Look at that, autoplay true. If you remember OKCupid's blog, they don't really do it anymore, but the research group at the dating site, OKCupid, they used to blog about things that they were learning from the aggregate data trends and they would just focus on sharing some sort of little insight into simple ways that an OKCupid user could use their dating site better, right? User used to blog about their data too, crucially different a few things. There's was not about improving customers' experience of the service. There's was about invading people's privacy in order to pass judgment on their sex lives. This is not a predictable consequence of signing up for a transport service. You don't expect that you're going to have someone being drawing conclusions about your private life. Again, it's so easy to look at something like Ashley Madison say, OK, well those people have some sort of plausible deniability of I didn't do what you said I did. Uber is saying we know for sure. We think our data definitely predicts that you were a slut last night. This is again a lot of opportunity for consequences for the user. Did they expect it? No. What do consent like Uber? Architect for real consent. Architect for informed consent. So what does that mean? Informed consent is permission freely granted where no is a totally consequence free alternative and it's the default value. And it's giving informed appreciation and understanding ahead of time of the facts, implications, and consequences involved in giving a yes. And if that sounds overly burdensome, come on, look at the TOS. We can set aside a little bit of information to genuinely say like, yo, when you sign up, we also do this really fun stuff where we look at your data and we say interesting things about it, you OK with that? This doesn't have to be tremendously hard, but we do have to make some sort of effort to give people better disclosure of what they're getting into. Google AdWords is another really interesting example. There was Harvard Research into basically it took two sets of names, one that's highly correlated with people who are black, one that is highly correlated with people who are white, and then found real names of working academics who had those first names and some sort of last name. So if we say that LaToya is a name highly correlated with black women, then we say, OK, so LaToya Adams is a real academic and we do some search center. And what they found were that when you search for names of white people versus black people, you have ads far more likely to show up that imply that the black person has an arrest record. Now think about what that means because AdWords algorithm is focused on predicting behavior and that's it. It doesn't care about content. It just has a template to fill in and select which one. Its job is just to figure out what is going to make us click, what based on what it knows about us and what it knows about others' behavior with those same ads. So what is this reflecting back on us? Deep learning is also really useful for doing photo analysis. We're already seeing some uses. Facial recognition, facial recognition, this actually was not using deep learning. But we're familiar with some of the interesting trends and possibilities that you can see, right? It's getting a lot better. iPhoto is more helpful now. But we're also seeing that even with far more sophisticated implementations that are using deep learning, this is from just two months ago. This is not, in fact, a jungle gym. This is Auschwitz. This is not an animal. This is Flickr doing auto tagging as well. This is Google doing auto tagging. To their credit, they acted on it really quickly once told, not to their credit. This was a month after Flickr had had that happen with black people being apes. So learn, pay attention to the news. They should have been acting on that a month sooner. And they did get it taken care of within 24 hours. But that says that 29 days earlier, this could have never happened. So why does something like this happen at all? And the answer goes back to the 1950s when Kodak was first developing ways to have really calibrated photos. So in order to make sure that photos were consistently rendered with nice, accurate colors, they created what are called Shirley cards. And these were cards used to color balance the printer. And you might notice something about them. Shirley cards were of white women. Film stock was optimized to accurately reproduce details in white-skinned people. The digital sensors of today are still trying to mimic the results of film processing. If we saw something radically different, we'd be complaining that the sensors suck. The problem is that that means that we have decade after decade of legacy data and current data that is all really poor data if it's of dark-skinned people. We think we have objective, vast data sets, but the data set is polluted with noise. I'm going to have to skip a few because we're running low on time, so I'm going to jump ahead a bit. So rationales for an algorithm can only be seen from inside of that black box that I talked about. And so that's great. Let's look at the rationale. That's a picture of the inside of the black box. It's in there, I swear. The thing about making decisions inside that black box is there's no accountability. There's no way to check our work, and there's no recourse for people when they tell us we've gotten it wrong. So there are some things that we can do to take some lessons from that. And there actually are professional ethicists, including ones in our own profession. I sure hadn't heard about them, and probably you haven't either. I adapted a few rules from various sources, including the Association for Computing Machinery. So let's take a look at a few of those. One, avoid harm to others. This might sound a lot like, say, medical ethics rules. How do we do that? Well, consider decisions' potential impact on others while we're doing development, while we're making decisions. Project the likelihood of consequences. Take to human well-being, minimize negative consequences to others. It's not enough to just say, ah, this is a problem. Minimize, minimize. What is the least invasive solution? We need to be really honest and trustworthy, even more so, because we are going to screw up at some point. We're going to need to be able to sincerely apologize and be believed. We need to have engendered enough trust that we can keep moving forward after we've made a fail. We need to provide others with full disclosure of the limitations of what we do and call attention to signs of risk. We also need to actively counter bias and inequality, because code is made by people. It's not objective. It reflects our tunnel vision. It replicates our flaws. Algorithms always have underlying assumptions about meaning, about accuracy, about the world in which they're generated, about how code should assign meaning to data. Indicating assumptions influence outcomes and consequences being generated always. So we need to do things like audit outcomes. You remember that adage, trust, but verify? No. I tell it to apply here. That's insufficient, because photo of black box. Instead, what we have to do is not trust, but audit constantly. This has to be part of our practices of checking what's coming out. Is it what it should be? Is it what's expected? Is it unbiased? So we're in this arms race right now, because Google, Microsoft, Apple, Facebook, they're all deeply investing in these technologies, and they're doubling down, and they're already rolling out implementations that are fascinating, but they're moving forward very quickly. And the problem with that is that they are both getting more precise in their correctness and more damaging in their wrongness. So we need to be really thoughtful about how we move forward with this to make sure that we are focused on being more precise in our correctness and using that correct information very carefully. We have to insist on getting this right because we want to be empathetic. So this is what we need to do. We have to have decision making authority in the hands of highly diverse teams who can anticipate diverse ways to screw up. And finally, we have to commit to transparency. We have to be able to apply our own expertise to say no to things that are going to harm users at all. We have to be able to say no over and over again to pitfalls like these as much as is necessary because we understand the consequences. Refuse to play along with it. Thank you. Thank you.
specific examples of uncritical programming. So to be talking about painful results of doing things in ways that are perfectly benignly people all the time. We're able to extract such remarkably precise insights about any individual. we have a right to know what they didn't consent to share with us, even when they willingly share other data that ends up leading us there? And we mitigate against unintended consequences of that drawing intuitions about similar data sets. So generate. So that's something that we're going to of pregnancy, something that specific. But how do they did not really predict well. Thanks, that people who have had not nearly enough a horrible violation of their privacy and their facial recognition, this actually was not using rationales for an algorithm can only be seen from so that's great. Let's look at the rationale. rules. How do we do that? Well, consider harm users at all. We have to be able to say no