 You send a robot into a hostile environment where decisions have to be made quickly and the algorithm will have to take care of that. The question is will there be humans in the loop or will they be on the loop or will they be out of the loop? Killer robots are weapons that would make drones look primitive. At least with a drone there is a human being who looks at a computer screen, sees the target and pushes the buttons to fire the missiles and kill. As we begin to approach the possibility of having machines select and engage targets, we want to be very careful not to cross that line without high-level policy review. As technology races ahead, as we achieve these fantastical advances, what decisions are we going to be comfortable delegating to machines and what kinds of decisions are we going to insist on reserving for the exercise of human judgment? We're a long way off from the day when armies of robot soldiers will march in perfect formation shooting lasers across enemy lines. But the U.S. military has stated its intent to use robots on the battlefield where they can theoretically carry supplies, bring cameras into dangerous places and yes, even kill. Some experts are calling robotics the new arms race. For that reason, an international group of scientists, professors and activists, including the Human Rights Watch and the International Committee for Robot Arms Control, are calling for a debate on the questions that inevitably arise with robots that can be used in war. I'm Mark Guberude and I'm a member of the International Committee for Robot Arms Control. I'm a physicist by training and I proposed a ban on autonomous weapons as early as 1988. We go to war and when we see what's going on, we decide there's a point we don't want to go beyond that point. If you look at the history of the Cold War, there are many incidents where people interrupted the chain of events. If you look at the crisis decisions at the highest level that were made during the human missile crisis or other major international crises, there's always a point where somebody says no, don't take the next step. And if we automate everything, there's not going to be that human intervention. Nobody always has to do that, otherwise wars will never end. Either you won or you lost or it's just too much blood and it's not worth it anymore. At some point people say stop. But if we outsource war, if we outsource the process of conflict, if we make that all a matter of machine decision, then we're not going to have that intervention of the human heart. This is going to be just the program that's running. Good morning everybody. For all of us in the DARPA community, there's no place we'd rather be than right here today. And the reason for that is that at DARPA, our mission is about breakthrough technologies for the future to help make our nation and our world a safer, a more secure place. DARPA is a 55-year-old agency in the Defense Department. We were started in the wake of Sputnik. It was a real wake-up call for the United States. It was a huge surprise. And then as now we understood that technology is the cornerstone of our national security. People were pretty clear that we did not want to go through that kind of surprise again. So DARPA was created specifically to live outside of the rest of the way that we do our science and technology investments, to be a projects agency with a specific mission on breakthrough technologies for national security. The general desire is to build machines that amplify the effectiveness of people. During the Iraq War, there was a very difficult problem that the Defense Department faced with improvised explosive devices. And so we had funded a number of robots before, and DARPA helped to see that those machines could be adapted for getting rid of IEDs. Really, when you think about what robots are capable of today, it's very, very, very early. And here at the DARPA Robotics Challenge, I think that really comes home to you when you see robots taking 30 minutes and not being able, even in that much time, being able to do the things that we as humans would find very, very simple to do. But focusing on disaster relief in the context of this challenge, I think, really allows us to push the technology, first to see what we're able to do today, and then start pushing it to the next level so that eventually we will have robots that have far more capable features for disaster relief, and I think for many other applications as well. We're not building weapons systems here. We're building the underlying technologies that can be used in many, many applications. Our main focus here is recovery and disaster relief. But just like DARPA worked on the internet, GPS, fiber optics, all of which we use every day, and so does the military, robotics will be the same. We will use robotics in our homes, in our work, and some of that will be military, but not all of it, to be sure. If somebody came up to me and asked me to build a robot that could fire a gun, I'd probably say no. I like what I do now. I like the research, because the research is fun. The technology itself doesn't say what it's going to be used for. In certain cases, of course, like a particular weapon and a bullet and things like that, those have primary uses for military systems and are not dual use. But these robotic systems are very general purpose, and you really need to understand that they are neutral with regard to the concern of are they military or non-military. So whether the Defense Department funds them or some other company funds them for whatever use, that development of technology is in fact possible to end up in any kind of system. Given the outrage over the military using unmanned drones and lethal strikes, many people may not be comfortable with the idea of robots as weapons. And most people in the robotics community seem to agree that now is the time to have that conversation before we have killer robots, not after. Really it comes down to how people use technology. It's a matter of human wisdom and being thoughtful about how technology gets applied. I'm very confident that we will have the wisdom to use robotics for mankind's benefit. But we can't assume that'll happen. We have to make that happen. This is an issue that, well, when I first started talking about it 25 years ago, people would just kind of stare at me. And then 10 years ago they would just say, oh yeah, Terminator, yeah, I saw that, you know, and it was a big joke. Just in the last five years, you know, the giggles have stopped. People are realizing this is serious. Our job at DARPA is to invest in advanced technologies and we pursue them because of their promise. But robotics is a great example of an area where we also recognize that in driving those technologies forward, we're also raising a whole host of very important broader societal questions. I don't have a position in terms of whether the U.S. should or should not sign any kind of ban. I think that the directive that the DOD has itself signed with regard to lethal autonomy has been, is really good. It's been very carefully thought through. It says that the primary concern is one of reliability. You want to make sure that if these systems decide within a particular set of instructions from a human operator whether to go to one place or the other, that that choice is always in line with the intent of the operator. So it really reflects back to a human being's choice. If someone like me comes along and says, we should have a hard red line that we should not cross, that's what they don't want to hear. Instead what they want is, well, we should think about what the ethical rules are. Which is just a way of deflecting the concern and saying, we're going to do it. We have, there are rules about how we're going to do it, but we're going to do it. Some people will say, oh, well, you know, where do you draw the line? And I always say, well, you draw it somewhere. That's the important point.
 You send a robot into a hostile environment where decisions have to be made quickly and
 the algorithm will have to take care of that.
 The question is will there be humans in the loop or will they be on the loop or will they
 be out of the loop?
 Killer robots are weapons that would make drones look primitive.
 At least with a drone there is a human being who looks at a computer screen, sees the target
 and pushes the buttons to fire the missiles and kill.
 As we begin to approach the possibility of having machines select and engage targets,
 we want to be very careful not to cross that line without high-level policy review.
 As technology races ahead, as we achieve these fantastical advances, what decisions are we
 going to be comfortable delegating to machines and what kinds of decisions are we going to
 insist on reserving for the exercise of human judgment?
 We're a long way off from the day when armies of robot soldiers will march in perfect formation
 shooting lasers across enemy lines.
 But the U.S. military has stated its intent to use robots on the battlefield where they
 can theoretically carry supplies, bring cameras into dangerous places and yes, even kill.
 Some experts are calling robotics the new arms race.
 For that reason, an international group of scientists, professors and activists, including
 the Human Rights Watch and the International Committee for Robot Arms Control, are calling
 for a debate on the questions that inevitably arise with robots that can be used in war.
 I'm Mark Guberude and I'm a member of the International Committee for Robot Arms Control.
 I'm a physicist by training and I proposed a ban on autonomous weapons as early as 1988.
 We go to war and when we see what's going on, we decide there's a point we don't want
 to go beyond that point.
 If you look at the history of the Cold War, there are many incidents where people interrupted
 the chain of events.
 If you look at the crisis decisions at the highest level that were made during the human
 missile crisis or other major international crises, there's always a point where somebody
 says no, don't take the next step.
 And if we automate everything, there's not going to be that human intervention.
 Nobody always has to do that, otherwise wars will never end.
 Either you won or you lost or it's just too much blood and it's not worth it anymore.
 At some point people say stop.
 But if we outsource war, if we outsource the process of conflict, if we make that all a
 matter of machine decision, then we're not going to have that intervention of the human
 heart.
 This is going to be just the program that's running.
 Good morning everybody.
 For all of us in the DARPA community, there's no place we'd rather be than right here today.
 And the reason for that is that at DARPA, our mission is about breakthrough technologies
 for the future to help make our nation and our world a safer, a more secure place.
 DARPA is a 55-year-old agency in the Defense Department.
 We were started in the wake of Sputnik.
 It was a real wake-up call for the United States.
 It was a huge surprise.
 And then as now we understood that technology is the cornerstone of our national security.
 People were pretty clear that we did not want to go through that kind of surprise again.
 So DARPA was created specifically to live outside of the rest of the way that we do
 our science and technology investments, to be a projects agency with a specific mission
 on breakthrough technologies for national security.
 The general desire is to build machines that amplify the effectiveness of people.
 During the Iraq War, there was a very difficult problem that the Defense Department faced
 with improvised explosive devices.
 And so we had funded a number of robots before, and DARPA helped to see that those machines
 could be adapted for getting rid of IEDs.
 Really, when you think about what robots are capable of today, it's very, very, very early.
 And here at the DARPA Robotics Challenge, I think that really comes home to you when
 you see robots taking 30 minutes and not being able, even in that much time, being able to
 do the things that we as humans would find very, very simple to do.
 But focusing on disaster relief in the context of this challenge, I think, really allows
 us to push the technology, first to see what we're able to do today, and then start pushing
 it to the next level so that eventually we will have robots that have far more capable
 features for disaster relief, and I think for many other applications as well.
 We're not building weapons systems here.
 We're building the underlying technologies that can be used in many, many applications.
 Our main focus here is recovery and disaster relief.
 But just like DARPA worked on the internet, GPS, fiber optics, all of which we use every
 day, and so does the military, robotics will be the same.
 We will use robotics in our homes, in our work, and some of that will be military, but
 not all of it, to be sure.
 If somebody came up to me and asked me to build a robot that could fire a gun, I'd probably
 say no.
 I like what I do now.
 I like the research, because the research is fun.
 The technology itself doesn't say what it's going to be used for.
 In certain cases, of course, like a particular weapon and a bullet and things like that,
 those have primary uses for military systems and are not dual use.
 But these robotic systems are very general purpose, and you really need to understand
 that they are neutral with regard to the concern of are they military or non-military.
 So whether the Defense Department funds them or some other company funds them for whatever
 use, that development of technology is in fact possible to end up in any kind of system.
 Given the outrage over the military using unmanned drones and lethal strikes, many
 people may not be comfortable with the idea of robots as weapons.
 And most people in the robotics community seem to agree that now is the time to have
 that conversation before we have killer robots, not after.
 Really it comes down to how people use technology.
 It's a matter of human wisdom and being thoughtful about how technology gets applied.
 I'm very confident that we will have the wisdom to use robotics for mankind's benefit.
 But we can't assume that'll happen.
 We have to make that happen.
 This is an issue that, well, when I first started talking about it 25 years ago, people
 would just kind of stare at me.
 And then 10 years ago they would just say, oh yeah, Terminator, yeah, I saw that, you
 know, and it was a big joke.
 Just in the last five years, you know, the giggles have stopped.
 People are realizing this is serious.
 Our job at DARPA is to invest in advanced technologies and we pursue them because of
 their promise.
 But robotics is a great example of an area where we also recognize that in driving those
 technologies forward, we're also raising a whole host of very important broader societal
 questions.
 I don't have a position in terms of whether the U.S. should or should not sign any kind
 of ban.
 I think that the directive that the DOD has itself signed with regard to lethal autonomy
 has been, is really good.
 It's been very carefully thought through.
 It says that the primary concern is one of reliability.
 You want to make sure that if these systems decide within a particular set of instructions
 from a human operator whether to go to one place or the other, that that choice is always
 in line with the intent of the operator.
 So it really reflects back to a human being's choice.
 If someone like me comes along and says, we should have a hard red line that we should
 not cross, that's what they don't want to hear.
 Instead what they want is, well, we should think about what the ethical rules are.
 Which is just a way of deflecting the concern and saying, we're going to do it.
 We have, there are rules about how we're going to do it, but we're going to do it.
 Some people will say, oh, well, you know, where do you draw the line?
 And I always say, well, you draw it somewhere.
 That's the important point.