Towards Universal Access to All Knowledge: Internet Archive
Kahle, Brewster
Advances in computing and communications mean that we can cost-effectively store every book, sound recording, movie, software package, and public web page ever created, and provide access to these collections via the Internet to students and adults all over the world. By mostly using existing institutions and funding sources, we can build this as well as compensate authors within what is current worldwide library budget. 20 petabytes and growing, the Internet Archive is the 250th most popular website. This talk will discuss the technology and politics of building a large library.
 We've got Bruce DeKale right here. He did a very good talk last afternoon about distributing the web. And we had a good turnout there. It was really awesome talk. Thank you. Right now, he's going to be talking about the Internet Archive, which you probably know. He's the one that founded it. So he knows a lot about it, I think. Let's hear it. Here's Bruce DeKale. A guy that's sleeping, snoring very loudly right next to my tent. So I know that not everybody is awake yet. This has been completely great. So I'm going to talk about basic. You can think of the Internet Archive as hacking the copyright system or trying to get institutions to do things that they're not used to doing, or the way I like to look at it is let's go back to the Library of Alexandria and do it again. And let's go and do the Library of Alexandria that's available to everybody. Can we make all the books, music, video, web pages, software ever created by humans available to anybody that wanted to have access to it? Can we do this? And it turns out technologically, actually, you can. That between the storage of what we have on computers now and the internet in terms of getting it to people, you can do it. So you say, well, why hasn't it happened? And there's a lot of institutional issues of trying to get this to all happen that has taken a lot longer than I thought. But we're getting there. So what I'd like to suggest is that universal access to all knowledge is within our grasp, and we're getting there. But we need a lot more help to be able to get there. So who are we? The Internet Archive is a nonprofit library. Is this showing? Yeah. It's a nonprofit library in San Francisco. Please visit us. And we've been around for almost 20 years. And the idea is to try to do the pieces of the internet that haven't gotten there yet. So as people are going and making things available on the net, they're mostly forgetting about the old things and the like. We see ourselves in the tradition of libraries. I like looking at what people carve in stone. What they carved in stone above the library in Boston was free to all. And this was put there by the robber barons. These are the capitalists that were not nice men. These guys were all about property and mine, mine, mine. Yet they carved free to all above the library that was their legacy. And why? Because information serves a different purpose than just selling stuff back and forth. So this is the tradition that we're in. Now, I'm an engineer. So I go out of any problem from an engineer's perspective. You go and say, OK, if we want all books, music, video, web pages up there, you have to say, well, how big is it? How hard a problem is it? Where do we get it from? How do we make it all work? So that's the structure of this talk. All right, so let's start with books. We say, all right, we want to do all books. The biggest book library by far is the Library of Congress. And they say they've got 28 million books. So 28 million books is by far the largest library ever made in the world. A book is about a megabyte if you have it in Microsoft Word. So 28 million megabytes, megagigatera, 28 terabytes. At 28 terabytes, that's four hard drives that you can buy at a local store. So you've got to have all of the words in the Library of Congress in the shopping cart for less than you pay in a month's rent. Something has changed. Something's happened. We could actually think about having all of this history easily accessible. Then the question is, would you want to do it? And the answer is yes. Actually, we're getting pretty used to having books on screens, even scanned books. Scanned books, so it's not on Kindles, but images, the pages, the screens are good enough. You get these beautiful books. But you can also take it another step. In some places, they say, well, we don't all have screens. We're not all online. Can you print it back out again? So we made a bookmobile. It's a print-on-demand bookmobile. So we put a satellite dish, a printer, cutter, binder, and kids make their own books. It costs about a euro a book to go and download, print, and bind a book. Cheap. So it's actually cheaper to do that than to lend it from a library. A study at Harvard said it cost $3 just administratively to lend a book. So for small books, you can actually do, you can make things available as long as people don't yell at you. In India, we went and made a couple of them. This is the first day at the Library of Alexandria in Egypt, engineer working with a kid, happy kid with his own book. And we did it even in Uganda. This is the first book this girl has ever owned. So we could take not only our books and music and video and make it available to us, but we could make it available even another step out there, which is pretty cool. There are these Rube Goldberg machines. There are these oddball things that go and do it on demand, and they can make books, and they come out a shoot. But I think the real way things are going, as we all know, is more on the area of screens. And the screens are getting so good that we can actually do beautiful books that are pleasure to read and go and take our books and make them available in lots of different formats. My favorite on here is in the bottom right. It is a little talking machine for the blind and dyslexic. It talks a little bit like this. But they now have access to millions of books that they never had before. OK, so now you're convinced maybe that it's a good thing to have it up there. We can go and have the storage to be able to have it up there. So how do you get it done? Well, we've been doing these different things, like putting scanning centers up. This is at the Library of Alexandria. This is a guy who doesn't look too happy. Well, anyway, they've scanned about 170,000 Arabic books there, and it's been continuing along. Then we designed and built our own scanner called the Scribe. We made these scanning centers. This is the one in San Francisco where they're doing microfilm down the center. And gotten it so that it's fairly efficient to basically turn the pages. You say, well, shouldn't you use robots? And we've tried the robots. They don't work very well. They tear the books, and they're expensive, and they don't work very well. I think they could work well. But the investment hasn't been made either by us or anybody else. But so we've been doing it basically by hand and getting beautiful books done. This is rare books from Korea. This is biology books out of China by working with the Chinese Academy of Sciences. And we've now set up 33 scanning centers in eight countries where libraries are doing this. You say, well, Google has already done actually a lot more than we have, about 10 times more than we have. But they have a lot more money than we do, and they locked it up. And so they basically took even the public domain and made it property again. And this is wrong. I mean, if there's a sin in our world is locking up the public domain, the public domain is small enough as it is. We should be arguing maybe about what's in copyright. So if they're the Microsoft, we're the Linux. And we're digitizing pretty fast. [? APPLAUSE?] So I've been going around and asking different places, can we get everything ever written in a particular language? So I got to meet with folks in Greece, but they were kind of busy imploding. There is Iceland. And we got yes out of parliamentarians. We got yeses out of the head of the libraries. And out of 300,000 people in Iceland, there was somebody that decided they were in charge of the no department. So they said no, and it all ground to a halt. But Bali said yes. So we basically started working with Balinese to go and digitize everything ever written in Balinese. We just want to do it all. So let's go and see if we can get whole languages. It turns out the way the Balinese write is not on paper, but on palm leaves. They scratch it into palm leaves. Now it's completely cool. And so these are these priests that worked with us to go and digitize these things by photographing them. They're just completely beautiful. And so now we've gone and digitized and photographed everything written in Balinese. [? What's your?] [? how??] [? OK,?] [? don't.?] [? But?] [? there?] [? put?] [? all my?] [? back down??] When we ask them, how do you read your palm leaves? They say, well, most people don't read. It's either the priests or there are these cool performances that are their culture. So shadow puppets or performances. And so we started videotaping these and starting to make them go online. So I'd like to just give a round of applause for the Balinese to be the first culture to go completely online. Don't you think we should do this with Turkish or Dutch or Danish materials? We can basically go and do this in such a way that their businesses still work and still come up right. So scanning centers, we're doing about 1,000 books every day in these scanning centers all over the world. We've got about 3 million free ebooks that are public domain. And we have modern books that are available for the blind and dyslexic, modern, meaning probably in copyright. But we're also doing a lending system. OK, here's hack number one on how to go and get things available to people, even though they're in copyright. So we try to buy books from publishers so that we can lend them one person at a time. The publishers in general have said no so far. So we've been digitizing our books, and we still lend them one person at a time. So where Google got into trouble by getting into lawsuits and the like, we've gotten into no lawsuits. And so the way that this works is you can go to openlibrary.org, click on a book, say this HTML5 for beginners, which is actually a book that we bought. But you'd see that it's checked out by somebody. So then you have to put it on your wait list. But if you go for a less popular book, say like this history of Mayflower descendants from the Boston Public Library, surprise, nobody's checked it out. OK, so you can go and say, OK, I want to borrow this book. You have a choice of formats, and then you borrow this book. And another thing that's cool about this is it's borrowing it from the Boston Public Library. So these real libraries are digitizing books that are in copyright, non-rights-cleared books. Digitizing them and lending them just like we are a library. And this has been going on for four years, and it's been just fine. So it's a mechanism of trying to be respectful of those that are trying to make money off of this stuff, but still having access and having it happen. And we've been trying out this whole approach of how far can you go and working with publishers, but not working for them, and basically building a library system and making it work. So we've been able to get books by the hundreds of thousands that are current books and make them available. OK, books. Let's go on to another media type, music. So what if we want it? This is an area that has more lawyers than business people, it seems. I mean, this is an area that just people like to sue each other in the whole music area. So we've had to be a little bit more careful about how we've gone about this. And we first started with rock and roll bands that wanted to be distributed. So it turns out the Grateful Dead started a tradition of allowing people to record their concerts and then trade them on cassettes with other people as long as no one made any money. That's been a key thing that I've found in all of this, is no one made any money. And so as these moved online, the bands were up for being distributed online. So we asked some level of permission. And it's usually the fans going and saying, is it OK to put your concerts on the archive? And somebody has to say yes. Maybe it's the drummer or somebody in that community says yes. This is a lot less than what lawyers would like with signatures and all of that stuff. It's like, nah, is it OK? Yeah. And if it ever becomes not OK, then we take it back down again. But you know, that's only happened once. And we now have 6,000 bands up there and 130,000 concerts. And everything the Grateful Dead's ever done. So the idea of getting music up there and out there is starting to work as well. There are these old collections that were on old websites. Before, mp3.com was a format that was standardized. It was the battle days of AIFF. Anybody remember AIFF? Yeah, bad news. Anyway, there were these sites that were trying to go and do these and distribute music. One of them was the Internet Underground Music Archive. And so they died long time ago. And so we're now up. We have them up. We have got lots of net labels that are using us for free hosting. And we've been working along with lots of different record producers now to start to go and do CD digitization. And there's some engineers actually here in Amsterdam that have been doing CD digitization software to try to help get all of this stuff to work well. And we're starting to get donations of LP's 78 RPM records and the like. And starting to do mass digitization of these different formats. So why do this? Well, we haven't just gone and put everything up on the net. We'd like to maybe do 30 seconds and then point to Amazon.com or something like that. But so far what we've done is made it available to researchers and the like and listening rooms so that at least on campus you can have full access to it. So we're starting to get better at music and getting better at the whole areas of music. So we've now added to our collections of other audio recordings that are freely hosted on the internet. So the idea of having infinite storage, infinite bandwidth forever for free for some communities is a very compelling offer. So basically make things available and put them up. So even audio is doable. Moving images. Most people think of movies as Hollywood films. And we're not that good at collecting this stuff yet. So mostly we've been doing old films that haven't been particularly distributed like through Hollywood. Those old films you saw in high school when they had a substitute teacher, they'd wheel in this projector and they'd show you why to be a typesetter or these old, are you ready for marriage? Anyway, these. So we've digitized these and made them available. And people love them. I'm not quite sure why. But they're there and we've been making these things available and people have been uploading things long before YouTube. YouTube really ran away with the whole area of video hosting. They own it. But there's still people about 1,000 a day, people putting things up in the internet archive because they want them maybe more permanent or some other reason. So we've been doing digitization. Even VHS tapes, which almost all have rights problems. But we find if we do ones that aren't on DVD, nobody gets mad at us. So we're getting better at digitizing. Even television is doable. We started archiving 20 channels of television in the year 2000, Russian, Chinese, Japanese, Iraqi, Al Jazeera, BBC, CNN, 24 hours a day, DVD quality. The idea is to at least hold on to it. And so we've got 9-11 collections. But we're now starting to lend television. So if you go to archive.org, you can basically go and search on what people said. If it's in US television news, because it's the only thing that we've got the closed captions, that's the transcripts of what people said. And you can type in and go and say, I want to see things about Edward Snowden and find all of the clips that have Edward Snowden in them. And you can then take those clips and put them in party rear blog, or request a DVD of the whole program if you want to make a documentary. And this is working. Even though it's an enormous amount of materials, the publishers, the networks, are happy about this. So we're actually able to make steps of making things widely available. We want everyone to be a John Stuart research department. Like the Comedy Central, where they go and say, here's what a politician said before, and now here's what they said now, and doesn't match. That type of thing, to have people think critically about television. So even moving images is doable. So if you do this kind of thing that we're doing, to go and offer free hosting, sometimes you attract attention from people you don't like. So the FBI gave us one of these nasty letters called a national security letter. A national security letter is when they demand information about our patrons of the internet archive, our users of the internet archive. And we can't even say to anybody that we've ever gotten this request. So we got one of these things. And we got our lawyers, the Electronic Frontier Foundation. Oh, hooray for the Electronic Frontier Foundation. I said, what can we do about this? And they said, well, you have to comply. Well, can I talk to my board about it? No. Can I talk to anybody about it? No. Can I ever talk to anybody about it? No. What happens if I don't do it? Jail. Is there anything we can do? No. Really? Well, you can sue the United States government. So we sue the United States government. And one. So there have been hundreds of thousands of these letters, hundreds of thousands of these letters sent out. There have been only three organizations that have publicly gone and pushed back on the government. And they've been all libraries. What's great about being a library is you're allowed to go and say, no. There's a long history of people being rounded up for what it is they've read and bad things happening to them. And people remember this. And so where Google doesn't have, at least publicly, hasn't said no, libraries are sort of our role makes it so that it's not an embarrassing thing to do. So we find that being a library is a good thing. Software. So there's a lot of software out there. And we're getting better at going and reproducing this software by running emulators in the browser. This is a real mind blow of taking a C emulators of old Apple or Commodore or Atari software and cross compiling it within script and into JavaScript. And it runs in your browser. So you click and it actually boots an old IBM PC in your browser and you're running your old game. It turns out this is very, very popular. Because I guess a lot of people spend their early days playing games. But anyway, they're now back. So Oregon Trail and all these other games are very popular. What we're probably best known for is crawling the worldwide web. So how many people have used the Wayback machine? Yay. So the Wayback machine is a way you can see the web as it was. That so many people are pouring their lives into the web. But web pages only on average last 100 days. So we go through and we try to archive them. And we started when it was pretty small. And it's now getting pretty freaking big. We archive about a billion pages every week to be able to create this Wayback machine. This is what Yahoo looked like in 1996. Pets.com with the little sock guy, dorky old web design. And I looked up what the, I thought, oh, why don't I go and look at what Chaos Communications Camp looked like. So this is the Chaos website from 1997. But it actually looks a whole hell of a lot like the current one. So there's a little retro thing going on. So it's not quite as dramatic. There's another thing that this has been used for. A user came back and said, hey, Brewster, there's been a change that your web collection has the only place that can show it. This is a press release by the United States government of the president being on an aircraft carrier saying, mission accomplished about war in Iraq. And it says that the president announces combat operations in Iraq have ended. Then a couple days later, they changed it. And they put in major combat operations have changed. They didn't make any notice that they changed a press release. It's George Orwell, right? To be able to go and redo press releases from the past. It's living in the day. So now this is an example of why we want something like a Wayback machine. Another is we have a web archiving tool that's a subscription-based service that a lot of companies and libraries and museums pay us to do, which is helpful. It keeps our lights on. And we now have 1,700 curated collections, like the Japanese disaster, where people have come together to go and say, you should archive this because soon it's offline. Or archive these things soon they're offline. So it's been a community project to be able to go and work together to build up what are the key things about an event to make sure that we really have it done well. So even web is doable. So the worldwide web collection of ours is probably about 10 petabytes of data. It's growing at a few petabytes every year. We have about 450 billion web pages. We get about 600,000 people a day using it. It's a database of 450 billion pages that gets queried about 2,000 times a second. And so that's sort of what the Wayback machine is. And it's been much more popular than we thought it would be, which has just been great. But even rare books and letters, you can go and do these things by photographing and breaking them available. Next up for us is personal digital archives. How are we going to do this stuff that's splintered on all sorts of places? So people don't even have them. It's not like boxes in people's basements anymore. It's not even hard drives that you have. It's these Flickr sites and these other places that you've gone and put your memories and guaranteed these guys are going down. Even the rich companies like Google, Google Video, ever heard of Google Video? Well, it used to exist. There were 6 million videos on it, but they took it away. Yahoo Video is now gone. GeoCities is famously gone. Apple Computer, the most valuable company in the whole world, couldn't figure out how to run 200 terabytes of mobile me on a continuous basis. So we archive that and making it available. So don't count on these places. They don't have your best interest in mind. They'll turn it off whenever they want to. So how do you go and preserve this stuff, both the physical and the digital? If we want to build a library of Alexandria version 2, well, what's the lesson out of the library of Alexandria version 1? What's it best known for? Burning. It's best known for not being here anymore. So how do we go and make it so that we don't do that again? Well, let's have multiple copies. If we put multiple copies in multiple places that will have different fault modes, then I think we have a better shot at it. So we gave a copy back in 2002 to the new library of Alexandria. And this is actually, they redid their first floor. If you get to Alexandria, go. It's a completely great city and a library. And so this is what it looked like in 2002. Access for All gave us free space, and they have for the last 10 years. Thank you, Access for All. So there's a partial copy in Amsterdam. This is what it looked like in 2008. This is what the Wayback Machine. Oh, we came up with this idea of doing data centers in shipping containers. Sun ran with it, and they gave us one of these. So the Wayback Machine was this. And so, oh, I get to ask, how big is the web? You have to ask me, how big is the web? The web is 8 feet by 8 feet by 20 feet. For several years, when you use the Wayback Machine, you are actually using this shipping container that sat outside on Sun's campus, which is pretty great. We've now made these prettier machines because we bought this cool church. And so there are these blinking lights inside. That's just how we've been able to scale up. We've also started to scale up physical collections, because libraries are throwing things away. So we've basically taken our ideas of how to do really compact, and we've used shipping containers inside warehouses. So we have books or music or video that are in boxes, protected by a box, protected by a shipping container, protected by a warehouse, protected by nonprofits. So the idea is to try to have layers of protection against certain types of attacks. But we're not there yet. Because, OK, we've got a copy in an earthquake zone, the Middle East, and a flood zone. What could go wrong? So I think we need some other copies in other places and more participation towards keeping our whole society and cultural heritage safe. A couple other things that we're worried about at the Internet Archive is some of the ways of trying to be sustainable in the era of corporations. Corporations are basically becoming real strangleholds on certain types of things, like end-user access. So we've been doing free public Wi-Fi and the like. So every time we get a building, we put free Wi-Fi with no passwords and stuff up. And that causes people to get free Wi-Fi, in some cases, and mad in other cases, both of which are perfectly fine with us. We've even gone and tried to apply open source ideas to housing. Housing has become a real problem because of the debt burden. So we're starting to try to transition housing to be able to support nonprofit workers that are debt-free. If you're interested in this idea, we're really trying to figure out some of these things. If you're going to go and build a new housing system for people that are working in the open source world, then you're going to need an organization like financial institutions. We started a credit union and been trying to work with some Bitcoin companies. Oh, did that make regulators mad? Anyway, so there's the idea of to take not just trying to make the data sustainable and copying it forward, but how do you go and get the communities around these materials to be able to have lifetimes that work? We're way into Bitcoin. People have been donating Bitcoin. If you've donated Bitcoin, then thank you very much. We pay our employees partially in Bitcoin, and that's all around working. Project that needs help. There's a lot. Everything needs help. But here's a few of my top. We're redesigning the Wayback machine and could really use some help on trying to make this new and different and neat. How do you do search at a level that we can actually do? Because searching 450 billion pages is too much for us. But maybe site-based search and the like. We're trying to get elastic search to be able to make all of our stuff more accessible, the full text much easier to use. And we're trying to build software that can be distributed, that people can archive their CDs, LPs, and books in a distributed way to go and participate in bringing these things online. There's a couple of programmers in Amsterdam looking for more programmers to help. And the distributed web is sort of a new idea that if there's some mechanism to go and weave it together to go and make a next-generation web that we don't have to archive by taking snapshots, we can actually archive working websites so that they'll live on for tens or hundreds of years after when the original administrators are gone. So in conclusion, universal access to all knowledge. It's possible to do. I think it could be one of the greatest things humans have ever done. I think it could be one of the things that our generation gets to go and offer the world, kind of like the man on the moon or the library of Alexandria in Asians' days, that we can pull this together. We have the technology to be able to pull it off. We've got the political will to live in an open environment as long as we don't lose it. And we have to act pretty fast, I'd say, because most kids have turned to screens instead of books or old materials, be able to find out information. They're learning from whatever it is they could get a hold of. And the best that we have to offer is not on the net yet. So we need to be bolder than we've been to go and make universal access to all knowledge happen. Thank you very much. Thank you. I'm sorry. No. I'm sorry. So I'm going to be hanging out. I'm at the food hacking camp, and I'm just going to be hanging out here. But thank you very much for coming. Have a great day. Is this? Yeah, it's on now. I'm very sorry that we didn't have time. OK. One more for Bruce.
 We've got Bruce DeKale right here.  He did a very good talk last afternoon about distributing the web.  And we had a good turnout there.  It was really awesome talk.  Thank you.  Right now, he's going to be talking about the Internet  Archive, which you probably know.  He's the one that founded it.  So he knows a lot about it, I think.  Let's hear it.  Here's Bruce DeKale.  A guy that's sleeping, snoring very loudly right next to my tent.  So I know that not everybody is awake yet.  This has been completely great.  So I'm going to talk about basic.  You can think of the Internet Archive  as hacking the copyright system or trying  to get institutions to do things that they're not  used to doing, or the way I like to look at it  is let's go back to the Library of Alexandria  and do it again.  And let's go and do the Library of Alexandria that's  available to everybody.  Can we make all the books, music, video, web pages,  software ever created by humans available to anybody  that wanted to have access to it?  Can we do this?  And it turns out technologically, actually,  you can.  That between the storage of what we have on computers now  and the internet in terms of getting it to people,  you can do it.  So you say, well, why hasn't it happened?  And there's a lot of institutional issues  of trying to get this to all happen that  has taken a lot longer than I thought.  But we're getting there.  So what I'd like to suggest is that universal access  to all knowledge is within our grasp,  and we're getting there.  But we need a lot more help to be able to get there.  So who are we?  The Internet Archive is a nonprofit library.  Is this showing?  Yeah.  It's a nonprofit library in San Francisco.  Please visit us.  And we've been around for almost 20 years.  And the idea is to try to do the pieces of the internet that  haven't gotten there yet.  So as people are going and making things available on the net,  they're mostly forgetting about the old things and the like.  We see ourselves in the tradition of libraries.  I like looking at what people carve in stone.  What they carved in stone above the library in Boston  was free to all.  And this was put there by the robber barons.  These are the capitalists that were not nice men.  These guys were all about property and mine, mine, mine.  Yet they carved free to all above the library that  was their legacy.  And why?  Because information serves a different purpose  than just selling stuff back and forth.  So this is the tradition that we're in.  Now, I'm an engineer.  So I go out of any problem from an engineer's perspective.  You go and say, OK, if we want all books, music, video, web  pages up there, you have to say, well, how big is it?  How hard a problem is it?  Where do we get it from?  How do we make it all work?  So that's the structure of this talk.  All right, so let's start with books.  We say, all right, we want to do all books.  The biggest book library by far is the Library of Congress.  And they say they've got 28 million books.  So 28 million books is by far the largest library ever  made in the world.  A book is about a megabyte if you have it in Microsoft Word.  So 28 million megabytes, megagigatera, 28 terabytes.  At 28 terabytes, that's four hard drives  that you can buy at a local store.  So you've got to have all of the words in the Library of Congress  in the shopping cart for less than you pay in a month's rent.  Something has changed.  Something's happened.  We could actually think about having all of this history  easily accessible.  Then the question is, would you want to do it?  And the answer is yes.  Actually, we're getting pretty used to having books on screens,  even scanned books.  Scanned books, so it's not on Kindles,  but images, the pages, the screens are good enough.  You get these beautiful books.  But you can also take it another step.  In some places, they say, well, we don't all have screens.  We're not all online.  Can you print it back out again?  So we made a bookmobile.  It's a print-on-demand bookmobile.  So we put a satellite dish, a printer, cutter, binder,  and kids make their own books.  It costs about a euro a book to go and download, print,  and bind a book.  Cheap.  So it's actually cheaper to do that than to lend it  from a library.  A study at Harvard said it cost $3 just administratively  to lend a book.  So for small books, you can actually do,  you can make things available as long as people  don't yell at you.  In India, we went and made a couple of them.  This is the first day at the Library of Alexandria in Egypt,  engineer working with a kid, happy kid with his own book.  And we did it even in Uganda.  This is the first book this girl has ever owned.  So we could take not only our books and music and video  and make it available to us, but we  could make it available even another step out there,  which is pretty cool.  There are these Rube Goldberg machines.  There are these oddball things that go and do it on demand,  and they can make books, and they come out a shoot.  But I think the real way things are going, as we all know,  is more on the area of screens.  And the screens are getting so good  that we can actually do beautiful books that  are pleasure to read and go and take our books  and make them available in lots of different formats.  My favorite on here is in the bottom right.  It is a little talking machine for the blind and dyslexic.  It talks a little bit like this.  But they now have access to millions of books  that they never had before.  OK, so now you're convinced maybe that it's  a good thing to have it up there.  We can go and have the storage to be able to have it up there.  So how do you get it done?  Well, we've been doing these different things,  like putting scanning centers up.  This is at the Library of Alexandria.  This is a guy who doesn't look too happy.  Well, anyway, they've scanned about 170,000 Arabic books  there, and it's been continuing along.  Then we designed and built our own scanner called the Scribe.  We made these scanning centers.  This is the one in San Francisco where they're doing microfilm  down the center.  And gotten it so that it's fairly efficient to basically turn  the pages.  You say, well, shouldn't you use robots?  And we've tried the robots.  They don't work very well.  They tear the books, and they're expensive,  and they don't work very well.  I think they could work well.  But the investment hasn't been made either by us  or anybody else.  But so we've been doing it basically by hand  and getting beautiful books done.  This is rare books from Korea.  This is biology books out of China  by working with the Chinese Academy of Sciences.  And we've now set up 33 scanning centers in eight countries  where libraries are doing this.  You say, well, Google has already  done actually a lot more than we have, about 10 times more  than we have.  But they have a lot more money than we do, and they locked it up.  And so they basically took even the public domain  and made it property again.  And this is wrong.  I mean, if there's a sin in our world  is locking up the public domain, the public domain  is small enough as it is.  We should be arguing maybe about what's in copyright.  So if they're the Microsoft, we're the Linux.  And we're digitizing pretty fast.  [? APPLAUSE?]  So I've been going around and asking different places,  can we get everything ever written in a particular language?  So I got to meet with folks in Greece,  but they were kind of busy imploding.  There is Iceland.  And we got yes out of parliamentarians.  We got yeses out of the head of the libraries.  And out of 300,000 people in Iceland,  there was somebody that decided they were in charge  of the no department.  So they said no, and it all ground to a halt.  But Bali said yes.  So we basically started working with Balinese  to go and digitize everything ever written in Balinese.  We just want to do it all.  So let's go and see if we can get whole languages.  It turns out the way the Balinese write is not on paper,  but on palm leaves.  They scratch it into palm leaves.  Now it's completely cool.  And so these are these priests that  worked with us to go and digitize these things  by photographing them.  They're just completely beautiful.  And so now we've gone and digitized and photographed  everything written in Balinese.  [? What's your?]  [? how??]  [? OK,?]  [? don't.?]  [? But?]  [? there?]  [? put?]  [? all my?]  [? back down??]  When we ask them, how do you read your palm leaves?  They say, well, most people don't read.  It's either the priests or there are these cool performances  that are their culture.  So shadow puppets or performances.  And so we started videotaping these  and starting to make them go online.  So I'd like to just give a round of applause  for the Balinese to be the first culture  to go completely online.  Don't you think we should do this with Turkish or Dutch  or Danish materials?  We can basically go and do this in such a way  that their businesses still work and still come up right.  So scanning centers, we're doing about 1,000 books every day  in these scanning centers all over the world.  We've got about 3 million free ebooks  that are public domain.  And we have modern books that are available for the blind  and dyslexic, modern, meaning probably in copyright.  But we're also doing a lending system.  OK, here's hack number one on how to go and get things  available to people, even though they're in copyright.  So we try to buy books from publishers  so that we can lend them one person at a time.  The publishers in general have said no so far.  So we've been digitizing our books,  and we still lend them one person at a time.  So where Google got into trouble by getting into lawsuits  and the like, we've gotten into no lawsuits.  And so the way that this works is you  can go to openlibrary.org, click on a book, say this HTML5  for beginners, which is actually a book that we bought.  But you'd see that it's checked out by somebody.  So then you have to put it on your wait list.  But if you go for a less popular book,  say like this history of Mayflower descendants  from the Boston Public Library, surprise,  nobody's checked it out.  OK, so you can go and say, OK, I want to borrow this book.  You have a choice of formats, and then you borrow this book.  And another thing that's cool about this  is it's borrowing it from the Boston Public Library.  So these real libraries are digitizing books  that are in copyright, non-rights-cleared books.  Digitizing them and lending them just like we are a library.  And this has been going on for four years,  and it's been just fine.  So it's a mechanism of trying to be respectful of those  that are trying to make money off of this stuff,  but still having access and having it happen.  And we've been trying out this whole approach of how far can  you go and working with publishers, but not working  for them, and basically building a library system  and making it work.  So we've been able to get books by the hundreds of thousands  that are current books and make them available.  OK, books.  Let's go on to another media type, music.  So what if we want it?  This is an area that has more lawyers than business people,  it seems.  I mean, this is an area that just people  like to sue each other in the whole music area.  So we've had to be a little bit more careful about how  we've gone about this.  And we first started with rock and roll bands  that wanted to be distributed.  So it turns out the Grateful Dead started a tradition  of allowing people to record their concerts  and then trade them on cassettes with other people  as long as no one made any money.  That's been a key thing that I've found in all of this,  is no one made any money.  And so as these moved online, the bands  were up for being distributed online.  So we asked some level of permission.  And it's usually the fans going and saying,  is it OK to put your concerts on the archive?  And somebody has to say yes.  Maybe it's the drummer or somebody in that community  says yes.  This is a lot less than what lawyers would like with signatures  and all of that stuff.  It's like, nah, is it OK?  Yeah.  And if it ever becomes not OK, then we take it back down again.  But you know, that's only happened once.  And we now have 6,000 bands up there and 130,000 concerts.  And everything the Grateful Dead's ever done.  So the idea of getting music up there and out there  is starting to work as well.  There are these old collections that were on old websites.  Before, mp3.com was a format that was standardized.  It was the battle days of AIFF.  Anybody remember AIFF?  Yeah, bad news.  Anyway, there were these sites that were trying to go and do  these and distribute music.  One of them was the Internet Underground Music Archive.  And so they died long time ago.  And so we're now up.  We have them up.  We have got lots of net labels that are using us for free  hosting.  And we've been working along with lots of different record  producers now to start to go and do CD digitization.  And there's some engineers actually here in Amsterdam  that have been doing CD digitization software to try  to help get all of this stuff to work well.  And we're starting to get donations of LP's 78 RPM  records and the like.  And starting to do mass digitization of these different  formats.  So why do this?  Well, we haven't just gone and put everything up on the net.  We'd like to maybe do 30 seconds and then point to  Amazon.com or something like that.  But so far what we've done is made it available to researchers  and the like and listening rooms so that at least on  campus you can have full access to it.  So we're starting to get better at music and getting better  at the whole areas of music.  So we've now added to our collections of other audio  recordings that are freely hosted on the internet.  So the idea of having infinite storage, infinite bandwidth  forever for free for some communities is a very  compelling offer.  So basically make things available and put them up.  So even audio is doable.  Moving images.  Most people think of movies as Hollywood films.  And we're not that good at collecting this stuff yet.  So mostly we've been doing old films that haven't been  particularly distributed like through Hollywood.  Those old films you saw in high school when they had a  substitute teacher, they'd wheel in this projector and  they'd show you why to be a typesetter or these old, are  you ready for marriage?  Anyway, these.  So we've digitized these and made them available.  And people love them.  I'm not quite sure why.  But they're there and we've been making these things  available and people have been uploading things long  before YouTube.  YouTube really ran away with the whole area of video  hosting.  They own it.  But there's still people about 1,000 a day, people  putting things up in the internet archive because they  want them maybe more permanent or some other reason.  So we've been doing digitization.  Even VHS tapes, which almost all have rights problems.  But we find if we do ones that aren't on DVD, nobody gets  mad at us.  So we're getting better at digitizing.  Even television is doable.  We started archiving 20 channels of television in the  year 2000, Russian, Chinese, Japanese, Iraqi, Al Jazeera,  BBC, CNN, 24 hours a day, DVD quality.  The idea is to at least hold on to it.  And so we've got 9-11 collections.  But we're now starting to lend television.  So if you go to archive.org, you can basically go and search  on what people said.  If it's in US television news, because it's the only thing  that we've got the closed captions, that's the transcripts  of what people said.  And you can type in and go and say, I want to see things  about Edward Snowden and find all of the clips that have  Edward Snowden in them.  And you can then take those clips and put them in  party rear blog, or request a DVD of the whole program if  you want to make a documentary.  And this is working.  Even though it's an enormous amount of materials, the  publishers, the networks, are happy about this.  So we're actually able to make steps of making things  widely available.  We want everyone to be a John Stuart research department.  Like the Comedy Central, where they go and say, here's what  a politician said before, and now here's what they said now,  and doesn't match.  That type of thing, to have people think critically  about television.  So even moving images is doable.  So if you do this kind of thing that we're doing, to go and  offer free hosting, sometimes you attract attention from  people you don't like.  So the FBI gave us one of these nasty letters called a  national security letter.  A national security letter is when they demand information  about our patrons of the internet archive, our users of  the internet archive.  And we can't even say to anybody that we've ever gotten  this request.  So we got one of these things.  And we got our lawyers, the Electronic Frontier Foundation.  Oh, hooray for the Electronic Frontier Foundation.  I said, what can we do about this?  And they said, well, you have to comply.  Well, can I talk to my board about it?  No.  Can I talk to anybody about it?  No.  Can I ever talk to anybody about it?  No.  What happens if I don't do it?  Jail.  Is there anything we can do?  No.  Really?  Well, you can sue the United States government.  So we sue the United States government.  And one.  So there have been hundreds of thousands of these letters,  hundreds of thousands of these letters sent out.  There have been only three organizations that have  publicly gone and pushed back on the government.  And they've been all libraries.  What's great about being a library is you're allowed to go  and say, no.  There's a long history of people being rounded up for what  it is they've read and bad things happening to them.  And people remember this.  And so where Google doesn't have, at least publicly,  hasn't said no, libraries are sort of our role makes it so  that it's not an embarrassing thing to do.  So we find that being a library is a good thing.  Software.  So there's a lot of software out there.  And we're getting better at going and reproducing this  software by running emulators in the browser.  This is a real mind blow of taking a C emulators of old  Apple or Commodore or Atari software and cross compiling  it within script and into JavaScript.  And it runs in your browser.  So you click and it actually boots an old IBM PC in your  browser and you're running your old game.  It turns out this is very, very popular.  Because I guess a lot of people spend their early days  playing games.  But anyway, they're now back.  So Oregon Trail and all these other games are very popular.  What we're probably best known for is crawling  the worldwide web.  So how many people have used the Wayback machine?  Yay.  So the Wayback machine is a way you can see the web as it  was.  That so many people are pouring their lives into the web.  But web pages only on average last 100 days.  So we go through and we try to archive them.  And we started when it was pretty small.  And it's now getting pretty freaking big.  We archive about a billion pages every week to be able to  create this Wayback machine.  This is what Yahoo looked like in 1996.  Pets.com with the little sock guy, dorky old web design.  And I looked up what the, I thought, oh, why don't I go  and look at what Chaos Communications Camp looked  like.  So this is the Chaos website from 1997.  But it actually looks a whole hell of a lot  like the current one.  So there's a little retro thing going on.  So it's not quite as dramatic.  There's another thing that this has been used for.  A user came back and said, hey, Brewster, there's been a  change that your web collection has the only  place that can show it.  This is a press release by the United States government of  the president being on an aircraft carrier saying,  mission accomplished about war in Iraq.  And it says that the president announces combat  operations in Iraq have ended.  Then a couple days later, they changed it.  And they put in major combat operations have changed.  They didn't make any notice that they changed a press  release.  It's George Orwell, right?  To be able to go and redo press releases from the past.  It's living in the day.  So now this is an example of why we want something like a  Wayback machine.  Another is we have a web archiving tool that's a  subscription-based service that a lot of companies and  libraries and museums pay us to do, which is helpful.  It keeps our lights on.  And we now have 1,700 curated collections, like the  Japanese disaster, where people have come together to go  and say, you should archive this because soon it's  offline.  Or archive these things soon they're offline.  So it's been a community project to be able to go and  work together to build up what are the key things about an  event to make sure that we really have it done well.  So even web is doable.  So the worldwide web collection of ours is probably about 10  petabytes of data.  It's growing at a few petabytes every year.  We have about 450 billion web pages.  We get about 600,000 people a day using it.  It's a database of 450 billion pages that gets queried  about 2,000 times a second.  And so that's sort of what the Wayback machine is.  And it's been much more popular than we thought it  would be, which has just been great.  But even rare books and letters, you can go and do these  things by photographing and breaking them available.  Next up for us is personal digital archives.  How are we going to do this stuff that's splintered on all  sorts of places?  So people don't even have them.  It's not like boxes in people's basements anymore.  It's not even hard drives that you have.  It's these Flickr sites and these other places that you've  gone and put your memories and guaranteed these guys are  going down.  Even the rich companies like Google, Google Video, ever  heard of Google Video?  Well, it used to exist.  There were 6 million videos on it, but they took it away.  Yahoo Video is now gone.  GeoCities is famously gone.  Apple Computer, the most valuable company in the whole  world, couldn't figure out how to run 200 terabytes of  mobile me on a continuous basis.  So we archive that and making it available.  So don't count on these places.  They don't have your best interest in mind.  They'll turn it off whenever they want to.  So how do you go and preserve this stuff, both the physical  and the digital?  If we want to build a library of Alexandria version 2, well,  what's the lesson out of the library of Alexandria version  1?  What's it best known for?  Burning.  It's best known for not being here anymore.  So how do we go and make it so that we don't do that again?  Well, let's have multiple copies.  If we put multiple copies in multiple places that will have  different fault modes, then I think we have a better shot  at it.  So we gave a copy back in 2002 to the new library of  Alexandria.  And this is actually, they redid their first floor.  If you get to Alexandria, go.  It's a completely great city and a library.  And so this is what it looked like in 2002.  Access for All gave us free space, and they have for the  last 10 years.  Thank you, Access for All.  So there's a partial copy in Amsterdam.  This is what it looked like in 2008.  This is what the Wayback Machine.  Oh, we came up with this idea of doing data centers in  shipping containers.  Sun ran with it, and they gave us one of these.  So the Wayback Machine was this.  And so, oh, I get to ask, how big is the web?  You have to ask me, how big is the web?  The web is 8 feet by 8 feet by 20 feet.  For several years, when you use the Wayback Machine, you are  actually using this shipping container that sat outside on  Sun's campus, which is pretty great.  We've now made these prettier machines because we bought this  cool church.  And so there are these blinking lights inside.  That's just how we've been able to scale up.  We've also started to scale up physical collections, because  libraries are throwing things away.  So we've basically taken our ideas of how to do really  compact, and we've used shipping containers  inside warehouses.  So we have books or music or video that are in boxes,  protected by a box, protected by a shipping container,  protected by a warehouse, protected by nonprofits.  So the idea is to try to have layers of protection against  certain types of attacks.  But we're not there yet.  Because, OK, we've got a copy in an earthquake zone, the  Middle East, and a flood zone.  What could go wrong?  So I think we need some other copies in other places and  more participation towards keeping our whole society and  cultural heritage safe.  A couple other things that we're worried about at the  Internet Archive is some of the ways of trying to be  sustainable in the era of corporations.  Corporations are basically becoming real strangleholds  on certain types of things, like end-user access.  So we've been doing free public Wi-Fi and the like.  So every time we get a building, we put free Wi-Fi with no  passwords and stuff up.  And that causes people to get free Wi-Fi, in some cases, and  mad in other cases, both of which are perfectly fine with us.  We've even gone and tried to apply open source ideas to  housing.  Housing has become a real problem because of the debt  burden.  So we're starting to try to transition housing to be able  to support nonprofit workers that are debt-free.  If you're interested in this idea, we're really trying to  figure out some of these things.  If you're going to go and build a new housing system for  people that are working in the open source world, then you're  going to need an organization like financial  institutions.  We started a credit union and been trying to work with some  Bitcoin companies.  Oh, did that make regulators mad?  Anyway, so there's the idea of to take not just trying to  make the data sustainable and copying it forward, but how do  you go and get the communities around these materials to be  able to have lifetimes that work?  We're way into Bitcoin.  People have been donating Bitcoin.  If you've donated Bitcoin, then thank you very much.  We pay our employees partially in Bitcoin, and that's all  around working.  Project that needs help.  There's a lot.  Everything needs help.  But here's a few of my top.  We're redesigning the Wayback machine and could really use  some help on trying to make this new and different and neat.  How do you do search at a level that we can actually do?  Because searching 450 billion pages is too much for us.  But maybe site-based search and the like.  We're trying to get elastic search to be able to make all  of our stuff more accessible, the full text much easier to use.  And we're trying to build software that can be distributed,  that people can archive their CDs, LPs, and books in a  distributed way to go and participate in bringing these  things online.  There's a couple of programmers in Amsterdam looking for more  programmers to help.  And the distributed web is sort of a new idea that if there's  some mechanism to go and weave it together to go and make a  next-generation web that we don't have to archive by  taking snapshots, we can actually archive working websites  so that they'll live on for tens or hundreds of years after  when the original administrators are gone.  So in conclusion, universal access to all knowledge.  It's possible to do.  I think it could be one of the greatest things humans have  ever done.  I think it could be one of the things that our generation gets  to go and offer the world, kind of like the man on the moon or  the library of Alexandria in Asians' days, that we can pull  this together.  We have the technology to be able to pull it off.  We've got the political will to live in an open environment as  long as we don't lose it.  And we have to act pretty fast, I'd say, because most kids have  turned to screens instead of books or old materials, be able  to find out information.  They're learning from whatever it is they could get a hold of.  And the best that we have to offer is not on the net yet.  So we need to be bolder than we've been to go and make  universal access to all knowledge happen.  Thank you very much.  Thank you.  I'm sorry.  No.  I'm sorry.  So I'm going to be hanging out.  I'm at the food hacking camp, and I'm just going to be hanging out  here.  But thank you very much for coming.  Have a great day.  Is this?  Yeah, it's on now.  I'm very sorry that we didn't have time.  OK.  One more for Bruce.
Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_0_1715.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_1716_1763.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_1764_2039.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_2040_3431.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_3432_3551.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_3552_4439.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_4440_4787.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_4788_5591.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_5592_6671.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_6672_8351.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_8352_10163.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_10164_12491.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_12492_12695.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_12696_12755.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_12756_14627.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_14628_14687.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_14688_15323.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_15324_16703.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_16704_17267.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_17268_19475.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_19476_20435.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_20436_27923.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_27924_28979.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_28980_29423.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_29424_29663.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_29664_31295.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_31296_34535.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_34536_35207.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_35208_35783.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_35784_38879.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_38880_39623.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_39624_41027.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_41028_41519.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_41520_41579.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_41580_41843.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_41844_42191.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_42192_42407.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_42408_45815.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_45816_46091.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_46092_46799.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_46800_47327.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_47328_48083.png Brewster_Kahle__Towards_Universal_Access_to_All_Knowledge__Internet_Archive_lKvoVxUQKD0_48084_48995.png
We've got Bruce DeKale right here. He did a very good talk last afternoon about distributing the web. And we had a good turnout there. It was really awesome talk. Thank you. Right now, he's going to be talking about the Internet Archive, which you probably know. He's the one that founded it. So he knows a lot about it, I think. Let's hear it. Here's Bruce DeKale. A guy that's sleeping, snoring very loudly right next to my tent. So I know that not everybody is awake yet. This has been completely great. So I'm going to talk about basic. You can think of the Internet Archive as hacking the copyright system or trying to get institutions to do things that they're not used to doing, or the way I like to look at it is let's go back to the Library of Alexandria and do it again. And let's go and do the Library of Alexandria that's available to everybody. Can we make all the books, music, video, web pages, software ever created by humans available to anybody that wanted to have access to it? Can we do this? And it turns out technologically, actually, you can. That between the storage of what we have on computers now and the internet in terms of getting it to people, you can do it. So you say, well, why hasn't it happened? And there's a lot of institutional issues of trying to get this to all happen that has taken a lot longer than I thought. But we're getting there. So what I'd like to suggest is that universal access to all knowledge is within our grasp, and we're getting there. But we need a lot more help to be able to get there. So who are we? The Internet Archive is a nonprofit library. Is this showing? Yeah. It's a nonprofit library in San Francisco. Please visit us. And we've been around for almost 20 years. And the idea is to try to do the pieces of the internet that haven't gotten there yet. So as people are going and making things available on the net, they're mostly forgetting about the old things and the like. We see ourselves in the tradition of libraries. I like looking at what people carve in stone. What they carved in stone above the library in Boston was free to all. And this was put there by the robber barons. These are the capitalists that were not nice men. These guys were all about property and mine, mine, mine. Yet they carved free to all above the library that was their legacy. And why? Because information serves a different purpose than just selling stuff back and forth. So this is the tradition that we're in. Now, I'm an engineer. So I go out of any problem from an engineer's perspective. You go and say, OK, if we want all books, music, video, web pages up there, you have to say, well, how big is it? How hard a problem is it? Where do we get it from? How do we make it all work? So that's the structure of this talk. All right, so let's start with books. We say, all right, we want to do all books. The biggest book library by far is the Library of Congress. And they say they've got 28 million books. So 28 million books is by far the largest library ever made in the world. A book is about a megabyte if you have it in Microsoft Word. So 28 million megabytes, megagigatera, 28 terabytes. At 28 terabytes, that's four hard drives that you can buy at a local store. So you've got to have all of the words in the Library of Congress in the shopping cart for less than you pay in a month's rent. Something has changed. Something's happened. We could actually think about having all of this history easily accessible. Then the question is, would you want to do it? And the answer is yes. Actually, we're getting pretty used to having books on screens, even scanned books. Scanned books, so it's not on Kindles, but images, the pages, the screens are good enough. You get these beautiful books. But you can also take it another step. In some places, they say, well, we don't all have screens. We're not all online. Can you print it back out again? So we made a bookmobile. It's a print-on-demand bookmobile. So we put a satellite dish, a printer, cutter, binder, and kids make their own books. It costs about a euro a book to go and download, print, and bind a book. Cheap. So it's actually cheaper to do that than to lend it from a library. A study at Harvard said it cost $3 just administratively to lend a book. So for small books, you can actually do, you can make things available as long as people don't yell at you. In India, we went and made a couple of them. This is the first day at the Library of Alexandria in Egypt, engineer working with a kid, happy kid with his own book. And we did it even in Uganda. This is the first book this girl has ever owned. So we could take not only our books and music and video and make it available to us, but we could make it available even another step out there, which is pretty cool. There are these Rube Goldberg machines. There are these oddball things that go and do it on demand, and they can make books, and they come out a shoot. But I think the real way things are going, as we all know, is more on the area of screens. And the screens are getting so good that we can actually do beautiful books that are pleasure to read and go and take our books and make them available in lots of different formats. My favorite on here is in the bottom right. It is a little talking machine for the blind and dyslexic. It talks a little bit like this. But they now have access to millions of books that they never had before. OK, so now you're convinced maybe that it's a good thing to have it up there. We can go and have the storage to be able to have it up there. So how do you get it done? Well, we've been doing these different things, like putting scanning centers up. This is at the Library of Alexandria. This is a guy who doesn't look too happy. Well, anyway, they've scanned about 170,000 Arabic books there, and it's been continuing along. Then we designed and built our own scanner called the Scribe. We made these scanning centers. This is the one in San Francisco where they're doing microfilm down the center. And gotten it so that it's fairly efficient to basically turn the pages. You say, well, shouldn't you use robots? And we've tried the robots. They don't work very well. They tear the books, and they're expensive, and they don't work very well. I think they could work well. But the investment hasn't been made either by us or anybody else. But so we've been doing it basically by hand and getting beautiful books done. This is rare books from Korea. This is biology books out of China by working with the Chinese Academy of Sciences. And we've now set up 33 scanning centers in eight countries where libraries are doing this. You say, well, Google has already done actually a lot more than we have, about 10 times more than we have. But they have a lot more money than we do, and they locked it up. And so they basically took even the public domain and made it property again. And this is wrong. I mean, if there's a sin in our world is locking up the public domain, the public domain is small enough as it is. We should be arguing maybe about what's in copyright. So if they're the Microsoft, we're the Linux. And we're digitizing pretty fast. [? APPLAUSE?] So I've been going around and asking different places, can we get everything ever written in a particular language? So I got to meet with folks in Greece, but they were kind of busy imploding. There is Iceland. And we got yes out of parliamentarians. We got yeses out of the head of the libraries. And out of 300,000 people in Iceland, there was somebody that decided they were in charge of the no department. So they said no, and it all ground to a halt. But Bali said yes. So we basically started working with Balinese to go and digitize everything ever written in Balinese. We just want to do it all. So let's go and see if we can get whole languages. It turns out the way the Balinese write is not on paper, but on palm leaves. They scratch it into palm leaves. Now it's completely cool. And so these are these priests that worked with us to go and digitize these things by photographing them. They're just completely beautiful. And so now we've gone and digitized and photographed everything written in Balinese. [? What's your?] [? how??] [? OK,?] [? don't.?] [? But?] [? there?] [? put?] [? all my?] [? back down??] When we ask them, how do you read your palm leaves? They say, well, most people don't read. It's either the priests or there are these cool performances that are their culture. So shadow puppets or performances. And so we started videotaping these and starting to make them go online. So I'd like to just give a round of applause for the Balinese to be the first culture to go completely online. Don't you think we should do this with Turkish or Dutch or Danish materials? We can basically go and do this in such a way that their businesses still work and still come up right. So scanning centers, we're doing about 1,000 books every day in these scanning centers all over the world. We've got about 3 million free ebooks that are public domain. And we have modern books that are available for the blind and dyslexic, modern, meaning probably in copyright. But we're also doing a lending system. OK, here's hack number one on how to go and get things available to people, even though they're in copyright. So we try to buy books from publishers so that we can lend them one person at a time. The publishers in general have said no so far. So we've been digitizing our books, and we still lend them one person at a time. So where Google got into trouble by getting into lawsuits and the like, we've gotten into no lawsuits. And so the way that this works is you can go to openlibrary.org, click on a book, say this HTML5 for beginners, which is actually a book that we bought. But you'd see that it's checked out by somebody. So then you have to put it on your wait list. But if you go for a less popular book, say like this history of Mayflower descendants from the Boston Public Library, surprise, nobody's checked it out. OK, so you can go and say, OK, I want to borrow this book. You have a choice of formats, and then you borrow this book. And another thing that's cool about this is it's borrowing it from the Boston Public Library. So these real libraries are digitizing books that are in copyright, non-rights-cleared books. Digitizing them and lending them just like we are a library. And this has been going on for four years, and it's been just fine. So it's a mechanism of trying to be respectful of those that are trying to make money off of this stuff, but still having access and having it happen. And we've been trying out this whole approach of how far can you go and working with publishers, but not working for them, and basically building a library system and making it work. So we've been able to get books by the hundreds of thousands that are current books and make them available. OK, books. Let's go on to another media type, music. So what if we want it? This is an area that has more lawyers than business people, it seems. I mean, this is an area that just people like to sue each other in the whole music area. So we've had to be a little bit more careful about how we've gone about this. And we first started with rock and roll bands that wanted to be distributed. So it turns out the Grateful Dead started a tradition of allowing people to record their concerts and then trade them on cassettes with other people as long as no one made any money. That's been a key thing that I've found in all of this, is no one made any money. And so as these moved online, the bands were up for being distributed online. So we asked some level of permission. And it's usually the fans going and saying, is it OK to put your concerts on the archive? And somebody has to say yes. Maybe it's the drummer or somebody in that community says yes. This is a lot less than what lawyers would like with signatures and all of that stuff. It's like, nah, is it OK? Yeah. And if it ever becomes not OK, then we take it back down again. But you know, that's only happened once. And we now have 6,000 bands up there and 130,000 concerts. And everything the Grateful Dead's ever done. So the idea of getting music up there and out there is starting to work as well. There are these old collections that were on old websites. Before, mp3.com was a format that was standardized. It was the battle days of AIFF. Anybody remember AIFF? Yeah, bad news. Anyway, there were these sites that were trying to go and do these and distribute music. One of them was the Internet Underground Music Archive. And so they died long time ago. And so we're now up. We have them up. We have got lots of net labels that are using us for free hosting. And we've been working along with lots of different record producers now to start to go and do CD digitization. And there's some engineers actually here in Amsterdam that have been doing CD digitization software to try to help get all of this stuff to work well. And we're starting to get donations of LP's 78 RPM records and the like. And starting to do mass digitization of these different formats. So why do this? Well, we haven't just gone and put everything up on the net. We'd like to maybe do 30 seconds and then point to Amazon.com or something like that. But so far what we've done is made it available to researchers and the like and listening rooms so that at least on campus you can have full access to it. So we're starting to get better at music and getting better at the whole areas of music. So we've now added to our collections of other audio recordings that are freely hosted on the internet. So the idea of having infinite storage, infinite bandwidth forever for free for some communities is a very compelling offer. So basically make things available and put them up. So even audio is doable. Moving images. Most people think of movies as Hollywood films. And we're not that good at collecting this stuff yet. So mostly we've been doing old films that haven't been particularly distributed like through Hollywood. Those old films you saw in high school when they had a substitute teacher, they'd wheel in this projector and they'd show you why to be a typesetter or these old, are you ready for marriage? Anyway, these. So we've digitized these and made them available. And people love them. I'm not quite sure why. But they're there and we've been making these things available and people have been uploading things long before YouTube. YouTube really ran away with the whole area of video hosting. They own it. But there's still people about 1,000 a day, people putting things up in the internet archive because they want them maybe more permanent or some other reason. So we've been doing digitization. Even VHS tapes, which almost all have rights problems. But we find if we do ones that aren't on DVD, nobody gets mad at us. So we're getting better at digitizing. Even television is doable. We started archiving 20 channels of television in the year 2000, Russian, Chinese, Japanese, Iraqi, Al Jazeera, BBC, CNN, 24 hours a day, DVD quality. The idea is to at least hold on to it. And so we've got 9-11 collections. But we're now starting to lend television. So if you go to archive.org, you can basically go and search on what people said. If it's in US television news, because it's the only thing that we've got the closed captions, that's the transcripts of what people said. And you can type in and go and say, I want to see things about Edward Snowden and find all of the clips that have Edward Snowden in them. And you can then take those clips and put them in party rear blog, or request a DVD of the whole program if you want to make a documentary. And this is working. Even though it's an enormous amount of materials, the publishers, the networks, are happy about this. So we're actually able to make steps of making things widely available. We want everyone to be a John Stuart research department. Like the Comedy Central, where they go and say, here's what a politician said before, and now here's what they said now, and doesn't match. That type of thing, to have people think critically about television. So even moving images is doable. So if you do this kind of thing that we're doing, to go and offer free hosting, sometimes you attract attention from people you don't like. So the FBI gave us one of these nasty letters called a national security letter. A national security letter is when they demand information about our patrons of the internet archive, our users of the internet archive. And we can't even say to anybody that we've ever gotten this request. So we got one of these things. And we got our lawyers, the Electronic Frontier Foundation. Oh, hooray for the Electronic Frontier Foundation. I said, what can we do about this? And they said, well, you have to comply. Well, can I talk to my board about it? No. Can I talk to anybody about it? No. Can I ever talk to anybody about it? No. What happens if I don't do it? Jail. Is there anything we can do? No. Really? Well, you can sue the United States government. So we sue the United States government. And one. So there have been hundreds of thousands of these letters, hundreds of thousands of these letters sent out. There have been only three organizations that have publicly gone and pushed back on the government. And they've been all libraries. What's great about being a library is you're allowed to go and say, no. There's a long history of people being rounded up for what it is they've read and bad things happening to them. And people remember this. And so where Google doesn't have, at least publicly, hasn't said no, libraries are sort of our role makes it so that it's not an embarrassing thing to do. So we find that being a library is a good thing. Software. So there's a lot of software out there. And we're getting better at going and reproducing this software by running emulators in the browser. This is a real mind blow of taking a C emulators of old Apple or Commodore or Atari software and cross compiling it within script and into JavaScript. And it runs in your browser. So you click and it actually boots an old IBM PC in your browser and you're running your old game. It turns out this is very, very popular. Because I guess a lot of people spend their early days playing games. But anyway, they're now back. So Oregon Trail and all these other games are very popular. What we're probably best known for is crawling the worldwide web. So how many people have used the Wayback machine? Yay. So the Wayback machine is a way you can see the web as it was. That so many people are pouring their lives into the web. But web pages only on average last 100 days. So we go through and we try to archive them. And we started when it was pretty small. And it's now getting pretty freaking big. We archive about a billion pages every week to be able to create this Wayback machine. This is what Yahoo looked like in 1996. Pets.com with the little sock guy, dorky old web design. And I looked up what the, I thought, oh, why don't I go and look at what Chaos Communications Camp looked like. So this is the Chaos website from 1997. But it actually looks a whole hell of a lot like the current one. So there's a little retro thing going on. So it's not quite as dramatic. There's another thing that this has been used for. A user came back and said, hey, Brewster, there's been a change that your web collection has the only place that can show it. This is a press release by the United States government of the president being on an aircraft carrier saying, mission accomplished about war in Iraq. And it says that the president announces combat operations in Iraq have ended. Then a couple days later, they changed it. And they put in major combat operations have changed. They didn't make any notice that they changed a press release. It's George Orwell, right? To be able to go and redo press releases from the past. It's living in the day. So now this is an example of why we want something like a Wayback machine. Another is we have a web archiving tool that's a subscription-based service that a lot of companies and libraries and museums pay us to do, which is helpful. It keeps our lights on. And we now have 1,700 curated collections, like the Japanese disaster, where people have come together to go and say, you should archive this because soon it's offline. Or archive these things soon they're offline. So it's been a community project to be able to go and work together to build up what are the key things about an event to make sure that we really have it done well. So even web is doable. So the worldwide web collection of ours is probably about 10 petabytes of data. It's growing at a few petabytes every year. We have about 450 billion web pages. We get about 600,000 people a day using it. It's a database of 450 billion pages that gets queried about 2,000 times a second. And so that's sort of what the Wayback machine is. And it's been much more popular than we thought it would be, which has just been great. But even rare books and letters, you can go and do these things by photographing and breaking them available. Next up for us is personal digital archives. How are we going to do this stuff that's splintered on all sorts of places? So people don't even have them. It's not like boxes in people's basements anymore. It's not even hard drives that you have. It's these Flickr sites and these other places that you've gone and put your memories and guaranteed these guys are going down. Even the rich companies like Google, Google Video, ever heard of Google Video? Well, it used to exist. There were 6 million videos on it, but they took it away. Yahoo Video is now gone. GeoCities is famously gone. Apple Computer, the most valuable company in the whole world, couldn't figure out how to run 200 terabytes of mobile me on a continuous basis. So we archive that and making it available. So don't count on these places. They don't have your best interest in mind. They'll turn it off whenever they want to. So how do you go and preserve this stuff, both the physical and the digital? If we want to build a library of Alexandria version 2, well, what's the lesson out of the library of Alexandria version 1? What's it best known for? Burning. It's best known for not being here anymore. So how do we go and make it so that we don't do that again? Well, let's have multiple copies. If we put multiple copies in multiple places that will have different fault modes, then I think we have a better shot at it. So we gave a copy back in 2002 to the new library of Alexandria. And this is actually, they redid their first floor. If you get to Alexandria, go. It's a completely great city and a library. And so this is what it looked like in 2002. Access for All gave us free space, and they have for the last 10 years. Thank you, Access for All. So there's a partial copy in Amsterdam. This is what it looked like in 2008. This is what the Wayback Machine. Oh, we came up with this idea of doing data centers in shipping containers. Sun ran with it, and they gave us one of these. So the Wayback Machine was this. And so, oh, I get to ask, how big is the web? You have to ask me, how big is the web? The web is 8 feet by 8 feet by 20 feet. For several years, when you use the Wayback Machine, you are actually using this shipping container that sat outside on Sun's campus, which is pretty great. We've now made these prettier machines because we bought this cool church. And so there are these blinking lights inside. That's just how we've been able to scale up. We've also started to scale up physical collections, because libraries are throwing things away. So we've basically taken our ideas of how to do really compact, and we've used shipping containers inside warehouses. So we have books or music or video that are in boxes, protected by a box, protected by a shipping container, protected by a warehouse, protected by nonprofits. So the idea is to try to have layers of protection against certain types of attacks. But we're not there yet. Because, OK, we've got a copy in an earthquake zone, the Middle East, and a flood zone. What could go wrong? So I think we need some other copies in other places and more participation towards keeping our whole society and cultural heritage safe. A couple other things that we're worried about at the Internet Archive is some of the ways of trying to be sustainable in the era of corporations. Corporations are basically becoming real strangleholds on certain types of things, like end-user access. So we've been doing free public Wi-Fi and the like. So every time we get a building, we put free Wi-Fi with no passwords and stuff up. And that causes people to get free Wi-Fi, in some cases, and mad in other cases, both of which are perfectly fine with us. We've even gone and tried to apply open source ideas to housing. Housing has become a real problem because of the debt burden. So we're starting to try to transition housing to be able to support nonprofit workers that are debt-free. If you're interested in this idea, we're really trying to figure out some of these things. If you're going to go and build a new housing system for people that are working in the open source world, then you're going to need an organization like financial institutions. We started a credit union and been trying to work with some Bitcoin companies. Oh, did that make regulators mad? Anyway, so there's the idea of to take not just trying to make the data sustainable and copying it forward, but how do you go and get the communities around these materials to be able to have lifetimes that work? We're way into Bitcoin. People have been donating Bitcoin. If you've donated Bitcoin, then thank you very much. We pay our employees partially in Bitcoin, and that's all around working. Project that needs help. There's a lot. Everything needs help. But here's a few of my top. We're redesigning the Wayback machine and could really use some help on trying to make this new and different and neat. How do you do search at a level that we can actually do? Because searching 450 billion pages is too much for us. But maybe site-based search and the like. We're trying to get elastic search to be able to make all of our stuff more accessible, the full text much easier to use. And we're trying to build software that can be distributed, that people can archive their CDs, LPs, and books in a distributed way to go and participate in bringing these things online. There's a couple of programmers in Amsterdam looking for more programmers to help. And the distributed web is sort of a new idea that if there's some mechanism to go and weave it together to go and make a next-generation web that we don't have to archive by taking snapshots, we can actually archive working websites so that they'll live on for tens or hundreds of years after when the original administrators are gone. So in conclusion, universal access to all knowledge. It's possible to do. I think it could be one of the greatest things humans have ever done. I think it could be one of the things that our generation gets to go and offer the world, kind of like the man on the moon or the library of Alexandria in Asians' days, that we can pull this together. We have the technology to be able to pull it off. We've got the political will to live in an open environment as long as we don't lose it. And we have to act pretty fast, I'd say, because most kids have turned to screens instead of books or old materials, be able to find out information. They're learning from whatever it is they could get a hold of. And the best that we have to offer is not on the net yet. So we need to be bolder than we've been to go and make universal access to all knowledge happen. Thank you very much. Thank you. I'm sorry. No. I'm sorry. So I'm going to be hanging out. I'm at the food hacking camp, and I'm just going to be hanging out here. But thank you very much for coming. Have a great day. Is this? Yeah, it's on now. I'm very sorry that we didn't have time. OK. One more for Bruce.
talk about basic. You can think of the Internet software ever created by humans available to access to all knowledge is within our grasp, and Internet Archive is a nonprofit library. Is this these scanning centers. This is the one in San locking up the public domain, the public domain scanning centers all over the world. We've got for them, and basically building a library system there and out there is starting to work as well. together to build up what are the key things worldwide web collection of ours is probably petabytes every year. We have about 450 billion days, that we can pull this together. We have the