Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. 
 
 What’s in the data set: 
 
 
 
 Crawl start date: 09 March, 2011 
 
 Crawl end date: 23 December, 2011 
 
 Number of captures: 2,713,676,341 
 
 Number of unique URLs: 2,273,840,159 
 
 Number of hosts: 29,032,069 
 
 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites. 
 
 However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed. 
 
 We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content. 
 
 If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered. ||||| Tammar wallaby young spend nine months in their mother's pouch 
 
 Researchers have laid bare the DNA of a kangaroo species for the first time. 
 
 An international team of scientists, writing in the Biomed Central journal, Genome Biology, say they have even indentified a gene responsible for the kangaroo's hop. 
 
 The group focussed on a small species of kangaroo that inhabits islands off Australia's south and western coasts. 
 
 The tammar wallaby (Macropus eugenii) is only the third marsupial to have its genome sequenced. 
 
 Making up the trio are the Tasmanian devil and the South American opossum. 
 
 The team says the first kangaroo genome is a milestone in the study of mammalian evolution. The ancestors of kangaroos and other marsupials diverged from other mammals at least 130 million years ago. 
 
 Professor Marilyn Renfree of the University of Melbourne, a lead researcher on the project, said: "The tammar wallaby sequencing project has provided us with many possibilities for understanding how marsupials are different from us." 
 
 Key marsupial traits 
 
 Dr Elizabeth Murchison, a marsupial specialist at the Wellcome Trust Sanger Institute in Cambridge, UK, described the study as "a wonderful tool for studying the evolution of marsupials, and mammals in general, and an impressive piece of work looking at one of Australia's iconic species." 
 
 Aside from identifying the "hop" gene, the researchers pinpointed the genes responsible for other key marsupial traits. 
 
 For example, tammar wallaby young are only the size of a grain of rice when born. They spend the next stage of their development in their mother's pouch, feeding on her milk. 
 
 The pouch is external to the mother's body and therefore the developing animal comes under attack from many pathogens. Antibiotics in the mother's milk are key to the survival of the offspring. 
 
 This is only the third marsupial genome to be sequenced 
 
 Dr Murchison told BBC News: "It's always really valuable to look at an organism that is a bit different to understand how humans and other mammals have evolved. 
 
 "It gives you a perspective on mammalian evolution by looking at mammals that have diverged fairly early on, like the kangaroo. 
 
 The researchers also studied the way tammar wallaby genes are turned "on" or "off" at different stages of the animal's life cycle and in different parts of the body. 
 
 They hope their work may help produce future treatments for human disease. 
 
 "This is the first genome project to be led by Australian scientists," commented Dr Murchison. 
 
 "It's been a huge project. It's been going on for almost my entire career. I'm from Australia and it was going on when I was an undergraduate there 10 years ago. 
 
 "It was a very early conceived genome project and it has evolved through various different stages as genome sequencing technologies have changed a lot. The team has adapted and kept up and so they have a huge amount of data." e.
– What makes the kangaroo hop? That’s just one of the questions answered by international researchers who’ve decoded the genome of a kangaroo species, the BBC reports. The genome research team—the first to be led by Australian scientists—sequenced the genome in 2008 but finally completed its analysis of it and published those findings last week. "It's been a huge project," says one scientist. "I'm from Australia and it was going on when I was an undergraduate there 10 years ago." The decoding gives scientists a clearer picture of mammalian evolution and could help cure diseases, both in humans and other mammals. Kangaroo milk has antibiotic properties, and researchers have copied it to synthesize a protein that kills bacteria already resistant to antibiotics, the Australian reports. The findings could lead to a vaccine to fend off a facial tumor that has ravaged Tasmanian devil populations—though such a medication could be a decade away.