The Coral Content Distribution Network

News

20 Aug 2012: We're still here! While active development has been stopped for a while, we continue to operate CoralCDN as an open, free service. It's now been running continuously for more than 8 years (since March 2004), and continues to get a few million users per day at last check. Enjoy continuing to use the service!

11 Jul 2012: CoralCDN used with special URL shortener during the 2011 Japan earthquake/tsunami. Read more on the ACM's Digital Library.

Overview

The availability of content on the Internet is to a large degree a
function of the cost shouldered by the publisher. A well-funded web
site can reach huge numbers of people through some combination of
load-balanced servers, fast network connections, and commercial
content distribution networks (CDNs). Publishers who cannot afford
such amenities are limited in the size of audience and type of content
they can serve. Moreover, their sites risk sudden overload following
publicity, a phenomenon nicknamed the ``Slashdot'' effect, after a
popular web site that periodically links to under-provisioned servers,
driving unsustainable levels of traffic to them. Thus, even
struggling content providers are often forced to expend significant
resources on content distribution.

Fortunately, at least with static content, there is an easy way for
popular data to reach many more people than publishers can afford to
serve themselves---volunteers can mirror the data on their own servers
and networks. Indeed, the Internet has a long history of
organizations with good network connectivity mirroring data they
consider to be of value. More recently, peer-to-peer file sharing has
demonstrated the willingness of even individual broadband users to
dedicate upstream bandwidth to redistribute content the users
themselves enjoy. Additionally, organizations that mirror popular
content reduce their downstream bandwidth utilization and improve the
latency for local users accessing the mirror.

What CoralCDN offers

CoralCDN is a decentralized, self-organizing,
peer-to-peer web-content distribution network. CoralCDN leverages the
aggregate bandwidth of volunteers running the software to absorb and
dissipate most of the traffic for web sites using the system. In so
doing, CoralCDN replicates content in proportion to the content's
popularity, regardless of the publisher's resources---in effect
democratizing content publication.

To use CoralCDN, a content publisher---or someone posting a link to
a high-traffic portal---simply appends
.nyud.net
to the hostname in a URL. Through DNS redirection, oblivious clients
with unmodified web browsers are transparently redirected to nearby
Coral web caches. These caches cooperate to transfer data from
nearby peers whenever possible, minimizing both the load on the origin
web server and the end-to-end latency experienced by browsers.

CoralCDN is built on top of a novel key/value indexing
infrastructure called Coral. Two properties make Coral ideal
for CDNs. First, Coral allows nodes to locate nearby cached copies
of web objects without querying more distant nodes. Second, Coral
prevents hot spots in the infrastructure, even under degenerate loads.
For instance, if every node repeatedly stores the same key, the rate
of requests to the most heavily-loaded machine is still only
logarithmic in the total number of nodes.

Coral exploits overlay routing techniques recently popularized by a
number of peer-to-peer distributed hash tables (DHTs). However, Coral
differs from DHTs in several ways. First, Coral's locality and hot-spot
prevention properties are not possible for DHTs. Second, Coral's
architecture is based on clusters of well-connected machines.
Clusters are exposed in the interface to higher-level software, and in
fact form a crucial part of the DNS redirection mechanism. Finally,
to achieve its goals, Coral provides weaker consistency than
traditional DHTs. For that reason, we call its indexing abstraction a
distributed sloppy hash table or DSHT.

CoralCDN makes a number of contributions. It enables people to
publish content that they previously could not or would not because of
distribution costs. It is the first completely decentralized and
self-organizing web-content distribution network. Coral, the
indexing infrastructure, provides a new abstraction potentially of use
to any application that needs to locate nearby instances of resources
on the network. Coral also introduces an epidemic clustering
algorithm that exploits distributed network measurements.
Furthermore, Coral is the first peer-to-peer key/value index that
can scale to many stores of the same key without hot-spot congestion,
thanks to a new rate-limiting technique. Finally, CoralCDN
contains the first peer-to-peer DNS redirection infrastructure,
allowing the system to inter-operate with unmodified web browsers.