Jujuby: Design and Deployment of a Crawler for Twitch CDN Mapping

Content Delivery Networks (CDNs) deliver much of the world’s content on the Internet today. As demand for content rises, the role of CDNs is as crucial as ever, making it imperative to understand the internal workings of such systems. Focusing on Twitch, one of the most popular live video services, we present in this work (1) the design of a Twitch CDN crawler, (2) two deployment methodologies, (3) a study of the data collected, and (4) a comparison of the deployment methodologies. Towards a Twitch CDN crawler, we sniffed the traffic in a Twitch session. A closer look into the traffic revealed that a load-balancer, named Usher, is in charge of redirecting the clients to their appropriate video servers. This enabled us to design and implement a programmable crawler that can be deployed to discover Twitch’s CDN at a large scale. Previous works have deployed their crawlers on platforms such as PlanetLab and the global Web proxies for broad network coverage. As these platforms become less available, we experimented on two fee-based platforms, i.e., the cloud computing and the virtual private network (VPN) services, for better availability into the future. By running our crawler on the two platforms, we collected data showing that Twitch serves its users in geographically wide but regionally concentrated clusters. These clusters are located primarily in North America, Europe, Asia, and Oceania. We were able to compare as well how the two crawling platforms are suitable for different crawling purposes.