62758879
Exploiting Underlying Structure for Detailed Reconstruction of an Internet-scale Event
Network "telescopes" that record packets sent to unused blocks of Internet address space have emerged as an important tool for observing Internet-scale events such as the spread of worms and the backscatter from flooding attacks that use spoofed source addresses. Current telescope analyses produce detailed tabulations of packet rates, victim population, and evolution over time. While such cataloging is a crucial first step in studying the telescope observations, incorporating an understanding of the underlying processes generating the observations allows us to construct detailed inferences about the broader "universe" in which the Internet-scale activity occurs, greatly enriching and deepening the analysis in the process. In this work we apply such an analysis to the propagation of the Witty worm, a malicious and well-engineered worm that when released in March 2004 infected more than 12,000 hosts world-wide in 75 minutes. We show that by carefully exploiting the structure of the worm, especially its pseudo-random number generation, from limited and imperfect telescope data we can with high fidelity: extract the individual rate at which each infectee injected packets into the network prior to loss; correct distortions in the telescope data due to the worm's volume overwhelming the monitor; reveal the worm's inability to fully reach all of its potential victims; determine the number of disks attached to each infected machine; compute when each infectee was last booted, to sub-second accuracy; explore the "who infected whom" infection tree; uncover that the worm specifically targeted hosts at a US military base; and pinpoint Patient Zero, the initial point of infection, i.e., the IP address of the system the attacker used to unleash Witty.
2987975
Data streaming algorithms for efficient and accurate estimation of flow size distribution
Knowing the distribution of the sizes of traffic flows passing through a network link helps a network operator to characterize network resource usage, infer traffic demands, detect traffic anomalies, and accommodate new traffic demands through better traffic engineering. Previous work on estimating the flow size distribution has been focused on making inferences from sampled network traffic. Its accuracy is limited by the (typically) low sampling rate required to make the sampling operation affordable. In this paper we present a novel data streaming algorithm to provide much more accurate estimates of flow distribution, using a "lossy data structure" which consists of an array of counters fitted well into SRAM. For each incoming packet, our algorithm only needs to increment one underlying counter, making the algorithm fast enough even for 40 Gbps (OC-768) links. The data structure is lossy in the sense that sizes of multiple flows may collide into the same counter. Our algorithm uses Bayesian statistical methods such as Expectation Maximization to infer the most likely flow size distribution that results in the observed counter values after collision. Evaluations of this algorithm on large Internet traces obtained from several sources (including a tier-1 ISP) demonstrate that it has very high measurement accuracy (within 2%). Our algorithm not only dramatically improves the accuracy of flow distribution measurement, but also contributes to the field of data streaming by formalizing an existing methodology and applying it to the context of estimating the flow-distribution.
18222685
XCHOKe: malicious source control for congestion avoidance at Internet gateways
The paper describes an algorithm to control flows from nonadaptive sources in the Internet in a simple and lightweight fashion. The algorithm, called XCHOKe, is an extension to CHOKe and RED. The paper presents experimental results from simulations that demonstrate that the algorithm outperforms current well-known algorithms for buffer management in providing better protection for TCP flows under varying degrees of attack from non-adaptive flows.
7207895
Space-code bloom filter for efficient traffic flow measurement
Per-flow traffic measurement is critical for usage accounting, traffic engineering, and anomaly detection. Previous methodologies are either based on random sampling (e.g., Cisco's NetFlow), which is inaccurate, or only account for the "elephants". Our paper introduces a novel technique for measuring per-flow traffic approximately, for all flows regardless of their sizes, at very high-speed (say, OC192+). The core of this technique is a novel data structure called Space Code Bloom Filter (SCBF). A SCBF is an approximate representation of a multiset; each element in this multiset is a traffic flow and its multiplicity is the number of packets in the flow. SCBF employs a Maximum Likelihood Estimation (MLE) method to measure the multiplicity of an element in the multiset. Through parameter tuning, SCBF allows for graceful tradeoff between measurement accuracy and computational and storage complexity. SCBF also contributes to the foundation of data streaming by introducing a new paradigm called blind streaming. We evaluated the performance of SCBF on packet traces gathered from a tier-1 ISP backbone and through mathematical analysis. Our preliminary results demonstrate that SCBF achieves reasonable mea-surement accuracy with very low storage and computational complexity.
2583878
A data streaming algorithm for estimating subpopulation flow size distribution
Statistical information about the flow sizes in the traffic passing through a network link helps a network operator to characterize network resource usage, infer traffic demands, detect traffic anomalies, and improve network performance through traffic engineering. Previous work on estimating the flow size distribution for the complete population of flows has produced techniques that either make inferences from sampled network traffic, or use data streaming approaches. In this work, we identify and solve a more challenging problem of estimating the size distribution and other statistical information about arbitrary subpopulations of flows. Inferring subpopulation flow statistics is more challenging than the complete population counterpart, since subpopulations of interest are often specified a posteriori (i.e., after the data collection is done), making it impossible for the data collection module to "plan in advance".Our solution consists of a novel mechanism that combines data streaming with traditional packet sampling to provide highly accurate estimates of subpopulation flow statistics. The algorithm employs two data collection modules operating in parallel --- a NetFlow-like packet sampler and a streaming data structure made up of an array of counters. Combining the data collected by these two modules, our estimation algorithm uses a statistical estimation procedure that correlates and decodes the outputs (observations) from both data collection modules to obtain flow statistics for any arbitrary subpopulation. Evaluations of this algorithm on real-world Internet traffic traces demonstrate its high measurement accuracy.
7308298
Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices
The traffic volume between origin/destination (OD) pairs in a network, known as traffic matrix, is essential for efficient network provisioning and traffic engineering. Existing approaches of estimating the traffic matrix, based on statistical inference and/or packet sampling, usually cannot achieve very high estimation accuracy. In this work, we take a brand new approach in attacking this problem. We propose a novel data streaming algorithm that can process traffic stream at very high speed (e.g., 40 Gbps) and produce traffic digests that are orders of magnitude smaller than the traffic stream. By correlating the digests collected at any OD pair using Bayesian statistics, the volume of traffic flowing between the OD pair can be accurately determined. We also establish principles and techniques for optimally combining this streaming method with sampling, when sampling is necessary due to stringent resource constraints. In addition, we propose another data streaming algorithm that estimates flow matrix, a finer-grained characterization than traffic matrix. Flow matrix is concerned with not only the total traffic between an OD pair (traffic matrix), but also how it splits into flows of various sizes. Through rigorous theoretical analysis and extensive synthetic experiments on real Internet traffic, we demonstrate that these two algorithms can produce very accurate estimation of traffic matrix and flow matrix respectively.
6238404
Fixed point analysis of single cell IEEE 802.11e WLANs: uniqueness, multistability and throughput differentiation
We consider the vector fixed point equations arising out of the analysis of the saturation throughput of a single cell IEEE 802.11e wireless local area network with nodes that have different back-off parameters, including different Arbitration InterFrame Space (AIFS) values. We consider balanced and unbalanced solutions of the fixed point equations arising in homogeneous and nonhomogeneous networks. We are concerned, in particular, with (i) whether the fixed point is balanced within a class, and (ii) whether the fixed point is unique. Our simulations show that when multiple unbalanced fixed points exist in a homogeneous system then the time behaviour of the system demonstrates severe short term unfairness (or multistability). Implications for the use of the fixed point formulation for performance analysis are also discussed. We provide a condition for the fixed point solution to be balanced within a class, and also a condition for uniqueness. We then provide an extension of our general fixed point analysis to capture AIFS based differentiation; again a condition for uniqueness is established. An asymptotic analysis of the fixed point is provided for the case in which packets are never abandoned, and the number of nodes goes to ∞. Finally the fixed point equations are used to obtain insights into the throughput differentiation provided by different initial back-offs, persistence factors, and AIFS, for finite number of nodes, and for differentiation parameter values similar to those in the standard.
17904994
Using omnidirectional structure from motion for registration of range images of minimal overlap
We propose a novel method of merging a series of range images with a minimal overlap between any two consecutive range images. We rigidly mount a parabolic catadioptric camera to the range scanner. Using two omniviews we are able to accurately estimate the relative displacement between two range views. The resultant motion is used for the registration of all range data to the same coordinate system. An additional perspective camera calibrated with respect to the scanner is used for texture mapping.
14310908
HPP: A Reliable Causal Broadcast Protocol for Large-Scale Replication
This paper describes a fast, reliable, scalable and efficient broadcast protocol called HPP (hierarchical propagation protocol) for weak-consistency replica management. It is based on organizing the nodes in a network into a logical hierarchy and maintaining a limited amount of state information at each node. It ensures that messages are not lost due to failures or partitions and minimizes redundancy. Furthermore, the protocol allows messages to be diffused while nodes are down provided the parent and child nodes of a failed node are alive. Moreover, the protocol allows nodes to be moved in the logical hierarchy and the network to be restructured dynamically in order to improve performance, while still ensuring that no messages are lost while the switch takes place and without disturbing normal operation. A performance study of the protocol in terms of availability and propagation delay indicates that the protocol reduces the delay by a factor of four compared to a protocol that does not diffuse messages past failed nodes.
1630993
Join-distinct aggregate estimation over update streams
There is growing interest in algorithms for processing andquerying continuous data streams (i.e., data that is seenonly once in a fixed order) with limited memory resources.Providing (perhaps approximate) answers to queries over suchstreams is a crucial requirement for many application environments;examples include large IP network installations where performancedata from different parts of the network needs to be continuouslycollected and analyzed.
The ability to estimate the number of distinct (sub)tuples inthe result of a join operation correlating two data streams (i.e.,the cardinality of a projection with duplicate elimination over ajoin) is an important requirement for several data-analysisscenarios. For instance, to enable real-time traffic analysis andload balancing, a network-monitoring application may need toestimate the number of distinct (<i>source</i>,destination) IP-address pairs occurring in the stream of IP packetsobserved by router <i>R</i><inf>1</inf>,where the source address is also seen in packets routed through adifferent router <i>R</i><inf>2</inf>.Earlier work has presented solutions to the individual problems ofdistinct counting and join-size estimation (without duplicateelimination) over streams. These solutions, however, arefundamentally different and extending or combining them to handleour more complex "Join-Distinct" estimation problem is far fromobvious. In this paper, we propose the <i>first</i>space-efficient algorithmic solution to the general Join-Distinctestimation problem over continuous data streams (our techniques canactually handle general <i>update streams</i>comprising tuple deletions as well as insertions). Our estimatorsare probabilistic in nature and rely on novel algorithms forbuilding and combining a new class of hash-based synopses (termed"JD <i>sketches</i>") for individual update streams. Wedemonstrate that our algorithms can provide low error,high-confidence Join-Distinct estimates using only small space andsmall processing time per update. In fact, we present lower boundsshowing that the space usage of our estimators is within smallfactors of the best possible for the Join-Distinct problem.Preliminary experimental results verify the effectiveness of ourapproach.
208013096
New insights from a fixed point analysis of single cell IEEE 802.11 WLANs
We study a fixed point formalisation of the well known analysis of Bianchi. We provide a significant simplification and generalisation of the analysis. In this more general framework, the fixed point solution and performance measures resulting from it are studied. Uniqueness of the fixed point is established. Simple and general throughput formulas are provided. It is shown that the throughput of any flow will be bounded by the one with the smallest transmission rate. The aggregate throughput is bounded by the reciprocal of the harmonic mean of the transmission rates. In an asymptotic regime with a large number of nodes, explicit formulas for the collision probability, the aggregate attempt rate and the aggregate throughput are provided. The results from the analysis are compared with ns2 simulations, and also with an exact Markov model of the back-off process. It is shown how the saturated network analysis can be used to obtain TCP transfer throughputs in some cases.