Meeting Coflow Deadlines in Data Center Networks With Policy-Based Selective Completion

—Recently, the abstraction of coﬂow is introduced to capture the collective data transmission patterns among modern distributed data-parallel applications. During processing, coﬂows generally act as barriers; accordingly, time-sensitive applications prefer their coﬂows to complete within deadlines, and deadline- aware coﬂow scheduling becomes very crucial. Regarding these data-parallel applications, we notice that many of them, including large-scale query systems , distributed iterative training , and erasure codes enabled storage , are able to tolerate loss-bounded incomplete inputs by design. This tolerance indeed brings a ﬂexible design space for the schedule of their coﬂows: when getting overloaded, the network can trade coﬂow completeness for the timeliness, and balance the completeness of different coﬂows on demand. Unfortunately, existing coﬂow schedulers neglect this tolerance, resulting in inﬂexible and inefﬁcient bandwidth allocations. In this paper, we explore this fundamental trade-off and design P OCO , a POlicy-based COﬂow scheduler, along with a transport layer enhancement scheme, to achieve customizable selective coﬂow completion for emerging time-sensitive distributed applications. Internally, P OCO employs a suite of novel designs along with admission controls to make ﬂexible , work-conserving , and performance-guaranteed rate allocation to online coﬂow requests very efﬁciently. Extensive trace-based simulations indicate that P OCO is highly ﬂexible and achieves optimal coﬂow schedules respecting the requirements speciﬁed by applications. transport layer enhancement enables P OCO to tolerate the impacts of foreground and background trafﬁc greatly. For instance, with and without protocol enhancement designs, the size of a burst foreground ﬂow that the P OCO -enabled concurrent model training can tolerate are about 27MB and 1.3MB, respectively, yielding a performance gap larger than 20 × in this case.