Thats a unique idea.You should consider the following idea to enhance your idea so that it would impact uniquely: In the era of global-scale services, big data analytical queries are often required to process datasets that span multiple data centers (DCs). In this setting, cross-DC bandwidth is often the scarcest, most volatile, and/or most expensive resource. Hover, current widely deployed big data analytics frameworks make no attempt to minimize the traffic traversing these links.#R##N##R##N#,  present Pixida, a scheduler that aims to minimize data movement across resource constrained links. To achieve this,  introduce a new abstraction called Silo, which is key to modeling Pixida's scheduling goals as a graph partitioning problem. Furthermore,  show that existing graph partitioning problem formulations do not map to how big data jobs work, causing their solutions to miss opportunities for avoiding data movement. To address this,  formulate a new graph partitioning problem and propose a novel algorithm to solve it.  integrated Pixida in Spark and our experiments show that, when compared to existing schedulers, Pixida achieves a significant traffic reduction of up to ~ 9x on the aforementioned links.
['Konstantinos Kloudas', 'Margarida Mamede', 'Nuno M. Preguiça', 'Rodrigo Rodrigues']
I am writing a research paper on a idea Pixida: optimizing data parallel jobs in wide-area data analytics
[{"from": "human", "value": "I am writing a research paper on a idea Pixida: optimizing data parallel jobs in wide-area data analytics"}, {"from": "gpt", "value": "Thats a unique idea.You should consider the following idea to enhance your idea so that it would impact uniquely: In the era of global-scale services, big data analytical queries are often required to process datasets that span multiple data centers (DCs). In this setting, cross-DC bandwidth is often the scarcest, most volatile, and/or most expensive resource. Hover, current widely deployed big data analytics frameworks make no attempt to minimize the traffic traversing these links.#R##N##R##N#,  present Pixida, a scheduler that aims to minimize data movement across resource constrained links. To achieve this,  introduce a new abstraction called Silo, which is key to modeling Pixida's scheduling goals as a graph partitioning problem. Furthermore,  show that existing graph partitioning problem formulations do not map to how big data jobs work, causing their solutions to miss opportunities for avoiding data movement. To address this,  formulate a new graph partitioning problem and propose a novel algorithm to solve it.  integrated Pixida in Spark and our experiments show that, when compared to existing schedulers, Pixida achieves a significant traffic reduction of up to ~ 9x on the aforementioned links."}]