As part of the Epsilon DMS Data Organization, the Data Pipeline team's responsibilities include but is not limited to maintaining streaming and batched data pipelines collecting hundreds of billions of data rows daily, near real time aggregation of a variety of data sets (using Spark Structured Streaming) and maintaining system of record raw and aggregate data sets (HDFS and HIVE)