For a large relation in a file system residing on a plurality of processing modules, if the large relation is local (for instance a table or intermediate result relation stored locally on the processing modules) or hash redistributed (for instance on equality join conditions or GROUP BY expressions), the processing modules may have an uneven (or “skewed”) number of rows to join or aggregate. This skew in the distribution of the rows in the processing modules can cause an imbalance in join or aggregation processing since some processing modules may have many more rows to process than other processing modules. The processing modules with fewer rows may finish first and be unused for further processing while the other processing modules continue processing. This leads to parallel inefficiency and a longer elapsed time to process the join or aggregation. Reducing the skew to achieve an improvement in parallel efficiency is a challenge.
USPTO Backgrounds
USPTO Backgrounds