There are four different sharding mechanisms for large-scale embedding tables for recommendation: data parallel(DP), row-wise (RW), column-wise (CW) and tablewise (TW). However, data parallel technique requires extra memory to store gradient-related information. Can you expand on this?
user
{"evolved_from": null, "kind": "human", "source": "ewof/sharegpt-instruct-unfiltered-deduped"}
There are four different sharding mechanisms for large-scale embedding tables for recommendation: data parallel(DP), row-wise (RW), column-wise (CW) and tablewise (TW). However, data parallel technique requires extra memory to store gradient-related information. Can you expand on this?
user
DUMMY RESPONSE
assistant