DeepSpeed
A distributed training library whose ZeRO optimizer shards model states across devices; standardizing on it determines which parallelism layouts your stack can express without rewriting models.
A distributed training library whose ZeRO optimizer shards model states across devices; standardizing on it determines which parallelism layouts your stack can express without rewriting models.