Support DistributedDataParallel

Closes #116 (closed)

With this fix, we can

• Train on multiple GPUs
• Continue a training started with 1 GPU on multiple GPUs
• Continue a training started with multiple GPUs on 1 GPU
• Load any pre-trained model (trained with 0, 1 or more GPUs)

It also closes #143 (closed) and #118 (closed)

changed milestone to %DAN-P5: Improve training

added P2 label

assigned to @mboillet

Before merging, I need to

• Add more details in the documentation
• Train a model in real conditions with multiple GPUs on Jean Zay

added 2 commits

• 32ce7f17 - 1 commit from branch main
• c553bbe0 - Support DistributedDataParallel

Compare with previous version

changed the description

added 1 commit

• 69461e35 - Document training on multiple GPUs

Compare with previous version

The training with 2 GPUs works as expected, similar to training with a single GPU.

requested review from @yschneider

added 4 commits

• 69461e35...d38bab3e - 2 commits from branch main
• 2aaf98aa - Support DistributedDataParallel
• e528e935 - Document training on multiple GPUs

Compare with previous version

marked this merge request as ready

added 1 commit

• c4a9f9a4 - Add fix_ddp_layers_names function

Compare with previous version

added 4 commits

• 0773855e - 1 commit from branch main
• bc660a06 - Support DistributedDataParallel
• 27001cba - Document training on multiple GPUs
• 41872136 - Add fix_ddp_layers_names function

Compare with previous version

resolved all threads

Also Closes #118 (closed)

approved this merge request

enabled an automatic merge when the pipeline for 41872136 succeeds

changed the description

merged