Skip to content

Fix dataloader

Solene Tarride requested to merge fix-dataloader-num_workers into master

We have an issue with the worker: some processes are blocked in "Running" status for ages.

We've made some tests, and it looks like this issue only occurs when num_workers>0.

  • num_workers=0 - working fine
  • num_workers=2 - blocked
  • num_workers=4 - blocked

This is a parameter from Pytorch's dataloader:

num_workers (int, optional) – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. (default: 0)

So the aim of this MR is to :

  • set num_workers=0 by default
  • remove the key num_workers from user configuration

Merge request reports