Skip to content

Debug/remove PiecewiseAffine augmentation transform

I got the following error while training three different models (RIMES-lines, RIMES-pages, READ-lines).

This is due to the piecewise_affine augmentation. I need to find out why we are getting this error, and either:

  • restrict parameters to valid ranges
  • or remove this transform
EPOCH 95/710:  29%|██▉       | 1084/3700 [01:58<04:46,  9.12it/s, values={'loss_ce': 0.1277, 'cer': 0.0357, 'wer': 0.1623, 'wer_no_punct': 0.1553}]
Traceback (most recent call last):
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/bin/teklia-dan", line 33, in <module>
    sys.exit(load_entry_point('teklia-dan', 'console_scripts', 'teklia-dan')())
  File "/gpfsdswork/projects/rech/rxm/ubz97wr/dan_p4/dan/cli.py", line 26, in main
    status = args.pop("func")(**args)
  File "/gpfsdswork/projects/rech/rxm/ubz97wr/dan_p4/dan/ocr/document/train.py", line 275, in run
    start_training(config, mlflow_logging=False)
  File "/gpfsdswork/projects/rech/rxm/ubz97wr/dan_p4/dan/ocr/document/train.py", line 258, in start_training
    train_and_test(0, config, mlflow_logging)
  File "/gpfsdswork/projects/rech/rxm/ubz97wr/dan_p4/dan/ocr/document/train.py", line 45, in train_and_test
    model.train(mlflow_logging=mlflow_logging)
  File "/gpfsdswork/projects/rech/rxm/ubz97wr/dan_p4/dan/manager/training.py", line 570, in train
    for ind_batch, batch_data in enumerate(self.dataset.train_loader):
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 634, in __next__
    data = self._next_data()
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1346, in _next_data
    return self._process_data(data)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data
    data.reraise()
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
TypeError: Caught TypeError in DataLoader worker process 3.
Original Traceback (most recent call last):
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/gpfsdswork/projects/rech/rxm/ubz97wr/dan_p4/dan/manager/ocr.py", line 79, in __getitem__
    sample["img"] = self.augmentation_transforms(image=np.array(sample["img"]))[
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/albumentations/core/composition.py", line 359, in __call__
    data = t(force_apply=True, **data)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/albumentations/core/composition.py", line 326, in __call__
    data = t(force_apply=True, **data)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/albumentations/core/transforms_interface.py", line 118, in __call__
    return self.apply_with_params(params, **kwargs)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/albumentations/core/transforms_interface.py", line 131, in apply_with_params
    res[key] = target_function(arg, **dict(params, **target_dependencies))
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/albumentations/augmentations/geometric/transforms.py", line 948, in apply
    return F.piecewise_affine(img, matrix, self.interpolation, self.mode, self.cval)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/albumentations/augmentations/utils.py", line 68, in wrapped_function
    return clip(func(img, *args, **kwargs), dtype, maxval)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/albumentations/augmentations/geometric/functional.py", line 698, in piecewise_affine
    return skimage.transform.warp(
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/skimage/transform/_warps.py", line 937, in warp
    coords = warp_coords(coord_map, output_shape)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/skimage/transform/_warps.py", line 624, in warp_coords
    tf_coords = coord_map(tf_coords)
  File "/gpfsscratch/rech/rxm/ubz97wr/env_dan_p4/lib/python3.10/site-packages/skimage/transform/_warps.py", line 928, in coord_map
    return inverse_map(*args, **map_args)
TypeError: 'NoneType' object is not callable

Exception in thread Thread-117 (_pin_memory_loop):
Edited by Solene Tarride