Memory leak: Catch error and retry

Refs https://redmine.teklia.com/issues/8765

added P3 label

assigned to @mblanco

added 1 commit

• 420f8405 - TODO: Try this on straton

Compare with previous version

added 1 commit

• f39e42af - Try something else (again)

Compare with previous version

added 1 commit

• 7bfa9468 - Use autocast

Compare with previous version

added 1 commit

• 5d71fac5 - Remove augmentation (to speed up leak)

Compare with previous version

added 2 commits

• 39310dfb - Store memory snapshots
• d5c42a88 - Always save snapshot

Compare with previous version

added 1 commit

• 627d3782 - Restart full training

Compare with previous version

added 1 commit

• 338059f4 - Read config

Compare with previous version

added 1 commit

• c0666f05 - Nits

Compare with previous version

changed the description

added 10 commits

• c0666f05...4e504aea - 4 commits from branch main
• c06bf2e5 - Memory leak: Catch error and retry on smaller batches
• 62b906a6 - Try something else
• 87c3cef6 - Try something else (again)
• da9d1c43 - Remove augmentation (to speed up leak)
• 24f82368 - Store memory snapshots
• 273c019f - Restart full training

Compare with previous version

added 1 commit

• 347cc27c - Store memory snapshots

Compare with previous version

added 1 commit

• a1eba543 - Try something else

Compare with previous version

added 1 commit

• 4cdd91ad - start training in subprocess

Compare with previous version

added 1 commit

• 35336bd9 - Some cleanup

Compare with previous version

added 1 commit

• 41325f18 - Nits

Compare with previous version

added 7 commits

• 44217c0e - Store memory snapshots
• a48a7895 - Restart full training
• 96eb3024 - Store memory snapshots
• 895a7d1f - Try something else
• ff57bd1c - start training in subprocess
• 6cf8e620 - Some cleanup
• 359ce276 - Nits

Compare with previous version

changed the description

changed the description