Skip to content

Remove DataFile hash

Erwan Rouchet requested to merge remove-datafile-hash into master

!777 (merged) 🤑

This is some preparatory work for frontend#233 (closed). To help the frontend upload files directly to S3 using CreateDataFile instead of UploadDataFile and bypass time limits, it has been decided to remove the MD5 hash check on DataFiles, making MD5 hashes useless to them. Therefore, MD5 hashes are completely removed from the DataFile.

This implies a little bit of refactoring to arkindex.project.aws to remove a now useless mixin, as well as updating the file checking algorithm to not update the attributes on a file, but just to check that what the client said the content type and length would be is what they actually are ("trust, but verify"). This also implies a lot of updates to unit tests, which took most of the time spent on this MR.

An issue has been noticed with the way the IIIF uploads are handled: When a JSON file gets uploaded, its @context is checked and it is only treated as an IIIF manifest if it has the expected context. This is only detected when using UploadDataFile and not CreateDataFile, therefore exclusively using the S3 upload would break IIIF imports. This will be fixed in a later MR.

Edited by Erwan Rouchet

Merge request reports

Loading