Skip to content

CreateMultipartUpload endpoint

https://redmine.teklia.com/issues/9153

To start supporting uploads of DataFiles and ModelVersions larger than 5GB, we need to implement proxies between an API client and the S3 API to support multipart uploads. The first step is declaring a new multipart upload, under which parts will be uploaded one by one.

A new CreateMultipartUpload endpoint should be introduced as a POST /api/v1/multipart/. As all of the multipart endpoints will have some shared parameters, two serializers need to be declared:

  • MultipartUploadSerializer, the shared serializer

    • object_type: write-only and required EnumField accepting the values model_version and data_file
    • object_id: write-only and required UUIDField
    • upload_id: required CharField for the multipart upload ID returned by S3
    • The validate() method should return an updated data dict with an extra object key. This key contains a ModelVersion or a DataFile instance, retrieved from the object_type and object_id.
    • The serializer validation raises HTTP 404 when:
      • the object does not exist
      • the user does not have read access to the DataFile's project or the ModelVersion's model
      • the ModelVersion's model is archived
    • The serializer validation raises HTTP 403 when the object exists and the user has read access, but not contributor access to the DataFile's project or the ModelVersion's model.

  • MultipartUploadCreateSerializer, used by this specific endpoint

    • upload_id must be marked as read-only since we will be generating it in this endpoint
    • checksum_algorithm is an optional, nullable EnumField accepting the names of the multipart upload checksum algorithms
    • the create() method must:
      • call object.s3_object.initiate_multipart_upload(ChecksumAlgorithm=checksum_algorithm) (docs)
      • handle botocore.exceptions.ClientError exceptions, raising a ValidationError with the e.response["Error"]["Message"] when e.response["Error"]["Code"] == "400"
      • return {"upload_id": id}, from the id of the object returned by initiate_multipart_upload.