S3 ClientError (HTTP 416) when requesting the latest logs on a task with a log file of zero bytes on Ceph

Sentry issue: ARKINDEX-BACKEND-1QN

This cannot be reproduced with MinIO or with Amazon S3; this only occurs with the Ceph S3 gateway.

The Task.short_logs property (renamed to Task.logs.latest in !2114 (merged)) gets an HTTP 416 Range Not Satisfiable when asking for the last 10 kilobytes of logs for a Ponos task that has a log file of zero bytes. When the log file has one byte, it gets one byte back. When the log file does not exist, it gets an HTTP 404, which it handles to return an empty string.

>>> from arkindex.ponos.models import Task
>>> task = Task.objects.first()
>>> task.s3_logs.delete()
>>> task.short_logs
''
>>> task.s3_logs.put(Body=b'a')
>>> task.short_logs
'a'
>>> task.s3_logs.put(Body=b'')
>>> task.short_logs
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/share/arkindex/ponos/models.py", line 544, in short_logs
  File "/usr/local/lib/python3.10/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 388, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 708, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRange) when calling the GetObject operation: Unknown
>>> try: t.short_logs
... except Exception as e: print(vars(e))
... 
{'response': {'Error': {'Code': 'InvalidRange', 'BucketName': 'preprod-ponos-logs'}, 'ResponseMetadata': {'RequestId': 'tx000004be22fe42de4d856-0064f9bd04-184c8b-default', 'HostId': '', 'HTTPStatusCode': 416, 'HTTPHeaders': {'content-length': '232', 'x-amz-request-id': 'tx000004be22fe42de4d856-0064f9bd04-184c8b-default', 'accept-ranges': 'bytes', 'content-type': 'application/xml', 'date': 'Thu, 07 Sep 2023 12:07:32 GMT'}, 'RetryAttempts': 0}}, 'operation_name': 'GetObject'}

An HTTP 416 should be treated just like an HTTP 404 and result in returning the empty string.