Add ListBuckets endpoint

Closes #1051 (closed)

The settings for the Minio client are weird. There cannot be a scheme in the endpoint, unlike boto3, and the region is irrelevant. Since we are working with multiple buckets, they could be on different regions, so the client just detects the region by itself when listing a specific bucket.

This results in the following settings:

s3:
  endpoint: https://somewhere.com  # optional, defaults to AWS
  access_key_id: blah
  secret_access_key: blah
  region: blah
ingest:
  endpoint: somewhere.com  # required
  access_key_id: blah
  secret_access_key: blah

requested review from @babadie

assigned to @erwanrouchet

changed the description

added 1 commit

• eeb0aa8d - Fix config tests

Compare with previous version

added 1 commit

• 3bdce30e - Rename module because Python imports are a mess

Compare with previous version

added 3 commits

• 6451cdfc - Friendship ended with minio
• 70999169 - Add botocore system check
• 92a29996 - Use moto for ingest tests

Compare with previous version

added 2 commits

• edfd03a1 - Add makemigrations to TEST_ENV
• d54b7f3c - Fix failing test due to checking settings too hard

Compare with previous version

added 1 commit

• 12ba5ad2 - Fix S3 mock in test_image

Compare with previous version

added 1 commit

• c7df11ae - Finding the balance between system checks and CI

Compare with previous version

added 21 commits

• c7df11ae...b1ae81a1 - 3 commits from branch master
• 332b1c5e - Add new ingest settings
• edfea468 - Add system check
• a2a92e59 - Add new S3 ingest scope
• c0493010 - Remove unused OpenAPI tag
• a0146f92 - Add ingest OpenAPI tag
• 04273169 - Add notes about wiki docs
• 1a331fda - Add getter for boto3 resource with ingest settings
• 8cae242a - Add bucket list endpoint
• 7d8639b9 - Switch to Minio client
• 077d569f - Fix config tests
• 329e8ecb - Rename module because Python imports are a mess
• ef76c69c - Friendship ended with minio
• 860a8a07 - Add botocore system check
• ad66e027 - Use moto for ingest tests
• b7dad7ad - Add makemigrations to TEST_ENV
• 7cce6ec8 - Fix failing test due to checking settings too hard
• bc5ccb9e - Fix S3 mock in test_image
• f56666a1 - Finding the balance between system checks and CI

Compare with previous version

changed title from Add Minio client and ListBuckets endpoint to Add ListBuckets endpoint

added 1 commit

• f5a8af7a - Add ListBuckets

Compare with previous version

added 20 commits

• f5a8af7a...c2c77e59 - 2 commits from branch master
• fee12f46 - Add new ingest settings
• 3bbaab67 - Add system check
• ec2d3b7b - Add new S3 ingest scope
• 3f819b38 - Remove unused OpenAPI tag
• 7aca9ae4 - Add ingest OpenAPI tag
• 62798e90 - Add notes about wiki docs
• dfe3cc2c - Add getter for boto3 resource with ingest settings
• 73481f11 - Add bucket list endpoint
• b5a831b9 - Switch to Minio client
• 259a8c02 - Fix config tests
• ad8c464b - Rename module because Python imports are a mess
• 58af00bf - Friendship ended with minio
• 678e3f56 - Add botocore system check
• f7562526 - Use moto for ingest tests
• 8ff1d45e - Add makemigrations to TEST_ENV
• ac37eee6 - Fix failing test due to checking settings too hard
• 1d853d5c - Fix S3 mock in test_image
• 05136bbc - Finding the balance between system checks and CI

Compare with previous version

Works immediately with ceph

mentioned in commit 213b4cdb

merged