Upload files to S3

Includes some fixes to the import tasks that haven't actually been tested. Those fixes were made only to make unit tests work.

changed the description

added 1 commit

• a8235e94 - Fix flake8

Compare with previous version

added 23 commits

• a8235e94...40d9a9bf - 2 commits from branch master
• 5c657ea8 - Turn Thumbnail.create into an instance method
• e086603c - Add boto3
• 32d216c9 - Add AWS settings
• a5f9f2a1 - Thumbnail creation over S3
• af17498f - Remove prefix in thumbnail names
• 81ce94fc - Override AWS settings for tests
• d910b127 - Do not run thumbnail gen in test_iiif
• 58396c04 - Test s3_check
• ed4da0b9 - Add tests for Thumbnail
• 89e04f60 - Update base image version
• 6ebb9c2d - Ignore local imageserver for thumbnail tests
• 135d9f32 - Use s3.meta instead of s3.Object().meta
• 947a4355 - Use BytesIO instead of TemporaryFile
• 1d4b2076 - Add staging bucket name setting
• ff1cb9c1 - Update system check
• d59ac4fd - Upload files to S3
• d4774b44 - Ensure no DataFile is saved when the upload fails
• 3a41fd5d - Update DataFile model for S3
• d16a1dbd - Update unit tests
• d254c375 - Fixes for other tests
• b488289c - Fix flake8

Compare with previous version

Codecov Report

Merging #252 into master will decrease coverage by 0.1%. The diff coverage is 71.87%.

@@            Coverage Diff             @@
##           master     #252      +/-   ##
==========================================
- Coverage   79.02%   78.92%   -0.11%     
==========================================
  Files         153      153              
  Lines        9132     9169      +37     
==========================================
+ Hits         7217     7237      +20     
- Misses       1915     1932      +17

Impacted Files	Coverage Δ
arkindex/project/checks.py	76.29% <ø> (ø)
arkindex/project/settings.py	0% <0%> (ø)
arkindex/dataimport/tests/test_files.py	100% <100%> (ø)
arkindex/dataimport/tasks/image.py	88.23% <100%> (ø)
arkindex/dataimport/tests/test_pdf.py	100% <100%> (ø)
arkindex/project/tests/test_checks.py	100% <100%> (ø)
arkindex/dataimport/api.py	84.5% <100%> (-0.08%)
arkindex/dataimport/tests/test_image.py	100% <100%> (ø)
arkindex/dataimport/tasks/pdf.py	94.11% <100%> (+1.8%)
arkindex/dataimport/models.py	21.89% <22.72%> (-0.26%)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update bde1261...95ec336. Read the comment docs.

added 9 commits

• 7f7bda9e - Bump
• 241c0eee - Add staging bucket name setting
• facaf9ba - Update system check
• eec7e13a - Upload files to S3
• 0514a9b1 - Ensure no DataFile is saved when the upload fails
• 788854a2 - Update DataFile model for S3
• 35c200b9 - Update unit tests
• cb810ef2 - Fixes for other tests
• a103393d - Fix flake8

Compare with previous version

added 13 commits

• c3f79d65 - Pour some timeout sauce onto Image.pillow_open
• 13cca92b - Send AWS secrets over Ponos
• b1ad6714 - Update tests
• a7edd670 - Use arkindex.project.aws to have a single S3 client
• e6b427fb - Add staging bucket name setting
• 24c36476 - Update system check
• f62fb394 - Upload files to S3
• 7dda18ee - Ensure no DataFile is saved when the upload fails
• cb7c58a2 - Update DataFile model for S3
• 00f81a0c - Update unit tests
• a3e8e191 - Fixes for other tests
• 11441ef3 - Fix flake8
• 73d5b9e9 - Use centralized s3 client

Compare with previous version

added 10 commits

• 48caf4f5 - Add a setting for AWS region
• bad3ef9d - Add staging bucket name setting
• 96f7807b - Update system check
• efe1639f - Upload files to S3
• 2009fc84 - Ensure no DataFile is saved when the upload fails
• bacd6785 - Update DataFile model for S3
• 81379b3f - Update unit tests
• f4a396a4 - Fixes for other tests
• c2b407a1 - Fix flake8
• 357b3a6f - Use centralized s3 client

Compare with previous version

added 11 commits

• 357b3a6f...bde1261b - 2 commits from branch master
• 11ebb063 - Add staging bucket name setting
• 87372c7b - Update system check
• 79ee0a1a - Upload files to S3
• ba08ace2 - Ensure no DataFile is saved when the upload fails
• b9ccfeff - Update DataFile model for S3
• 8d803a96 - Update unit tests
• f95ff466 - Fixes for other tests
• b6c2909e - Fix flake8
• 11a3edd4 - Use centralized s3 client

Compare with previous version

added 1 commit

• 95ec3364 - Comments

Compare with previous version

resolved all discussions

merged

mentioned in commit 337ac5e4