Vore Ponos server-side code

Closes #1423 (closed)

I used the following script to merge the repository into this one:

sudo apt install git-filter-repo

git clone --no-tags git@gitlab.com:teklia/arkindex/ponos /tmp/ponos
cd /tmp/ponos
git filter-repo --path=ponos/ --path=tests/server/ --path=tests/fixtures/ --path=requirements-server.txt
mkdir arkindex
mv ponos arkindex/
mv requirements-server.txt arkindex/ponos/
mv tests/server arkindex/ponos/tests
mv tests/fixtures arkindex/ponos/tests/fixtures
rmdir tests
git add -A
git commit -m 'Move Ponos server-side code to Arkindex app'

workon backend
git checkout --branch vore-ponos
git remote add ponos /tmp/ponos
git fetch ponos
git merge --allow-unrelated-histories ponos/master
git remote rm ponos
rm -rf /tmp/ponos

TODO

• Fix all the unit tests
• Move arkindex.ponos.urls to a arkindex.project.ponos_v1 (this will later be merged with arkindex.project.api_v1)
• Merge the custom views, OpenAPI overrides, permissions defined in arkindex.project.views with the API endpoints in arkindex.ponos.api
• Check the OpenAPI schema
• Test the unmodified Ponos agent a lot
• Write instructions for developers to remove the Ponos setup from their backend environments

Cleaning up the dev environment

• To remove Ponos entirely from your backend environment: pip uninstall ponos_server ponos-agent
• To check whether or not you have Ponos properly uninstalled: pip freeze | grep -i ponos should return nothing.

Follow-ups

• Some of various simplifications discussed on Redmine will be done in this MR because they make the integration easier, but others might need to be done in other merge requests.
• Resetting the migrations will probably be necessary in order to fully cleanup the code, in another merge request, with instructions on what SQL queries to run manually to reset django_migrations properly.
• The switch to /api/v1/ponos will better be done in a separate merge request, as this will require some work on the agent side (and maybe should be done after we switch to an OpenAPI-based client).
• Prevent Ponos agents from accessing any endpoint that isn't related to Ponos, by editing the authorization classes so that the JWT agent authentication is only set on Ponos endpoints (meaning this will also be nicely documented in OpenAPI)
• Found #1426 (closed) while checking the OpenAPI schema!
• Found #1427 (closed) #1428 (closed) #1429 (closed) frontend#1219 (closed) frontend#1220 (closed) while testing the Ponos agent!

changed milestone to %Arkindex 1.4.0

assigned to @erwanrouchet

changed the description

changed the description

changed the description

added 1 commit

• 9225e4fb - Fix tests, merge URLs and views

Compare with previous version

marked the checklist item Fix all the unit tests as completed

marked the checklist item Move arkindex.ponos.urls to a arkindex.project.ponos_v1 (this will later be merged with arkindex.project.api_v1) as completed

marked the checklist item Merge the custom views, OpenAPI overrides, permissions defined in arkindex.project.views with the API endpoints in arkindex.ponos.api as completed

changed the description

marked the checklist item Check the OpenAPI schema as completed

marked this merge request as ready

marked the checklist item Test the unmodified Ponos agent a lot as completed

marked the checklist item Write instructions for developers to remove the Ponos setup from their backend environments as completed

requested review from @babadie

mentioned in issue #1427 (closed)

changed the description

added 211 commits

• 9225e4fb...f3679bd9 - 5 commits from branch master
• b42efe28 - Empty structures
• 154e7e87 - Base models
• 864776a4 - Setup CI
• 5f06c69e - Docker base support
• e603465d - Write & display logs
• 042cfb4d - Trigger children tasks
• efb1e18a - Support Docker custom CMD
• bb076227 - Delete container when final + Retry
• 616b798e - Run in docker.
• 578c4922 - Artifacts
• ff2fc56b - Arkindex integration
• 17b69ad6 - Workflow and task details APIs
• 585fee63 - Optional environment variables in recipes
• bab16f68 - Network support
• c9e23632 - Handle container exit code
• 1897e246 - Start children when all level tasks are completed.
• 1e295260 - First implementation of PonosCommand
• 43f2d951 - Handle paths inside Docker using PathArgument
• 3e42579b - Check the workflow's state instead of tasks when retrying
• 0acbdc36 - Methods to get task ID and run ID
• ccfbe5f5 - Handle API errors when inspecting a container
• 1f23ceb3 - Handle args with spaces in PonosCommand
• 04541da0 - Tail on long logs and raw/download endpoint
• 6e0903a9 - Optimize workflow state
• 141c4387 - Django admin optimizations and enhancements
• 09085e2f - Use APIView to prevent breaking compatibility with OpenAPI schema generation
• cf7b3c3e - Different path on host for data volume
• 37c7b78d - Fix HTTP 500 on task log
• f98c4f94 - Use yaml.safe_load instead of deprecated yaml.load
• bfebe9df - Flush logs to stdout
• 390ccd53 - Handle children tasks in complex graphs
• a7ba1a97 - Recipe validation
• 7808e81f - Prevent empty task slugs
• 0c0d82be - Allow an optional Docker network
• 1bff5b33 - Fix for OpenAPI
• 841ce2de - Remove old recipe options
• 0b3824d3 - Allow empty GPU names field on agent
• 8271fd66 - Revert "Allow empty GPU names field on agent"
• 31e70068 - Revert "Revert "Allow empty GPU names field on agent""
• 71bd283e - Multiple parents for each task
• 24825ff1 - Task and agent tags
• 8254f917 - ECDH agent authentication
• cd6314ed - JSON Web Tokens
• 3be95ac4 - Handle missing Ponos settings during tests
• 2721a524 - Remove shortcuts in ponos.keys
• fffcee92 - Add admin for Farm
• ec8e9a6b - Fix FieldError in Agent admin
• 4d1163e6 - Stop ponos task
• 6eb35b65 - Display the agent
• 0518123c - Ponos agent unit tests
• 0c398e36 - Add missing __init__ for test subpackages
• 7f9dabb7 - test settings
• 058c4571 - Authentication unit tests
• 92376796 - List task parents using Python when not using PostgreSQL
• b941772d - test for artifact, task and action object
• 4b93f671 - Advanced agent filtering
• 59892e32 - Prevent agents stepping on each other when starting tasks
• 449388c4 - Require agent authentication only for agent actions
• 363f2e7e - Add agent tags in admin
• da17262f - Remove temporary files in unit tests
• ac2d220e - Make a running task override all other workflow states
• 1c8d8564 - Fix editing agents and workflows in admin
• 77e0c8b1 - Fix errors on workflow state when there are Completed and Pending tasks
• 7d121d0f - Re-enable task editing in admin with the proper fix
• 3e9516a7 - Un-ignore some linting rules
• bb2712a0 - Pin cryptography to 2.7
• 44466a57 - Remove PonosCommand
• e03d7077 - Unpin cryptography
• 093c0cbc - Handle UnicodeDecodeError on sliced off Unicode characters
• be92175a - Make Stopped a final state on tasks
• c709655a - Prevent upgrading to new major dependency versions
• 412d8501 - Remove the seed from FarmSerializer
• c7db0941 - Docstrings everywhere
• e2f4ce83 - Enhance task filtering and agent display
• 60215f86 - Prevent deleting an agent with non-final tasks
• 7a4b8cec - Check agent ID against task definition to handle scheduling issues
• d3775a1f - Use Tox for unit tests
• 88b973b0 - Artifact model & API
• 21f1d72d - ArtifactInline is readonly, fixes #48 (closed)
• deed66c9 - Add new fields on Task to store Docker definition
• 1f5ed5f8 - Add workflow_id env variable when starting a container
• 37759d0d - Add Dict serializer for env attribute on Task
• b827f9b4 - Add endpoint to create a task
• 423e1dfe - Expose docker socket
• 11ded738 - Handle multi-database setups in RunPython
• 835f2a4e - Add Workflow environment variables handling in CreateNewTask endpoint
• 45d3e2c6 - Change permissions on ListArtifacts endpoint
• 98a7ec24 - Fix is_agent check priority
• d76da61c - Stale read fixes
• 12637102 - pre-commit linting
• ff8c8341 - List all next actions in a single transaction
• b210ccb5 - Support artifacts as a docker image for a task, in the backend.
• b2edfe06 - Switch to the new JSONField
• 679bf022 - Bump cryptography dependency
• 6cc0f9f9 - Handle large artifacts
• 95177a58 - Bump to Django 3.1.1
• 26f55507 - Bump to black 20.8b1
• 42cae864 - Store secrets
• 016e85c4 - Export secrets through the API
• fd3c7560 - Define tasks ordering in ponos.models
• ab8e2bb6 - Bump Python requirement Django to 3.1.2
• db305e89 - Bump Python requirement pyyaml to 5.3.1
• 84f4ef9a - Bump Python requirement pytz to 2020.1
• 81f6ddb7 - Manual start of a workflow
• e92886fa - Repeat StartTask actions to prevent infinitely pending tasks
• caeef613 - Add blank=True to nullable fields
• 0463789f - Prevent UnicodeDecodeError on task logs
• ef192d93 - Bump Python requirement pytz to 2020.4
• 05b1ee76 - Store agent state when attributing actions
• fcb2c3bf - Drop included and excluded tags per agent
• ec371027 - Human readable Agent hardware in the admin
• 78baad5e - Bump Python requirement Django to 3.1.3
• 9e18f671 - Revert "Always use current agent" and fix last_ping update
• a581213d - Bump Python requirement djangorestframework-simplejwt to 4.5.0
• a593e810 - List agents state endpoint
• d454f385 - Agent details
• ddb26ffb - Order agents by hostname
• 6f3355bc - Store GPU infos from agents in Database
• 5f8fba77 - Bump Python requirement djangorestframework-simplejwt to 4.6.0
• 6ee5b6a8 - Assign a GPU to a task
• 5b3389d2 - Bump Python requirement Django to 3.1.4
• 500d0ee5 - Use a DB field to require GPU
• cdb11064 - Add simple type hints for Spectacular
• 3a37b790 - Bump Python requirement Django to 3.1.5
• 60ca4f06 - Bump Python requirement pytz to 2020.5
• 28b06a80 - Allow a task creator to stop or restart his tasks
• 97f95709 - Avoid stale read on children state updates
• c24886e9 - Add object level permission check for artifact download redirection
• 4b651bf6 - Bump Python requirement Django to 3.1.6
• 3e3f22c1 - Bump Python requirement pytz to 2021.1
• c00758c1 - Bump Python requirement djangorestframework to 3.12.2
• e607cb96 - Bump Python requirement pyyaml to 5.4.1
• 1c738f1c - Avoid stale read when an agent update one of its assigned tasks
• e5ac6ae4 - Add drf-spectacular
• 14431b2e - Bump Python requirement boto3 to 1.17.11
• 54c965c2 - Bump Python requirement cryptography to 3.4.6
• 049443fc - Bump to Django 3.1.7
• 1145ab6a - Avoid duplicate queries in WorkflowDetails
• cdb0b4cb - Add an endpoint to list farms
• a0b973c5 - Bump Python requirement django-enumfields to 2.1.1
• 1ed09535 - Revert "Apply 1 suggestion(s) to 1 file(s)"
• 6a658a0e - Bump Python requirement cryptography to 3.4.7
• d6d95380 - Bump Python requirement djangorestframework to 3.12.4
• 3d0932d6 - Bump Python requirement djangorestframework-simplejwt to 4.7.1
• 61c74ba8 - Bump Python requirement drf-spectacular to 0.16.0
• 4fef7b5d - Bump Python requirement Django to 3.2.3
• 090eb39f - Bump Python requirement Django to 3.2.4
• 3adb0301 - Bump Python requirement drf-spectacular to 0.17.0
• 2a82e799 - Bump Python requirement drf-spectacular to 0.17.2
• cf546e62 - Bump Python requirement Django to 3.2.5
• 7f99aeda - Bump Python requirement boto3 to 1.17.105
• c719c90e - Bump Python requirement djangorestframework-simplejwt to 4.7.2
• 5ec53ded - Restrict artifact size from 1 byte to 5 GiB
• b5d17d87 - Remove deprecated default_app_config
• 4009d47c - Handle existing GPUs when registering an agent
• bae06465 - Add priority attribute on Task to sort them in next_tasks()
• 025c3a21 - Bump Python requirement drf-spectacular to 0.17.3
• 2de75dc5 - Bump Python requirement Django to 3.2.6
• a77b1ab2 - Bump Python requirement boto3 to 1.18.13
• 67992da9 - Bump Python requirement djangorestframework-simplejwt to 4.8.0
• 26ed86f0 - Handle duplicate slug errors in CreateTask
• 3cfc8264 - Bump Python requirement drf-spectacular to 0.18.2
• 9ba2d5f6 - Pin PyJWT to 2.1.0
• 17669282 - Bump Python requirement pytz to 2021.3
• 3e564ede - Requires gpu from recipe
• 17af363d - Bump Python requirement djangorestframework-simplejwt to 5.0.0
• e5ff8bbb - Bump to Python 3.8 and Django 4
• 5a80493b - Bump Python requirement Django to 4.0.2
• bd414f27 - Prevent making one SQL query per row in the admin's Task list
• 7f27af1f - Add a completion date on workflows
• 47e32e7a - Bump black in pre-commit config
• 0959099a - Add workflow and task expiry
• 3353d3e8 - Bump Python requirement Django to 4.0.4
• 1b5e46b2 - Bump Python requirement pytz to 2022.1
• 75e849aa - Support arbitrary files to populate task context
• b241d870 - Allow empty dicts in extra_files
• 7010718a - Restrict Task.env and Task.extra_files to strings
• 78046765 - Bump Python requirement pyyaml to 6.0
• 5b2f134a - Bump Python requirement djangorestframework-simplejwt to 5.2.0
• 44cbae43 - Bump Python requirement pytz to 2022.2.1
• 8f92da48 - Manually set last_run when retrying a workflow
• 9e8f9da6 - Handle empty runs in Workflow.get_state
• 17274e3e - Add a signal on task failure
• 3cf409c6 - Add shared memory size
• 5e94da93 - Prevent agent creation from the admin
• 2392622e - Bump Python requirement pytz to 2022.6
• d1c2e4e6 - Bump Python requirement djangorestframework-simplejwt to 5.2.2
• fe0ffe80 - Bump Python requirement Django to 4.1.3
• 8d201a09 - Bump Python requirement Django to 4.1.4
• 8b475db3 - Bump Python requirement Django to 4.1.5
• 51c0ce0b - Make the maximum Artifact file size configurable
• caac0329 - Check that artifact IDs are UUIDs in recipes
• 1bed4dc3 - Move Ponos server-side code to Arkindex app
• 35ae63cb - Fix gitignore
• 2f2692d1 - Merge requirements
• 1f61a4a4 - Remove Ponos git clone from build system
• 43b22718 - Update imports
• 61416350 - Fix README
• bbdd9766 - Fix more imports
• ef687a18 - Remove duplicate test
• 9f5efd9a - Reintroduce testing helper
• 7cbb5605 - Fix sorting of cryptography imports
• f65c32b5 - Fix arkindex.ponos.tests.test_admin
• 0db7f31a - Add AgentUser.is_internal
• a6181512 - Fix PONOS_PRIVATE_KEY errors in test_api
• 9ba514ab - Fix tests, merge URLs and views

Compare with previous version