New DatasetSet model

This includes a temporary fix for Process Datasets because I really would like to put that in another merge request as this is already breaking my brain as is; similarly I'm going to create a new issue for the new dataset set mgmt API endpoints.

This looks like it works but there's still some very weird things that I can't figure out. For example, if you troubleshoot using the arkindex.training.tests.test_datasets_api.TestDatasetsAPI.test_partial_update_ponos_task_state_update test, you will notice that adding prefetch_related("sets") here

@extend_schema(tags=["datasets"])
class DatasetUpdate(ACLMixin, RetrieveUpdateDestroyAPIView):
    permission_classes = (IsVerified,)
    serializer_class = DatasetSerializer

    def get_queryset(self):
        queryset = Dataset.objects.filter(corpus__in=Corpus.objects.readable(self.request.user))
        return queryset.prefetch_related("sets").select_related("corpus", "creator")

does not remove any queries but actually adds some, when there already are duplicated / triplicated / etc queries that retrieve dataset sets when using that endpoint (only when updating / when there is a Dataset instance, I think?) and I can't figure out why, at first I though maybe I had used something other than dataset.sets.all() somewhere but it actually looks like the prefetch_related doesn't work at all.

marked the checklist item create process dataset issue as completed

marked the checklist item create new API endpoints issue as completed

mentioned in issue callico/callico#395 (closed)

added 13 commits

e2c5b6e5...4632b510 - 5 commits from branch master
0229c525 - update models + data migration
5fe13227 - wip API endpoints
2a64e13d - WIP fix API endpoints :'(
f2ced181 - fixed datasets API
5b2fcba9 - temp dataset process fix
94822ae1 - fix delete corpus
b9b314c7 - fix api schema
99806a78 - put set back in destroydatasetelement

Compare with previous version

added 1 commit

e96d0a25 - regen fixtures + fix dataset process sets testing

Compare with previous version

added 12 commits

e96d0a25...f03910d6 - 3 commits from branch master
fdd611c8 - update models + data migration
e64bc82d - wip API endpoints
3b4b7072 - WIP fix API endpoints :'(
7e4fc11d - fixed datasets API
f5103b78 - temp dataset process fix
b4f601b1 - fix delete corpus
c87f3ed3 - fix api schema
58367d27 - put set back in destroydatasetelement
1b7da324 - regen fixtures + fix dataset process sets testing

Compare with previous version

added 1 commit

c367fd09 - ensure temp process dataset sets fix migrations runs BEFORE the dataset set migration

Compare with previous version

mentioned in merge request !2264 (merged)

added 11 commits

c6405687 - 1 commit from branch master
54cb800b - update models + data migration
2e89a321 - wip API endpoints
ca9ea129 - WIP fix API endpoints :'(
39bd4dbd - fixed datasets API
0b9d79bf - temp dataset process fix
2217829b - fix delete corpus
9fc815d1 - fix api schema
c35b0191 - put set back in destroydatasetelement
ef55a9fc - regen fixtures + fix dataset process sets testing
760834d8 - ensure temp process dataset sets fix migrations runs BEFORE the dataset set migration

Compare with previous version

added 9 commits

0abeda52 - Make the set_elements field opt-in
d67c18b0 - Add prefetch for both sets and set_elements
f183c5c3 - Avoid filling up the RAM with dataset elements when cloning
9bf0e8d3 - Make the migration reversible
8727e94b - Add helpers to play with Django prefetches
6cafe501 - Restore set_elements on create responses
f0543742 - Restore set_elements on update responses
9f41df7b - Update query counts
2df4601f - Always sort sets by name

Compare with previous version

merged

mentioned in issue #1698 (closed)

New DatasetSet model

Merged by Erwan Rouchet 11 months ago (Mar 26, 2024 11:21am UTC) 11 months ago

Activity

New DatasetSet model

Merge request reports

Merged by Erwan Rouchet 11 months ago (Mar 26, 2024 11:21am UTC) 11 months ago

Activity