Optimize element deletion using raw SQL
Deleting a single element could call 26 SQL queries, including some SELECT
that could fill up the RAM. This uses raw SQL to perform the deletion in 13 queries without loading anything in RAM, saving 50% of deletion time.
This overrides .delete
on the model itself to avoid Django's own cascading (9 extra queries). This bypasses all signals if there are any, so the deletion signal has been replaced with an error to avoid future issues with queryset deletion. Element.objects.all().delete()
would cause a SELECT
, loading all elements in RAM, then 13 queries per element.
Corpus.delete
, a method that used to avoid a ProtectedError
when deleting a corpus without deleting its elements first, has been removed as it triggered the signal and is no longer used anywhere; it was only used for the delete button on the corpus admin, which has been removed.
Follow-ups:
- Augment the raw SQL to use
VALUES ()
(like forhas_children
) and handle a large amount of element IDs at once, then override.delete
on element querysets to use that, allowing a new "delete this selection" option #518 (closed) - Rewrite
CorpusConsumer
to delete a whole corpus with similar queries #519 (closed) - Use
assertExactQueries
to assert harder #516 (closed)
Merge request reports
Activity
Codecov Report
Merging #1059 into master will decrease coverage by
0.03%
. The diff coverage is74.54%
.@@ Coverage Diff @@ ## master #1059 +/- ## ========================================== - Coverage 87.95% 87.91% -0.04% ========================================== Files 236 237 +1 Lines 15283 15280 -3 ========================================== - Hits 13442 13434 -8 - Misses 1841 1846 +5
Impacted Files Coverage Δ arkindex/documents/management/commands/delete.py 0.00% <0.00%> (ø)
arkindex/documents/signals.py 0.00% <0.00%> (-73.34%)
arkindex/documents/tests/test_corpus.py 100.00% <ø> (ø)
arkindex/documents/models.py 33.44% <60.00%> (-0.12%)
arkindex/documents/deletion.py 78.26% <78.26%> (ø)
arkindex/documents/api/elements.py 98.80% <100.00%> (+<0.01%)
.../documents/tests/consumers/test_corpus_consumer.py 100.00% <100.00%> (ø)
arkindex/documents/tests/test_destroy_elements.py 100.00% <100.00%> (ø)
arkindex/documents/tests/test_retrieve_elements.py 100.00% <100.00%> (ø)
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update c8affdb...757aa5f. Read the comment docs.Edited by Bastien Abadieassigned to @babadie and unassigned @erwanrouchet
mentioned in commit 5461b5f8
mentioned in issue #518 (closed)
mentioned in issue #519 (closed)