Skip to content

Reconnect on default db after export is uploaded

Refs https://redmine.teklia.com/issues/6380

Large exports using the remote server tend to lose their default database connection.

The export is correctly created and published on S3/ceph, but the final sql query to update the CorpusExport state fails:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/share/arkindex/documents/export/__init__.py", line 187, in export_corpus
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 812, in save
    self.save_base(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 863, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 976, in _save_table
    updated = self._do_update(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 1040, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1216, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/django/__init__.py", line 616, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 301, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 269, in create_cursor
    cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/rq/worker.py", line 1431, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1280, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1317, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/usr/share/arkindex/documents/export/__init__.py", line 196, in export_corpus
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 812, in save
    self.save_base(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 863, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 976, in _save_table
    updated = self._do_update(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 1040, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1216, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1396, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 323, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 300, in _cursor
    with self.wrap_database_errors:
  File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 301, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 269, in create_cursor
    cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed

From https://arkindex.teklia.com/rq/queues/1/8a0a56f0-48c4-42dd-8423-57dc66283e88/