Skip to content

Possible memory leak in the binary build

Sentry Issue: ARKINDEX-BACKEND-4E

SystemExit: 1
(32 additional frame(s) were not displayed)
...
  File "arkindex_common/ml_tool.py", line 257, in iter
  File "glob.py", line 72, in _iglob
    for name in glob_in_dir(dirname, basename, dironly):
  File "glob.py", line 92, in _glob0
    if os.path.lexists(os.path.join(dirname, basename)):
  File "posixpath.py", line 181, in lexists
    os.lstat(path)
  File "gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)

Gunicorn workers in preprod have been killed a few times after the preprod ran out of memory, causing it to go down for about 10 minutes at a time, while runaway tasks (ponos#12) were flooding the CreateClassifications endpoint. There was a slow, but visible, growth in memory usage from backend Gunicorn workers before the tasks were killed. While there were other API calls during that time,

Status pages for the three runaway tasks that were running during the OOM kills:

Edited by Erwan Rouchet