Issue #3456: OperationalError: database is locked - Pulp

Actions

Send by e-mail Copy link

Issue #3456

closed

OperationalError: database is locked

Added by daviddavis about 6 years ago. Updated almost 5 years ago.

Status:

CLOSED - WONTFIX

Priority:

Normal

Assignee:

bmbouter

Category:

Sprint/Milestone:

3.0.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Sprint:

Quarter:

Description

Sometimes, we encounter the following error on sqlite:

Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/travis/build/pulp/pulp/pulpcore/pulpcore/tasking/tasks.py", line 293, in __call__
    return super(UserFacingTask, self).__call__(*args, **kwargs)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/pulp_file/app/tasks/synchronizing.py", line 72, in synchronize
    for report in changeset.apply():
  File "/home/travis/build/pulp/pulp/plugin/pulpcore/plugin/changeset/main.py", line 196, in apply
    for report in itertools.chain(self._apply_additions(), self._apply_removals()):
  File "/home/travis/build/pulp/pulp/plugin/pulpcore/plugin/changeset/main.py", line 155, in _apply_additions
    self._add_content(content)
  File "/home/travis/build/pulp/pulp/plugin/pulpcore/plugin/changeset/main.py", line 104, in _add_content
    self.repository_version.add_content(content.stored_model)
  File "/home/travis/build/pulp/pulp/pulpcore/pulpcore/app/models/repository.py", line 392, in add_content
    association.save()
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/models/base.py", line 905, in _save_table
    forced_update)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/models/base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/models/query.py", line 664, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 889, in execute_sql
    cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked

More info from django:

https://docs.djangoproject.com/en/2.0/ref/databases/#database-is-locked-errors

Related issues

Actions

Copy link

Updated by amacdona@redhat.com about 6 years ago

What situations does this happen? Can it be avoided by "being gentle"?

I see 2 options:

1. Accept that developing Pulp is inherently not "lightweight" and stop claiming to support SQLite.
2. Add something like this to the contributor documentation. "SQLite should be used for lightweight development, but may fail when running concurrent tasks. If you run into this problem, it is time to switch to Postgres."

Actions

Copy link

Updated by dalley about 6 years ago

Project changed from File Support to Pulp
Triaged changed from No to Yes

We should try increasing the timeout first.

Actions

Copy link

Updated by dalley about 6 years ago

Status changed from NEW to ASSIGNED
Assignee set to dalley

Actions

Copy link

Updated by dalley about 6 years ago

Status changed from ASSIGNED to POST

PR: https://github.com/pulp/pulp/pull/3376

Added by dalley about 6 years ago

Revision 3537757c | View on GitHub

Set a 20 second timeout for sqlite

closes #3456 https://pulp.plan.io/issues/3456

Added by dalley about 6 years ago

Revision 3537757c | View on GitHub

Set a 20 second timeout for sqlite

closes #3456 https://pulp.plan.io/issues/3456

Actions

Copy link

Updated by dalley about 6 years ago

Status changed from POST to MODIFIED

Applied in changeset pulp|3537757c2e8ab8b4a97196f9e6b5a20f227b3884.

Actions

Copy link

Updated by dalley about 6 years ago

Related to Task #3505: The vagrant environment should use Postgres as a DB added

Actions

Copy link

Updated by dalley about 6 years ago

Status changed from MODIFIED to NEW

Reopening because this is still a big problem with certain plugins like the Python plugin, which downloads every release package for each project specified.

Syncing only a couple of projects from PyPI will fail due to this issue.

Actions

Copy link

Updated by dalley about 6 years ago

Assignee deleted (~~dalley~~)

Actions

Copy link

Updated by kersom about 6 years ago

I have been found this problem a few times when running smash tests.

[root@f27-pulp3 ~]# journalctl -xe
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return Database.Cursor.execute(self, query, params)
Apr 03 13:24:18 f27-pulp3 celery[3149]: sqlite3.OperationalError: database is locked
Apr 03 13:24:18 f27-pulp3 celery[3149]: The above exception was the direct cause of the following exception:
Apr 03 13:24:18 f27-pulp3 celery[3149]: Traceback (most recent call last):
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/celery/app/trace.py", line 374, in trace_task
Apr 03 13:24:18 f27-pulp3 celery[3149]:     R = retval = fun(*args, **kwargs)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/devel/pulp/pulpcore/pulpcore/tasking/tasks.py", line 273, in __call__
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return super().__call__(*args, **kwargs)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/celery/app/trace.py", line 629, in __protected_call__
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return self.run(*args, **kwargs)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/pulp_file/app/tasks/synchronizing.py", line 72, in synchronize
Apr 03 13:24:18 f27-pulp3 celery[3149]:     for report in changeset.apply():
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/main.py", line 196, in apply
Apr 03 13:24:18 f27-pulp3 celery[3149]:     for report in itertools.chain(self._apply_additions(), self._apply_removals()):
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/main.py", line 155, in _apply_additions
Apr 03 13:24:18 f27-pulp3 celery[3149]:     self._add_content(content)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/main.py", line 104, in _add_content
Apr 03 13:24:18 f27-pulp3 celery[3149]:     self.repository_version.add_content(content.stored_model)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/devel/pulp/pulpcore/pulpcore/app/models/repository.py", line 394, in add_content
Apr 03 13:24:18 f27-pulp3 celery[3149]:     association.save()
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 729, in save
Apr 03 13:24:18 f27-pulp3 celery[3149]:     force_update=force_update, update_fields=update_fields)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 759, in save_base
Apr 03 13:24:18 f27-pulp3 celery[3149]:     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 823, in _save_table
Apr 03 13:24:18 f27-pulp3 celery[3149]:     forced_update)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 872, in _do_update
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return filtered._update(values) > 0
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 709, in _update
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return query.get_compiler(self.db).execute_sql(CURSOR)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
Apr 03 13:24:18 f27-pulp3 celery[3149]:     cursor = super().execute_sql(result_type)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1066, in execute_sql
Apr 03 13:24:18 f27-pulp3 celery[3149]:     cursor.execute(sql, params)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return super().execute(sql, params)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return executor(sql, params, many, context)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return self.cursor.execute(sql, params)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
Apr 03 13:24:18 f27-pulp3 celery[3149]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return self.cursor.execute(sql, params)
Apr 03 13:24:18 f27-pulp3 celery[3149]:   File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 303, in execute
Apr 03 13:24:18 f27-pulp3 celery[3149]:     return Database.Cursor.execute(self, query, params)
Apr 03 13:24:18 f27-pulp3 celery[3149]: django.db.utils.OperationalError: database is locked
Apr 03 13:25:51 f27-pulp3 audit[1501]: USER_START pid=1501 uid=0 auid=0 ses=2 msg='op=login id=0 exe="/usr/sbin/sshd" hostname=? addr=192.168.122.1 terminal=/dev/pts/1 res=success'
Apr 03 13:25:51 f27-pulp3 audit[1501]: CRYPTO_KEY_USER pid=1501 uid=0 auid=0 ses=2 msg='op=destroy kind=server fp=SHA256:e2:23:f1:b4:55:25:b5:c5:b9:6c:91:db:28:3b:7a:15:c8:26:fd:e7:f3:ea:07:98:46:55:2b:6e:e0:05:

Actions

Copy link

#10

Updated by daviddavis about 6 years ago

Status changed from NEW to MODIFIED
Assignee set to dalley

https://github.com/pulp/pulp/pull/3405

Actions

Copy link

#11

Updated by dalley about 6 years ago

Status changed from MODIFIED to NEW

A significant percentage of our travis jobs are hitting this.

Actions

Copy link

#12

Updated by bmbouter about 6 years ago

Status changed from NEW to ASSIGNED
Assignee changed from dalley to bmbouter

At the meeting today we determined we want some more info on this issue. I'm taking as ASSIGNED so I can investigate and post what I find. With more info we can determine if it's possible to run Pulp on sqlite without error.

dalley, I am hoping to work w/ you to analyze whatever I find.

Actions

Copy link

#13