Task #3505
closedThe vagrant environment should use Postgres as a DB
100%
Description
Justification is written here: https://www.redhat.com/archives/pulp-dev/2018-April/msg00204.html
Sqlite falls over very quickly when loaded. Despite the recent change to add a 20 second timeout, Pulp still frequently gets "OperationalError: database is locked" errors when handling a couple dozen to a couple hundred packages, with the liklihood of a sync failure due to db lock being almost 100% after about 300 packages.
This is immediately apparent with the Python plugin, which downloads every release package for a provided project name by default.
In practical terms, syncing the "Django" and "pulpcore" projects works about half the time, and syncing "Django", "pulpcore", and "scipy" usually fails.
This makes developing some plugins, such as the Python plugin, particularly difficult. We can keep sqlite support for very simple plugin development but we shouldn't leave it the default in our own development environment.
Related issues
Updated by dalley over 6 years ago
- Related to Issue #3456: OperationalError: database is locked added
Updated by bmbouter over 6 years ago
With the PR for 3456 being merged I don't think sqlite is emitting OperationDatabaseLocked anymore.
Updated by dalley over 6 years ago
I still think we should make the change because developers doing hand testing may need to run very heavy loads in the process of debugging certain issues.
Updated by bmbouter over 6 years ago
- Tags Dev Environment added
- Tags deleted (
Pulp 3 MVP, Pulp 3 installer)
Couldn't it be more frustrating if a workload runs well in the dev environment, but fails on Travis due to sqlite CI? I think having it fail directly in front of the developer would actually be better.
Retagging to match that this issue is specific to a Pulp3 dev environment.
Updated by dalley over 6 years ago
I think it's still the other way around because it's much easier to switch from postgres to sqlite than to switch from sqlite to postgres.
And if we reach a point where our ** smoke tests are failing on sqlite due to sqlite limitations rather than logic flaws, we should seriously reconsider supporting sqlite to begin with.
Updated by bmbouter over 6 years ago
Each time we've hit a problem with sqlite it's been an opportunity for us to improve Pulp's ability to run in more places. With the recent sqlite timing changes, I don't know of any current workflows that raise OperationalError. If Pulp+sqlite is broken somehow and it is unfixable that would be very useful info.
Updated by dkliban@redhat.com over 6 years ago
The problem seems to be that even though SQLite supports a better concurrency model[0] already, Django does not take advantage of it[1].
The Django issue has a suggestion[2] to use the 'connection_created' signal to make Pulp use sqlite with the WAL mode. We should probably do exactly that.
[0] http://www2.sqlite.org/wal.html
[1] https://code.djangoproject.com/ticket/24018
[2] https://stackoverflow.com/questions/4534992/place-to-set-sqlite-pragma-option-in-django-project/6843199#6843199
Updated by bmbouter over 6 years ago
- Description updated (diff)
We need to switch back to postgreSQL. I outlined the reasons why here: https://www.redhat.com/archives/pulp-dev/2018-April/msg00204.html
Updated by daviddavis over 6 years ago
- Status changed from NEW to POST
- Groomed changed from No to Yes
- Sprint set to Sprint 36
Added by daviddavis over 6 years ago
Updated by daviddavis over 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset devel|09fe3dfe218aac3d401bbd31d250e6ece2459aa7.
Updated by bmbouter almost 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Revert "Adding sqlite binary to dev env"
This reverts commit fe68c0a850ffe0d7361ae5956bebdc5ac886fe9b.
fixes #3505 https://pulp.plan.io/issues/3505