Task #3505

The vagrant environment should use Postgres as a DB

Added by dalley almost 3 years ago. Updated about 1 year ago.

Start date:
Due date:
% Done:


Estimated time:
Platform Release:
Sprint Candidate:
Dev Environment
Sprint 36


Justification is written here:

Sqlite falls over very quickly when loaded. Despite the recent change to add a 20 second timeout, Pulp still frequently gets "OperationalError: database is locked" errors when handling a couple dozen to a couple hundred packages, with the liklihood of a sync failure due to db lock being almost 100% after about 300 packages.

This is immediately apparent with the Python plugin, which downloads every release package for a provided project name by default.

In practical terms, syncing the "Django" and "pulpcore" projects works about half the time, and syncing "Django", "pulpcore", and "scipy" usually fails.

This makes developing some plugins, such as the Python plugin, particularly difficult. We can keep sqlite support for very simple plugin development but we shouldn't leave it the default in our own development environment.

Related issues

Related to Pulp - Issue #3456: OperationalError: database is lockedCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 09fe3dfe View on GitHub
Added by daviddavis over 2 years ago

Revert "Adding sqlite binary to dev env"

This reverts commit fe68c0a850ffe0d7361ae5956bebdc5ac886fe9b.

fixes #3505


#1 Updated by dalley almost 3 years ago

  • Sprint Candidate changed from No to Yes

#2 Updated by dalley almost 3 years ago

  • Related to Issue #3456: OperationalError: database is locked added

#3 Updated by bmbouter almost 3 years ago

With the PR for 3456 being merged I don't think sqlite is emitting OperationDatabaseLocked anymore.

#4 Updated by dalley almost 3 years ago

I still think we should make the change because developers doing hand testing may need to run very heavy loads in the process of debugging certain issues.

#5 Updated by bmbouter almost 3 years ago

  • Tags Dev Environment added
  • Tags deleted (Pulp 3 MVP, Pulp 3 installer)

Couldn't it be more frustrating if a workload runs well in the dev environment, but fails on Travis due to sqlite CI? I think having it fail directly in front of the developer would actually be better.

Retagging to match that this issue is specific to a Pulp3 dev environment.

#6 Updated by dalley almost 3 years ago

I think it's still the other way around because it's much easier to switch from postgres to sqlite than to switch from sqlite to postgres.

And if we reach a point where our ** smoke tests are failing on sqlite due to sqlite limitations rather than logic flaws, we should seriously reconsider supporting sqlite to begin with.

#7 Updated by bmbouter almost 3 years ago

Each time we've hit a problem with sqlite it's been an opportunity for us to improve Pulp's ability to run in more places. With the recent sqlite timing changes, I don't know of any current workflows that raise OperationalError. If Pulp+sqlite is broken somehow and it is unfixable that would be very useful info.

#8 Updated by over 2 years ago

The problem seems to be that even though SQLite supports a better concurrency model[0] already, Django does not take advantage of it[1].

The Django issue has a suggestion[2] to use the 'connection_created' signal to make Pulp use sqlite with the WAL mode. We should probably do exactly that.


#9 Updated by bmbouter over 2 years ago

  • Description updated (diff)

We need to switch back to postgreSQL. I outlined the reasons why here:

#10 Updated by daviddavis over 2 years ago

  • Status changed from NEW to POST
  • Groomed changed from No to Yes
  • Sprint set to Sprint 36

#11 Updated by daviddavis over 2 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#12 Updated by dalley over 2 years ago

  • Assignee set to daviddavis

#13 Updated by daviddavis over 1 year ago

  • Sprint/Milestone set to 3.0.0

#14 Updated by bmbouter over 1 year ago

  • Tags deleted (Pulp 3)

#15 Updated by bmbouter about 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF