Project

Profile

Help

Task #3505

The vagrant environment should use Postgres as a DB

Added by dalley over 1 year ago. Updated 8 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Dev Environment
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 36

Description

Justification is written here: https://www.redhat.com/archives/pulp-dev/2018-April/msg00204.html

Sqlite falls over very quickly when loaded. Despite the recent change to add a 20 second timeout, Pulp still frequently gets "OperationalError: database is locked" errors when handling a couple dozen to a couple hundred packages, with the liklihood of a sync failure due to db lock being almost 100% after about 300 packages.

This is immediately apparent with the Python plugin, which downloads every release package for a provided project name by default.

In practical terms, syncing the "Django" and "pulpcore" projects works about half the time, and syncing "Django", "pulpcore", and "scipy" usually fails.

This makes developing some plugins, such as the Python plugin, particularly difficult. We can keep sqlite support for very simple plugin development but we shouldn't leave it the default in our own development environment.


Related issues

Related to Pulp - Issue #3456: OperationalError: database is locked CLOSED - WONTFIX Actions

Associated revisions

Revision 09fe3dfe View on GitHub
Added by daviddavis over 1 year ago

Revert "Adding sqlite binary to dev env"

This reverts commit fe68c0a850ffe0d7361ae5956bebdc5ac886fe9b.

fixes #3505
https://pulp.plan.io/issues/3505

History

#1 Updated by dalley over 1 year ago

  • Sprint Candidate changed from No to Yes

#2 Updated by dalley over 1 year ago

  • Related to Issue #3456: OperationalError: database is locked added

#3 Updated by bmbouter over 1 year ago

With the PR for 3456 being merged I don't think sqlite is emitting OperationDatabaseLocked anymore.

https://github.com/pulp/pulp/pull/3405

#4 Updated by dalley over 1 year ago

I still think we should make the change because developers doing hand testing may need to run very heavy loads in the process of debugging certain issues.

#5 Updated by bmbouter over 1 year ago

  • Tags Dev Environment added
  • Tags deleted (Pulp 3 MVP, Pulp 3 installer)

Couldn't it be more frustrating if a workload runs well in the dev environment, but fails on Travis due to sqlite CI? I think having it fail directly in front of the developer would actually be better.

Retagging to match that this issue is specific to a Pulp3 dev environment.

#6 Updated by dalley over 1 year ago

I think it's still the other way around because it's much easier to switch from postgres to sqlite than to switch from sqlite to postgres.

And if we reach a point where our ** smoke tests are failing on sqlite due to sqlite limitations rather than logic flaws, we should seriously reconsider supporting sqlite to begin with.

#7 Updated by bmbouter over 1 year ago

Each time we've hit a problem with sqlite it's been an opportunity for us to improve Pulp's ability to run in more places. With the recent sqlite timing changes, I don't know of any current workflows that raise OperationalError. If Pulp+sqlite is broken somehow and it is unfixable that would be very useful info.

#8 Updated by dkliban@redhat.com over 1 year ago

The problem seems to be that even though SQLite supports a better concurrency model0 already, Django does not take advantage of it1.

The Django issue has a suggestion2 to use the 'connection_created' signal to make Pulp use sqlite with the WAL mode. We should probably do exactly that.

[0] http://www2.sqlite.org/wal.html
[1] https://code.djangoproject.com/ticket/24018
[2] https://stackoverflow.com/questions/4534992/place-to-set-sqlite-pragma-option-in-django-project/6843199#6843199

#9 Updated by bmbouter over 1 year ago

  • Description updated (diff)

We need to switch back to postgreSQL. I outlined the reasons why here: https://www.redhat.com/archives/pulp-dev/2018-April/msg00204.html

#10 Updated by daviddavis over 1 year ago

  • Status changed from NEW to POST
  • Groomed changed from No to Yes
  • Sprint set to Sprint 36

#11 Updated by daviddavis over 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#12 Updated by dalley over 1 year ago

  • Assignee set to daviddavis

#13 Updated by daviddavis 8 months ago

  • Sprint/Milestone set to 3.0

#14 Updated by bmbouter 8 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF