Project

Profile

Help

Task #3505

closed

The vagrant environment should use Postgres as a DB

Added by dalley over 6 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Dev Environment
Sprint:
Sprint 36
Quarter:

Description

Justification is written here: https://www.redhat.com/archives/pulp-dev/2018-April/msg00204.html

Sqlite falls over very quickly when loaded. Despite the recent change to add a 20 second timeout, Pulp still frequently gets "OperationalError: database is locked" errors when handling a couple dozen to a couple hundred packages, with the liklihood of a sync failure due to db lock being almost 100% after about 300 packages.

This is immediately apparent with the Python plugin, which downloads every release package for a provided project name by default.

In practical terms, syncing the "Django" and "pulpcore" projects works about half the time, and syncing "Django", "pulpcore", and "scipy" usually fails.

This makes developing some plugins, such as the Python plugin, particularly difficult. We can keep sqlite support for very simple plugin development but we shouldn't leave it the default in our own development environment.


Related issues

Related to Pulp - Issue #3456: OperationalError: database is lockedCLOSED - WONTFIXbmbouterActions
Actions #1

Updated by dalley over 6 years ago

  • Sprint Candidate changed from No to Yes
Actions #2

Updated by dalley over 6 years ago

  • Related to Issue #3456: OperationalError: database is locked added
Actions #3

Updated by bmbouter over 6 years ago

With the PR for 3456 being merged I don't think sqlite is emitting OperationDatabaseLocked anymore.

https://github.com/pulp/pulp/pull/3405

Actions #4

Updated by dalley over 6 years ago

I still think we should make the change because developers doing hand testing may need to run very heavy loads in the process of debugging certain issues.

Actions #5

Updated by bmbouter over 6 years ago

  • Tags Dev Environment added
  • Tags deleted (Pulp 3 MVP, Pulp 3 installer)

Couldn't it be more frustrating if a workload runs well in the dev environment, but fails on Travis due to sqlite CI? I think having it fail directly in front of the developer would actually be better.

Retagging to match that this issue is specific to a Pulp3 dev environment.

Actions #6

Updated by dalley over 6 years ago

I think it's still the other way around because it's much easier to switch from postgres to sqlite than to switch from sqlite to postgres.

And if we reach a point where our ** smoke tests are failing on sqlite due to sqlite limitations rather than logic flaws, we should seriously reconsider supporting sqlite to begin with.

Actions #7

Updated by bmbouter over 6 years ago

Each time we've hit a problem with sqlite it's been an opportunity for us to improve Pulp's ability to run in more places. With the recent sqlite timing changes, I don't know of any current workflows that raise OperationalError. If Pulp+sqlite is broken somehow and it is unfixable that would be very useful info.

Actions #8

Updated by dkliban@redhat.com over 6 years ago

The problem seems to be that even though SQLite supports a better concurrency model[0] already, Django does not take advantage of it[1].

The Django issue has a suggestion[2] to use the 'connection_created' signal to make Pulp use sqlite with the WAL mode. We should probably do exactly that.

[0] http://www2.sqlite.org/wal.html
[1] https://code.djangoproject.com/ticket/24018
[2] https://stackoverflow.com/questions/4534992/place-to-set-sqlite-pragma-option-in-django-project/6843199#6843199

Actions #9

Updated by bmbouter over 6 years ago

  • Description updated (diff)

We need to switch back to postgreSQL. I outlined the reasons why here: https://www.redhat.com/archives/pulp-dev/2018-April/msg00204.html

Actions #10

Updated by daviddavis over 6 years ago

  • Status changed from NEW to POST
  • Groomed changed from No to Yes
  • Sprint set to Sprint 36

Added by daviddavis over 6 years ago

Revision 09fe3dfe | View on GitHub

Revert "Adding sqlite binary to dev env"

This reverts commit fe68c0a850ffe0d7361ae5956bebdc5ac886fe9b.

fixes #3505 https://pulp.plan.io/issues/3505

Actions #11

Updated by daviddavis over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #12

Updated by dalley over 6 years ago

  • Assignee set to daviddavis
Actions #13

Updated by daviddavis over 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #14

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3)
Actions #15

Updated by bmbouter almost 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF