Project

Profile

Help

Task #1803

Plan replacement of mongodb with postgres

Added by Anonymous over 3 years ago. Updated 5 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
API
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Documentation, Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 5

Related issues

Related to Pulp - Task #1872: Profile Django ORM instantiation cost CLOSED - CURRENTRELEASE Actions
Related to Pulp - Task #1873: Plan REST API for 3.0 CLOSED - CURRENTRELEASE Actions
Related to Pulp - Task #1874: Plan User/Auth system for 3.0 CLOSED - DUPLICATE Actions

History

#1 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from NEW to ASSIGNED

#2 Updated by semyers over 3 years ago

I'm in the process of writing all this in more detail in an etherpad, but for now I'll outline the work so far.

pcreech and I started by basically just writing out all of models we had, and trying to organize them in a diagram.

This went poorly, because pulp has four distinct categories of models, and the tool we were using didn't make it very easy to diagram them all together. With all the pulp releasing I've been doing, and all the RHUI pcreech has been doing, all of this finally came together toward the end of last week while jortel was on PTO. Before unleashing the confusing and questionably useful diagrams on the team, I wanted to get jortel's feedback. I was not disappointed.

With his help, we've identified the aformentioned data categories: Repositories, Content Units, RBAC, Tasks. I'm writing up a doc right now that explains what collections we currently have in mongo, to which category they belong, how the currently relate to each other, and some speculation about how we can migrate the data in that category to a more relational design. Since I'll be speculating, this doc will go up in an etherpad for review and improvement by the small team focused on this at the moment (jortel, pcreech, me), before being submitted to the team at large.

Depending on how quickly I get this done and reviewed, the link to that etherpad should most likely be appearing here tomorrow.

#3 Updated by semyers over 3 years ago

  • Private changed from No to Yes

The "Relational Pulp" etherpad has been drafted and edited. It is available for internal comment at http://pulp.etherpad.corp.redhat.com/244 by the entirety of the pulp team.

#4 Updated by semyers over 3 years ago

  • Private changed from Yes to No

#5 Updated by semyers over 3 years ago

  • Related to Task #1872: Profile Django ORM instantiation cost added

#6 Updated by mhrivnak over 3 years ago

  • Related to Task #1873: Plan REST API for 3.0 added

#7 Updated by mhrivnak over 3 years ago

  • Related to Task #1874: Plan User/Auth system for 3.0 added

#8 Updated by mhrivnak over 3 years ago

  • Sprint/Milestone changed from 19 to 20

#9 Updated by semyers over 3 years ago

The migration plan is largely "ratified", in that objections to it have been (or are currently being) addressed. I feel like we've reached a point where what we don't know outweighs what we do know, and the best way I can think of to bridge the knowledge gap is to finishing modeling pulp out on postgres and start trying to use it. I'll be modifying my relational pulp project to this end over the coming days.

https://github.com/seandst/relational-pulp/

#10 Updated by mhrivnak over 3 years ago

  • Sprint/Milestone changed from 20 to 21

#11 Updated by semyers over 3 years ago

Quick update, progress is still being made. pcreech has converted the project from docker to Vagrant, which is awesome. The pulp platform models related to repos and units have been written down, and I'm currently porting RPM's units and repo-related models now so we can start looking at migrating data from nonrel-pulp to rel-pulp to see what explodes and needs to be revisited. :)

#12 Updated by mhrivnak over 3 years ago

  • Sprint/Milestone changed from 21 to 22

#13 Updated by mhrivnak about 3 years ago

  • Sprint/Milestone changed from 22 to 23

#14 Updated by semyers about 3 years ago

  • Status changed from ASSIGNED to CLOSED - CURRENTRELEASE

This is largely done (and by done, I mean now we can start to do it? :D). The rel-pulp doc and related repo are generally accepted by the team, and are now available in the pulp namespace on github:

https://github.com/pulp/relational-pulp - check out the db-translation-guide.md doc.

Most of the work was done by a cabal made up of jortel, pcreech, and myself with lots of great ideas and insights coming from all over the place, including stakeholders not on the pulp team. Thanks to all involved. More meetings will ensue, followed by more redmine tasks.

#15 Updated by bmbouter over 1 year ago

  • Sprint set to Sprint 5

#16 Updated by bmbouter over 1 year ago

  • Sprint/Milestone deleted (23)

#17 Updated by bmbouter 5 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF