Plan replacement of mongodb with postgres
#2 Updated by semyers over 4 years ago
I'm in the process of writing all this in more detail in an etherpad, but for now I'll outline the work so far.
pcreech and I started by basically just writing out all of models we had, and trying to organize them in a diagram.
This went poorly, because pulp has four distinct categories of models, and the tool we were using didn't make it very easy to diagram them all together. With all the pulp releasing I've been doing, and all the RHUI pcreech has been doing, all of this finally came together toward the end of last week while jortel was on PTO. Before unleashing the confusing and questionably useful diagrams on the team, I wanted to get jortel's feedback. I was not disappointed.
With his help, we've identified the aformentioned data categories: Repositories, Content Units, RBAC, Tasks. I'm writing up a doc right now that explains what collections we currently have in mongo, to which category they belong, how the currently relate to each other, and some speculation about how we can migrate the data in that category to a more relational design. Since I'll be speculating, this doc will go up in an etherpad for review and improvement by the small team focused on this at the moment (jortel, pcreech, me), before being submitted to the team at large.
Depending on how quickly I get this done and reviewed, the link to that etherpad should most likely be appearing here tomorrow.
#9 Updated by semyers about 4 years ago
The migration plan is largely "ratified", in that objections to it have been (or are currently being) addressed. I feel like we've reached a point where what we don't know outweighs what we do know, and the best way I can think of to bridge the knowledge gap is to finishing modeling pulp out on postgres and start trying to use it. I'll be modifying my relational pulp project to this end over the coming days.
#11 Updated by semyers about 4 years ago
Quick update, progress is still being made. pcreech has converted the project from docker to Vagrant, which is awesome. The pulp platform models related to repos and units have been written down, and I'm currently porting RPM's units and repo-related models now so we can start looking at migrating data from nonrel-pulp to rel-pulp to see what explodes and needs to be revisited. :)
#14 Updated by semyers about 4 years ago
- Status changed from ASSIGNED to CLOSED - CURRENTRELEASE
This is largely done (and by done, I mean now we can start to do it? :D). The rel-pulp doc and related repo are generally accepted by the team, and are now available in the pulp namespace on github:
https://github.com/pulp/relational-pulp - check out the db-translation-guide.md doc.
Most of the work was done by a cabal made up of jortel, pcreech, and myself with lots of great ideas and insights coming from all over the place, including stakeholders not on the pulp team. Thanks to all involved. More meetings will ensue, followed by more redmine tasks.
Please register to edit this issue