Project

Profile

Help

Task #2413

Fix dev environment setup for Pulp 3.0

Added by semyers almost 3 years ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 17

Description

The setup scripts (including vagrant and ansible playbooks) for a pulp 3.0 dev environment evolved out of pulp 2, and have done so in an unstructured way. As a result, these scripts are in need of improvement and organization.


Checklist


Related issues

Blocks Pulp - Task #2479: Update systemd tasking service files to use new Pulp 3 locations MODIFIED Actions

Associated revisions

Revision 9142a6ac View on GitHub
Added by semyers almost 3 years ago

Overhaul ansible playbook and Vagrantfile for 3.0

This isn't complete, largely because pulp's own 3.0-dev branch isn't
itself complete (we haven't yet removed all subpackages provided in
pulp 2, plugins aren't converted to pulp 3 yet, etc.).

The vagrant setup script has been destroyed, and its functionality
has been implemented in ansible. The ansible pulp_facts module has
been significantly improved, and should be much easier to maintain
by (and be in general less scary to) Pulp developers in general.

Additionally, support for the pulpproject.org repository has been
added (ruby deps are installed, and bundler is run on that repo's
Gemfile)

re #2413
https://pulp.plan.io/issues/2413

History

#1 Updated by mhrivnak almost 3 years ago

  • Sprint/Milestone set to 29
  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#2 Updated by semyers almost 3 years ago

There are things that we like to have in the dev environments that aren't needed at all for "production" pulp. This includes things like "django-crispy-forms" for adding prettyness to the browseable API, installing IPython to make "manage.py shell" be way more usable, etc, so I've added this as a checklist item.

#3 Updated by semyers almost 3 years ago

  • Checklist item developer-only dependencies should be clearly separate from pulp dependencies added

#4 Updated by semyers almost 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to semyers

#5 Updated by semyers almost 3 years ago

I'm currently taking some time and getting to know ansible better than I currently do in hopes of getting the entire dev setup process to work entirely in ansible. Ansible 2.2 recently came out, and possibly allows for working entirely in python3, so we might no longer need to bootstrap with python2 + the python2 selinux libs package.

In addition to the errors that we see as a result of attempting to install pulp2 stuff, the pulp3 dev env is currently suffering from sphinx being installed into the python 2 system env, so it's not available to py3 envs. Also, pulp3 subpackages (common, exceptions, tasking) aren't being installed by the current scripts, so they need to be managed; hopefully in an easy to implement way that we can also implement in the plugins?

Since so many folks keep getting hit by the uninstalled subpackages, I'll likely just put a bandaid on that first and then dig into the deeper ansible and documentation improvements afterward.

#6 Updated by semyers almost 3 years ago

semyers wrote:

I'm currently taking some time and getting to know ansible better than I currently do in hopes of getting the entire dev setup process to work entirely in ansible. Ansible 2.2 recently came out, and possibly allows for working entirely in python3, so we might no longer need to bootstrap with python2 + the python2 selinux libs package.

Ansible works perfectly, so far, with only py3 installed on the guest vm.

In addition to the errors that we see as a result of attempting to install pulp2 stuff, the pulp3 dev env is currently suffering from sphinx being installed into the python 2 system env, so it's not available to py3 envs. Also, pulp3 subpackages (common, exceptions, tasking) aren't being installed by the current scripts, so they need to be managed; hopefully in an easy to implement way that we can also implement in the plugins?

I'm experimenting with a yaml file for this, something like 'dev-env.yaml' that is able to identify which directories contain setup.pys that need to be installed, and also provide a way to identify additional requirements that for whatever reason shouldn't be installed in the packaged product (things like ipython come to mind, and also build-only requirements like sphinx)

Since so many folks keep getting hit by the uninstalled subpackages, I'll likely just put a bandaid on that first and then dig into the deeper ansible and documentation improvements afterward.

I lied :(

It was easy enough to get ansible running on py3 that I've actually been breaking the existing playbook up into more roles as appropriate, getting rid of as much vagrant-specific behavior as I can, and converting the "vagrant-setup" script to ansible. That's almost done, and will be the first PR for this effort, at which point I hope to follow up with some related quality of life fixes that should become trivial once all the work lives in ansible. The difficulty with this now is making sure ansible and any bash scripts have the same information. To date, we've solved this by hardcoding paths, like ~/devel, but once they're all in ansible it might be trivial to make this a customizeable value.

#7 Updated by semyers almost 3 years ago

As I'd hoped, I was able to get my playbook working yesterday, which completely removed the need to have the vagrant setup script. Once that was working, I started trying to break it; this did not take as long as I would have liked. Short story: Ansible loops are not pretty things. Adding conditionals to them makes them even less pretty, and doing crazy things like nesting them or looping over more exotic datatypes (like, say, a dict), makes them downright unreadable. In order for ansible to correctly set up the virtualenvs for our plugins, we need to do a lot of these un-pretty things if we put all the logic in our playbook tasks.

Fortunately, we don't have to do that! We've already got the pulp_facts module0 doing some work to pre-render some conditional results for use later in the playbook. Today, I've been cleaning that guy up (encapsulating each fact-getter into a function, using ansible's python interface1 to define the module, etc) and adding in the conditional looping logic that I need in the tasks, so rather than trying to do all the work in the tasks, it gets done beforehand and the tasks can just iterate.

#8 Updated by semyers almost 3 years ago

  • Checklist item errors related to broken Pulp 2 services should not appear set to Done
  • Checklist item the setup scripts should only attempt to install Pulp 3 packages and services set to Done
  • Checklist item the pulp 3 development environment should be functional once the scripts finish set to Done

#9 Updated by semyers almost 3 years ago

  • Checklist item the setup scripts should only attempt to install Pulp 3 packages and services set to Done

#10 Updated by semyers almost 3 years ago

The playbook works now, cleanly handling missing plugins/side projects (e.g. you didn't clone crane, pulp-smash, pulp-python, etc). It's a big pile of WIP commits right now, so tomorrow I'll do a little more testing, get those commits crushed down and pushed up to devel, and then address the checklist items that have to happen over in the pulp repo.

#11 Updated by semyers almost 3 years ago

pushed for testing: https://github.com/pulp/devel/pull/21

I'm still making minor tweaks and getting the documentation together, but it should be functional now.

#12 Updated by semyers almost 3 years ago

Note that once that's accepted, there are probably still two more PRs:
- docs PR for the 4th checkbox
- other changes to the pulp repo to take advantage of the new playbook functionality, mainly by separating out dev-only dependencies like ipython, crispy forms, etc, completing the 5th checklist item
- likely a trivial PR to devel to let the playbook know about the new requirements file locations

#13 Updated by mhrivnak almost 3 years ago

  • Sprint/Milestone changed from 29 to 30

#14 Updated by semyers almost 3 years ago

  • Blocks Task #2479: Update systemd tasking service files to use new Pulp 3 locations added

#15 Updated by semyers over 2 years ago

  • Priority changed from Normal to High

With the 2.12.2 beta out the door and the team poised to return to 3.0 this week, this has become a fair bit more important. :)

#17 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone changed from 30 to 36

#19 Updated by bmbouter over 1 year ago

  • Sprint set to Sprint 17

#20 Updated by bmbouter over 1 year ago

  • Sprint/Milestone deleted (36)

#21 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#22 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF