Pulp

There are things that we like to have in the dev environments that aren't needed at all for "production" pulp. This includes things like "django-crispy-forms" for adding prettyness to the browseable API, installing IPython to make "manage.py shell" be way more usable, etc, so I've added this as a checklist item.

I'm currently taking some time and getting to know ansible better than I currently do in hopes of getting the entire dev setup process to work entirely in ansible. Ansible 2.2 recently came out, and possibly allows for working entirely in python3, so we might no longer need to bootstrap with python2 + the python2 selinux libs package.

In addition to the errors that we see as a result of attempting to install pulp2 stuff, the pulp3 dev env is currently suffering from sphinx being installed into the python 2 system env, so it's not available to py3 envs. Also, pulp3 subpackages (common, exceptions, tasking) aren't being installed by the current scripts, so they need to be managed; hopefully in an easy to implement way that we can also implement in the plugins?

Since so many folks keep getting hit by the uninstalled subpackages, I'll likely just put a bandaid on that first and then dig into the deeper ansible and documentation improvements afterward.

semyers wrote:

I'm currently taking some time and getting to know ansible better than I currently do in hopes of getting the entire dev setup process to work entirely in ansible. Ansible 2.2 recently came out, and possibly allows for working entirely in python3, so we might no longer need to bootstrap with python2 + the python2 selinux libs package.

Ansible works perfectly, so far, with only py3 installed on the guest vm.

In addition to the errors that we see as a result of attempting to install pulp2 stuff, the pulp3 dev env is currently suffering from sphinx being installed into the python 2 system env, so it's not available to py3 envs. Also, pulp3 subpackages (common, exceptions, tasking) aren't being installed by the current scripts, so they need to be managed; hopefully in an easy to implement way that we can also implement in the plugins?

I'm experimenting with a yaml file for this, something like 'dev-env.yaml' that is able to identify which directories contain setup.pys that need to be installed, and also provide a way to identify additional requirements that for whatever reason shouldn't be installed in the packaged product (things like ipython come to mind, and also build-only requirements like sphinx)

Since so many folks keep getting hit by the uninstalled subpackages, I'll likely just put a bandaid on that first and then dig into the deeper ansible and documentation improvements afterward.

I lied :(

It was easy enough to get ansible running on py3 that I've actually been breaking the existing playbook up into more roles as appropriate, getting rid of as much vagrant-specific behavior as I can, and converting the "vagrant-setup" script to ansible. That's almost done, and will be the first PR for this effort, at which point I hope to follow up with some related quality of life fixes that should become trivial once all the work lives in ansible. The difficulty with this now is making sure ansible and any bash scripts have the same information. To date, we've solved this by hardcoding paths, like ~/devel, but once they're all in ansible it might be trivial to make this a customizeable value.

As I'd hoped, I was able to get my playbook working yesterday, which completely removed the need to have the vagrant setup script. Once that was working, I started trying to break it; this did not take as long as I would have liked. Short story: Ansible loops are not pretty things. Adding conditionals to them makes them even less pretty, and doing crazy things like nesting them or looping over more exotic datatypes (like, say, a dict), makes them downright unreadable. In order for ansible to correctly set up the virtualenvs for our plugins, we need to do a lot of these un-pretty things if we put all the logic in our playbook tasks.

Fortunately, we don't have to do that! We've already got the pulp_facts module[0] doing some work to pre-render some conditional results for use later in the playbook. Today, I've been cleaning that guy up (encapsulating each fact-getter into a function, using ansible's python interface[1] to define the module, etc) and adding in the conditional looping logic that I need in the tasks, so rather than trying to do all the work in the tasks, it gets done beforehand and the tasks can just iterate.

The playbook works now, cleanly handling missing plugins/side projects (e.g. you didn't clone crane, pulp-smash, pulp-python, etc). It's a big pile of WIP commits right now, so tomorrow I'll do a little more testing, get those commits crushed down and pushed up to devel, and then address the checklist items that have to happen over in the pulp repo.

pushed for testing: https://github.com/pulp/devel/pull/21

I'm still making minor tweaks and getting the documentation together, but it should be functional now.

Note that once that's accepted, there are probably still two more PRs:
- docs PR for the 4th checkbox
- other changes to the pulp repo to take advantage of the new playbook functionality, mainly by separating out dev-only dependencies like ipython, crispy forms, etc, completing the 5th checklist item
- likely a trivial PR to devel to let the playbook know about the new requirements file locations