Project

Profile

Help

Story #2840

Publish ansible roles to install pulp3

Added by bizhang over 2 years ago. Updated 8 months ago.

Status:
MODIFIED
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 25

Description

Motivation

Pulp3 alpha needs to be fully installable, the easiest way of doing this is upload the necessary ansible roles onto ansible galaxy.

A lot of the roles from pulp/devel can be used as the basis for the galaxy roles including:

  • db [0]
  • systemd [1]

A new role would need to be written to install pulp from PyPI.
We can use jytr's roles for rabbitmq and qpidd [2]

All the roles needs to work in RHEL7 and Fedora 25 (preferably be platform agnostic)

This playbook should be published on ansible galaxy, and instructions should be published in the pulp3 installation document [3].

[0] https://github.com/pulp/devel/tree/de6a8351c2aba5a0ded3228ecce0448844101e4d/ansible/roles/db
[1] https://github.com/pulp/devel/tree/de6a8351c2aba5a0ded3228ecce0448844101e4d/ansible/roles/systemd
[2] https://galaxy.ansible.com/jtyr/
[3] https://github.com/pulp/pulp/blob/589f0e74f17f25442d6720b30e09e40b839a292e/docs/installation.rst


Checklist


Related issues

Related to External - Task #2694: document how to deploy Pulp using roles on Ansible Galaxy CLOSED - WONTFIX Actions

Associated revisions

Revision 28599f48 View on GitHub
Added by ipanova@redhat.com about 2 years ago

Publish ansible galaxy playbook for pulp3 pypi installation.

closes #2840
https://pulp.plan.io/issues/2840

History

#1 Updated by bizhang over 2 years ago

  • Description updated (diff)

#2 Updated by bizhang over 2 years ago

  • Subject changed from Publish ansible playbook to install pulp3 to Publish ansible roles to install pulp3
  • Description updated (diff)

#3 Updated by bmbouter over 2 years ago

There are also Ansible roles (including Pulp2 installers) made by a user called jtyr on galaxy: https://galaxy.ansible.com/jtyr/ He has a rabbitMQ and Qpid role. Could we publish a playbook that calls his roles and then runs the to-be-created role that installs Pulp from PyPI.

Then the role published through this task would:
  • install Pulp via pip
  • Install the systemd unit files
  • install the httpd or nginx configs

I'm not sure what provides the server.conf. I imagine it would probably be included via MANIFEST.in? If so then Ansible would get it when it pip installs Pulp from source via github.

#4 Updated by bizhang over 2 years ago

  • Description updated (diff)

server.conf should be included in the pulpcore python package.

Using jtyr's packages sounds good to me. I've updated the description to match.

#5 Updated by bmbouter over 2 years ago

  • Groomed changed from No to Yes

Thank you @bizhang for writing this up!

#6 Updated by mhrivnak over 2 years ago

This will also require httpd and a new config file for httpd. We should not re-use the httpd config from Pulp 2.

#7 Updated by mhrivnak over 2 years ago

  • Related to Task #2694: document how to deploy Pulp using roles on Ansible Galaxy added

#8 Updated by bmbouter over 2 years ago

Adding a checklist item to create the /var/lib/pulp directory as part of the Ansible playbook

#9 Updated by bmbouter over 2 years ago

Adding a checklist item to add the 'pulp' user to the Ansible playbook.

#10 Updated by bizhang over 2 years ago

  • Tags Pulp 3 Plugin Writer Alpha added

#11 Updated by amacdona@redhat.com over 2 years ago

  • Blocks Task #2908: Build a plugin writer's Vagrant VM and publish on Vagrant cloud on a Pulp account added

#12 Updated by mhrivnak over 2 years ago

  • Priority changed from Normal to Low

#13 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone set to 42

#14 Updated by ipanova@redhat.com over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#15 Updated by mhrivnak over 2 years ago

  • Priority changed from Low to Normal

#16 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone changed from 42 to 43

#17 Updated by jortel@redhat.com over 2 years ago

  • Sprint/Milestone changed from 43 to 44

#18 Updated by ipanova@redhat.com about 2 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100

#20 Updated by Ichimonji10 about 2 years ago

These playbooks fail on RHEL 7:

$ ansible-playbook deploy-pulp3.yml -e pulp3_broker=rabbitmq -i rhel-7-test,                 

PLAY [all] ************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************
ok: [rhel-7-test]            

TASK [Require minimal Python and Ansible versions] ********************************************************************
fatal: [rhel-7-test]: FAILED! => {                         
    "assertion": "ansible_python_version|version_compare('3.5.0', operator='ge')",                                     
    "changed": false,        
    "evaluated_to": false,   
    "failed": true,          
    "msg": "This role and those that depend on it require at least Python 3.5 and Ansible 2.2.\n"                      
}                            
        to retry, use: --limit @/home/ichimonji10/code/devel/ansible/deploy-pulp3.retry                                

PLAY RECAP ************************************************************************************************************
rhel-7-test                : ok=1    changed=0    unreachable=0    failed=1

I believe the issue is that RHEL 7 provides Python 3.4:

$ yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id                                                                                                         repo name                                                                                                                status
epel/x86_64                                                                                                     Extra Packages for Enterprise Linux 7 - x86_64                                                                           11,947
rhel-7-server-extras-rpms/x86_64                                                                                Red Hat Enterprise Linux 7 Server - Extras (RPMs)                                                                           639
rhel-7-server-optional-rpms/7Server/x86_64                                                                      Red Hat Enterprise Linux 7 Server - Optional (RPMs)                                                                      12,809
rhel-7-server-rpms/7Server/x86_64                                                                               Red Hat Enterprise Linux 7 Server (RPMs)                                                                                 17,215
repolist: 42,610

#21 Updated by Ichimonji10 about 2 years ago

These playbooks don't function on Fedora 25. The failures don't become obvious until after 30+ minutes of runtime. Here's a summary of what eventually goes wrong:

[root@fedora-25-test ~]# systemctl --state failed
  UNIT                          LOAD   ACTIVE SUB    DESCRIPTION
● pulp_resource_manager.service loaded failed failed Pulp Resource Manager
● pulp_worker@1.service         loaded failed failed Pulp Celery Worker
● pulp_worker@2.service         loaded failed failed Pulp Celery Worker

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

3 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

The issue appears to be that pulp_resource_manager can't connect to RabbitMQ. I suspect using a third-party role to install and configure RabbitMQ is overkill.

# journalctl -u pulp_resource_manager
-- Logs begin at Tue 2017-09-19 11:54:18 EDT, end at Tue 2017-09-19 14:37:49 EDT. --
Sep 19 12:07:17 fedora-25-test systemd[1]: Started Pulp Resource Manager.
Sep 19 12:07:22 fedora-25-test celery[26720]: [2017-09-19 16:07:22,334: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Sep 19 12:07:22 fedora-25-test celery[26720]: Trying again in 2.00 seconds...
Sep 19 12:07:24 fedora-25-test celery[26720]: [2017-09-19 16:07:24,371: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Sep 19 12:07:24 fedora-25-test celery[26720]: Trying again in 4.00 seconds...
Sep 19 12:07:28 fedora-25-test celery[26720]: [2017-09-19 16:07:28,387: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Sep 19 12:07:28 fedora-25-test celery[26720]: Trying again in 6.00 seconds...
Sep 19 12:07:34 fedora-25-test celery[26720]: [2017-09-19 16:07:34,424: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Sep 19 12:07:34 fedora-25-test celery[26720]: Trying again in 8.00 seconds...
Sep 19 12:07:42 fedora-25-test celery[26720]: [2017-09-19 16:07:42,451: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Sep 19 12:07:42 fedora-25-test celery[26720]: Trying again in 10.00 seconds...
Sep 19 12:07:52 fedora-25-test celery[26720]: [2017-09-19 16:07:52,482: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Sep 19 12:07:52 fedora-25-test celery[26720]: Trying again in 12.00 seconds...
Sep 19 12:08:04 fedora-25-test celery[26720]: [2017-09-19 16:08:04,505: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Sep 19 12:08:04 fedora-25-test celery[26720]: Trying again in 14.00 seconds...
Sep 19 12:08:18 fedora-25-test celery[26720]: [2017-09-19 16:08:18,530: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.

(snip!)

Sep 19 12:56:46 fedora-25-test celery[26720]: [2017-09-19 16:56:46,885: CRITICAL/MainProcess] Unrecoverable error: OperationalError('[Errno 111] Connection refused',)
Sep 19 12:56:46 fedora-25-test celery[26720]: Traceback (most recent call last):
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
Sep 19 12:56:46 fedora-25-test celery[26720]:     yield
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/kombu/connection.py", line 405, in ensure_connection
Sep 19 12:56:46 fedora-25-test celery[26720]:     callback)
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
Sep 19 12:56:46 fedora-25-test celery[26720]:     return fun(*args, **kwargs)
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/kombu/connection.py", line 261, in connect
Sep 19 12:56:46 fedora-25-test celery[26720]:     return self.connection
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/kombu/connection.py", line 802, in connection
Sep 19 12:56:46 fedora-25-test celery[26720]:     self._connection = self._establish_connection()
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/kombu/connection.py", line 757, in _establish_connection
Sep 19 12:56:46 fedora-25-test celery[26720]:     conn = self.transport.establish_connection()
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
Sep 19 12:56:46 fedora-25-test celery[26720]:     conn.connect()
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/amqp/connection.py", line 282, in connect
Sep 19 12:56:46 fedora-25-test celery[26720]:     self.transport.connect()
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/amqp/transport.py", line 109, in connect
Sep 19 12:56:46 fedora-25-test celery[26720]:     self._connect(self.host, self.port, self.connect_timeout)
Sep 19 12:56:46 fedora-25-test celery[26720]:   File "/home/pulp/pulpvenv/lib/python3.5/site-packages/amqp/transport.py", line 150, in _connect
Sep 19 12:56:46 fedora-25-test celery[26720]:     self.sock.connect(sa)
Sep 19 12:56:46 fedora-25-test celery[26720]: ConnectionRefusedError: [Errno 111] Connection refused
Sep 19 12:56:46 fedora-25-test celery[26720]: During handling of the above exception, another exception occurred:

(snip!)

#22 Updated by ipanova@redhat.com about 2 years ago

on el7 python3.5 is available trough software collections, maybe worth explicitly mentioning that in docs.
w/r to the broker probably some config parameter needs to be tweaked, maybe the defaults ones do not satisfy our specific install.

#23 Updated by ipanova@redhat.com about 2 years ago

We decided for now not to include the el7 support.
The broker connection issue was fixed, thanks for testing and pointing this out

#24 Updated by bmbouter almost 2 years ago

  • Tags deleted (Pulp 3 Plugin Writer Alpha)

Cleaning up Redmine tags

#25 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 25

#26 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (44)

#27 Updated by dkliban@redhat.com over 1 year ago

  • Blocks deleted (Task #2908: Build a plugin writer's Vagrant VM and publish on Vagrant cloud on a Pulp account)

#28 Updated by daviddavis 8 months ago

  • Sprint/Milestone set to 3.0

#29 Updated by bmbouter 8 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF