Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2020-09-09T13:50:14ZPulp
Planio Pulp - Issue #7479 (NEW): pulp_installer source-upgrade CI is failing on pkg_resources.Contextual...https://pulp.plan.io/issues/74792020-09-09T13:50:14Zmdepaulo@redhat.com
<p>This occurs in debian-10 when running collect static after pulp_devel, not when running it after installing pulp via <code>pulp_all_services</code>.</p>
<p>It started occurring on daily CI and pull requests on 2020-09-8. I re-ran the overnight CI from the prior day (which originally succeeded), and when it did, it failed.</p>
<p>This error stands out as a dependency issue. Sometimes a package other than toml is listed though:</p>
<pre><code>pkg_resources.ContextualVersionConflict: (toml 0.10.1 (/usr/local/lib/pulp/lib/python3.7/site-packages), Requirement.parse('toml<=0.10.0'), {'dynaconf'})
</code></pre>
<p><a href="https://github.com/pulp/pulp_installer/runs/1091080108?check_suite_focus=true#step:7:1467" class="external">https://github.com/pulp/pulp_installer/runs/1091080108?check_suite_focus=true#step:7:1467</a></p>
<p>Full error:</p>
<pre><code> RUNNING HANDLER [pulp_common : Collect static content] *************************
Wednesday 09 September 2020 12:42:43 +0000 (0:00:02.148) 0:08:16.417 ***
fatal: [debian-10]: FAILED! => {
"changed": true,
"cmd": [
"/usr/local/lib/pulp/bin/django-admin",
"collectstatic",
"--noinput",
"--link"
],
"delta": "0:00:00.358256",
"end": "2020-09-09 12:42:43.925665",
"rc": 1,
"start": "2020-09-09 12:42:43.567409"
}
STDERR:
Traceback (most recent call last):
File "/usr/local/lib/pulp/bin/django-admin", line 8, in <module>
sys.exit(execute_from_command_line())
File "/usr/local/lib/pulp/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/usr/local/lib/pulp/lib/python3.7/site-packages/django/core/management/__init__.py", line 325, in execute
settings.INSTALLED_APPS
File "/usr/local/lib/pulp/lib/python3.7/site-packages/django/conf/__init__.py", line 79, in __getattr__
self._setup(name)
File "/usr/local/lib/pulp/lib/python3.7/site-packages/django/conf/__init__.py", line 66, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/pulp/lib/python3.7/site-packages/django/conf/__init__.py", line 157, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/pulp/lib/python3.7/site-packages/pulpcore/app/settings.py", line 76, in <module>
plugin_app_config = entry_point.load()
File "/usr/local/lib/pulp/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2410, in load
self.require(*args, **kwargs)
File "/usr/local/lib/pulp/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2433, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/lib/pulp/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (toml 0.10.1 (/usr/local/lib/pulp/lib/python3.7/site-packages), Requirement.parse('toml<=0.10.0'), {'dynaconf'})
MSG:
non-zero return code
changed: [fedora-31]
</code></pre> Pulp - Issue #7443 (ASSIGNED): pulp installer does not set ownership and permissions correctly be...https://pulp.plan.io/issues/74432020-09-02T10:23:03Zipanova@redhat.comipanova@redhat.com
<p>Some steps are skipped because user apache cannot be found and added to the pulp group <a href="https://github.com/pulp/pulp_installer/blob/master/roles/pulp_common/tasks/install.yml#L107-L133" class="external">https://github.com/pulp/pulp_installer/blob/master/roles/pulp_common/tasks/install.yml#L107-L133</a></p>
<pre><code>TASK [pulp_common : Find the nologin executable] *******************************
ok: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Make sure pulp group exists] *******************************
ok: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Create user vagrant] ***************************************
skipping: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Add user vagrant to extra groups] **************************
skipping: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Add user vagrant to pulp group] ****************************
changed: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Make sure /var/lib/pulp is world executable, and exists] ***
changed: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Create cache dir for Pulp] *********************************
changed: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Check if we have Pulp 2 installed] *************************
ok: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Add user 'apache' to 'pulp' group if it exists] ************
skipping: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Set permissions on '/var/lib/pulp' if pulp2 is installed] ***
skipping: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Find subdirs without setgid] *******************************
skipping: [pulp2-nightly-pulp3-source-centos7]
TASK [pulp_common : Set setgid on the /var/lib/pulp subdirs] *******************
skipping: [pulp2-nightly-pulp3-source-centos7]
</code></pre>
<p>After install finishes</p>
<pre><code>$ stat /var/lib/pulp
File: ‘/var/lib/pulp’
Size: 184 Blocks: 0 IO Block: 4096 directory
Device: fd01h/64769d Inode: 5121737 Links: 9
Access: (0775/drwxrwxr-x) Uid: ( 1000/ vagrant) Gid: ( 1001/ pulp)
Context: system_u:object_r:httpd_sys_rw_content_t:s0
Access: 2020-09-02 09:59:45.951659170 +0000
Modify: 2020-09-02 09:59:39.995633259 +0000
Change: 2020-09-02 09:59:39.995633259 +0000
Birth: -
$ ll /var/lib/pulp
total 8
-rw-r--r--. 1 apache apache 2 Sep 1 19:18 0005_puppet_module_name_change.txt
drwxrwxr-x. 7 vagrant vagrant 103 Sep 1 19:30 assets
-rw-r--r--. 1 root root 0 Sep 1 19:18 db_initialized.flag
drwxrwxr-x. 7 apache pulp 73 Sep 1 19:18 published
drwxr-xr-x. 3 vagrant pulp 25 Sep 1 19:25 pulpcore_static
drwxrwxr-x. 2 apache pulp 25 Sep 1 19:18 static
drwxrwxr-x. 7 vagrant pulp 4096 Sep 1 19:24 tmp
drwxrwxr-x. 2 apache pulp 6 Jul 13 15:40 uploads
</code></pre>
<p>There is no /var/lib/pulp/content because this is a fresh install. I have created and synced a pulp2 repo.
Directory is created however it does not belong to the pulp group, in addition the setgid is missing and there is no write permission for the group.</p>
<pre><code>
$ ll /var//lib/pulp
total 8
-rw-r--r--. 1 apache apache 2 Sep 1 19:18 0005_puppet_module_name_change.txt
drwxrwxr-x. 7 vagrant vagrant 103 Sep 1 19:30 assets
drwxr-xr-x. 3 apache apache 19 Sep 2 07:32 content
-rw-r--r--. 1 root root 0 Sep 1 19:18 db_initialized.flag
drwxrwxr-x. 7 apache pulp 73 Sep 1 19:18 published
drwxr-xr-x. 3 vagrant pulp 25 Sep 1 19:25 pulpcore_static
drwxrwxr-x. 2 apache pulp 25 Sep 1 19:18 static
drwxrwxr-x. 7 vagrant pulp 4096 Sep 1 19:24 tmp
drwxrwxr-x. 2 apache pulp 6 Jul 13 15:40 uploads
</code></pre>
<p>This makes it impossible to create hard link during the migration <a href="https://pulp.plan.io/issues/7244" class="external">https://pulp.plan.io/issues/7244</a></p> Pulp - Task #7281 (NEW): Update docs to state that installer can only install one cluster at a timehttps://pulp.plan.io/issues/72812020-08-05T14:39:19Zdkliban@redhat.com
<p>The documentation needs to have a "Known limitations" section. One of the items should state that that the installer can only install one Pulp cluster at a time.</p> Pulp - Story #7247 (NEW): As a pulp_installer developer-user, the pulp_rpm signing service will b...https://pulp.plan.io/issues/72472020-07-30T19:56:47Zmdepaulo@redhat.com
<p>The current way pulp_rpm's signing service needs to be installed is a temporary.</p>
<p>So let's add the current ansible-based solution I already developed. I developed it as part of the selinux el8 dev env, and it's in the pulp_devel (not meant for end users.)</p> Pulp - Issue #7136 (ASSIGNED): Requirement conflict when running RUNNING HANDLER [pulp.pulp_insta...https://pulp.plan.io/issues/71362020-07-14T17:07:05Zzen42@linux.com
<p>Tried to follow the install instructions here: <a href="https://docs.pulpproject.org/installation/instructions.html" class="external">https://docs.pulpproject.org/installation/instructions.html</a></p>
<p>I have done so on 2 diffrent boxes, one was RHEL7.8 the other a vagrant centos7</p>
<p>On both I hit the below error during the ansible run:</p>
<p>RUNNING HANDLER [pulp.pulp_installer.pulp_common : Collect static content] **********************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/local/lib/pulp/bin/django-admin", "collectstatic", "--noinput", "--link"], "delta": "0:00:00.262311", "end": "2020-07-14 16:46:52.375639", "msg": "non-zero return code", "rc": 1, "start": "2020-07-14 16:46:52.113328", "stderr": "Traceback (most recent call last):\n File "/usr/local/lib/pulp/bin/django-admin", line 8, in \n sys.exit(execute_from_command_line())\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/<strong>init</strong>.py", line 381, in execute_from_command_line\n utility.execute()\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/<strong>init</strong>.py", line 325, in execute\n settings.INSTALLED_APPS\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/<strong>init</strong>.py", line 79, in <strong>getattr</strong>\n self._setup(name)\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/<strong>init</strong>.py", line 66, in _setup\n self._wrapped = Settings(settings_module)\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/<strong>init</strong>.py", line 157, in <strong>init</strong>\n mod = importlib.import_module(self.SETTINGS_MODULE)\n File "/usr/lib64/python3.6/importlib/<strong>init</strong>.py", line 126, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File "", line 994, in _gcd_import\n File "", line 971, in _find_and_load\n File "", line 955, in _find_and_load_unlocked\n File "", line 665, in _load_unlocked\n File "", line 678, in exec_module\n File "", line 219, in _call_with_frames_removed\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py", line 73, in \n plugin_app_config = entry_point.load()\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/<strong>init</strong>.py", line 2317, in load\n self.require(*args, **kwargs)\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/<strong>init</strong>.py", line 2340, in require\n items = working_set.resolve(reqs, env, installer, extras=self.extras)\n File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/<strong>init</strong>.py", line 779, in resolve\n raise VersionConflict(dist, req).with_context(dependent_req)\npkg_resources.VersionConflict: (pulpcore 3.5.0 (/usr/local/lib/pulp/lib/python3.6/site-packages), Requirement.parse('pulpcore<3.5,>=3.4'))", "stderr_lines": ["Traceback (most recent call last):", " File "/usr/local/lib/pulp/bin/django-admin", line 8, in ", " sys.exit(execute_from_command_line())", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/<strong>init</strong>.py", line 381, in execute_from_command_line", " utility.execute()", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/<strong>init</strong>.py", line 325, in execute", " settings.INSTALLED_APPS", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/<strong>init</strong>.py", line 79, in <strong>getattr</strong>", " self._setup(name)", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/<strong>init</strong>.py", line 66, in _setup", " self._wrapped = Settings(settings_module)", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/<strong>init</strong>.py", line 157, in <strong>init</strong>", " mod = importlib.import_module(self.SETTINGS_MODULE)", " File "/usr/lib64/python3.6/importlib/<strong>init</strong>.py", line 126, in import_module", " return _bootstrap._gcd_import(name[level:], package, level)", " File "", line 994, in _gcd_import", " File "", line 971, in _find_and_load", " File "", line 955, in _find_and_load_unlocked", " File "", line 665, in _load_unlocked", " File "", line 678, in exec_module", " File "", line 219, in _call_with_frames_removed", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py", line 73, in ", " plugin_app_config = entry_point.load()", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/<strong>init</strong>.py", line 2317, in load", " self.require(*args, **kwargs)", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/<strong>init</strong>.py", line 2340, in require", " items = working_set.resolve(reqs, env, installer, extras=self.extras)", " File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/<strong>init</strong>.py", line 779, in resolve", " raise VersionConflict(dist, req).with_context(dependent_req)", "pkg_resources.VersionConflict: (pulpcore 3.5.0 (/usr/local/lib/pulp/lib/python3.6/site-packages), Requirement.parse('pulpcore<3.5,>=3.4'))"], "stdout": "", "stdout_lines": []}</p>
<p>The interesting bit is on the last line: Requirement.parse('pulpcore<3.5,>=3.4'))"]</p>
<p>I have tried and failed to find where this requirement is being stored but clearly it need updated for 3.5</p> Pulp - Story #7007 (NEW): As a user, I do not have to worry about Pulp being accidentally upgrade...https://pulp.plan.io/issues/70072020-06-18T15:40:06Zmdepaulo@redhat.com
<p>We should pursue using dnf versionlock to accomplish this.</p>
<p>This is needed because handlers/tasks "Run database migrations" will not be run if users run <code>dnf update</code>. Pulp would be broken until users re-run the installer.</p> Pulp - Task #6904 (NEW): Document using https://pypi.org/project/pulpcore-releases/ for the insta...https://pulp.plan.io/issues/69042020-06-03T15:25:07Zbmbouterbmbouter@redhat.com
<p>The Pulp Dependency Checker is a great tool to show compatibility between a pulpcore version and various concerns.</p>
<p>We should do three things:</p>
<ol>
<li>
<p>Move the pdc tool to the pulp org.</p>
</li>
<li>
<p>Add a very obvious link to the pulp_installer docs recommending users to use the tool to determine pulpcore and plugin compatibility</p>
</li>
<li>
<p>Update the error message that the installer puts out when the pre-flight check fails. Have that error message point users to specifically check which plugins are compatible with the pulpcore version the installer is trying to install.</p>
</li>
</ol> Pulp - Issue #6896 (NEW): [pulp_installer] Parametrize postgres package in roles/pulp_databasehttps://pulp.plan.io/issues/68962020-06-03T11:03:10Zspredzy
<p>As a user, while the default postgresql package is called rh-postgrelsql96 - which is fine - I would like to be able to specify a different package name, as we can specify various diffferents parameters name.</p>
<p>It is currently hardcoded at the playbook level[1] which prevents any overloading.</p>
<p>[1] <a href="https://github.com/pulp/pulp_installer/blob/master/roles/pulp_database/tasks/install_postgres.yml#L13" class="external">https://github.com/pulp/pulp_installer/blob/master/roles/pulp_database/tasks/install_postgres.yml#L13</a></p> Pulp - Task #6798 (NEW): Document the new guidelines for plugin installation logichttps://pulp.plan.io/issues/67982020-05-21T18:47:54Zmdepaulo@redhat.com
<p>There are 3 places they could be:</p>
<ol>
<li>A role in a separate git repo and on galaxy.</li>
<li>A separate role in the pulp_installer repo (pulp_rpm will be this.)</li>
<li>Conditional logic within the pulp_installer's other roles.</li>
</ol> Pulp - Story #6797 (ASSIGNED): [epic] As a user, I can consume all the plugin prereq roles in the...https://pulp.plan.io/issues/67972020-05-21T18:45:22Zmdepaulo@redhat.com
<p>pulp_rpm_prerequisites exists because the installer has had a plugin neutral policy.</p>
<p>This policy was for very long misunderstood: It's not about avoiding favoritism to any plugins, it's about not tying the installer (which is tied to pulpcore releases) to plugin releases. So that say pulpcore 3.3 logic would be in pulp_installer 3.3 release, and so that pulp_cardboardbox 0.7 logic would be in the pulp_cardboardbox_prerequisites 0.7 role.</p>
<p>The team now agrees that this policy is counter-productive because:</p>
<ol>
<li>Having a role in a separate repo (not part of the pulp_installer collection) is extra work for developers, and for users.</li>
<li>The only plugin that currently needs a prereq role, pulp_rpm, has version numbers and releases that correspond to pulpcore releases. pulp_rpm 3.3.z needs pulpcore 3.3.z, etc. So the pulp_rpm specific installation logic can be safely bundled in pulp_installer 99% of the time.</li>
</ol> Pulp - Task #6747 (NEW): Demo video for pulp_installerhttps://pulp.plan.io/issues/67472020-05-14T21:48:07Zfao89
<ul>
<li>Video should not have audio</li>
<li>
<a href="https://asciinema.org/" class="external">https://asciinema.org/</a> - records terminal output and can be embedded in our docs and in the README on github</li>
<li>include RPM and Container plugins</li>
</ul> Pulp - Story #6688 (NEW): pulp_installer: preflight check and system-wide packages are incompatiblehttps://pulp.plan.io/issues/66882020-05-08T14:40:15Zmdepaulo@redhat.com
<p>Part of the pre-flight check does not understand system-wide packages, but another part is still affected by them.</p>
<p>This leads to false positives (enforcements) in addition to false negatives in the preflight check.</p>
<p>We no longer need system-wide packages, so we should remove support for it, and migrate user installs off of it, as safely as possible.</p> Pulp - Task #6306 (ASSIGNED): Request EPEL8 versions of packages in the pulp-devel rolehttps://pulp.plan.io/issues/63062020-03-06T21:22:23Zmdepaulo@redhat.com
<p>This PR has to do some workarounds for EL8 support, because the packages were not in EPEL8 yet:
<a href="https://github.com/pulp/ansible-pulp/pull/243/files#" class="external">https://github.com/pulp/ansible-pulp/pull/243/files#</a></p>
<p>Ignoring some helpful developing tools packages:
jnettop
fd-find
fzf</p>
<p>and Installing F28 (Python 3.6) versions of a package we needt:
python3-virtualenvwrapper</p>
<p>and its deps:
python3-virtualenv-clone
python3-stevedore</p>
<p>We should request that they be packaged for EPEL8.
See "## Consumer request for packages"
<a href="https://lists.fedoraproject.org/archives/list/epel-announce@lists.fedoraproject.org/thread/KXMMLYSAXAVHDKFFBVEFYYZHPJBWXOQQ/" class="external">https://lists.fedoraproject.org/archives/list/epel-announce@lists.fedoraproject.org/thread/KXMMLYSAXAVHDKFFBVEFYYZHPJBWXOQQ/</a></p>
<p>And then added to the list of packages to install as normal.</p> Pulp - Issue #6274 (NEW): Problem: existing Pulp deployments with system-wide packages conflict w...https://pulp.plan.io/issues/62742020-03-03T19:51:59Zmdepaulo@redhat.com
<p>Note: This issue is specificaly about running ansible-pulp against existing deployments of Pulp.</p>
<p>Manual testing with <a href="https://quay.io/repository/pulp/pulp_rpm-ci-c7" class="external">centos 7 pulp_rpm images</a> reveals that we can modify /usr/local/lib/pulp/pyvenv.cfg to not use system-wide packages anymore, and then let the <code>pip install</code> commands fix the missing dependencies. So this is the approach that ansible-pulp should take when <code>pulp_use_system_wide_pkgs</code> is false (or perhaps whenever centos 7 is in use.)</p> Pulp - Story #5618 (NEW): As a user, I can download & run a version of the ansible installer that...https://pulp.plan.io/issues/56182019-10-25T08:37:28Zmdepaulo@redhat.com
<p>Currently users are encouraged to get the latest ansible-pulp roles via git cloning. Later on, Ansible Galaxy.</p>
<p>The only stable tag ever done was 3.0.0rc1. Presumably we will create them for 3.0.0 and later.<br>
<a href="https://github.com/pulp/ansible-pulp/releases" class="external">https://github.com/pulp/ansible-pulp/releases</a></p>
<p>However, consider the following scenario (hypothetical release dates):<br>
1. They download the roles (either method) on Apr 1. They are versioned as 3.0.3 and install pulp 3.0.3<br>
2. They run them against their test env and it works.<br>
3. Pulp 3.1.0 & ansible-pulp 3.1.0 are released on Apr 15.<br>
4. They run the 3.0.3 roles against their prod env on May 1.<br>
5. The 3.0.3 roles try to install pulp 3.1.0 from pip, but fails due to the lack of new logic.</p>
<p>It would make sense to have a variable for the pulp version to install, that defaults to the same version as the roles, but can be overriden (but doing so is discouraged.)</p>
<p>Plugin versions would also be an issue. Let's discuss how this can be handled.</p>
<p>Also, I am not sure if there is an existing task for publishing the roles (other than pulp_rpm_prerequisites) to Ansible Galaxy (pulp project on it.):<br>
<a href="https://galaxy.ansible.com/pulp" class="external">https://galaxy.ansible.com/pulp</a></p>