https://pulp.plan.io/https://pulp.plan.io/favicon.ico2017-06-02T03:58:53ZPulpRPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=199812017-06-02T03:58:53Zrmcgoverrmcgover@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/19981/diff?detail_id=20587">diff</a>)</li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=199892017-06-02T12:39:17Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>The publishes that will get slower, were meant to be done with rsync distributor.</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=199902017-06-02T12:59:31Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>i had a chat with <a class="user active" href="https://pulp.plan.io/users/24">jluza</a>, we believe that If we fix this issue <a href="https://pulp.plan.io/issues/2783" class="external">https://pulp.plan.io/issues/2783</a>, the current one will not have slowdown in the rsync publish, because the file would have preserved the mitime and would not get rsynced</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=199982017-06-02T14:44:36Zttereshcttereshc@redhat.com
<ul><li><strong>Triaged</strong> changed from <i>No</i> to <i>Yes</i></li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=200152017-06-04T23:44:07Zrmcgoverrmcgover@redhat.com
<ul></ul><p>Both yum and rsync publishes will become slower due to this (though rsync is worse affected).</p>
<p>For example, we had a repo with a repodata directory containing about 28GB worth of sqlite files and that's enough to noticeably slow down the "copy files" step in yum publishes.</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=201612017-06-12T12:33:13Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>jluza</i></li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=202932017-06-16T14:33:38Zjluzajluza@redhat.com
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>POST</i></li></ul><p>PR <a href="https://github.com/pulp/pulp_rpm/pull/1056" class="external">https://github.com/pulp/pulp_rpm/pull/1056</a></p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=216442017-08-15T15:33:55Zjluzajluza@redhat.com
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>MODIFIED</i></li></ul><p>Applied in changeset <a class="changeset" title="RemoveOldRepodataStep for yum publisher New publish step removes repodata older than threshold w..." href="https://pulp.plan.io/projects/pulp_rpm/repository/9/revisions/7457f4ab3fae861ac590ad584b64d655d8c91074">7457f4ab3fae861ac590ad584b64d655d8c91074</a>.</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=216452017-08-15T15:49:05Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Tracker</strong> changed from <i>Issue</i> to <i>Story</i></li><li><strong>Subject</strong> changed from <i>sqlite files are retained indefinitely</i> to <i>As a user i can configure removal of old published repodata </i></li><li><strong>% Done</strong> set to <i>0</i></li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=229982017-11-29T02:22:01Zpcreechpcreech@redhat.com
<ul><li><strong>Platform Release</strong> set to <i>2.15.0</i></li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=230662017-11-30T18:29:32Zpcreechpcreech@redhat.com
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>5</i></li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=232742017-12-12T22:25:00ZIchimonji10jerebear@protonmail.com
<ul><li><strong>Status</strong> changed from <i>5</i> to <i>ASSIGNED</i></li></ul><p>Tested against Fedora 25 and Fedora 26 with Pulp 2.15 beta 2 installed. Here's the relevant packages on F25:</p>
<pre><code>[root@fedora-25-pulp-2-15-beta ~]# rpm -qa | grep pulp | sort
pulp-admin-client-2.15.0-0.2.beta.fc25.noarch
pulp-deb-admin-extensions-1.6.0-0.2.beta.fc25.noarch
pulp-deb-plugins-1.6.0-0.2.beta.fc25.noarch
pulp-docker-admin-extensions-3.1.0-0.3.beta.fc25.noarch
pulp-docker-plugins-3.1.0-0.3.beta.fc25.noarch
pulp-ostree-admin-extensions-1.3.0-1.fc25.noarch
pulp-ostree-plugins-1.3.0-1.fc25.noarch
pulp-puppet-admin-extensions-2.15.0-0.2.beta.fc25.noarch
pulp-puppet-plugins-2.15.0-0.2.beta.fc25.noarch
pulp-puppet-tools-2.15.0-0.2.beta.fc25.noarch
pulp-python-admin-extensions-2.0.2-1.fc25.noarch
pulp-python-plugins-2.0.2-1.fc25.noarch
pulp-rpm-admin-extensions-2.15.0-0.2.beta.fc25.noarch
pulp-rpm-plugins-2.15.0-0.2.beta.fc25.noarch
pulp-selinux-2.15.0-0.2.beta.fc25.noarch
pulp-server-2.15.0-0.2.beta.fc25.noarch
python-kombu-3.0.33-8.pulp.fc25.noarch
python-pulp-bindings-2.15.0-0.2.beta.fc25.noarch
python-pulp-client-lib-2.15.0-0.2.beta.fc25.noarch
python-pulp-common-2.15.0-0.2.beta.fc25.noarch
python-pulp-deb-common-1.6.0-0.2.beta.fc25.noarch
python-pulp-docker-common-3.1.0-0.3.beta.fc25.noarch
python-pulp-oid_validation-2.15.0-0.2.beta.fc25.noarch
python-pulp-ostree-common-1.3.0-1.fc25.noarch
python-pulp-puppet-common-2.15.0-0.2.beta.fc25.noarch
python-pulp-python-common-2.0.2-1.fc25.noarch
python-pulp-repoauth-2.15.0-0.2.beta.fc25.noarch
python-pulp-rpm-common-2.15.0-0.2.beta.fc25.noarch
python-pulp-streamer-2.15.0-0.2.beta.fc25.noarch
</code></pre>
<p>And F26:</p>
<pre><code>[root@fedora-26-pulp-2-15-beta ~]# rpm -qa | grep pulp | sort
pulp-admin-client-2.15.0-0.2.beta.fc26.noarch
pulp-deb-admin-extensions-1.6.0-0.2.beta.fc26.noarch
pulp-deb-plugins-1.6.0-0.2.beta.fc26.noarch
pulp-docker-admin-extensions-3.1.0-0.3.beta.fc26.noarch
pulp-docker-plugins-3.1.0-0.3.beta.fc26.noarch
pulp-ostree-admin-extensions-1.3.0-1.fc26.noarch
pulp-ostree-plugins-1.3.0-1.fc26.noarch
pulp-puppet-admin-extensions-2.15.0-0.2.beta.fc26.noarch
pulp-puppet-plugins-2.15.0-0.2.beta.fc26.noarch
pulp-puppet-tools-2.15.0-0.2.beta.fc26.noarch
pulp-python-admin-extensions-2.0.2-1.fc26.noarch
pulp-python-plugins-2.0.2-1.fc26.noarch
pulp-rpm-admin-extensions-2.15.0-0.2.beta.fc26.noarch
pulp-rpm-plugins-2.15.0-0.2.beta.fc26.noarch
pulp-selinux-2.15.0-0.2.beta.fc26.noarch
pulp-server-2.15.0-0.2.beta.fc26.noarch
python-pulp-bindings-2.15.0-0.2.beta.fc26.noarch
python-pulp-client-lib-2.15.0-0.2.beta.fc26.noarch
python-pulp-common-2.15.0-0.2.beta.fc26.noarch
python-pulp-deb-common-1.6.0-0.2.beta.fc26.noarch
python-pulp-docker-common-3.1.0-0.3.beta.fc26.noarch
python-pulp-oid_validation-2.15.0-0.2.beta.fc26.noarch
python-pulp-ostree-common-1.3.0-1.fc26.noarch
python-pulp-puppet-common-2.15.0-0.2.beta.fc26.noarch
python-pulp-python-common-2.0.2-1.fc26.noarch
python-pulp-repoauth-2.15.0-0.2.beta.fc26.noarch
python-pulp-rpm-common-2.15.0-0.2.beta.fc26.noarch
python-pulp-streamer-2.15.0-0.2.beta.fc26.noarch
</code></pre>
<p>Here's the test script:</p>
<pre><code>#!/usr/bin/bash
set -euo pipefail
readonly rpms=(
'https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned/bear-4.1-1.noarch.rpm'
'https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned/camel-0.1-1.noarch.rpm'
'https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned/cat-1.0-1.noarch.rpm'
)
counts=()
pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id foo --generate-sqlite true
for rpm in "${rpms[@]}"; do
wget "$rpm"
pulp-admin rpm repo uploads rpm --repo-id foo --file "$(basename "$rpm")"
pulp-admin rpm repo publish run --repo-id foo
find /var/lib/pulp/published/yum/master -type f -name '*-primary.sqlite.bz2'
counts+=("$(find /var/lib/pulp/published/yum/master -type f -name '*-primary.sqlite.bz2' | wc --lines)")
done
pulp-admin rpm repo delete --repo-id foo
echo "${counts[*]}"
</code></pre>
<p>Here's the final output on F25:</p>
<pre><code>1 2 3
</code></pre>
<p>And on F26:</p>
<pre><code>1 2 3
</code></pre> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=232752017-12-12T22:26:27ZIchimonji10jerebear@protonmail.com
<ul><li><strong>File</strong> <a href="/attachments/351">fedora-26-pulp-2-15-beta.txt</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/351/fedora-26-pulp-2-15-beta.txt">fedora-26-pulp-2-15-beta.txt</a> added</li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=232762017-12-13T14:51:04Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>@jeremy did you configure and notice the new options for the distributor? <a href="https://github.com/pulp/pulp_rpm/pull/1056/files#diff-7cb9179c56cd2d7232d3e801a56ffceaR631" class="external">https://github.com/pulp/pulp_rpm/pull/1056/files#diff-7cb9179c56cd2d7232d3e801a56ffceaR631</a><br>
You did not mention any of those in your testing.</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=232772017-12-13T14:56:53ZIchimonji10jerebear@protonmail.com
<ul></ul><p>No, I didn't use any of those new options. I ran the script listed in my previous comment. I'll re-test.</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=232782017-12-13T14:58:47ZIchimonji10jerebear@protonmail.com
<ul></ul><p>I performed testing in the way that I did because of what the issue states:</p>
<blockquote>
<p>Expected result:</p>
<p>After first publish, 1 primary sqlite file exists. After second publish:</p>
<ul>
<li>If <a href="https://pulp.plan.io/issues/1684" class="external">https://pulp.plan.io/issues/1684</a> is implemented, sqlite files should be subject to the configured retention options. Thus it depends on the age of the existing sqlite file.</li>
<li>Otherwise, to be consistent with XML handling, the old sqlite files should probably be deleted, so only 1 primary sqlite file exists.</li>
</ul>
</blockquote>
<p>Is it fair to say that Pulp 2.15 doesn't act according to this description? If so, that's OK. I just want to have that explicit confirmation.</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=232912017-12-13T21:42:54ZIchimonji10jerebear@protonmail.com
<ul></ul><p>Fail, again. When one attempts to configure a distributor with the <code>remove_old_repodata</code> or <code>remove_old_repodata_threshold</code> options, an error is returned. Peeking in the logs shows errors like:</p>
<pre><code>pulp.plugins.pulp_rpm.plugins.distributors.yum.configuration:ERROR: Configuration key [remove_old_repodata] is not supported
pulp.server.controllers.repository:ERROR: (2257-64544) Exception adding distributor to repo [06acfbc4-96b4-4e6a-a418-32bed2f55bf4]; the repo will be deleted
pulp.server.controllers.repository:ERROR: (2257-64544) Traceback (most recent call last):
pulp.server.controllers.repository:ERROR: (2257-64544) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 433, in create_repo
pulp.server.controllers.repository:ERROR: (2257-64544) dist_controller.add_distributor(repo_id, type_id, plugin_config, auto_publish, dist_id)
pulp.server.controllers.repository:ERROR: (2257-64544) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 77, in add_distributor
pulp.server.controllers.repository:ERROR: (2257-64544) raise exceptions.PulpDataException(message)
pulp.server.controllers.repository:ERROR: (2257-64544) PulpDataException: Configuration key [remove_old_repodata] is not supported
pulp.server.webservices.middleware.exception:INFO: Configuration key [remove_old_repodata] is not supported
</code></pre>
<p>and:</p>
<pre><code>pulp.plugins.pulp_rpm.plugins.distributors.yum.configuration:ERROR: Configuration key [remove_old_repodata_threshold] is not supported
pulp.server.controllers.repository:ERROR: (2256-33152) Exception adding distributor to repo [9d542ad1-7ca6-4a90-9906-7ee23a775932]; the repo will be deleted
pulp.server.controllers.repository:ERROR: (2256-33152) Traceback (most recent call last):
pulp.server.controllers.repository:ERROR: (2256-33152) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 433, in create_repo
pulp.server.controllers.repository:ERROR: (2256-33152) dist_controller.add_distributor(repo_id, type_id, plugin_config, auto_publish, dist_id)
pulp.server.controllers.repository:ERROR: (2256-33152) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 77, in add_distributor
pulp.server.controllers.repository:ERROR: (2256-33152) raise exceptions.PulpDataException(message)
pulp.server.controllers.repository:ERROR: (2256-33152) PulpDataException: Configuration key [remove_old_repodata_threshold] is not supported
pulp.server.webservices.middleware.exception:INFO: Configuration key [remove_old_repodata_threshold] is not supported
</code></pre>
<p>Have a look at <a href="https://github.com/PulpQE/pulp-smash/pull/826" class="external">Pulp Smash 826</a>, and let me know if something is wrong with the test case.</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=232922017-12-14T12:57:36Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>new distributor parameters where not added to the validation config <a href="https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/distributors/yum/configuration.py#L58" class="external">https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/distributors/yum/configuration.py#L58</a></p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=233772017-12-15T14:23:28Zpcreechpcreech@redhat.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li></ul><p>WIth the failing of verification, this issue is being considered a blocker for 2.15.0 until further notice</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=234232017-12-18T17:20:55ZIchimonji10jerebear@protonmail.com
<ul></ul><p>This issue has been fixed in 2.15 beta 3. See:</p>
<ul>
<li><a href="https://github.com/PulpQE/pulp-smash/pull/826" class="external">https://github.com/PulpQE/pulp-smash/pull/826</a></li>
<li><a href="https://github.com/PulpQE/pulp-smash/issues/681" class="external">https://github.com/PulpQE/pulp-smash/issues/681</a></li>
</ul>
<p>Here's a sample of the RPMs installed when doing testing:</p>
<pre><code>[root@fedora-26-pulp-2-15-beta ~]# rpm -qa | grep pulp | sort
pulp-admin-client-2.15.0-0.2.beta.fc26.noarch
pulp-deb-admin-extensions-1.6.0-0.2.beta.fc26.noarch
pulp-deb-plugins-1.6.0-0.2.beta.fc26.noarch
pulp-docker-admin-extensions-3.1.0-0.3.beta.fc26.noarch
pulp-docker-plugins-3.1.0-0.3.beta.fc26.noarch
pulp-ostree-admin-extensions-1.3.0-1.fc26.noarch
pulp-ostree-plugins-1.3.0-1.fc26.noarch
pulp-puppet-admin-extensions-2.15.0-0.2.beta.fc26.noarch
pulp-puppet-plugins-2.15.0-0.2.beta.fc26.noarch
pulp-puppet-tools-2.15.0-0.2.beta.fc26.noarch
pulp-python-admin-extensions-2.0.2-1.fc26.noarch
pulp-python-plugins-2.0.2-1.fc26.noarch
pulp-rpm-admin-extensions-2.15.0-0.3.beta.fc26.noarch
pulp-rpm-plugins-2.15.0-0.3.beta.fc26.noarch
pulp-selinux-2.15.0-0.2.beta.fc26.noarch
pulp-server-2.15.0-0.2.beta.fc26.noarch
python-pulp-bindings-2.15.0-0.2.beta.fc26.noarch
python-pulp-client-lib-2.15.0-0.2.beta.fc26.noarch
python-pulp-common-2.15.0-0.2.beta.fc26.noarch
python-pulp-deb-common-1.6.0-0.2.beta.fc26.noarch
python-pulp-docker-common-3.1.0-0.3.beta.fc26.noarch
python-pulp-oid_validation-2.15.0-0.2.beta.fc26.noarch
python-pulp-ostree-common-1.3.0-1.fc26.noarch
python-pulp-puppet-common-2.15.0-0.2.beta.fc26.noarch
python-pulp-python-common-2.0.2-1.fc26.noarch
python-pulp-repoauth-2.15.0-0.2.beta.fc26.noarch
python-pulp-rpm-common-2.15.0-0.3.beta.fc26.noarch
python-pulp-streamer-2.15.0-0.2.beta.fc26.noarch
</code></pre>
<p>Notice that the RPM-related RPMs have had their version numbers bumped.</p> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=240032018-01-17T21:08:53Zpcreechpcreech@redhat.com
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=286302018-06-03T22:27:37Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-11 priority-6 priority-default closed" href="/issues/3551">Issue #3551</a>: RemoveOldRepodataStep for yum publisher not checking repomd.xml to remove old files</i> added</li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=382542019-04-15T20:17:49Zbmbouterbmbouter@redhat.com
<ul><li><strong>Tags</strong> <i>Pulp 2</i> added</li></ul> RPM Support - Story #2788: As a user i can configure removal of old published repodata https://pulp.plan.io/issues/2788?journal_id=485032019-10-16T17:32:11Zmihai.ibanescu@gmail.commihai.ibanescu@gmail.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-11 priority-6 priority-default closed" href="/issues/5573">Issue #5573</a>: Publish won't create multiple checkecksummed copies of primary.xml, fileliststs.xml etc even when in fast-forward mode</i> added</li></ul>