Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-08-11T22:27:01ZPulp
Planio RPM Support - Issue #9233 (CLOSED - DUPLICATE): Downloaded content seems to be removed when a tas...https://pulp.plan.io/issues/92332021-08-11T22:27:01Zlmjachky
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2294":<a href="https://github.com/pulp/pulp_rpm/issues/2294" class="external">https://github.com/pulp/pulp_rpm/issues/2294</a></p>
<hr>
<p>For large repositories, this means that when performing <strong>immediate</strong> synchronization, all the downloaded content is lost after the task's failure. Pulp is then trying to download the content (artifacts) once again from scratch when issuing re-syncing, rather than associating the orphaned artifacts with corresponding content units.</p>
<p>More info can be found here: <a href="https://community.theforeman.org/t/oracle-linux-8-appstream-sync-fails/24676/9" class="external">https://community.theforeman.org/t/oracle-linux-8-appstream-sync-fails/24676/9</a>. The following part is the most important to us:</p>
<blockquote>
<p>If you are performing one of these large syncs, and get part way through before the sync fails with the timeout, it seems everything which was downloaded into the tmp directory is thrown away, and not kept. So, the next time the sync is tried, it has to download everything again.</p>
</blockquote>
<p>Used versions: pulpcore 3.14.3 and pulp_rpm 3.14.0</p>
<p>This behaviour could not be reproduced for pulp_file (<a href="https://fixtures.pulpproject.org/file/" class="external">https://fixtures.pulpproject.org/file/</a>; pulpcore 3.15.0.dev and pulp_file 1.9.0.dev); therefore, we should focus on the pulp_rpm plugin.</p> RPM Support - Issue #9208 (CLOSED - DUPLICATE): Published .treeinfo metadata not matching expecta...https://pulp.plan.io/issues/92082021-08-04T04:48:45Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2292":<a href="https://github.com/pulp/pulp_rpm/issues/2292" class="external">https://github.com/pulp/pulp_rpm/issues/2292</a></p>
<hr>
<ol>
<li>"packages" field on variants may not match with reality in the mirror case. You can see that the "repository" and "packages" fields for the "variant-External" variant which previously had both the repository and packages listed as "../rpm-signed/" now maps to "External" and "External/Packages", respectively. But in the event of mirror sync, this is incorrect.</li>
</ol>
<p>Likewise, the Land variant is wrong - but this may be a fixture issue. There is no such "Packages" directory. <a href="https://fixtures.pulpproject.org/rpm-distribution-tree/variants/land/" class="external">https://fixtures.pulpproject.org/rpm-distribution-tree/variants/land/</a></p>
<pre><code> ('change', 'variant-Land.packages', ('Packages', 'Land/Packages')),
('change', 'variant-Land.repository', ('variants/land', 'Land')),
('change',
'variant-External.packages',
('../rpm-signed/', 'External/Packages')),
('change', 'variant-External.repository', ('../rpm-signed/', 'External'))]
</code></pre>
<ol start="2">
<li>The order of the "variants" is different from the original - since productmd sometimes places significance on the first variant in the list, this could possibly have an impact in some cases.</li>
</ol>
<pre><code>[('change', 'general.variants', ('Land,Sea,External', 'External,Land,Sea')),
('change', 'tree.variants', ('Land,Sea,External', 'External,Land,Sea')),
</code></pre>
<p><code>test_publish.py::DistributionTreeMetadataTestCase</code> needs to be updated for both of these</p> Migration Plugin - Issue #9191 (CLOSED - DUPLICATE): Migrated advisories with dates-as-timestamps...https://pulp.plan.io/issues/91912021-08-02T19:37:16Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp-2to3-migration/issues/483": <a href="https://github.com/pulp/pulp-2to3-migration/issues/483" class="external">https://github.com/pulp/pulp-2to3-migration/issues/483</a></p> Pulp - Issue #9009 (CLOSED - DUPLICATE): CacheBasicOperationsTestCase::test_04_basic_expires unit...https://pulp.plan.io/issues/90092021-07-03T14:52:45Zdkliban@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/2023":<a href="https://github.com/pulp/pulpcore/issues/2023" class="external">https://github.com/pulp/pulpcore/issues/2023</a></p>
<hr>
<p>I am seeing the following in our CI from time to time:</p>
<pre><code># Run unit tests.
cmd_prefix bash -c "PULP_DATABASES__default__USER=postgres django-admin test --noinput /usr/local/lib/python3.6/site-packages/pulpcore/tests/unit/"
Creating test database for alias 'default'...
Access policy for groups/users created.
Access policy for groups created.
Access policy for tasks updated.
Deleting Guardians' AnonymousUser
System check identified no issues (0 silenced).
.............................F.......s.............................................pulp [None]: asyncio:ERROR: Task was destroyed but it is pending!
task: <Task pending coro=<ArtifactDownloader._handle_content_unit() running at /usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py:207> wait_for=<_GatheringFuture pending cb=[<TaskWakeupMethWrapper object at 0x7f8fd9a270d8>()]>>
pulp [None]: asyncio:ERROR: Task was destroyed but it is pending!
task: <Task pending coro=<ArtifactDownloader._handle_content_unit() running at /usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py:207> wait_for=<_GatheringFuture pending cb=[<TaskWakeupMethWrapper object at 0x7f8fd9a27468>()]>>
.............
======================================================================
FAIL: test_04_basic_expires (pulpcore.tests.unit.test_cache.CacheBasicOperationsTestCase)
Tests setting values with expiration times
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pulpcore/tests/unit/test_cache.py", line 42, in test_04_basic_expires
self.assertIsNone(ret)
AssertionError: b'hi' is not None
----------------------------------------------------------------------
Ran 96 tests in 6.627s
</code></pre> RPM Support - Issue #8992 (CLOSED - DUPLICATE): Memory usage when destroy repository; The memory ...https://pulp.plan.io/issues/89922021-06-30T07:11:25Zwilful
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2280":<a href="https://github.com/pulp/pulp_rpm/issues/2280" class="external">https://github.com/pulp/pulp_rpm/issues/2280</a></p>
<hr>
<p>When i try to destroy one of repositories, pulp (RQ basically) are using more 3G RAM and crashed by OOM-Killer</p>
<pre><code class="c syntaxhl" data-language="c"><span class="n">Jun</span> <span class="mi">30</span> <span class="mo">06</span><span class="o">:</span><span class="mi">59</span><span class="o">:</span><span class="mi">16</span> <span class="o">***</span> <span class="n">kernel</span><span class="o">:</span> <span class="n">Out</span> <span class="n">of</span> <span class="n">memory</span><span class="o">:</span> <span class="n">Kill</span> <span class="n">process</span> <span class="mi">81458</span> <span class="p">(</span><span class="n">rq</span><span class="p">)</span> <span class="n">score</span> <span class="mi">493</span> <span class="n">or</span> <span class="n">sacrifice</span> <span class="n">child</span>
<span class="n">Jun</span> <span class="mi">30</span> <span class="mo">06</span><span class="o">:</span><span class="mi">59</span><span class="o">:</span><span class="mi">16</span> <span class="o">***</span> <span class="n">kernel</span><span class="o">:</span> <span class="n">Killed</span> <span class="n">process</span> <span class="mi">81458</span> <span class="p">(</span><span class="n">rq</span><span class="p">),</span> <span class="n">UID</span> <span class="mi">1002</span><span class="p">,</span> <span class="n">total</span><span class="o">-</span><span class="n">vm</span><span class="o">:</span><span class="mi">4434448</span><span class="n">kB</span><span class="p">,</span> <span class="n">anon</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">3936104</span><span class="n">kB</span><span class="p">,</span> <span class="n">file</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">0</span><span class="n">kB</span><span class="p">,</span> <span class="n">shmem</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">0</span><span class="n">kB</span>
</code></pre>
<p>State before on my node:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="cp"># free -h
</span> <span class="n">total</span> <span class="n">used</span> <span class="n">free</span> <span class="n">shared</span> <span class="n">buff</span><span class="o">/</span><span class="n">cache</span> <span class="n">available</span>
<span class="n">Mem</span><span class="o">:</span> <span class="mi">7</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">1</span><span class="p">.</span><span class="mi">5</span><span class="n">G</span> <span class="mi">3</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">2</span><span class="p">.</span><span class="mi">2</span><span class="n">G</span> <span class="mi">2</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">3</span><span class="p">.</span><span class="mi">7</span><span class="n">G</span>
<span class="n">Swap</span><span class="o">:</span> <span class="mi">0</span><span class="n">B</span> <span class="mi">0</span><span class="n">B</span> <span class="mi">0</span><span class="n">B</span>
</code></pre>
<pre><code class="c syntaxhl" data-language="c"> <span class="s">"versions"</span><span class="o">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"core"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"rpm"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"python"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.4.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"file"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"1.8.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"deb"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"2.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"container"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"2.6.0"</span>
<span class="p">}</span>
<span class="p">],</span>
</code></pre> RPM Support - Test #8809 (CLOSED - DUPLICATE): Better tests for metadata mirroringhttps://pulp.plan.io/issues/88092021-05-24T19:54:30Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2277":<a href="https://github.com/pulp/pulp_rpm/issues/2277" class="external">https://github.com/pulp/pulp_rpm/issues/2277</a></p>
<hr>
<p>We need a fixture repository with some of the extra files, such as repomd.xml.asc (metadata signature), extra_files.json, .treeinfo, possibly licenses, multiple package directories / package locations, extra repomd entries that Pulp doesn't natively care about, etc. And then we need to test that mirroring works properly with such repos.</p> RPM Support - Issue #8720 (CLOSED - DUPLICATE): Published RPM metadata isn't sorted properlyhttps://pulp.plan.io/issues/87202021-05-07T22:09:20Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2274":<a href="https://github.com/pulp/pulp_rpm/issues/2274" class="external">https://github.com/pulp/pulp_rpm/issues/2274</a></p>
<hr>
<p>RPM metadata should be published in-order, which helps with compression efficency (see associated BZ). createrepo_c does this, but not via the library itself, so Pulp is still publishing unordered metadata.</p>
<p>Note that the metadata is "fine", it works, it's just inefficient to compress.</p>
<p>createrepo_c uses location_href as the sort key.</p>
<p>Problem: Pulp mixes location_href's together from many different repositories, and because they are meaningless, it basically ignores them. So we store useless data in the database.</p>
<p>We should remove the location_href and location_base fields (the latter is entirely unused), and replace them with just a filename, which we can possibly use to reconstruct a location_href if we need to keep it for backwards compatibility. Then we can properly sort by it, and we can use it directly in various places without needing to rewrite the value constantly.</p>
<p>It is not a "real" part of the RPM package metadata, only a value which createrepo_c happens to provide on the objects which we copied over. This probably shouldn't have been done.</p>
<p>We should therefore sort by the filename, which is basically equivalent to sorting by location_href since within a repository all the packages should have the same directory.</p> Pulp - Task #8226 (CLOSED - CURRENTRELEASE): Remove # coding=utf-8 from plugin-templatehttps://pulp.plan.io/issues/82262021-02-08T18:18:29Zggainey
<p>Python3 defaults to utf8 now.</p> Pulp - Task #7908 (CLOSED - CURRENTRELEASE): Make sure all exceptions live in pulpcore.plugin.exc...https://pulp.plan.io/issues/79082020-12-01T13:38:59Zggainey
<p>See discussion at <a href="https://github.com/pulp/pulpcore/pull/1027#discussion_r524551412" class="external">https://github.com/pulp/pulpcore/pull/1027#discussion_r524551412</a></p>
<p>This will require a deprecation cycle, with a given exception living "in two places" for one release. Once this issue is in MODIFIED, open a new issue to complete the move.</p> Pulp - Issue #6564 (CLOSED - CURRENTRELEASE): Export filename for pulp exports has dupe slasheshttps://pulp.plan.io/issues/65642020-04-23T17:47:05ZdaviddavisRPM Support - Story #5356 (CLOSED - CURRENTRELEASE): As a user, I can download a configuration fo...https://pulp.plan.io/issues/53562019-08-27T12:28:24Zdkliban@redhat.com
<p>It is common for repositories to include a repo file that can be used to add the repository to a client. This file can be manually placed into /etc/yum.repos.d/ or it can be specified as a parameter to yum/dnf. e.g.</p>
<pre><code>dnf config-manager --add-repo http://example.com/pulp/content/some/repo/path/config.repo
</code></pre> Pulp - Test #4362 (CLOSED - COMPLETE): Ansible installer for Pulp3 needs to add a systemd unit fi...https://pulp.plan.io/issues/43622019-01-30T14:41:28Zbherring
<a name="Testing"></a>
<h2 >Testing<a href="#Testing" class="wiki-anchor">¶</a></h2>
<ul>
<li>Need to investigate if there is an issue here or not (from Triage)</li>
</ul> Pulp - Issue #3889 (CLOSED - CURRENTRELEASE): psycopg2 warninghttps://pulp.plan.io/issues/38892018-07-30T19:48:06Zbmbouterbmbouter@redhat.com
<p>Whenever the Pulp3 database driver connects to Postgresql it emits this warning:</p>
<pre><code>/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
</code></pre>
<p>It would be great to start Pulp3 without these warnings</p> Pulp - Issue #2919 (CLOSED - CURRENTRELEASE): 500 error when creating repo with duplicate namehttps://pulp.plan.io/issues/29192017-07-17T21:12:27Zdkliban@redhat.com
<p>Looks like the serializer is failing to enforce uniqueness constraint during validation. As a result the database enforces it and the user receives a 500 response.</p>
<pre><code>Traceback:
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/core/handlers/base.py" in get_response
132. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/views/decorators/csrf.py" in wrapped_view
58. return view_func(*args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/rest_framework/viewsets.py" in view
86. return self.dispatch(request, *args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/rest_framework/views.py" in dispatch
489. response = self.handle_exception(exc)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/rest_framework/views.py" in handle_exception
449. self.raise_uncaught_exception(exc)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/rest_framework/views.py" in dispatch
486. response = handler(request, *args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/rest_framework/mixins.py" in create
21. self.perform_create(serializer)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/rest_framework/mixins.py" in perform_create
26. serializer.save()
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/rest_framework/serializers.py" in save
215. self.instance = self.create(validated_data)
File "/home/vagrant/devel/pulp/platform/pulpcore/app/serializers/base.py" in create
112. instance = super(ModelSerializer, self).create(validated_data)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/rest_framework/serializers.py" in create
916. instance = ModelClass.objects.create(**validated_data)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/manager.py" in manager_method
127. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/query.py" in create
348. obj.save(force_insert=True, using=self.db)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/base.py" in save
734. force_update=force_update, update_fields=update_fields)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/base.py" in save_base
762. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/base.py" in _save_table
846. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/base.py" in _do_insert
885. using=using, raw=raw)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/manager.py" in manager_method
127. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/query.py" in _insert
920. return query.get_compiler(using=using).execute_sql(return_id)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/models/sql/compiler.py" in execute_sql
974. cursor.execute(sql, params)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/backends/utils.py" in execute
79. return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/backends/utils.py" in execute
64. return self.cursor.execute(sql, params)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/utils.py" in __exit__
98. six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/utils/six.py" in reraise
685. raise value.with_traceback(tb)
File "/home/vagrant/.virtualenvs/pulp/lib/python3.5/site-packages/django/db/backends/utils.py" in execute
64. return self.cursor.execute(sql, params)
Exception Type: IntegrityError at /api/v3/repositories/
Exception Value: duplicate key value violates unique constraint "pulp_app_repository_name_key"
DETAIL: Key (name)=(test-repo) already exists.
</code></pre> Pulp - Story #176 (CLOSED - CURRENTRELEASE): [RFE] Refactor config.get_boolean to raise ValueErro...https://pulp.plan.io/issues/1762015-02-19T01:03:52Zrbarlow
<p>+<span>+ This bug was initially created as a clone of <a href="https://bugzilla.redhat.com/show_bug.cgi?id=915462" class="external">Bugzilla Bug #915462</a> +</span>+</p>
<p>Description of problem:</p>
<p>This method currently returns None when it cannot parse a boolean from its input. It would be more appropriate for it to raise ValueError.</p>
<p>--- Additional comment from <a href="mailto:cduryee@redhat.com" class="email">cduryee@redhat.com</a> at 11/18/2014 00:07:33 ---</p>
<p>This is still the behavior as of 2.4.3.</p>