Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-09T18:59:35ZPulp
Planio RPM Support - Issue #9627 (MODIFIED): publish fails on MD5-checksummed repos, on FIPShttps://pulp.plan.io/issues/96272021-12-09T18:59:35Zggainey
<p>See associated BZ for details, reproducer</p> Docker Support - Issue #9594 (MODIFIED): Sync repo from registry requiring authentication failshttps://pulp.plan.io/issues/95942021-12-01T04:09:02Zyuzhengyuzheng@redhat.com
<p>Sync from a private repo in quay.io with basic auth credentials:</p>
<pre><code class="text syntaxhl" data-language="text">$ curl -X POST -u admin:admin -d '{"override_config": {"feed": "https://quay.io", "upstream_name": "redhat-dev/test----repo114", "tags": ["latest"], "basic_auth_username": "redhat-dev+copy_test", "basic_auth_password": "'$quay_password'"}}' https://docker-pulp-httpd/pulp/api/v2/repositories/redhat-operators/actions/sync/
</code></pre>
<p>The sync task fails with log:</p>
<pre><code class="text syntaxhl" data-language="text">Dec 1 03:54:45 docker-pulp-httpd pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[ddd7a76a-87ab-4cde-a1f6-a5321f005ec5]
...
Dec 1 03:54:45 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: [ddd7a76a] Download succeeded: https://quay.io/v2/redhat-dev/test----repo114/tags/list.
...
Dec 1 03:54:45 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: [ddd7a76a] Download succeeded: https://quay.io/v2/redhat-dev/test----repo114/manifests/latest.
...
Dec 1 03:54:45 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: [ddd7a76a] Download succeeded: https://quay.io/v2/redhat-dev/test----repo114/manifests/sha256:37daa087fa88d8f5f0f97ba58e4facbba81e87513d6d6584bcf0f6417e994bd1.
...
Dec 1 03:54:46 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://quay.io/v2/redhat-dev/test----repo114/blobs/sha256:41c24cf3eae7653c4ff3c0e8762ce2ffb21b8da07715e874bc7b901cc3227e8b failed with code 401: UNAUTHORIZED
...
Dec 1 03:54:46 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://quay.io/v2/redhat-dev/test----repo114/blobs/sha256:41c24cf3eae7653c4ff3c0e8762ce2ffb21b8da07715e874bc7b901cc3227e8b failed with code 401: UNAUTHORIZED
...
Dec 1 03:54:47 docker-pulp-httpd pulp: pulp.server.async.tasks:INFO: [ddd7a76a] Task failed : [ddd7a76a-87ab-4cde-a1f6-a5321f005ec5] : Image download(s) from https://quay.io/v2/redhat-dev/test----repo114/blobs/sha256:41c24cf3eae7653c4ff3c0e8762ce2ffb21b8da07715e874bc7b901cc3227e8b failed. Sync task has failed to prevent a corrupted repository.
</code></pre>
<p>It can download tag list and manifests successfully, but has trouble to download blobs.</p>
<p>Pulp version:</p>
<pre><code class="text syntaxhl" data-language="text"># rpm -qa | grep pulp-docker
python-pulp-docker-common-3.2.3-3.el7eng.noarch
pulp-docker-plugins-3.2.3-3.el7eng.noarch
pulp-docker-admin-extensions-3.2.3-3.el7eng.noarch
</code></pre> RPM Support - Issue #9559 (MODIFIED): Unable to sync Fedora 35 repositoryhttps://pulp.plan.io/issues/95592021-11-08T02:09:29Zhyu
<p>Cloned from <a href="https://bugzilla.redhat.com/show_bug.cgi?id=2020473" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=2020473</a></p>
<h2>Description of problem:
When publishing the Fedora 35 repository, Pulp failed with the following error:</h2>
<h2>celery.app.trace:ERROR: [56fc62c9] (24505-91520) Task pulp.server.managers.repo.publish.publish[56fc62c9-29b0-41bf-a820-ba3625ef4a00] raised unexpected: TemplateSyntaxError(u'Empty variable tag on line 548',)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) Traceback (most recent call last):
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [56fc62c9] (24505-91520) R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(Task, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(PulpTask, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in <strong>protected_call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self.run(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1142, in publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1251, in check_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1303, in _do_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) publish_report = publish_repo(transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 901, in wrap_f
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return f(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 185, in publish_repo
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._publisher.process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) super(PluginStep, self).process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) step.process()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 240, in process
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self._process_block(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 302, in _process_block
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.process_main(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 500, in process_main
celery.app.trace:ERROR: [56fc62c9] (24505-91520) context.add_unit_metadata(unit)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/filelists.py", line 42, in add_unit_metadata
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.metadata_file_handle.write(unit.render_filelists(self.checksum_type))
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 869, in render_filelists
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._render(metadata, context)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 884, in _render
celery.app.trace:ERROR: [56fc62c9] (24505-91520) t = Template(template)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 191, in <strong>init</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.nodelist = self.compile_nodelist()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 230, in compile_nodelist
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return parser.parse()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 482, in parse
celery.app.trace:ERROR: [56fc62c9] (24505-91520) raise self.error(token, 'Empty variable tag on line %d' % token.lineno)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) TemplateSyntaxError: Empty variable tag on line 548</h2>
<p>This is because one of the rpm in the repo has as "{{" filename causing the Django template syntax error.</p>
<p>In "6dc8dae1be904c2613d5aa3667dacd2554d05077eb1ce4296b6edfa3c4db3a46-filelists.xml.gz"
/usr/lib/.build-id
/usr/lib/.build-id/42
/usr/lib/.build-id/42/5049f5a9e0d1ac25c21de795f28fe886bfa0ca
/usr/lib64/R/library/rlang
/usr/lib64/R/library/rlang/DESCRIPTION
/usr/lib64/R/library/rlang/INDEX
/usr/lib64/R/library/rlang/LICENSE
/usr/lib64/R/library/rlang/Meta
snip...
/usr/lib64/R/library/rlang/help/wref_key.html
/usr/lib64/R/library/rlang/help/wref_value.html
/usr/lib64/R/library/rlang/help/zap.html
/usr/lib64/R/library/rlang/help/zap_srcref.html
/usr/lib64/R/library/rlang/help/{{.html <================= TemplateSyntaxError: Empty variable tag on line 548\n",
/usr/lib64/R/library/rlang/help/{{}}.html <=================</p>
<p>$ rpm -qlp R-rlang-0.4.11-3.fc35.x86_64.rpm | grep "{{"
/usr/lib64/R/library/rlang/help/{{.html
/usr/lib64/R/library/rlang/help/{{}}.html</p>
<p>help]$ cat {{.html</p>
<p>help]$ cat {{}}.html</p>
<p>Steps to Reproduce:</p>
<ol>
<li>Create a custom repository call fedora 35 and add the following feed url</li>
</ol>
<p><a href="https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/" class="external">https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/</a></p>
<ol start="2">
<li>Sync the repo</li>
</ol>
<p>Actual results:
Failed to publish repo</p>
<p>Expected results:
Repo can be published successfully</p>
<p>Additional info:
Tested on Satellite 6.10 and the repository can be synced successfully without issue.</p> RPM Support - Issue #9553 (MODIFIED): Publishing repository with large metadata may consume high ...https://pulp.plan.io/issues/95532021-11-03T06:10:34Zhyu
<p>Pulp is consuming high memory when publishing RHEL 7 repository. This is happening when Pulp is calculating the checksum of the metadata. It reads the whole metadata file into memory at once to calculates the checksum. For example, the other.xml.gz (compressed ) for RHEL 7 repository is about 837MB size. Reading the entire file into memory will cause Pulp worker to consume more than 1GB for RAM.</p>
<p><a href="https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101" class="external">https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101</a>.</p>
<p>How to reproduce:</p>
<ol>
<li>Sync the RHEL 7 repository.</li>
<li>After that manually force full publish it and run the below command to observe the memory usage.</li>
</ol>
<p>watch -n 1 'ps -aux | grep resource_worker</p>
<ol start="3">
<li>The memory usage should be stable between 200MB to 350MB all the time, but will suddenly go up to about 1.1GB for about 3 seconds (around finalizing the publish rpms step) then back to 200MB+.</li>
</ol> Pulp - Issue #9360 (MODIFIED): Cleanup doesn't work when import_content raises exceptionhttps://pulp.plan.io/issues/93602021-09-09T03:21:52Zyuzhengyuzheng@redhat.com
<p>A repo sync task failed with error:</p>
<p>Content import of /var/lib/pulp/working/<a href="mailto:reserved_resource_worker-4@pulp-docker04.web.prod.ext.phx2.redhat.com" class="email">reserved_resource_worker-4@pulp-docker04.web.prod.ext.phx2.redhat.com</a>/005cfc84-9b50-42d2-8a04-28e60b16f5e5/sha256:801e265f146090ed6045b2d673922e7aef38bebecaeb65dba770ea84b351c6b2 failed - must be an existing file</p>
<p>But the unit sha256:801e265f146090ed6045b2d673922e7aef38bebecaeb65dba770ea84b351c6b2 should be removed in the task, but it's not.</p> RPM Support - Issue #8893 (MODIFIED): 3rd party repository sync fails with 'InvalidStringData: st...https://pulp.plan.io/issues/88932021-06-14T14:14:36Zggainey
<p>ModulemdDefaults are BSON-encoded before being saved into MongoDB because MongoDB (apparently) has restrictions on valid keys which are incompatible with the module data we need to store.</p>
<p><a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95" class="external">https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95</a></p>
<p>...But the serialized BSON strings being created by the encoding function we are using are not always valid UTF-8... Presumably, it works the vast majority of the time, enough so that it wasn't noticed.</p>
<p>Only specific permutations of the input data appear to trigger this. If I delete key from the profiles dictionary, all of a sudden it's UTF-8 compatible. This is presumably why the problem pops into and out of existence.</p>
<p>At some point we attempt to save this string to MongoDB, and Mongo decides to apply a UTF-8 validation to it, and it blows up.</p> RPM Support - Issue #8890 (MODIFIED): Publishing a repository can take longer time to finish if m...https://pulp.plan.io/issues/88902021-06-14T06:00:43Zhyu
<p>Description of problem:
Pulp can take more than an hour to publish a repository when a large number of repositories have been synced from upstream and same errata are existed in the synced repositories, such as RHEL 7.x, RHEL 7 EUS and different aches.</p>
<p>The more repositories have the same errata the more "erratum pkglist" entries will be created in the mongodb which can cause the performance degradation.</p>
<p>For example:</p>
<blockquote>
<p>db.erratum_pkglists.find({errata_id: "RHSA-2018:2557"}).count()
387
db.erratum_pkglists.find({errata_id: "RHBA-2019:2180"}).count()
217</p>
</blockquote>
<p>When publishing errata, Pulp will use the above query to get all package lists of the errata. This will take long time to process when they are many package lists returned by the query and each package list is consist of many packages.</p>
<p>As we can see below, the "Publish Errata" step is very slow. 53 minutes has passed, it has only processed about 2073 errata. It will take more than an hour to finish.
...
{
"description": "Publishing Errata",
"details": "",
"error_details": [],
"items_total": 4789,
"num_failures": 0,
"num_processed": 2073,
"num_success": 2073,
"state": "IN_PROGRESS",
"step_id": "2f09190d-013a-4300-9445-eccb52ad94fe",
"step_type": "errata"
},
...
"start_time": "2021-06-09T12:41:13Z",</p>
<a name="date"></a>
<h1 >date<a href="#date" class="wiki-anchor">¶</a></h1>
<p>Wed Jun 9 13:32:41 UTC 2021</p> Pulp - Issue #8778 (MODIFIED): Set pulp repo distributor last_publish to publish start time inste...https://pulp.plan.io/issues/87782021-05-19T11:16:53Zrbikar
<p>This is related ticket <a href="https://pulp.plan.io/issues/5951" class="external">https://pulp.plan.io/issues/5951</a> for previous fix of the problem. The current fix relates to <a href="https://projects.engineering.redhat.com/browse/RHELDST-5086" class="external">https://projects.engineering.redhat.com/browse/RHELDST-5086</a>.</p>
<p>The problematic content type here is erratum that is shared with more than one repository. When this erratum gets updated, pulp will set last_unit_added to all affected repos, including those currently in publish (afaict only erratum type units behave like this). So the subsequent fast-forward publish will be skipped because the erratum was updated during publish.</p>
<p>This problem can be fixed by setting repo distributor last_publish to publish start time instead of finish time.</p> Pulp - Issue #8766 (MODIFIED): After restarting services, task is being re-run even if it's in a ...https://pulp.plan.io/issues/87662021-05-17T09:04:51Zttereshcttereshc@redhat.com
<p>When stopping the worker, the running task state is marked as error (or any completed state) but the message of the task remains in the worker queue. After restarting the worker, the task will be re-run. Same thing happen again, if you try to restart the worker over and over.</p>
<p>Steps to Reproduce:</p>
<ol>
<li>Sync any repository and copy the task id (the initial 8 characters), eg. "b3bd24be"</li>
<li>While syncing the repository, stop the workers</li>
</ol>
<p>systemctl stop pulp_workers</p>
<ol start="3">
<li>Check the /va/log/messages. you should see:</li>
</ol>
<p>pulp: pulp.server.async.tasks:INFO: [30a62ff1] Task failed : [b3bd24be-b805-42b9-9f81-69eb4ddaad55] : Worker terminated abnormally while processing task b3bd24be-b805-42b9-9f81-69eb4ddaad55. Check the logs for details
pulp: celery.app.trace:ERROR: [30a62ff1] (15661-20736) Task pulp.server.async.tasks._release_resource[30a62ff1-7d72-497f-ad25-661119917bbb] raised unexpected: AttributeError("'NoneType' object has no attribute 'top'",)</p>
<ol start="4">
<li>Then start the pulp_workers</li>
</ol>
<p>systemctl start pulp_workers</p>
<ol start="5">
<li>Check the /var/log/messages again. You should see to same task (b3bd24be) runs again:</li>
</ol>
<pre><code>pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[b3bd24be-b805-42b9-9f81-69eb4ddaad55]
pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[30a62ff1-7d72-497f-ad25-661119917bbb]
pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
...
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/.
pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Parsing metadata.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/.
pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Parsing metadata.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/.
pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Parsing metadata.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/.
pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Parsing metadata.
</code></pre>
<ol start="6">
<li>Repeat the steps again, the task will re-run again and again.</li>
</ol> Docker Support - Issue #8344 (MODIFIED): pulp2pulp sync fails when schema1 is not presenthttps://pulp.plan.io/issues/83442021-03-04T19:31:29Zipanova@redhat.comipanova@redhat.com
<ol>
<li>on pulp1 sync registry.redhat.io/dotnet/dotnetcore-10-rhel7</li>
<li>try with pulp2 sync from pulp1</li>
<li>observe error : PLP0000: 404 Client Error: 'Not Found' for url: <a href="https://localhost:5000/dotnetcore-10-rhel7/manifests/1.0" class="external">https://localhost:5000/dotnetcore-10-rhel7/manifests/1.0</a>
</li>
</ol> Pulp - Issue #8317 (MODIFIED): Allow the auth private key size to be configurablehttps://pulp.plan.io/issues/83172021-03-02T15:07:17Zmprahl
<p>Clients such as Fedora 33 have a default OpenSSL configuration that doesn't allow the usage of 1024 bit keys. To allow for backwards compatibility, allow this value to be configurable on the Pulp side.</p>
<p>See more information at:
<a href="https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2" class="external">https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2</a></p> Pulp - Issue #8251 (MODIFIED): Traceback in 2.21.5 when logging during task.on_failure()https://pulp.plan.io/issues/82512021-02-12T13:05:13Zbmbouterbmbouter@redhat.com
<p>A bugfix introduced in 2.21.5 for Issue <a href="https://pulp.plan.io/issues/5502" class="external">https://pulp.plan.io/issues/5502</a> does not fully resolve the issue. Since it's released, this new issue is tracking the remaining portion of that defect.</p>
<p>In the same failure scenario as Issue 5502, the user will experience this traceback instead:</p>
<pre><code>Feb 11 18:10:34 dhcp-2-174 pulp: py.warnings:WARNING: [ccf4ab5a] (33775-20800) "MongoClient opened before fork. Create MongoClient "
Feb 11 18:10:34 dhcp-2-174 pulp: py.warnings:WARNING: [ccf4ab5a] (33775-20800)
Feb 11 18:10:34 dhcp-2-174 pulp: pulp.server.async.tasks:INFO: [ccf4ab5a] Task failed : [18ac52ec-9f3e-47bc-b34b-98734bee3656] : Worker terminated abnormally while processing task 18ac52ec-9f3e-47bc-b34
b-98734bee3656. Check the logs for details
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) Task pulp.server.async.tasks._release_resource[ccf4ab5a-af22-4d41-84c9-255085b7eded] raised unexpected: UnboundLocalErro
r("local variable 'original_formatted_traceback' referenced before assignment",)
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) Traceback (most recent call last):
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) R = retval = fun(*args, **kwargs)
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 108, in __call__
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) return super(PulpTask, self).__call__(*args, **kwargs)
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) return self.run(*args, **kwargs)
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 376, in _release_resource
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) new_task.on_failure(exception, task_id, (), {}, MyEinfo)
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 779, in on_failure
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) _logger.debug(original_formatted_traceback)
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:ERROR: [ccf4ab5a] (33775-20800) UnboundLocalError: local variable 'original_formatted_traceback' referenced before assignment
Feb 11 18:10:34 dhcp-2-174 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[308a2cd5-f900-434c-8a02-d2aeb3e86992]
Feb 11 18:10:34 dhcp-2-174 pulp: celery.app.trace:INFO: [fe61a18e] Task pulp.server.managers.repo.unit_association.associate_from_repo[fe61a18e-5a65-4a40-a282-eb614e5e64ef] succeeded in 0.0295602829992s
: {'units_successful': [], 'units_failed_signature_filter': []}
</code></pre> Nectar - Issue #7911 (MODIFIED): Unable to sync large openshift docker reposhttps://pulp.plan.io/issues/79112020-12-02T11:24:03Zipanova@redhat.comipanova@redhat.com
<p>This is a regression of <a href="https://pulp.plan.io/issues/6471" class="external">https://pulp.plan.io/issues/6471</a></p>
<p>We are trying to sync the following docker images from satellite server and we got the error while doing sync
1.openshift3/mysql-apb
2.openshift3/mediawiki-apb
3.openshift3/mariadb-apb
4.openshift3/metrics-hawkular-metrics</p>
<p>Getting the below error, when syncing the repository from <a href="http://registry.redhat.io/https://registry.redhat.io" class="external">http://registry.redhat.io/https://registry.redhat.io</a></p>
<pre><code>
Error:
DKR1020: Image download(s) from http://registry.redhat.io/v2/openshift3/mysql-apb/blobs/sha256:1e3f598e03f841ddc4933fafb143d44393d4ce0517604c5cb2367024975e871c failed. Sync task has failed to prevent a corrupted repository.
DKR1020: Image download(s) from https://registry.redhat.io/v2/openshift3/mysql-apb/blobs/sha256:2a81c83bf7ad14e424acbfc7eecaf84aec2b273c25124807d820c4da607644e1 failed. Sync task has failed to prevent a corrupted repository.
Stacktrace:
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) (104, 'Connection reset by peer')
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) Traceback (most recent call last):
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 292, in _fetch
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) for chunk in chunks:
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/requests/models.py", line 656, in generate
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) for chunk in self.raw.stream(chunk_size, decode_content=True):
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 273, in stream
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) data = self.read(amt=amt, decode_content=decode_content)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 203, in read
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) data = self._fp.read(amt)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib64/python2.7/httplib.py", line 611, in read
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) s = self.fp.read(amt)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib64/python2.7/socket.py", line 380, in read
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) data = self._sock.recv(left)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 188, in recv
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) data = self.connection.recv(*args, **kwargs)
Apr 1 09:45:10 satellite pulp: nectar.downloaders.threaded:ERROR: (22861-60960) SysCallError: (104, 'Connection reset by peer')
The workaround is to limit the tags to sync but it is not feasible for some users because they will have to manually update the tags to sync when the upstream repo is updated
</code></pre> Packaging - Issue #4281 (MODIFIED): 2.19 Nighlty FIPS post-install issue with Apachehttps://pulp.plan.io/issues/42812018-12-19T18:42:33Zbherring
<a name="Notes"></a>
<h2 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h2>
<p>Nightly Jenkins [0] experienced sanity failures starting last night, 12/19</p>
<p>Only FIPs is affected.</p>
<p>Local test re-creations on 2.18-stable (fips/non-fips) and 2.19 non-fips experienced no issues with pulp-admin status smoke test or test harness execution</p>
<p>Still investigating, but could use some help to understand why `pulp-admin status` is returning errors and where to dig into the pulp code or additional logs to find the source of error.</p>
<a name="Re-creation"></a>
<h2 >Re-creation<a href="#Re-creation" class="wiki-anchor">¶</a></h2>
<ol>
<li>Install the latest Pulp2-nightly.</li>
<li>Verify pulp is functioning with `pulp-admin status` or run some tests, if you are so inclined.</li>
<li>Enable FIPs</li>
<li>Post-install, run `pulp-admin status`.</li>
<li>Verify the errors seen below and a failure return.</li>
<li>Bonus: Return back to non-FIPs and note Pulp2 functions. Back to FIPs, and the error is reproducible locally.
<ol>
<li>Verifying FIPs state with: `cat /proc/sys/crypto/fips_enabled`</li>
</ol>
</li>
</ol>
<a name="Test-Harness"></a>
<h2 >Test Harness<a href="#Test-Harness" class="wiki-anchor">¶</a></h2>
<ul>
<li>SW: Pulp2-nightly (2.19a1)</li>
<li>FIPs Enabled: Yes</li>
</ul>
<a name="django-version"></a>
<h3 >django version<a href="#django-version" class="wiki-anchor">¶</a></h3>
<pre><code>[root@rhel76-fips-219 ~]# rpm -qa | grep -i django
python2-django-1.11.13-4.el7.noarch
python-django-bash-completion-1.11.13-4.el7.noarch
[root@rhel76-fips-219 ~]#
</code></pre>
<a name="journalctl"></a>
<h3 >journalctl<a href="#journalctl" class="wiki-anchor">¶</a></h3>
<pre><code>Dec 19 11:49:57 rhel76-fips-219 dracut[7120]: *** Creating initramfs image file '/boot/initramfs-3.10.0-957.1.3.el7.x86_64kdump.img' done ***
Dec 19 11:49:58 rhel76-fips-219 kdumpctl[4859]: kexec: loaded kdump kernel
Dec 19 11:49:58 rhel76-fips-219 kdumpctl[4859]: Starting kdump: [OK]
Dec 19 11:49:58 rhel76-fips-219 systemd[1]: Started Crash recovery kernel arming.
Dec 19 11:49:58 rhel76-fips-219 systemd[1]: Startup finished in 490ms (kernel) + 3.557s (initrd) + 44.199s (userspace) = 48.248s.
Dec 19 11:51:41 rhel76-fips-219 chronyd[4292]: Source 216.229.0.49 replaced with 72.14.181.128
Dec 19 11:56:54 rhel76-fips-219 auditd[4260]: Audit daemon rotating log files
Dec 19 12:01:01 rhel76-fips-219 systemd[1]: Started Session 2 of user root.
Dec 19 12:01:01 rhel76-fips-219 CROND[16396]: (root) CMD (run-parts /etc/cron.hourly)
Dec 19 12:01:01 rhel76-fips-219 run-parts(/etc/cron.hourly)[16399]: starting 0anacron
Dec 19 12:01:01 rhel76-fips-219 run-parts(/etc/cron.hourly)[16405]: finished 0anacron
Dec 19 12:04:11 rhel76-fips-219 systemd[1]: Starting Cleanup of Temporary Directories...
Dec 19 12:04:11 rhel76-fips-219 systemd[1]: Started Cleanup of Temporary Directories.
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) Internal Server Error: /pulp/api/v2/status/
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) Traceback (most recent call last):
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 131, in get_response
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) response = middleware_method(request, response)
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) File "/usr/lib/python2.7/site-packages/django/middleware/http.py", line 23, in process_response
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) set_response_etag(response)
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) File "/usr/lib/python2.7/site-packages/django/utils/cache.py", line 109, in set_response_etag
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) response['ETag'] = quote_etag(hashlib.md5(response.content).hexdigest())
Dec 19 12:07:15 rhel76-fips-219 pulp[5043]: django.request:ERROR: (5043-60928) ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Dec 19 12:14:27 rhel76-fips-219 dhclient[4644]: DHCPREQUEST on ens3 to 192.168.122.1 port 67 (xid=0x2ec2a980)
Dec 19 12:14:27 rhel76-fips-219 dhclient[4644]: DHCPACK from 192.168.122.1 (xid=0x2ec2a980)
Dec 19 12:14:27 rhel76-fips-219 NetworkManager[4342]: <info> [1545239667.2262] dhcp4 (ens3): address 192.168.122.210
Dec 19 12:14:27 rhel76-fips-219 NetworkManager[4342]: <info> [1545239667.2267] dhcp4 (ens3): plen 24 (255.255.255.0)
Dec 19 12:14:27 rhel76-fips-219 NetworkManager[4342]: <info> [1545239667.2268] dhcp4 (ens3): gateway 192.168.122.1
Dec 19 12:14:27 rhel76-fips-219 NetworkManager[4342]: <info> [1545239667.2268] dhcp4 (ens3): lease time 3600
Dec 19 12:14:27 rhel76-fips-219 NetworkManager[4342]: <info> [1545239667.2268] dhcp4 (ens3): hostname 'rhel76-fips-219'
Dec 19 12:14:27 rhel76-fips-219 NetworkManager[4342]: <info> [1545239667.2268] dhcp4 (ens3): nameserver '192.168.122.1'
Dec 19 12:14:27 rhel76-fips-219 NetworkManager[4342]: <info> [1545239667.2268] dhcp4 (ens3): state changed bound -> bound
Dec 19 12:14:27 rhel76-fips-219 dbus[4286]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Dec 19 12:14:27 rhel76-fips-219 systemd[1]: Starting Network Manager Script Dispatcher Service...
Dec 19 12:14:27 rhel76-fips-219 dhclient[4644]: bound to 192.168.122.210 -- renewal in 1278 seconds.
Dec 19 12:14:27 rhel76-fips-219 dbus[4286]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Dec 19 12:14:27 rhel76-fips-219 nm-dispatcher[16418]: req:1 'dhcp4-change' [ens3]: new request (5 scripts)
Dec 19 12:14:27 rhel76-fips-219 nm-dispatcher[16418]: req:1 'dhcp4-change' [ens3]: start running ordered scripts...
Dec 19 12:14:27 rhel76-fips-219 systemd[1]: Started Network Manager Script Dispatcher Service.
Dec 19 12:19:31 rhel76-fips-219 pulp[4855]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Dec 19 12:19:31 rhel76-fips-219 pulp[4855]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Dec 19 12:19:31 rhel76-fips-219 pulp[4946]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[299291c5-991d-45dc-a47f-a66b772f300e]
Dec 19 12:19:31 rhel76-fips-219 pulp[5653]: kombu.transport.qpid:INFO: [299291c5] Connected to qpid with SASL mechanism ANONYMOUS
Dec 19 12:19:31 rhel76-fips-219 pulp[4946]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[2e90f887-1657-4e18-8dac-203e298f5927]
Dec 19 12:19:31 rhel76-fips-219 pulp[5653]: py.warnings:WARNING: [299291c5] (5653-50688) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient
Dec 19 12:19:31 rhel76-fips-219 pulp[5653]: py.warnings:WARNING: [299291c5] (5653-50688) "MongoClient opened before fork. Create MongoClient "
Dec 19 12:19:31 rhel76-fips-219 pulp[5653]: py.warnings:WARNING: [299291c5] (5653-50688)
Dec 19 12:19:31 rhel76-fips-219 pulp[5653]: celery.app.trace:INFO: [299291c5] Task pulp.server.controllers.repository.queue_download_deferred[299291c5-991d-45dc-a47f-a66b772f300e] succeeded in 0.104735261s: None
Dec 19 12:19:31 rhel76-fips-219 pulp[5653]: celery.app.trace:INFO: [2e90f887] Task pulp.server.controllers.repository.download_deferred[2e90f887-1657-4e18-8dac-203e298f5927] succeeded in 0.0167369139999s: None
Dec 19 12:26:04 rhel76-fips-219 chronyd[4292]: Source 204.11.201.12 replaced with 45.33.84.208
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) Internal Server Error: /pulp/api/v2/plugins/types/
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) Traceback (most recent call last):
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 131, in get_response
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) response = middleware_method(request, response)
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) File "/usr/lib/python2.7/site-packages/django/middleware/http.py", line 23, in process_response
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) set_response_etag(response)
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) File "/usr/lib/python2.7/site-packages/django/utils/cache.py", line 109, in set_response_etag
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) response['ETag'] = quote_etag(hashlib.md5(response.content).hexdigest())
Dec 19 12:32:53 rhel76-fips-219 pulp[5042]: django.request:ERROR: (5042-46336) ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Dec 19 12:35:45 rhel76-fips-219 dhclient[4644]: DHCPREQUEST on ens3 to 192.168.122.1 port 67 (xid=0x2ec2a980)
Dec 19 12:35:45 rhel76-fips-219 dhclient[4644]: DHCPACK from 192.168.122.1 (xid=0x2ec2a980)
Dec 19 12:35:45 rhel76-fips-219 NetworkManager[4342]: <info> [1545240945.6943] dhcp4 (ens3): address 192.168.122.210
Dec 19 12:35:45 rhel76-fips-219 NetworkManager[4342]: <info> [1545240945.6944] dhcp4 (ens3): plen 24 (255.255.255.0)
Dec 19 12:35:45 rhel76-fips-219 NetworkManager[4342]: <info> [1545240945.6944] dhcp4 (ens3): gateway 192.168.122.1
Dec 19 12:35:45 rhel76-fips-219 NetworkManager[4342]: <info> [1545240945.6944] dhcp4 (ens3): lease time 3600
Dec 19 12:35:45 rhel76-fips-219 NetworkManager[4342]: <info> [1545240945.6944] dhcp4 (ens3): hostname 'rhel76-fips-219'
Dec 19 12:35:45 rhel76-fips-219 NetworkManager[4342]: <info> [1545240945.6944] dhcp4 (ens3): nameserver '192.168.122.1'
Dec 19 12:35:45 rhel76-fips-219 NetworkManager[4342]: <info> [1545240945.6944] dhcp4 (ens3): state changed bound -> bound
Dec 19 12:35:45 rhel76-fips-219 dbus[4286]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Dec 19 12:35:45 rhel76-fips-219 systemd[1]: Starting Network Manager Script Dispatcher Service...
Dec 19 12:35:45 rhel76-fips-219 dhclient[4644]: bound to 192.168.122.210 -- renewal in 1584 seconds.
Dec 19 12:35:45 rhel76-fips-219 dbus[4286]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Dec 19 12:35:45 rhel76-fips-219 systemd[1]: Started Network Manager Script Dispatcher Service.
Dec 19 12:35:45 rhel76-fips-219 nm-dispatcher[16473]: req:1 'dhcp4-change' [ens3]: new request (5 scripts)
Dec 19 12:35:45 rhel76-fips-219 nm-dispatcher[16473]: req:1 'dhcp4-change' [ens3]: start running ordered scripts...
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) Internal Server Error: /pulp/api/v2/status/
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) Traceback (most recent call last):
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 131, in get_response
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) response = middleware_method(request, response)
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) File "/usr/lib/python2.7/site-packages/django/middleware/http.py", line 23, in process_response
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) set_response_etag(response)
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) File "/usr/lib/python2.7/site-packages/django/utils/cache.py", line 109, in set_response_etag
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) response['ETag'] = quote_etag(hashlib.md5(response.content).hexdigest())
Dec 19 12:38:23 rhel76-fips-219 pulp[5041]: django.request:ERROR: (5041-39040) ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Dec 19 12:49:31 rhel76-fips-219 pulp[4855]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Dec 19 12:49:31 rhel76-fips-219 pulp[4946]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[9db6c832-7553-4449-97a9-f444de4d65b5]
Dec 19 12:49:31 rhel76-fips-219 pulp[4946]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[f5cfb227-540d-45a6-b193-66552344d32b]
Dec 19 12:49:31 rhel76-fips-219 pulp[5653]: celery.app.trace:INFO: [9db6c832] Task pulp.server.controllers.repository.queue_download_deferred[9db6c832-7553-4449-97a9-f444de4d65b5] succeeded in 0.0134792559998s:
</code></pre>
<a name="httpderror_log"></a>
<h3 >httpd/error_log<a href="#httpderror_log" class="wiki-anchor">¶</a></h3>
<pre><code>[Wed Dec 19 11:44:38.226539 2018] [mpm_prefork:notice] [pid 18091] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Dec 19 11:45:03.164808 2018] [core:notice] [pid 4837] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Dec 19 11:45:03.177642 2018] [mime_magic:error] [pid 4837] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
[Wed Dec 19 11:45:03.190029 2018] [suexec:notice] [pid 4837] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.122.210. Set the 'ServerName' directive globally to suppress this message
[Wed Dec 19 11:45:03.696125 2018] [mime_magic:error] [pid 4837] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
[Wed Dec 19 11:45:03.696136 2018] [auth_digest:notice] [pid 4837] AH01757: generating secret for digest authentication ...
[Wed Dec 19 11:45:03.704563 2018] [lbmethod_heartbeat:notice] [pid 4837] AH02282: No slotmem from mod_heartmonitor
[Wed Dec 19 11:45:03.862247 2018] [mpm_prefork:notice] [pid 4837] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Wed Dec 19 11:45:03.862272 2018] [core:notice] [pid 4837] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Dec 19 11:47:33.138775 2018] [mpm_prefork:notice] [pid 4837] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Dec 19 11:47:55.430449 2018] [core:notice] [pid 3430] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Dec 19 11:47:55.440461 2018] [mime_magic:error] [pid 3430] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
[Wed Dec 19 11:47:55.456732 2018] [suexec:notice] [pid 3430] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.122.210. Set the 'ServerName' directive globally to suppress this message
[Wed Dec 19 11:47:55.999249 2018] [mime_magic:error] [pid 3430] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
[Wed Dec 19 11:47:55.999265 2018] [auth_digest:notice] [pid 3430] AH01757: generating secret for digest authentication ...
[Wed Dec 19 11:47:56.015928 2018] [lbmethod_heartbeat:notice] [pid 3430] AH02282: No slotmem from mod_heartmonitor
[Wed Dec 19 11:47:56.151806 2018] [mpm_prefork:notice] [pid 3430] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Wed Dec 19 11:47:56.151827 2018] [core:notice] [pid 3430] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Dec 19 11:48:56.230693 2018] [mpm_prefork:notice] [pid 3430] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Dec 19 11:49:20.875143 2018] [core:notice] [pid 4850] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Dec 19 11:49:20.890359 2018] [mime_magic:error] [pid 4850] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
[Wed Dec 19 11:49:20.895820 2018] [suexec:notice] [pid 4850] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.122.210. Set the 'ServerName' directive globally to suppress this message
[Wed Dec 19 11:49:21.436851 2018] [mime_magic:error] [pid 4850] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
[Wed Dec 19 11:49:21.436865 2018] [auth_digest:notice] [pid 4850] AH01757: generating secret for digest authentication ...
[Wed Dec 19 11:49:21.453957 2018] [lbmethod_heartbeat:notice] [pid 4850] AH02282: No slotmem from mod_heartmonitor
[Wed Dec 19 11:49:21.573101 2018] [mpm_prefork:notice] [pid 4850] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Wed Dec 19 11:49:21.573123 2018] [core:notice] [pid 4850] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
</code></pre>
<a name="219-nightly-pulp-admin-status-failure"></a>
<h2 >2.19-nightly pulp-admin status failure<a href="#219-nightly-pulp-admin-status-failure" class="wiki-anchor">¶</a></h2>
<pre><code>[root@rhel76-fips-219 ~]# pulp-admin status
+----------------------------------------------------------------------+
Status of the server
+----------------------------------------------------------------------+
There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache's log.
[root@rhel76-fips-219 ~]#
</code></pre>
<a name="Bonus-Non-fips-219-Nightly-Check"></a>
<h2 >Bonus: Non-fips 2.19 Nightly Check<a href="#Bonus-Non-fips-219-Nightly-Check" class="wiki-anchor">¶</a></h2>
<pre><code>[root@rhel76-fips-219 ~]# pulp-admin status
+----------------------------------------------------------------------+
Status of the server
+----------------------------------------------------------------------+
Api Version: 2
Database Connection:
Connected: True
Known Workers:
_id: scheduler@rhel76-fips-219
_ns: workers
Last Heartbeat: 2018-12-19T18:31:08Z
_id: resource_manager@rhel76-fips-219
_ns: workers
Last Heartbeat: 2018-12-19T18:31:12Z
_id: reserved_resource_worker-0@rhel76-fips-219
_ns: workers
Last Heartbeat: 2018-12-19T18:31:12Z
Messaging Connection:
Connected: True
Versions:
Platform Version: 2.19a1
[root@rhel76-fips-219 ~]# cat /proc/sys/crypto/fips_enabled
0
</code></pre>
<a name="References"></a>
<h2 >References:<a href="#References" class="wiki-anchor">¶</a></h2>
<p>[0] <a href="https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp%202%20-%20Master/job/pulp-2-master-dev-rhel7-fips/178/" class="external">https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp%202%20-%20Master/job/pulp-2-master-dev-rhel7-fips/178/</a></p>