Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-01T04:09:02ZPulp
Planio Docker Support - Issue #9594 (MODIFIED): Sync repo from registry requiring authentication failshttps://pulp.plan.io/issues/95942021-12-01T04:09:02Zyuzhengyuzheng@redhat.com
<p>Sync from a private repo in quay.io with basic auth credentials:</p>
<pre><code class="text syntaxhl" data-language="text">$ curl -X POST -u admin:admin -d '{"override_config": {"feed": "https://quay.io", "upstream_name": "redhat-dev/test----repo114", "tags": ["latest"], "basic_auth_username": "redhat-dev+copy_test", "basic_auth_password": "'$quay_password'"}}' https://docker-pulp-httpd/pulp/api/v2/repositories/redhat-operators/actions/sync/
</code></pre>
<p>The sync task fails with log:</p>
<pre><code class="text syntaxhl" data-language="text">Dec 1 03:54:45 docker-pulp-httpd pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[ddd7a76a-87ab-4cde-a1f6-a5321f005ec5]
...
Dec 1 03:54:45 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: [ddd7a76a] Download succeeded: https://quay.io/v2/redhat-dev/test----repo114/tags/list.
...
Dec 1 03:54:45 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: [ddd7a76a] Download succeeded: https://quay.io/v2/redhat-dev/test----repo114/manifests/latest.
...
Dec 1 03:54:45 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: [ddd7a76a] Download succeeded: https://quay.io/v2/redhat-dev/test----repo114/manifests/sha256:37daa087fa88d8f5f0f97ba58e4facbba81e87513d6d6584bcf0f6417e994bd1.
...
Dec 1 03:54:46 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://quay.io/v2/redhat-dev/test----repo114/blobs/sha256:41c24cf3eae7653c4ff3c0e8762ce2ffb21b8da07715e874bc7b901cc3227e8b failed with code 401: UNAUTHORIZED
...
Dec 1 03:54:46 docker-pulp-httpd pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://quay.io/v2/redhat-dev/test----repo114/blobs/sha256:41c24cf3eae7653c4ff3c0e8762ce2ffb21b8da07715e874bc7b901cc3227e8b failed with code 401: UNAUTHORIZED
...
Dec 1 03:54:47 docker-pulp-httpd pulp: pulp.server.async.tasks:INFO: [ddd7a76a] Task failed : [ddd7a76a-87ab-4cde-a1f6-a5321f005ec5] : Image download(s) from https://quay.io/v2/redhat-dev/test----repo114/blobs/sha256:41c24cf3eae7653c4ff3c0e8762ce2ffb21b8da07715e874bc7b901cc3227e8b failed. Sync task has failed to prevent a corrupted repository.
</code></pre>
<p>It can download tag list and manifests successfully, but has trouble to download blobs.</p>
<p>Pulp version:</p>
<pre><code class="text syntaxhl" data-language="text"># rpm -qa | grep pulp-docker
python-pulp-docker-common-3.2.3-3.el7eng.noarch
pulp-docker-plugins-3.2.3-3.el7eng.noarch
pulp-docker-admin-extensions-3.2.3-3.el7eng.noarch
</code></pre> Pulp - Issue #9360 (MODIFIED): Cleanup doesn't work when import_content raises exceptionhttps://pulp.plan.io/issues/93602021-09-09T03:21:52Zyuzhengyuzheng@redhat.com
<p>A repo sync task failed with error:</p>
<p>Content import of /var/lib/pulp/working/<a href="mailto:reserved_resource_worker-4@pulp-docker04.web.prod.ext.phx2.redhat.com" class="email">reserved_resource_worker-4@pulp-docker04.web.prod.ext.phx2.redhat.com</a>/005cfc84-9b50-42d2-8a04-28e60b16f5e5/sha256:801e265f146090ed6045b2d673922e7aef38bebecaeb65dba770ea84b351c6b2 failed - must be an existing file</p>
<p>But the unit sha256:801e265f146090ed6045b2d673922e7aef38bebecaeb65dba770ea84b351c6b2 should be removed in the task, but it's not.</p> Pulp - Issue #7185 (CLOSED - CURRENTRELEASE): force_full rsync publish is done unnecessarily in s...https://pulp.plan.io/issues/71852020-07-22T03:03:30Zyuzhengyuzheng@redhat.com
<p>A full rsync publish is done when some criteria are met (defined in <a href="https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/rsync/publish.py#L373" class="external">https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/rsync/publish.py#L373</a>). Some of the criteria don't need a full rsync publish to make the right content published to rsync distributor:</p>
<ul>
<li>When there's units removal since last rsync publish</li>
<li>When there's predistributor publish failure since last rsync publish</li>
</ul>
<p>Since it takes longer time to do full publish, especially for big repos containing lots of units, unnecessary full publish should be avoided by removing above criteria.</p> RPM Support - Issue #2774 (CLOSED - WONTFIX): Inconsistent Checksum and Checksumtype when uploadi...https://pulp.plan.io/issues/27742017-05-23T06:20:12Zyuzhengyuzheng@redhat.com
<pre><code># rpm -qa pulp-server
pulp-server-2.13.0-1.el7.noarch
</code></pre>
<p>Reproducing steps:<br>
1. Create a rpm repo</p>
<pre><code># pulp-admin rpm repo create --repo-id test-repo10
Successfully created repository [test-repo10]
</code></pre>
<p>2. Upload a drpm to the repo with --checksum-type=md5</p>
<pre><code># pulp-admin rpm repo uploads drpm --repo-id test-repo10 --file /root/test-alpha-1.1-2_1.1-3.noarch.drpm --checksum-type md5
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: test-alpha-1.1-2_1.1-3.noarch.drpm
... completed
Creating upload requests on the server...
[==================================================] 100%
Initializing: test-alpha-1.1-2_1.1-3.noarch.drpm
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.
Uploading: test-alpha-1.1-2_1.1-3.noarch.drpm
[==================================================] 100%
3304/3304 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Succeeded
Deleting the upload request...
... completed
</code></pre>
<p>3. Check uploaded drpm</p>
<pre><code># pulp-admin rpm repo content drpm --repo-id test-repo10
Arch: noarch
Checksum: f9476caa0f491dfa227a984b092e05f68b637471f7c7c2f648475f067f12fc2e
Checksumtype: md5
Epoch: 0
Filename: drpms/test-alpha-1.1-2_1.1-3.noarch.drpm
Release: 3
Version: 1.1
</code></pre>
<p>Checksumtype is md5, but Checksum shows a value of sha256 checksum.</p>
<pre><code># sha256sum /root/test-alpha-1.1-2_1.1-3.noarch.drpm
f9476caa0f491dfa227a984b092e05f68b637471f7c7c2f648475f067f12fc2e /root/test-alpha-1.1-2_1.1-3.noarch.drpm
</code></pre> Pulp - Issue #2687 (CLOSED - CURRENTRELEASE): Rsync publish removes symlinks and replace them wit...https://pulp.plan.io/issues/26872017-03-31T05:31:08Zyuzhengyuzheng@redhat.com
<p>Symlinks in rsync targeted storage will be replaced with directories when doing rsync publish.</p>
<p>Reproducing steps:</p>
<pre><code>1. Create 2 rpm repos with yum_distributor and rpm_rsync_distributor defined
[test-repo01]
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo01", "description": "test repo", "distributors": [{"distributor_type_id":"yum_distributor", "distributor_id":"yum_distributor", "distributor_config":{"http":true, "https":false, "relative_url":"content/dist/test-repo01"}},{"distributor_type_id":"rpm_rsync_distributor","distributor_id":"cdn_distributor","distributor_config":{"remote": {"auth_type": "publickey","ssh_user": "cdn", "ssh_identity_file": "/var/www/.ssh/id_rsa", "host": "10.66.136.68", "root": "/home/cdn/"}, "predistributor_id": "yum_distributor"}}], "notes": {"_repo-type": "rpm-repo", "relative_url": "content/dist/test-repo01"}, "importer_type_id": "yum_importer", "id": "test-repo01"}'
[test-repo02]
Create it in the same way as test-repo01.
2. Upload rpm1 to test-repo01, yum publish and rsync publish the repo
# pulp-admin rpm repo uploads rpm --repo-id test-repo01 --file /tmp/dog-4.23-1.noarch.rpm
# pulp-admin rpm repo publish run --repo-id test-repo01
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo01/actions/publish/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'
3. Create symlink on remote server
# sudo -u cdn ln -s /home/cdn/content/dist/test-repo01/ /home/cdn/content/dist/test-repo02
4. Upload rpm2 to test-repo02, yum publish and rsync publish the repo
# pulp-admin rpm repo uploads rpm --repo-id test-repo02 --file /tmp/fox-1.1-2.noarch.rpm
# pulp-admin rpm repo publish run --repo-id test-repo02
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo02/actions/publish/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'
</code></pre>
<p>Expected result:</p>
<pre><code>test-repo02 on remote server should contain content of both test-repo01 and test-repo02.
</code></pre>
<p>Actual result:</p>
<pre><code>Before rsync publish in step4, test-repo02 is a symlink.
# ll /home/cdn/content/dist/test-repo02
lrwxrwxrwx. 1 cdn cdn 35 Mar 31 13:04 /home/cdn/content/dist/test-repo02 -> /home/cdn/content/dist/test-repo01/
However, after the publish, it's replaced with a directory and loses existing content (content of test-repo01) in it.
# ll /home/cdn/content/dist/test-repo02
total 4
drwxr-xr-x. 3 cdn cdn 15 Mar 31 13:10 Packages
drwxr-xr-x. 2 cdn cdn 4096 Mar 31 13:10 repodata
</code></pre>
<p>Adding -K (--keep-dirlinks) for rsync will avoid this issue.</p>
<pre><code>/usr/lib/python2.7/site-packages/pulp/plugins/rsync/publish.py
65 def remote_mkdir(self, path):
66 """
67 Creates path on remote server. The path is rooted in distributor's remote_root directory.
...
78 args = ['rsync', '-avrK', '-f+ */']
</code></pre> Docker Support - Issue #2678 (CLOSED - CURRENTRELEASE): 'relative_repo_path' in docs should be 'r...https://pulp.plan.io/issues/26782017-03-29T10:04:27Zyuzhengyuzheng@redhat.com
<p>In <a href="https://github.com/pulp/pulp_docker/blob/master/docs/tech-reference/distributor.rst" class="external">https://github.com/pulp/pulp_docker/blob/master/docs/tech-reference/distributor.rst</a>:</p>
<pre><code>relative_repo_path
The relative path from the root where the repository will be published. Defaults to the repository id.
</code></pre>
<p>The right configuration should be 'repo_relative_path' according to the implementation in code.</p> RPM Support - Issue #2666 (CLOSED - CURRENTRELEASE): Rsync publish for RPM repo fails to run in f...https://pulp.plan.io/issues/26662017-03-27T04:20:25Zyuzhengyuzheng@redhat.com
<p>Reproducing steps:<br>
1. Create a RPM repo with default configurations. Upload some rpms/srpms to the repo.</p>
<pre><code>$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo38", "description": "test repo", "distributors": [{"distributor_id": "yum_distributor", "distributor_type_id": "yum_distributor", "distributor_config": {"http": true, "https": false, "relative_url": "content/dist/test-repo38"}}, {"distributor_id": "cdn_distributor", "distributor_type_id": "rpm_rsync_distributor", "distributor_config": {"remote": {"auth_type": "publickey", "ssh_user": "cdn", "host": "10.66.136.68", "root": "/home/cdn/", "ssh_identity_file": "/var/www/.ssh/id_rsa"}, "predistributor_id": "yum_distributor"}}], "notes": {"_repo-type": "rpm-repo", "relative_url": "content/dist/test-repo38"}, "importer_type_id": "yum_importer", "id": "test-repo38"}'
$ pulp-admin rpm repo uploads rpm --repo-id test-repo38 --file fox-1.1-2.noarch.rpm
$ pulp-admin rpm repo uploads srpm --repo-id test-repo38 --file test-srpm01-1.0-1.src.rpm
</code></pre>
<p>2. Run yum publish and rsync publish.</p>
<pre><code>$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo38/actions/publish/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "yum_distributor", "override_config": {}}'
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo38/actions/publish/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'
</code></pre>
<p>3. Repeat step 2, and check status of the 2nd rsync publish task.</p>
<pre><code>$ pulp-admin tasks details --task-id c64f9c6f-d150-4c4f-b8e2-67888f8380d3
+----------------------------------------------------------------------+
Task Details
+----------------------------------------------------------------------+
Operations: publish
Resources: test-repo38 (repository)
State: Successful
Start Time: 2017-03-27T04:08:19Z
Finish Time: 2017-03-27T04:08:20Z
Result:
Completed: 2017-03-27T04:08:20Z
Details:
Description:
Details:
Error Details:
Items Total: 2
Num Failures: 0
Num Processed: 2
Num Success: 2
</code></pre>
<p>No new unit has been added to the repo since last publish, and default publish mode should be fast forward. However, all units in the repo are processed here. The problem might be last_published is incorrectly set to None when last_deleted has no value.</p>
<pre><code>/usr/lib/python2.7/site-packages/pulp/plugins/rsync/publish.py
348 if self.last_deleted:
349 last_deleted = self.last_deleted.replace(tzinfo=None)
350 else:
351 last_published = None
</code></pre> RPM Support - Issue #2657 (CLOSED - CURRENTRELEASE): Publishing iso_rsync_distributor fails when ...https://pulp.plan.io/issues/26572017-03-23T08:50:43Zyuzhengyuzheng@redhat.com
<p>When ISO repo in Pulp is not published to https path, publishing iso_rsync_distributor fails due to trying to copy from a non-existing https path.</p>
<p>Reproducing steps:</p>
<pre><code>1. Create an ISO repo which only serves via http
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo23", "description": "test repo", "distributors": [{"distributor_id": "iso_distributor", "distributor_type_id": "iso_distributor", "distributor_config": {"serve_https": false, "serve_http": true, "relative_url": "content/dist/test-repo23"}}, {"distributor_id": "cdn_distributor", "distributor_type_id": "iso_rsync_distributor", "distributor_config": {"remote": {"auth_type": "publickey", "ssh_user": "cdn", "host": "10.66.136.68", "root": "/home/cdn/", "ssh_identity_file": "/var/www/.ssh/id_rsa"}, "predistributor_id": "iso_distributor"}}], "notes": {"rsync_repodata": "True", "_repo-type": "iso-repo", "relative_url": "content/dist/test-repo23"}, "importer_type_id": "iso_importer", "id": "test-repo23"}'
{"scratchpad": {}, "display_name": "test-repo23", "description": "test repo", "last_unit_added": null, "notes": {"rsync_repodata": "True", "_repo-type": "iso-repo", "relative_url": "content/dist/test-repo23"}, "last_unit_removed": null, "content_unit_counts": {}, "_ns": "repos", "_id": {"$oid": "58d3856020ed2e2185ee575b"}, "id": "test-repo23", "_href": "/pulp/api/v2/repositories/test-repo23/"}
2. Check distributors for created repo
db.repo_distributors.find({repo_id:"test-repo23"})
/* 0 */
{
"_id" : ObjectId("58d3856020ed2e2185ee575d"),
"repo_id" : "test-repo23",
"distributor_id" : "iso_distributor",
"distributor_type_id" : "iso_distributor",
"config" : {
"serve_https" : false,
"relative_url" : "content/dist/test-repo23",
"serve_http" : true
},
"auto_publish" : false,
"last_updated" : ISODate("2017-03-23T08:20:48.592Z"),
"last_override_config" : {},
"scratchpad" : {},
"_ns" : "repo_distributors",
"last_publish" : ISODate("2017-03-23T08:22:15.719Z")
}
/* 1 */
{
"_id" : ObjectId("58d3856020ed2e2185ee575e"),
"repo_id" : "test-repo23",
"distributor_id" : "cdn_distributor",
"distributor_type_id" : "iso_rsync_distributor",
"config" : {
"remote" : {
"auth_type" : "publickey",
"ssh_user" : "cdn",
"host" : "10.66.136.68",
"root" : "/home/cdn/",
"ssh_identity_file" : "/var/www/.ssh/id_rsa"
},
"predistributor_id" : "iso_distributor"
},
"auto_publish" : false,
"last_updated" : ISODate("2017-03-23T08:20:48.595Z"),
"last_override_config" : {},
"scratchpad" : {},
"_ns" : "repo_distributors"
}
3. Upload an ISO to this repo
# pulp-admin iso repo uploads upload --repo-id test-repo23 --file 3.iso
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
...
[\]
Running...
Task Succeeded
Deleting the upload request...
... completed
4. Run ISO publish
# pulp-admin iso repo publish run --repo-id test-repo23
+----------------------------------------------------------------------+
Publishing Repository [test-repo18]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
The repository was successfully published.
Task Succeeded
4. Check ISO repo in Pulp
# ll /var/lib/pulp/published/http/isos/content/dist/test-repo23/
total 4
lrwxrwxrwx. 1 apache apache 97 Mar 23 16:22 3.iso -> /var/lib/pulp/content/units/iso/da/3ce6e699362190cb518be82ea132c67a38dda031afe8d79011ee1a2d2cf377
-rw-r--r--. 1 apache apache 80 Mar 23 16:22 PULP_MANIFEST
# ll /var/lib/pulp/published/https/isos/content/dist/test-repo23/
ls: cannot access /var/lib/pulp/published/https/isos/content/dist/test-repo23/: No such file or directory
Content is only published to http path as expected.
5. Run Rsycn publish
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo23/actions/publish/ -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'
The publish fails. Error log in Pulp server:
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) ['rsync', '-rtKOzi', '--files-from', u'/var/cache/pulp/reserved_resource_worker-1@dhcp-136-68.nay.redhat.com/bd0c988e-0062-4556-9974-2c0c3427cb7b/f2360d71-7352-4abc-a967-f2bcaf1aa7a6', '--relative', '-e', u'ssh -l cdn -i /var/www/.ssh/id_rsa -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -S /tmp/rsync_distributor-%r@%h:%p -o "ControlMaster auto" -o "ControlPersist 10"', '--copy-links', u'/var/lib/pulp/published/https/isos/content/dist/test-repo23', u'cdn@10.66.136.68:/home/cdn/content/dist/test-repo23']
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) rsync: link_stat "/var/lib/pulp/published/https/isos/content/dist/test-repo23/PULP_MANIFEST" failed: No such file or directory (2)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) ['rsync', '-rtKOzi', '--files-from', u'/var/cache/pulp/reserved_resource_worker-1@dhcp-136-68.nay.redhat.com/bd0c988e-0062-4556-9974-2c0c3427cb7b/f2360d71-7352-4abc-a967-f2bcaf1aa7a6', '--relative', '-e', u'ssh -l cdn -i /var/www/.ssh/id_rsa -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -S /tmp/rsync_distributor-%r@%h:%p -o "ControlMaster auto" -o "ControlPersist 10"', '--copy-links', u'/var/lib/pulp/published/https/isos/content/dist/test-repo23', u'cdn@10.66.136.68:/home/cdn/content/dist/test-repo23']
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) rsync: link_stat "/var/lib/pulp/published/https/isos/content/dist/test-repo23/PULP_MANIFEST" failed: No such file or directory (2)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688)
</code></pre> RPM Support - Issue #2655 (CLOSED - CURRENTRELEASE): units_successful is empty for successful ISO...https://pulp.plan.io/issues/26552017-03-22T03:10:20Zyuzhengyuzheng@redhat.com
<p>Tested version:</p>
<pre><code># rpm -qa pulp-server
pulp-server-2.12.1-1.el7.noarch
</code></pre>
<p>Reproducing steps:</p>
<pre><code>1. Create 2 ISO repos
# pulp-admin iso repo create --repo-id test-repo4
Successfully created repository [test-repo4]
# pulp-admin iso repo create --repo-id test-repo5
Successfully created repository [test-repo5]
2. Upload an ISO to test-repo4
# pulp-admin iso repo uploads upload --repo-id test-repo4 --file 2.iso
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: 2.iso
... completed
Creating upload requests on the server...
[==================================================] 100%
Initializing: 2.iso
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.
Uploading: 2.iso
[==================================================] 100%
2097152/2097152 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Succeeded
Deleting the upload request...
... completed
3. Copy ISO from test-repo4 to test-repo5
# pulp-admin iso repo copy --from-repo-id test-repo4 --to-repo-id test-repo5
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Nothing found that matches the given criteria and repository configuration
4. Get the associating task from DB
db.task_status.find({task_id:"f22583f3-b9a8-4b3e-9532-1f5c0cc450a5"})
/* 0 */
{
"_id" : ObjectId("58d1e155a2374cc00811e375"),
"task_id" : "f22583f3-b9a8-4b3e-9532-1f5c0cc450a5",
"exception" : null,
"task_type" : "pulp.server.managers.repo.unit_association.associate_from_repo",
"tags" : [
"pulp:repository:test-repo5",
"pulp:repository:test-repo4",
"pulp:action:associate"
],
"finish_time" : "2017-03-22T02:28:37Z",
"_ns" : "task_status",
"traceback" : null,
"spawned_tasks" : [],
"progress_report" : {},
"worker_name" : "reserved_resource_worker-1@dhcp-136-68.nay.redhat.com",
"result" : {
"units_successful" : []
},
"error" : null,
"group_id" : null,
"id" : null,
"state" : "finished",
"start_time" : "2017-03-22T02:28:37Z"
}
</code></pre>
<p>Both the output of 'iso repo copy' and '"units_successful" : []' in task result show nothing's copied. But actually the ISO is copied successfully.</p>
<pre><code># pulp-admin iso repo list --repo-id test-repo5
+----------------------------------------------------------------------+
ISO Repositories
+----------------------------------------------------------------------+
Id: test-repo5
Display Name: None
Description: None
Content Unit Counts:
Iso: 1
</code></pre>
<p>This affects result checking which depends on searching units in 'units_successful'.</p> RPM Support - Issue #2627 (CLOSED - CURRENTRELEASE): Uploading drpm by pulp-admin with --checksum...https://pulp.plan.io/issues/26272017-03-09T03:14:25Zyuzhengyuzheng@redhat.com
<p>Tested version:</p>
<pre><code># rpm -qa|grep pulp-admin
pulp-admin-client-2.12.1-1.el7.noarch
</code></pre>
<p>Reproducing steps:</p>
<pre><code>1. Create a rpm repo
# pulp-admin rpm repo create --repo-id test-repo1
Successfully created repository [test-repo1]
2. Upload a drpm to the repo
# pulp-admin rpm repo uploads drpm --repo-id test-repo1 --file test-alpha-1.1-1_1.1-2.noarch.drpm --checksum-type sha256
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: test-alpha-1.1-1_1.1-2.noarch.drpm
... completed
Creating upload requests on the server...
[==================================================] 100%
Initializing: test-alpha-1.1-1_1.1-2.noarch.drpm
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.
Uploading: test-alpha-1.1-1_1.1-2.noarch.drpm
[==================================================] 100%
3268/3268 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Failed
The importer yum_importer indicated a failed response when uploading drpm unit
to repository test-repo1.
Deleting the upload request...
... completed
</code></pre>
<p>Log from Pulp server:</p>
<p>Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) unexpected error occurred importing uploaded file: The field 'checksum_type' does not exist on the document 'DRPM'<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) Traceback (most recent call last):<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/upload.py", line 118, in upload<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) handlers[type_id](repo, type_id, unit_key, metadata, file_path, conduit, config)<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/upload.py", line 418, in <em>handle_package<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) unit = model_class(**rpm_data)<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 154, ininit</em>_<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) super(NonMetadataPackage, self).__init__(*args, **kwargs)<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 55, in <em>init</em><br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) super(AutoRetryDocument, self).__init__(*args, **kwargs)<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) File "/usr/lib/python2.7/site-packages/mongoengine/base/document.py", line 80, in <em>init</em><br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) raise FieldDoesNotExist(msg)<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp_rpm.plugins.importers.yum.upload:ERROR: (18520-70880) FieldDoesNotExist: The field 'checksum_type' does not exist on the document 'DRPM'<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp.server.managers.content.upload:ERROR: (18520-70880) Error from the importer while importing uploaded unit to repository [test-repo1]<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp.server.managers.content.upload:ERROR: (18520-70880) Traceback (most recent call last):<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp.server.managers.content.upload:ERROR: (18520-70880) File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 223, in import_uploaded_unit<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp.server.managers.content.upload:ERROR: (18520-70880) unit_type=unit_type_id, summary=result['summary'], details=result['details']<br>
Mar 9 11:11:01 dhcp-136-68 pulp: pulp.server.managers.content.upload:ERROR: (18520-70880) PulpCodedException: The importer yum_importer indicated a failed response when uploading drpm unit to repository test-repo1.</p>
<p>Uploading without specifying --checksum-type works.</p> Pulp - Issue #1891 (CLOSED - NOTABUG): Distribution of 'import_uploaded_unit' tasks is unbalanced...https://pulp.plan.io/issues/18912016-05-04T07:03:49Zyuzhengyuzheng@redhat.com
<p><strong>How reproducible:</strong><br>
100%</p>
<p><strong>Builds:</strong><br>
pulp-server-2.7.1-1.el6.noarch<br>
pulp-rpm-plugins-2.7.1-1.el6.noarch</p>
<p><strong>Steps:</strong><br>
1. Prepare 10 uploaded rpms in /var/lib/pulp/uploads/<br>
2. Run 10 'POST /pulp/api/v2/repositories/{REPO_ID}/actions/import_upload/' requests in threads (thread_number=5) by attached script<br>
3. Check pulp task status from db client</p>
<p><strong>Actual result:</strong><br>
There were 4 running workers. But all 'import_uploaded_unit' tasks were allocated to the same pulp worker - reserved_resource_worker-3.</p>
<p>[root@dhcp-136-194 tmp]# service pulp_workers status<br>
celery init v10.0.<br>
Using config script: /etc/default/pulp_workers<br>
node reserved_resource_worker-0 (pid 845) is running...<br>
node reserved_resource_worker-1 (pid 870) is running...<br>
node reserved_resource_worker-2 (pid 899) is running...<br>
node reserved_resource_worker-3 (pid 927) is running...</p>
<p>db.task_status.find({task_type:"pulp.server.managers.content.upload.import_uploaded_unit"}).count()<br>
10<br>
db.task_status.find({task_type:"pulp.server.managers.content.upload.import_uploaded_unit", worker_name:"reserved_resource_worker-3@dhcp-136-194***"}).count()<br>
10</p>
<p><strong>Expected result:</strong><br>
Distribution of 10 tasks should be balanced on 4 workers. Otherwise, tasks can not be run in parallel.</p>