Issue #7995
closedAmazon 2 core repo sync fails
Added by mwaqas@bloombergindustry.com almost 4 years ago. Updated over 3 years ago.
Description
pulp unable to sync Amazon 2 core repo. getting following error:
"local variable 'result' referenced before assignment"
{"pulp_tasks"=> [{"pulp_href"=>"/pulp/api/v3/tasks/0bbaa197-efcd-41e2-b897-33e89487e966/", "pulp_created"=>"2020-12-11T20:30:14.878+00:00", "state"=>"failed", "name"=>"pulp_rpm.app.tasks.synchronizing.synchronize", "started_at"=>"2020-12-11T20:30:14.983+00:00", "finished_at"=>"2020-12-11T20:30:15.128+00:00", "error"=> {"traceback"=> " File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job\n" + " rv = job.perform()\n" + " File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform\n" + " self._result = self._execute()\n" + " File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute\n" + " return self.func(*self.args, **self.kwargs)\n" + " File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 213, in synchronize\n" + " remote_url = fetch_remote_url(remote)\n" + " File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 140, in fetch_remote_url\n" + " return fetch_mirror(remote)\n" + " File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 126, in fetch_mirror\n" + " repodata_exists = get_repomd_file(remote, match.group(2))\n" + " File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 109, in get_repomd_file\n" + " return result\n", "description"=>"local variable 'result' referenced before assignment"}, "worker"=>"/pulp/api/v3/workers/bea91eb8-e0a6-4b9c-ad7d-19ff490e31f8/", "child_tasks"=>[], "progress_reports"=>[], "created_resources"=>[]}], "create_version"=>true, "task_groups"=>[], "poll_attempts"=>{"total"=>1, "failed"=>1}}
Updated by ipanova@redhat.com almost 4 years ago
based on the discussions with a user and from looking at the traceback this is a pulp3 issue. potential offending line is https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/synchronizing.py#L111 possibly not catching some other exception that is happening. Asked user to provide more info and list of installed packages.
Updated by mwaqas@bloombergindustry.com almost 4 years ago
I see following pulp packages:
tfm-rubygem-pulpcore_client-3.6.0-1.el7.noarch python3-pulpcore-3.6.3-2.el7.noarch tfm-rubygem-pulp_rpm_client-3.6.2-1.el7.noarch
Updated by ggainey almost 4 years ago
The problem llies in Pulp3's handling of Amazon's reositories.
Pulp2 works:
713 pulp-admin login -u admin -p admin
714 pulp-admin rpm repo create --repo-id amazon --feed http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
715 pulp-admin rpm repo sync run --repo-id amazon
Pulp3 fails. If you point Pulp3 at a specific return from mirror.list, it still fails, but gets further:
BASE_ADDR=":"
#http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list"
AMAZON_URL="http://amazonlinux.us-east-1.amazonaws.com/2/core/2.0/x86_64/35c6cb980fb37ade35d6bf08c25249754c0833617182395eda0a5d6cf7e981d9/"
AMAZON_NAME="amazon4"
AMAZON_HREF=$(http POST $BASE_ADDR/pulp/api/v3/repositories/rpm/rpm/ name=$AMAZON_NAME | jq -r '.pulp_href')
echo "repo_href : " $AMAZON_HREF
if [ -z "$AMAZON_HREF" ]; then exit; fi
# add remote
http POST $BASE_ADDR/pulp/api/v3/remotes/rpm/rpm/ name=$AMAZON_NAME url=$AMAZON_URL policy='immediate' download_concurrency=7
# find remote's href
REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/rpm/rpm/ | jq -r ".results[] | select(.name == \"${AMAZON_NAME}\") | .pulp_href")
echo "remote_href : " $REMOTE_HREF
if [ -z "$REMOTE_HREF" ]; then exit; fi
# sync
TASK_URL=$(http POST $BASE_ADDR$AMAZON_HREF'sync/' remote=$REMOTE_HREF | jq -r '.task')
echo "Task url : " $TASK_URL
if [ -z "$TASK_URL" ]; then exit; fi
# wait for task
wait_until_task_finished $BASE_ADDR$TASK_URL
Task processes metadata, but fails (another 403) trying to access RPMs:
Still waiting...
{
"pulp_href": "/pulp/api/v3/tasks/27481944-929c-40b7-ae9c-4e60a5bdda9e/",
"pulp_created": "2020-12-15T13:51:48.773054Z",
"state": "failed",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"logging_cid": "73fc25e344da4449887b93f36a3bab5e",
"started_at": "2020-12-15T13:51:48.944922Z",
"finished_at": "2020-12-15T13:52:02.612576Z",
"error": {
"traceback": " File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 975, in perform_job\n rv = job.perform()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 696, in perform\n self._result = self._execute()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 719, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 272, in synchronize\n dv.create()\n File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n loop.run_until_complete(pipeline)\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py\", line 152, in run\n pb.done += task.result() # download_count\n File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py\", line 178, in _handle_content_unit\n await asyncio.gather(*downloaders_for_content)\n File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/models.py\", line 88, in download\n download_result = await downloader.run(extra_data=self.extra_data)\n File \"/home/vagrant/devel/pulpcore/pulpcore/download/base.py\", line 227, in run\n return await self._run(extra_data=extra_data)\n File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/downloaders.py\", line 89, in _run\n self.raise_for_status(response)\n File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/downloaders.py\", line 69, in raise_for_status\n response.raise_for_status()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py\", line 1005, in raise_for_status\n headers=self.headers,\n",
"description": "403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/7c996fe24814bbd6ec92af3c497a155667ffd2062965255e64e4bb44ccf0d7a0/java-11-amazon-corretto-javadoc-11.0.8+10-1.amzn2.x86_64.rpm')"
},
"worker": "/pulp/api/v3/workers/13f52012-2e41-4d3b-b59f-a2e11135ec87/",
"parent_task": null,
"child_tasks": [],
"task_group": null,
"progress_reports": [
{
"message": "Parsed Comps",
"code": "parsing.comps",
"state": "completed",
"total": 11,
"done": 11,
"suffix": null
},
{
"message": "Parsed Advisories",
"code": "parsing.advisories",
"state": "completed",
"total": 482,
"done": 482,
"suffix": null
},
{
"message": "Downloading Artifacts",
"code": "downloading.artifacts",
"state": "failed",
"total": null,
"done": 73,
"suffix": null
},
{
"message": "Parsed Packages",
"code": "parsing.packages",
"state": "canceled",
"total": 22852,
"done": 3004,
"suffix": null
},
{
"message": "Downloading Metadata Files",
"code": "downloading.metadata",
"state": "canceled",
"total": null,
"done": 5,
"suffix": null
},
{
"message": "Associating Content",
"code": "associating.content",
"state": "canceled",
"total": null,
"done": 500,
"suffix": null
}
],
"created_resources": [],
"reserved_resources_record": [
"/pulp/api/v3/repositories/rpm/rpm/8ae9cf17-6da7-43d8-b81c-f0d65fbab23c/",
"/pulp/api/v3/remotes/rpm/rpm/32fec855-7d7a-4877-9c9d-f8db3ea8cf0d/"
]
}
Task in final state: failed
Updated by ggainey almost 4 years ago
You can look at the primary.xml.gz for a specific mirror-entry with this wget:
http://amazonlinux.us-east-1.amazonaws.com/2/core/2.0/x86_64/35c6cb980fb37ade35d6bf08c25249754c0833617182395eda0a5d6cf7e981d9/repodata/primary.xml.gz
Some of the RPMs named there are accessible:
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$ wget http://amazonlinux.us-east-1.amazonaws.com/2/core/2.0/x86_64/35c6cb980fb37ade35d6bf08c25249754c0833617182395eda0a5d6cf7e981d9/../../../../../blobstore/b8cd88deba82ed0657d5c64bba64cf42cb97b21937810c125622e556e6f64040/glibc-headers-2.26-36.amzn2.x86_64.rpm
--2020-12-15 14:18:06-- http://amazonlinux.us-east-1.amazonaws.com/blobstore/b8cd88deba82ed0657d5c64bba64cf42cb97b21937810c125622e556e6f64040/glibc-headers-2.26-36.amzn2.x86_64.rpm
Resolving amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)... 52.216.229.237, 2600:1fa0:80e4:bc49:34d9:2bae::
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|52.216.229.237|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 521796 (510K) [binary/octet-stream]
Saving to: ‘glibc-headers-2.26-36.amzn2.x86_64.rpm.1’
100%[========================================================================================================================================>] 521,796 --.-K/s in 0.05s
2020-12-15 14:18:07 (10.5 MB/s) - ‘glibc-headers-2.26-36.amzn2.x86_64.rpm.1’ saved [521796/521796]
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$ file glibc-headers-2.26-36.amzn2.x86_64.rpm.1
glibc-headers-2.26-36.amzn2.x86_64.rpm.1: RPM v3.0 bin i386/x86_64 glibc-headers-2.26-36.amzn2
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$
The one from the failure in #note-4, alas, is not:
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$ wget http://amazonlinux.us-east-1.amazonaws.com/2/core/2.0/x86_64/35c6cb980fb37ade35d6bf08c25249754c0833617182395eda0a5d6cf7e981d9/../../../../../blobstore/7c996fe24814bbd6ec92af3c497a155667ffd2062965255e64e4bb44ccf0d7a0/java-11-amazon-corretto-javadoc-11.0.8+10-1.amzn2.x86_64.rpm
--2020-12-15 14:19:37-- http://amazonlinux.us-east-1.amazonaws.com/blobstore/7c996fe24814bbd6ec92af3c497a155667ffd2062965255e64e4bb44ccf0d7a0/java-11-amazon-corretto-javadoc-11.0.8+10-1.amzn2.x86_64.rpm
Resolving amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)... 52.216.227.171, 2600:1fa0:8094:9040:34d8:e675::
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|52.216.227.171|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-12-15 14:19:37 ERROR 403: Forbidden.
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$
So - even once we fix the mirror-list issue, that specific repo has a problem we can't fix, it looks like.
Regardless, the mirror-list response doesn't end in a '/', and the code at https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/synchronizing.py#L101 fails as a result (thanks @ipanova !)
Updated by fao89 almost 4 years ago
- Project changed from Pulp to RPM Support
- Description updated (diff)
Updated by fao89 almost 4 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 87
Updated by ggainey over 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ggainey
Updated by pulpbot over 3 years ago
- Status changed from ASSIGNED to POST
Updated by ggainey over 3 years ago
With the attached PR handling the mirror-list is fixed. The 403-forbidden isn't anything pulp3 can do anything about.
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ pulp rpm repository create az4
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ pulp rpm repository create --name az4
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ pulp rpm remote create --name az4 --url http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ pulp rpm repository sync --name az4 --remote az4
Started background task /pulp/api/v3/tasks/931aeab8-2ffd-47d5-a562-748eee930a0d/
................Error: Task /pulp/api/v3/tasks/931aeab8-2ffd-47d5-a562-748eee930a0d/ failed: '403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/55d75541d27a69176da43c5f62487809c8af032b85f1252eac3c58bc48e54e73/xmlrpc-c-c++-1.32.5-1905.svn2451.amzn2.0.1.x86_64.rpm')'
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ wget http://amazonlinux.us-east-1.amazonaws.com/blobstore/55d75541d27a69176da43c5f62487809c8af032b85f1252eac3c58bc48e54e73/xmlrpc-c-c++-1.32.5-1905.svn2451.amzn2.0.1.x86_64.rpm
--2021-03-16 13:51:38-- http://amazonlinux.us-east-1.amazonaws.com/blobstore/55d75541d27a69176da43c5f62487809c8af032b85f1252eac3c58bc48e54e73/xmlrpc-c-c++-1.32.5-1905.svn2451.amzn2.0.1.x86_64.rpm
Resolving amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)... 52.216.240.70, 2600:1fa0:8041:4041:34d8:8dfe::
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|52.216.240.70|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-03-16 13:51:38 ERROR 403: Forbidden.
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$
Updated by vchepkov over 3 years ago
That's great and I agree, we will be opening AWS support tickets to resolve permission issues
Added by ggainey over 3 years ago
Updated by ggainey over 3 years ago
- Status changed from POST to MODIFIED
Applied in changeset fd130be505d2adff140f29800da416fe178aed3f.
Updated by dalley over 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Teach pulp_rpm to only use urljoin() where it really needs urljoin().
fixes #7995