RPM Support

based on the discussions with a user and from looking at the traceback this is a pulp3 issue. potential offending line is https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/synchronizing.py#L111 possibly not catching some other exception that is happening. Asked user to provide more info and list of installed packages.

I see following pulp packages:

tfm-rubygem-pulpcore_client-3.6.0-1.el7.noarch python3-pulpcore-3.6.3-2.el7.noarch tfm-rubygem-pulp_rpm_client-3.6.2-1.el7.noarch

http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list

The problem llies in Pulp3's handling of Amazon's reositories.

Pulp2 works:

  713  pulp-admin login -u admin -p admin
  714  pulp-admin rpm repo create --repo-id amazon --feed http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
  715  pulp-admin rpm repo sync run --repo-id amazon

Pulp3 fails. If you point Pulp3 at a specific return from mirror.list, it still fails, but gets further:

BASE_ADDR=":"
#http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list"
AMAZON_URL="http://amazonlinux.us-east-1.amazonaws.com/2/core/2.0/x86_64/35c6cb980fb37ade35d6bf08c25249754c0833617182395eda0a5d6cf7e981d9/"
AMAZON_NAME="amazon4"
AMAZON_HREF=$(http POST $BASE_ADDR/pulp/api/v3/repositories/rpm/rpm/ name=$AMAZON_NAME | jq -r '.pulp_href')
echo "repo_href : " $AMAZON_HREF
if [ -z "$AMAZON_HREF" ]; then exit; fi
# add remote
http POST $BASE_ADDR/pulp/api/v3/remotes/rpm/rpm/ name=$AMAZON_NAME url=$AMAZON_URL  policy='immediate' download_concurrency=7
# find remote's href
REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/rpm/rpm/ | jq -r ".results[] | select(.name == \"${AMAZON_NAME}\") | .pulp_href")
echo "remote_href : " $REMOTE_HREF
if [ -z "$REMOTE_HREF" ]; then exit; fi
# sync
TASK_URL=$(http POST $BASE_ADDR$AMAZON_HREF'sync/' remote=$REMOTE_HREF | jq -r '.task')
echo "Task url : " $TASK_URL
if [ -z "$TASK_URL" ]; then exit; fi
# wait for task
wait_until_task_finished $BASE_ADDR$TASK_URL

Task processes metadata, but fails (another 403) trying to access RPMs:

Still waiting...
{
  "pulp_href": "/pulp/api/v3/tasks/27481944-929c-40b7-ae9c-4e60a5bdda9e/",
  "pulp_created": "2020-12-15T13:51:48.773054Z",
  "state": "failed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "73fc25e344da4449887b93f36a3bab5e",
  "started_at": "2020-12-15T13:51:48.944922Z",
  "finished_at": "2020-12-15T13:52:02.612576Z",
  "error": {
    "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 975, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 696, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 719, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 272, in synchronize\n    dv.create()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py\", line 152, in run\n    pb.done += task.result()  # download_count\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py\", line 178, in _handle_content_unit\n    await asyncio.gather(*downloaders_for_content)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/models.py\", line 88, in download\n    download_result = await downloader.run(extra_data=self.extra_data)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/download/base.py\", line 227, in run\n    return await self._run(extra_data=extra_data)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/downloaders.py\", line 89, in _run\n    self.raise_for_status(response)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/downloaders.py\", line 69, in raise_for_status\n    response.raise_for_status()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py\", line 1005, in raise_for_status\n    headers=self.headers,\n",
    "description": "403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/7c996fe24814bbd6ec92af3c497a155667ffd2062965255e64e4bb44ccf0d7a0/java-11-amazon-corretto-javadoc-11.0.8+10-1.amzn2.x86_64.rpm')"
  },
  "worker": "/pulp/api/v3/workers/13f52012-2e41-4d3b-b59f-a2e11135ec87/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Parsed Comps",
      "code": "parsing.comps",
      "state": "completed",
      "total": 11,
      "done": 11,
      "suffix": null
    },
    {
      "message": "Parsed Advisories",
      "code": "parsing.advisories",
      "state": "completed",
      "total": 482,
      "done": 482,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "downloading.artifacts",
      "state": "failed",
      "total": null,
      "done": 73,
      "suffix": null
    },
    {
      "message": "Parsed Packages",
      "code": "parsing.packages",
      "state": "canceled",
      "total": 22852,
      "done": 3004,
      "suffix": null
    },
    {
      "message": "Downloading Metadata Files",
      "code": "downloading.metadata",
      "state": "canceled",
      "total": null,
      "done": 5,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "canceled",
      "total": null,
      "done": 500,
      "suffix": null
    }
  ],
  "created_resources": [],
  "reserved_resources_record": [
    "/pulp/api/v3/repositories/rpm/rpm/8ae9cf17-6da7-43d8-b81c-f0d65fbab23c/",
    "/pulp/api/v3/remotes/rpm/rpm/32fec855-7d7a-4877-9c9d-f8db3ea8cf0d/"
  ]
}
Task in final state: failed

You can look at the primary.xml.gz for a specific mirror-entry with this wget:

http://amazonlinux.us-east-1.amazonaws.com/2/core/2.0/x86_64/35c6cb980fb37ade35d6bf08c25249754c0833617182395eda0a5d6cf7e981d9/repodata/primary.xml.gz

Some of the RPMs named there are accessible:

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$ wget http://amazonlinux.us-east-1.amazonaws.com/2/core/2.0/x86_64/35c6cb980fb37ade35d6bf08c25249754c0833617182395eda0a5d6cf7e981d9/../../../../../blobstore/b8cd88deba82ed0657d5c64bba64cf42cb97b21937810c125622e556e6f64040/glibc-headers-2.26-36.amzn2.x86_64.rpm
--2020-12-15 14:18:06--  http://amazonlinux.us-east-1.amazonaws.com/blobstore/b8cd88deba82ed0657d5c64bba64cf42cb97b21937810c125622e556e6f64040/glibc-headers-2.26-36.amzn2.x86_64.rpm
Resolving amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)... 52.216.229.237, 2600:1fa0:80e4:bc49:34d9:2bae::
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|52.216.229.237|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 521796 (510K) [binary/octet-stream]
Saving to: ‘glibc-headers-2.26-36.amzn2.x86_64.rpm.1’

100%[========================================================================================================================================>] 521,796     --.-K/s   in 0.05s   

2020-12-15 14:18:07 (10.5 MB/s) - ‘glibc-headers-2.26-36.amzn2.x86_64.rpm.1’ saved [521796/521796]

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$ file glibc-headers-2.26-36.amzn2.x86_64.rpm.1
glibc-headers-2.26-36.amzn2.x86_64.rpm.1: RPM v3.0 bin i386/x86_64 glibc-headers-2.26-36.amzn2
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$ 

The one from the failure in #note-4, alas, is not:

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$ wget http://amazonlinux.us-east-1.amazonaws.com/2/core/2.0/x86_64/35c6cb980fb37ade35d6bf08c25249754c0833617182395eda0a5d6cf7e981d9/../../../../../blobstore/7c996fe24814bbd6ec92af3c497a155667ffd2062965255e64e4bb44ccf0d7a0/java-11-amazon-corretto-javadoc-11.0.8+10-1.amzn2.x86_64.rpm
--2020-12-15 14:19:37--  http://amazonlinux.us-east-1.amazonaws.com/blobstore/7c996fe24814bbd6ec92af3c497a155667ffd2062965255e64e4bb44ccf0d7a0/java-11-amazon-corretto-javadoc-11.0.8+10-1.amzn2.x86_64.rpm
Resolving amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)... 52.216.227.171, 2600:1fa0:8094:9040:34d8:e675::
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|52.216.227.171|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-12-15 14:19:37 ERROR 403: Forbidden.

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 import_export]$ 

So - even once we fix the mirror-list issue, that specific repo has a problem we can't fix, it looks like.

Regardless, the mirror-list response doesn't end in a '/', and the code at https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/tasks/synchronizing.py#L101 fails as a result (thanks @ipanova !)

With the attached PR handling the mirror-list is fixed. The 403-forbidden isn't anything pulp3 can do anything about.

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ pulp rpm repository create az4
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ pulp rpm repository create --name az4

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ pulp rpm remote create --name az4 --url http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ pulp rpm repository sync --name az4 --remote az4
Started background task /pulp/api/v3/tasks/931aeab8-2ffd-47d5-a562-748eee930a0d/
................Error: Task /pulp/api/v3/tasks/931aeab8-2ffd-47d5-a562-748eee930a0d/ failed: '403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/55d75541d27a69176da43c5f62487809c8af032b85f1252eac3c58bc48e54e73/xmlrpc-c-c++-1.32.5-1905.svn2451.amzn2.0.1.x86_64.rpm')'
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ wget http://amazonlinux.us-east-1.amazonaws.com/blobstore/55d75541d27a69176da43c5f62487809c8af032b85f1252eac3c58bc48e54e73/xmlrpc-c-c++-1.32.5-1905.svn2451.amzn2.0.1.x86_64.rpm
--2021-03-16 13:51:38--  http://amazonlinux.us-east-1.amazonaws.com/blobstore/55d75541d27a69176da43c5f62487809c8af032b85f1252eac3c58bc48e54e73/xmlrpc-c-c++-1.32.5-1905.svn2451.amzn2.0.1.x86_64.rpm
Resolving amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)... 52.216.240.70, 2600:1fa0:8041:4041:34d8:8dfe::
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|52.216.240.70|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-03-16 13:51:38 ERROR 403: Forbidden.

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ 

That's great and I agree, we will be opening AWS support tickets to resolve permission issues