Project

Profile

Help

Story #6729

As user, I can sync SUSE repos which require auth-token

Added by abrix 5 months ago. Updated 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
No
Tags:
Sprint:
Sprint 75
Quarter:

Description

Hi,

with Pulp 3 the possibility of mirroring a SUSE or any other repository which requires an auth-token, does not seem to be available anymore as it was possible with Pulp 2 [1].

There's also no reference in the API documentation [2] to the use of an auth-token in the remotes.

Best regards,

Alex

403 error during repo sync:

{
    "created_resources": [],
    "error": {
        "description": "403, message='Forbidden', url=URL('https://updates.suse.com/SUSE/Products/SLE-SERVER/12-SP4/x86_64/product/.treeinfo')",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 884, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 125, in synchronize\n    treeinfo = get_treeinfo_data(remote)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/kickstart/treeinfo.py\", line 24, in get_treeinfo_data\n    result = downloader.fetch()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 154, in fetch\n    return done.pop().result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/base.py\", line 221, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/download/http.py\", line 185, in _run\n    response.raise_for_status()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py\", line 946, in raise_for_status\n    headers=self.headers)\n"
    },
    "finished_at": "2020-05-14T06:50:14.111981Z",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "progress_reports": [],
    "pulp_created": "2020-05-14T06:50:13.791631Z",
    "pulp_href": "/pulp/api/v3/tasks/ce86f248-93d3-4da7-b6ef-49e00693497e/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/rpm/rpm/5adfc022-9992-4e09-8d31-454a41f0f805/",
        "/pulp/api/v3/repositories/rpm/rpm/6193f212-d07a-43de-acc4-c3bb331c372b/"
    ],
    "started_at": "2020-05-14T06:50:13.898656Z",
    "state": "failed",
    "worker": "/pulp/api/v3/workers/85fb61f8-f74e-4f3a-a95d-07f8caa24c7d/"
}

[1] [[https://pulpproject.org/2015/12/09/syncing-and-publishing-suse-repositories-with-pulp/]]

[2] [[https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/remotes_rpm_rpm_create]]

Proposed solution:

  • Create new RpmRemote and its serializer with one extra field for a auth token
  • Create new downloader (and downloader chooser) which can use newly created remotes with auth-token

Related issues

Related to Migration Plugin - Story #6927: As a user, I can migrate Remotes for SLES reposCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 3f84d87b View on GitHub
Added by ppicka 4 months ago

SLES support with authentication token

Added support for SLES repository with use of authentication token. Also updated docs how to use it.

closes: #6729 https://pulp.plan.io/issues/6729

[nocoverage]

History

#1 Updated by ttereshc 5 months ago

  • Tracker changed from Issue to Story
  • Subject changed from Pulp 3 remotes rpm auth-token missing to As user, I can sync SUSE repos which require auth-token
  • % Done set to 0
  • Tags deleted (Pulp 3 RPM blocker)

#2 Updated by dkliban@redhat.com 5 months ago

Here is the story for when this feature was added to Pulp 2[0]. It does seem like this should be a separate Remote type that will use a custom downloader.

[0] https://pulp.plan.io/issues/1358

#3 Updated by abrix 5 months ago

I tried as suggested in issue 1358 [1] the following query via repositories

http --body PUT http://localhost:24817/pulp/api/v3/repositories/rpm/rpm/d465894f-c31a-4e31-8841-c89f09920895/ name=upstream-sles-12sp4-x86_64-pool importer_config:='{"query_auth_token": "SECRET"}'
{
    "importer_config": [
        "Unexpected field"
    ]
}

and more applicable in remotes (IMHO) but with the same result

http --body PUT http://localhost:24817"/pulp/api/v3/remotes/rpm/rpm/795524ff-6ee6-444d-bd49-8eb60938c368/" name=upstream-sles-12sp4-x86_64-pool importer_config:='{"query_auth_token": "SECRET"}' url="https://updates.suse.com/SUSE/Products/SLE-SERVER/12-SP4/x86_64/product"
{
    "importer_config": [
        "Unexpected field"
    ]
}

[1] https://pulp.plan.io/issues/1358#note-8

#4 Updated by ppicka 4 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ppicka

#5 Updated by ppicka 4 months ago

  • Description updated (diff)

#6 Updated by ppicka 4 months ago

  • Description updated (diff)

#7 Updated by dkliban@redhat.com 4 months ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 73

#8 Updated by rchan 4 months ago

  • Sprint changed from Sprint 73 to Sprint 74

#9 Updated by pulpbot 4 months ago

  • Status changed from ASSIGNED to POST

#10 Updated by ipanova@redhat.com 4 months ago

  • Related to Story #6927: As a user, I can migrate Remotes for SLES repos added

#11 Updated by rchan 4 months ago

  • Sprint changed from Sprint 74 to Sprint 75

#12 Updated by ppicka 4 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#13 Updated by ttereshc 2 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#14 Updated by ttereshc 2 months ago

  • Sprint/Milestone set to Pulp RPM 3.5.0

Please register to edit this issue

Also available in: Atom PDF