Project

Profile

Help

Issue #9328

closed

repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)

Added by mwaqas@bloombergindustry.com about 1 year ago. Updated 11 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 105
Quarter:

Description

I created multiple repos and selected default setting but metadata does not get published. Packages directory is missing from published repo. 'manually republishing metadata fixes it but I am afraid it will become stale.


Files

unpublished_metadata.png (47.6 KB) unpublished_metadata.png mwaqas@bloombergindustry.com, 09/01/2021 05:10 PM
repo_settings.png (355 KB) repo_settings.png mwaqas@bloombergindustry.com, 09/01/2021 05:10 PM

Related issues

Copied to RPM Support - Backport #9392: Backport #9328 "repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Actions #1

Updated by dalley about 1 year ago

  • Project changed from Pulp to RPM Support
  • Priority changed from Normal to High
  • Sprint set to Sprint 104
Actions #2

Updated by dalley about 1 year ago

@mwaqas@bloombergindustry.com Which versions of pulpcore and pulp_rpm are you using?

Actions #3

Updated by dalley about 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

Confirmed w/ master branch

Actions #4

Updated by dalley about 1 year ago

  • Triaged changed from No to Yes
Actions #5

Updated by ttereshc about 1 year ago

  • Subject changed from repository metadata not getting published to repository metadata not getting published for some repos sync from a mirror list
Actions #6

Updated by ttereshc about 1 year ago

  • Subject changed from repository metadata not getting published for some repos sync from a mirror list to repository metadata not getting published for some repos synced from a mirror list
Actions #8

Updated by mwaqas@bloombergindustry.com about 1 year ago

dalley wrote:

@mwaqas@bloombergindustry.com Which versions of pulpcore and pulp_rpm are you using?

rpm -qa |grep -i pulpcore
tfm-rubygem-pulpcore_client-3.14.1-1.el7.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python3-pulpcore-3.14.4-1.el7.noarch
rpm -qa |grep -i pulp_rpm
tfm-rubygem-pulp_rpm_client-3.13.3-1.el7.noarch

Actions #9

Updated by rchan about 1 year ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions #11

Updated by dalley about 1 year ago

  • Subject changed from repository metadata not getting published for some repos synced from a mirror list to repository metadata not getting published for some repos synced from behind Amazon CDN

https://packages.rundeck.com/pagerduty/rundeck/rpm_any/rpm_any/x86_64 is not a mirrorlist repo, so that is not the issue.

But both of these repos are hosted behind the Amazon Cloudfront CDN, which is doing some really crazy and annoying obfuscation (presumably for purposes of DDoS prevention). I don't have proof yet but I suspect that's part of the problem.

edit: I was wrong, that's not the reason. The reason is that these repos use location_href values that point backwards outside of the actual repo. Pulp cannot mirror these paths, so the only solution is to throw an error and tell the user to disable mirroring, so that the location_href values in the metadata can be rewritten inside the path namespace of the distribution.

Actions #12

Updated by dalley about 1 year ago

The solution is to disable "mirror-on-sync". And the "bugfix" here should be to add an hard-error that fails for repositories like this one, and tells the user to disable mirroring.

The "mirror" option makes a complete copy of the metadata and retains the same package layout as the original repo. But because the original layout keeps all of the package outside of the repo itself, mirror can't possibly work. It would be putting the files outside of the distribution base_path and polluting / interfering with the global namespace, and we cannot allow that.

Actions #13

Updated by dalley about 1 year ago

  • Status changed from ASSIGNED to POST
Actions #15

Updated by dalley about 1 year ago

  • Subject changed from repository metadata not getting published for some repos synced from behind Amazon CDN to repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)
Actions #16

Updated by dalley about 1 year ago

  • Sprint/Milestone set to 3.14.4

Added by dalley about 1 year ago

Revision 59d67f8a

Dealt with another situation incompatible with mirror syncing

Some repos use location_href values that point outside of the repository. We cannot support this in mirror mode because we cannot write paths outside of the repository namespace.

closes: #9328 https://pulp.plan.io/issues/9328

Actions #17

Updated by dalley about 1 year ago

  • Status changed from POST to MODIFIED
Actions #18

Updated by dalley about 1 year ago

  • Sprint/Milestone changed from 3.14.4 to 3.16.0
Actions #19

Updated by dalley about 1 year ago

  • Copied to Backport #9392: Backport #9328 "repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)" to 3.14.z added
Actions #20

Updated by pulpbot 11 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF