Project

Profile

Help

Issue #9328

repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)

Added by mwaqas@bloombergindustry.com about 1 month ago. Updated about 1 month ago.

Status:
MODIFIED
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 105
Quarter:

Description

I created multiple repos and selected default setting but metadata does not get published. Packages directory is missing from published repo. 'manually republishing metadata fixes it but I am afraid it will become stale.

unpublished_metadata.png (47.6 KB) unpublished_metadata.png mwaqas@bloombergindustry.com, 09/01/2021 05:10 PM
repo_settings.png (355 KB) repo_settings.png mwaqas@bloombergindustry.com, 09/01/2021 05:10 PM
250
250

Related issues

Copied to RPM Support - Backport #9392: Backport #9328 "repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)" to 3.14.zCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 59d67f8a View on GitHub
Added by dalley about 1 month ago

Dealt with another situation incompatible with mirror syncing

Some repos use location_href values that point outside of the repository. We cannot support this in mirror mode because we cannot write paths outside of the repository namespace.

closes: #9328 https://pulp.plan.io/issues/9328

History

#1 Updated by dalley about 1 month ago

  • Project changed from Pulp to RPM Support
  • Priority changed from Normal to High
  • Sprint set to Sprint 104

#2 Updated by dalley about 1 month ago

@mwaqas@bloombergindustry.com Which versions of pulpcore and pulp_rpm are you using?

#3 Updated by dalley about 1 month ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

Confirmed w/ master branch

#4 Updated by dalley about 1 month ago

  • Triaged changed from No to Yes

#5 Updated by ttereshc about 1 month ago

  • Subject changed from repository metadata not getting published to repository metadata not getting published for some repos sync from a mirror list

#6 Updated by ttereshc about 1 month ago

  • Subject changed from repository metadata not getting published for some repos sync from a mirror list to repository metadata not getting published for some repos synced from a mirror list

#8 Updated by mwaqas@bloombergindustry.com about 1 month ago

dalley wrote:

@mwaqas@bloombergindustry.com Which versions of pulpcore and pulp_rpm are you using?

rpm -qa |grep -i pulpcore
tfm-rubygem-pulpcore_client-3.14.1-1.el7.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python3-pulpcore-3.14.4-1.el7.noarch
rpm -qa |grep -i pulp_rpm
tfm-rubygem-pulp_rpm_client-3.13.3-1.el7.noarch

#9 Updated by rchan about 1 month ago

  • Sprint changed from Sprint 104 to Sprint 105

#11 Updated by dalley about 1 month ago

  • Subject changed from repository metadata not getting published for some repos synced from a mirror list to repository metadata not getting published for some repos synced from behind Amazon CDN

https://packages.rundeck.com/pagerduty/rundeck/rpm_any/rpm_any/x86_64 is not a mirrorlist repo, so that is not the issue.

But both of these repos are hosted behind the Amazon Cloudfront CDN, which is doing some really crazy and annoying obfuscation (presumably for purposes of DDoS prevention). I don't have proof yet but I suspect that's part of the problem.

edit: I was wrong, that's not the reason. The reason is that these repos use location_href values that point backwards outside of the actual repo. Pulp cannot mirror these paths, so the only solution is to throw an error and tell the user to disable mirroring, so that the location_href values in the metadata can be rewritten inside the path namespace of the distribution.

#12 Updated by dalley about 1 month ago

The solution is to disable "mirror-on-sync". And the "bugfix" here should be to add an hard-error that fails for repositories like this one, and tells the user to disable mirroring.

The "mirror" option makes a complete copy of the metadata and retains the same package layout as the original repo. But because the original layout keeps all of the package outside of the repo itself, mirror can't possibly work. It would be putting the files outside of the distribution base_path and polluting / interfering with the global namespace, and we cannot allow that.

#13 Updated by dalley about 1 month ago

  • Status changed from ASSIGNED to POST

#15 Updated by dalley about 1 month ago

  • Subject changed from repository metadata not getting published for some repos synced from behind Amazon CDN to repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)

#16 Updated by dalley about 1 month ago

  • Sprint/Milestone set to 3.14.4

#17 Updated by dalley about 1 month ago

  • Status changed from POST to MODIFIED

#18 Updated by dalley about 1 month ago

  • Sprint/Milestone changed from 3.14.4 to 3.16.0

#19 Updated by dalley about 1 month ago

  • Copied to Backport #9392: Backport #9328 "repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)" to 3.14.z added

Please register to edit this issue

Also available in: Atom PDF