Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #9328

closed

repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)

Added by mwaqas@bloombergindustry.com over 2 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
dalley
Sprint/Milestone:
3.16.0
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 105
Quarter:

Description

I created multiple repos and selected default setting but metadata does not get published. Packages directory is missing from published repo. 'manually republishing metadata fixes it but I am afraid it will become stale.

Files

Download all files
unpublished_metadata.png (47.6 KB) unpublished_metadata.png mwaqas@bloombergindustry.com, 09/01/2021 05:10 PM
repo_settings.png (355 KB) repo_settings.png mwaqas@bloombergindustry.com, 09/01/2021 05:10 PM

Related issues

Copied to RPM Support - Backport #9392: Backport #9328 "repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Actions
Copy link
 #1

Updated by dalley over 2 years ago

  • Project changed from Pulp to RPM Support
  • Priority changed from Normal to High
  • Sprint set to Sprint 104
Actions
Copy link
 #2

Updated by dalley over 2 years ago

@mwaqas@bloombergindustry.com Which versions of pulpcore and pulp_rpm are you using?

Actions
Copy link
 #3

Updated by dalley over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

Confirmed w/ master branch

Actions
Copy link
 #4

Updated by dalley over 2 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #5

Updated by ttereshc over 2 years ago

  • Subject changed from repository metadata not getting published to repository metadata not getting published for some repos sync from a mirror list
Actions
Copy link
 #6

Updated by ttereshc over 2 years ago

  • Subject changed from repository metadata not getting published for some repos sync from a mirror list to repository metadata not getting published for some repos synced from a mirror list
Actions
Copy link
 #8

Updated by mwaqas@bloombergindustry.com over 2 years ago

dalley wrote:

@mwaqas@bloombergindustry.com Which versions of pulpcore and pulp_rpm are you using?

rpm -qa |grep -i pulpcore
tfm-rubygem-pulpcore_client-3.14.1-1.el7.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python3-pulpcore-3.14.4-1.el7.noarch
rpm -qa |grep -i pulp_rpm
tfm-rubygem-pulp_rpm_client-3.13.3-1.el7.noarch

Actions
Copy link
 #9

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions
Copy link
 #11

Updated by dalley over 2 years ago

  • Subject changed from repository metadata not getting published for some repos synced from a mirror list to repository metadata not getting published for some repos synced from behind Amazon CDN

https://packages.rundeck.com/pagerduty/rundeck/rpm_any/rpm_any/x86_64 is not a mirrorlist repo, so that is not the issue.

But both of these repos are hosted behind the Amazon Cloudfront CDN, which is doing some really crazy and annoying obfuscation (presumably for purposes of DDoS prevention). I don't have proof yet but I suspect that's part of the problem.

edit: I was wrong, that's not the reason. The reason is that these repos use location_href values that point backwards outside of the actual repo. Pulp cannot mirror these paths, so the only solution is to throw an error and tell the user to disable mirroring, so that the location_href values in the metadata can be rewritten inside the path namespace of the distribution.

Actions
Copy link
 #12

Updated by dalley over 2 years ago

The solution is to disable "mirror-on-sync". And the "bugfix" here should be to add an hard-error that fails for repositories like this one, and tells the user to disable mirroring.

The "mirror" option makes a complete copy of the metadata and retains the same package layout as the original repo. But because the original layout keeps all of the package outside of the repo itself, mirror can't possibly work. It would be putting the files outside of the distribution base_path and polluting / interfering with the global namespace, and we cannot allow that.

Actions
Copy link
 #13

Updated by dalley over 2 years ago

  • Status changed from ASSIGNED to POST
Actions
Copy link
 #15

Updated by dalley over 2 years ago

  • Subject changed from repository metadata not getting published for some repos synced from behind Amazon CDN to repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)
Actions
Copy link
 #16

Updated by dalley over 2 years ago

  • Sprint/Milestone set to 3.14.4

Added by dalley over 2 years ago

Revision 59d67f8a | View on GitHub

Dealt with another situation incompatible with mirror syncing

Some repos use location_href values that point outside of the repository. We cannot support this in mirror mode because we cannot write paths outside of the repository namespace.

closes: #9328 https://pulp.plan.io/issues/9328

Actions
Copy link
 #17

Updated by dalley over 2 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #18

Updated by dalley over 2 years ago

  • Sprint/Milestone changed from 3.14.4 to 3.16.0
Actions
Copy link
 #19

Updated by dalley over 2 years ago

  • Copied to Backport #9392: Backport #9328 "repository metadata not getting published for some repos which have ".." in location_href (points outside of repo)" to 3.14.z added
Actions
Copy link
 #20

Updated by pulpbot over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Send by e-mail Copy link

Also available in: Atom PDF