Project

Profile

Help

Issue #4059

closed

During rpm upload filename should be preserved in the storage path

Added by ipanova@redhat.com over 5 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.21.0
OS:
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 58
Quarter:

Description

As of now if rpm is uploaded the storage path will look like:

    "_storage_path" : "/var/lib/pulp/content/units/rpm/d8/35d09b8cb744866dfd913f1f4fa64700683c1199351350c770e67760ce03a0/73744eb6-01df-4386-a647-2fe64cc95c8e",

Instead of the filename, the upload_id is used during the construction of storage path.

This will lead to the rsync publish distributor to publish files without trailing filename since it uses the _storage_path

https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/upload.py#L441
https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/publish_step.py#L1376


Related issues

Related to RPM Support - Issue #4798: Rpm file uploaded/published successfully but not accessibleCLOSED - WONTFIXActions
Related to RPM Support - Issue #5419: Regression - rpm_basefilename tests failuresCLOSED - CURRENTRELEASEggaineyActions
Actions #1

Updated by ipanova@redhat.com over 5 years ago

  • Description updated (diff)
Actions #2

Updated by CodeHeeler over 5 years ago

  • Triaged changed from No to Yes
Actions #3

Updated by bmbouter almost 5 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

Actions #4

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added
Actions #5

Updated by ggainey almost 5 years ago

  • Related to Issue #4798: Rpm file uploaded/published successfully but not accessible added
Actions #6

Updated by ggainey almost 5 years ago

  • Status changed from CLOSED - WONTFIX to ASSIGNED
  • Assignee set to ggainey
  • Sprint set to Sprint 53

Reopening, as this appears to be causing Other Issues (see #4798)

Actions #7

Updated by amacdona@redhat.com almost 5 years ago

  • Sprint changed from Sprint 53 to Sprint 54
Actions #8

Updated by ggainey almost 5 years ago

[notes from investigating, before I forget everything]

After a bunch of experimentation, I've discovered there are two problems with upload.

The first is exactly as explained in the description. An uploaded RPM is stored using its upload_id instead of its name, and that causes problems at rsync-distribution-time.

As a specific example - uploading 'rpm-with-non-ascii-1-1.fc25.noarch.rpm' leaves us with a path like

/var/lib/pulp/content/units/rpm/45/092320c072b594d5951ef9e8663c8b7ccfd529f1659d33c7087ec29606a4a4/989f7cbd-e61c-4183-92b6-8b31e9ded37e

instead of

/var/lib/pulp/content/units/rpm/59/e9b952c2a8ab9442864adf0379f5bcc002ae219eb679d1635cce6c08e6d54d/rpm-with-non-ascii-1-1.fc25.noarch.rpm

The second problem is when you attempt to upload an RPM to a deferred-download repository, where a) the repository has been sync'd at least once (so we know its RPMs), b) the RPM being uploaded is 'known' to the repository, and c) the feed for the deferred-repository is unreachable or otherwise broken - uploading an RPM 'directly' to get around the broken-feed problem. (see #4798)

Generally, uploading an RPM with the same NEVRA as one already in the repository won't do anything - if you want to replace a NEVRA, delete the old one first, then upload the new. If a broken feed means you can't get the 'original' RPM, then fix the feed and get it 'the right way'. However, as noted in #4798, #4265 prevents us from fixing the feed.

This doesn't mean we should add a special case to teach upload to notice that a feed is broken and allow uploading an existing NEVRA 'just this once'. Rather than complicate this issue with yet another edge-case, we'll fix the upload-filepath problem that breaks the rsync publisher here, and fix the broken-feed-for-deferred-repos problem under #4265.

Actions #9

Updated by ggainey almost 5 years ago

  • Status changed from ASSIGNED to CLOSED - WONTFIX

After further investigation, I'm back to closing this as WONTFIX. The original description is correct - RPM-upload does result in the actual RPM binary ending up with its upload-id as the filename, instead of the 'real' filename. However, the only place to see this is under the content/units portion of pulp's tree. The Packages symlink in the repo has the correct name, and points to the correct file; this works after export as well (tested rsycn and ISO exports)

As an example - I created a bare repo, empty , and uploaded a single RPM into it, rpm-with-non-ascii-1-1.fc25.noarch.rpm

pulp-admin rpm repo create --repo-id empty --relative-url empty
pulp-admin rpm repo uploads rpm --file rpm-with-non-ascii-1-1.fc25.noarch.rpm --repo-id empty
pulp-admin rpm repo publish run --repo-id empty

It clearly has the 'wrong' name on the filesystem:

[vagrant@pulp2 ~]$ sudo tree /var/lib/pulp
/var/lib/pulp
├── celery
├── content
│   └── units
│       └── rpm
│           └── 45
│               └── 092320c072b594d5951ef9e8663c8b7ccfd529f1659d33c7087ec29606a4a4
│                   └── aab88552-2c46-4b5d-a57d-cf134b4340f3
├── nodes
│   └── published

But the link in the published repo is named correctly and points to the right place:

├── published
│   └── yum
│       ├── http
│       ├── https
...
│       │   └── repos
│       │       ├── empty -> /var/lib/pulp/published/yum/master/yum_distributor/empty/1560350550.08
│       │       └── listing
│       └── master
...
│           └── yum_distributor
│               └── empty
│                   └── 1560350550.08
│                       ├── Packages
│                       │   └── r
│                       │       └── rpm-with-non-ascii-1-1.fc25.noarch.rpm -> /var/lib/pulp/content/units/rpm/45/092320c072b594d5951ef9e8663c8b7ccfd529f1659d33c7087ec29606a4a4/aab88552-2c46-4b5d-a57d-cf134b4340f3
│                       └── repodata
...

And I can retrieve the RPM using its correct name:

[vagrant@pulp2 ~]$ curl -L -k -o rpm-with-non-ascii.rpm https://localhost/pulp/repos/empty/Packages/r/rpm-with-non-ascii-1-1.fc25.noarch.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6432  100  6432    0     0   216k      0 --:--:-- --:--:-- --:--:--  216k
[vagrant@pulp2 ~]$ file rpm-with-non-ascii.rpm 
rpm-with-non-ascii.rpm: RPM v3.0 bin i386/x86_64 rpm-with-non-ascii-1-1.fc25
[vagrant@pulp2 ~]$ 

And when I export this repo:

[vagrant@pulp2 ~]$ http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/empty/distributors/ < rsync_distributor_config.json
[vagrant@pulp2 ~]$ http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/empty/actions/publish/ < publish_rsync.json

The resulting rsync'd tree on the destination machine is a correct repository:

~/RSYNC $ tree
.
├── content
│   └── units
│       └── rpm
│           └── 45
│               └── 092320c072b594d5951ef9e8663c8b7ccfd529f1659d33c7087ec29606a4a4
│                   └── aab88552-2c46-4b5d-a57d-cf134b4340f3
└── empty
    ├── Packages
    │   └── r
    │       └── rpm-with-non-ascii-1-1.fc25.noarch.rpm -> ../../../content/units/rpm/45/092320c072b594d5951ef9e8663c8b7ccfd529f1659d33c7087ec29606a4a4/aab88552-2c46-4b5d-a57d-cf134b4340f3
    └── repodata
        ├── 00e8c4132473d5f6e823d5b6b7c4850ec2eec5224f053a00d62cf73447fba90d-primary.xml.gz
        ├── 474a6fd724269774139856dadc69b675a47d35a4c21b666ae4d0244b64b391fe-updateinfo.xml.gz
        ├── a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
        ├── b72a89d18f6871e7009bf833a71d461e2ec766663991e4f8601fd3e73a386728-other.xml.gz
        ├── bc7b0ed20aefa013eb348dfb191c54fcdf5781c71c21ecb027b50a48b409b328-filelists.xml.gz
        └── repomd.xml

9 directories, 8 files
~/RSYNC $

All of the above being said - it's possible I'm missing where something is actually broken - please re-open with an explanation if so!

Actions #10

Updated by ggainey over 4 years ago

  • Status changed from CLOSED - WONTFIX to NEW
  • Sprint deleted (Sprint 54)

DELIVERY-5013 shows a use-case where this behavior is causing pain for our users - re-opening for consideration for future sprint

Actions #11

Updated by ggainey over 4 years ago

  • Status changed from NEW to POST
  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
  • Sprint set to Sprint 57

Figured out how to fix this.

https://github.com/pulp/pulp_rpm/pull/1421

Added by ggainey over 4 years ago

Revision 5154cebd | View on GitHub

Teach upload to use RPM filename instead of upload-id

closes #4059 https://pulp.plan.io/issues/4059

Actions #12

Updated by rchan over 4 years ago

  • Sprint changed from Sprint 57 to Sprint 58
Actions #13

Updated by ggainey over 4 years ago

  • Status changed from POST to MODIFIED
Actions #14

Updated by kersom over 4 years ago

  • Related to Issue #5419: Regression - rpm_basefilename tests failures added
Actions #15

Updated by dalley over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Platform Release set to 2.21.0

Also available in: Atom PDF