Project

Profile

Help

Issue #4856

Uploading ISOs of same name to a repo creates duplicates, makes repo unusable

Added by rmcgover over 2 years ago. Updated over 2 years ago.

Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Uploading an ISO file into a repo which already contains an ISO of the same name, but with different content, will cause the repo to end up with multiple units of the same name. This breaks the repo, as PULP_MANIFEST will contain two entries for the same file, making the repo unable to be synced by another Pulp instance.

Steps to reproduce

In dev env:

# Make an ISO repo
$ pulp-admin -u admin -p admin iso repo create --repo-id zoo-iso

# Upload a file
$ echo hamburger > food
$ pulp-admin -u admin -p admin iso repo uploads upload --repo-id zoo-iso -f food

# Upload another file of same name
$ echo pizza > food
$ pulp-admin -u admin -p admin iso repo uploads upload --repo-id zoo-iso -f food

# Publish it
$ pulp-admin -u admin -p admin iso repo publish run --repo-id zoo-iso

# Check generated PULP_MANIFEST
$ curl -k https://localhost/pulp/isos/zoo-iso/PULP_MANIFEST
food,8cec6a9cb8cb1941243b138e50ab8e2dd1da2a380dd2c7c63928003f82173d9b,10
food,fff8acd78f7528c143cb5a6971f911d3869368cbc177f3f4404d945c6accc08d,6

Actual behavior

- Repo contains two ISO units named "food"
- PULP_MANIFEST contains two entries named "food"
- Pulp can't sync from the published repo.

Expected behavior

- Repo contains one ISO unit named "food" with checksum fff8acd7... (the second uploaded file)
- PULP_MANIFEST contains one entry named "food"
- Pulp can sync from the published repo.

Additional info

Might be a regression from Pulp ~2.8 but I'm not sure.

Seems almost a duplicate of https://pulp.plan.io/issues/3047, but I only see sync being tested there, even though issue description mentions uploads.

Tested with pulp_rpm 0f5b2e7c.

History

#2 Updated by bherring over 2 years ago

  • Related to Issue #4857: Pulp 2 Nightly Regression in test_iso_crud.ISOUpdateTestCase.test_all added

#3 Updated by ipanova@redhat.com over 2 years ago

rmcgover do you happen to have the fix in place already? or could you possibly submit a PR?

#4 Updated by ipanova@redhat.com over 2 years ago

  • Triaged changed from No to Yes

#5 Updated by ttereshc over 2 years ago

  • Related to deleted (Issue #4857: Pulp 2 Nightly Regression in test_iso_crud.ISOUpdateTestCase.test_all)

#6 Updated by ttereshc over 2 years ago

  • Related to Issue #4857: Pulp 2 Nightly Regression in test_iso_crud.ISOUpdateTestCase.test_all added

#7 Updated by ttereshc over 2 years ago

  • Related to deleted (Issue #4857: Pulp 2 Nightly Regression in test_iso_crud.ISOUpdateTestCase.test_all)

#8 Updated by ipanova@redhat.com over 2 years ago

rmcgover it seems like Pr for this issue https://pulp.plan.io/issues/4857 is fixing the problem. I was not able to reproduce this from 2- master. Can you confirm and close?

[ipanova@ina pulp_rpm]$ pulp-admin -u admin -p admin iso repo create --repo-id zoo-iso
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

Successfully created repository [zoo-iso]

[ipanova@ina pulp_rpm]$ echo hamburger > food
[ipanova@ina pulp_rpm]$ pulp-admin -u admin -p admin iso repo uploads upload --repo-id zoo-iso -f food
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: food
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: food
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: food
[==================================================] 100%
10/10 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[ipanova@ina pulp_rpm]$ pulp-admin iso repo  isos --repo-id zoo-iso
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

Name:     food
Size:     10
Checksum: 8cec6a9cb8cb1941243b138e50ab8e2dd1da2a380dd2c7c63928003f82173d9b

[ipanova@ina pulp_rpm]$ echo pizza > food
[ipanova@ina pulp_rpm]$ pulp-admin -u admin -p admin iso repo uploads upload --repo-id zoo-iso -f food
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: food
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: food
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: food
[==================================================] 100%
6/6 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[ipanova@ina pulp_rpm]$ pulp-admin iso repo  isos --repo-id zoo-iso
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

Name:     food
Size:     6
Checksum: fff8acd78f7528c143cb5a6971f911d3869368cbc177f3f4404d945c6accc08d

[ipanova@ina pulp_rpm]$ pulp-admin -u admin -p admin iso repo publish run --repo-id zoo-iso
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                    Publishing Repository [zoo-iso]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The repository was successfully published.

Task Succeeded

[ipanova@ina pulp_rpm]$ curl -k https://localhost/pulp/isos/zoo-iso/PULP_MANIFEST
food,fff8acd78f7528c143cb5a6971f911d3869368cbc177f3f4404d945c6accc08d,6
[ipanova@ina pulp_rpm]$ sudo ls -l /var/lib/pulp/published/https/isos/zoo-iso/food 
lrwxrwxrwx. 1 apache apache 97 May 28 10:53 /var/lib/pulp/published/https/isos/zoo-iso/food -> /var/lib/pulp/content/units/iso/21/403956117c5aebaece1fd64e2c3ebb0d86b8123e31f9aad365ce97c7bef498
[ipanova@ina pulp_rpm]$ cat /var/lib/pulp/content/units/iso/21/403956117c5aebaece1fd64e2c3ebb0d86b8123e31f9aad365ce97c7bef498
pizza
[ipanova@ina pulp_rpm]$ echo tacos > food
[ipanova@ina pulp_rpm]$ pulp-admin -u admin -p admin iso repo uploads upload --repo-id zoo-iso -f food
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: food
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: food
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: food
[==================================================] 100%
6/6 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[ipanova@ina pulp_rpm]$ pulp-admin iso repo  isos --repo-id zoo-iso
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

Name:     food
Size:     6
Checksum: e3e27097e49379044410e50f80781f63b0606eb93577a2d8ee4380eebf9b38bd

[ipanova@ina pulp_rpm]$ pulp-admin -u admin -p admin iso repo publish run --repo-id zoo-iso
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                    Publishing Repository [zoo-iso]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The repository was successfully published.

Task Succeeded

[ipanova@ina pulp_rpm]$ curl -k https://localhost/pulp/isos/zoo-iso/PULP_MANIFEST
food,e3e27097e49379044410e50f80781f63b0606eb93577a2d8ee4380eebf9b38bd,6
[ipanova@ina pulp_rpm]$ sudo ls -l /var/lib/pulp/published/https/isos/zoo-iso/food 
lrwxrwxrwx. 1 apache apache 97 May 28 10:55 /var/lib/pulp/published/https/isos/zoo-iso/food -> /var/lib/pulp/content/units/iso/e1/a29a7a8b1e8ca8f06c32d2dd7c24ab337d56821a48f657965c4265e370dfd9
[ipanova@ina pulp_rpm]$ cat /var/lib/pulp/content/units/iso/e1/a29a7a8b1e8ca8f06c32d2dd7c24ab337d56821a48f657965c4265e370dfd9
tacos
[ipanova@ina pulp_rpm]$ 

#9 Updated by rmcgover over 2 years ago

Yes, I couldn't reproduce it now as well.

Actually, there still seems to be a closely related issue - corrupt repo can still be created if you put ISOs into the repo by copy, rather than direct upload. e.g. this process:

  • create empty repos: upload-repo, target-repo
  • upload food=hamburger to upload-repo
  • iso repo copy --from-repo-id upload-repo --to-repo-id target-repo
  • upload food=pizza to upload-repo
  • iso repo copy --from-repo-id upload-repo --to-repo-id target-repo

End result: upload-repo contains only one "food" (good) but target-repo contains two, and has broken PULP_MANIFEST.

Would it be clearer to close this issue and file a new one for that?

#10 Updated by ipanova@redhat.com over 2 years ago

  • Status changed from NEW to CLOSED - NOTABUG

rmcgover i will close this issue. I was able to reproduce copy scenario so I opened a new issue https://pulp.plan.io/issues/4892

Please register to edit this issue

Also available in: Atom PDF