Project

Profile

Help

Task #4028

Prevent duplicate files in repositories

Added by daviddavis about 1 year ago. Updated about 1 month ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Target Release - File:
Sprint:
Sprint 60

Description

Prevent the possibility of duplicate files (ie files with the same relative_path) in a single repo version. I think the places we need to check for dupes are after sync and associate.

Steps to reproduce:

http POST http://localhost:24817/pulp/api/v3/repositories/ name=foo
export REPO_HREF=$(http :24817/pulp/api/v3/repositories/ | jq -r '.results[] | select(.name == "foo") | ._href')
echo "blah1" > 1.iso
echo "blah2" > 2.iso
export ART1=$(http --form POST http://localhost:24817/pulp/api/v3/artifacts/ file@./1.iso | jq -r '._href')
export ART2=$(http --form POST http://localhost:24817/pulp/api/v3/artifacts/ file@./2.iso | jq -r '._href')
export CU1=$(http POST http://localhost:24817/pulp/api/v3/content/file/files/ relative_path=1.iso _artifact="$ART1" | jq -r '._href')
export CU2=$(http POST http://localhost:24817/pulp/api/v3/content/file/files/ relative_path=1.iso _artifact="$ART2" | jq -r '._href')
http POST ':24817'$REPO_HREF'versions/' add_content_units:="[\"$CU1\", \"$CU2\"]" 
sleep 5
http ":24817${REPO_HREF}versions/1/" 

You'll see the count is 2 which indicates that there are two 1.iso files in the repository.


Related issues

Related to File Support - Test #4738: File repositories shouldn't have duplicate files NEW Actions
Related to Pulp - Story #5008: As a user, I can enforce content specific uniqueness constaints in a repo version MODIFIED Actions
Blocked by Pulp - Story #3934: As a plugin writer, I can have a stage that removes duplicates MODIFIED Actions

Associated revisions

Revision 2551ede9 View on GitHub
Added by bmbouter about 1 month ago

Prevent duplicate `relative_path` files

When a RepositoryVersion has another `file.file` added with the same
`relative_path` the original one should be removed. This is shown
correctly on the RepositoryVersion `added`, `removed` fields.

This adds a test which uploads two distinct Artifacts, makes them into
File content, and then associates one, and then another with a Repo. The
second RepositoryVersion should have only 1 file.file content unit in
it.

Required PR: https://github.com/pulp/pulpcore/pull/331

https://pulp.plan.io/issues/4028
closes #4028

History

#1 Updated by daviddavis about 1 year ago

  • Blocked by Story #3934: As a plugin writer, I can have a stage that removes duplicates added

#2 Updated by daviddavis about 1 year ago

  • Tracker changed from Issue to Task
  • % Done set to 0

#3 Updated by daviddavis about 1 year ago

  • Subject changed from Remove duplicate files from repositories after sync to Prevent duplicate files in repositories
  • Description updated (diff)

#4 Updated by daviddavis 7 months ago

  • Description updated (diff)

#5 Updated by daviddavis 7 months ago

When publishing we sort by created when building the manifest file0 as #4731 points out. We should look at removing this code since it won't be needed anymore.

[0] https://github.com/pulp/pulp_file/blob/dd366601de3ae8741a7f0c2ee8f288f90f74d142/pulp_file/app/tasks/publishing.py#L73-L75

#6 Updated by daviddavis 7 months ago

  • Related to Test #4738: File repositories shouldn't have duplicate files added

#7 Updated by daviddavis 7 months ago

  • Blocked by Issue #4740: Pulpcore doesn't provide a way to guarantee uniqueness in repo versions added

#8 Updated by daviddavis 7 months ago

  • Blocked by deleted (Issue #4740: Pulpcore doesn't provide a way to guarantee uniqueness in repo versions)

#9 Updated by daviddavis 7 months ago

  • Blocked by Issue #3541: Core should not add/remove content to a repository or create a repository_version without plugin input added

#10 Updated by bmbouter 7 months ago

  • Tags deleted (Pulp 3)

#11 Updated by ttereshc 5 months ago

  • Related to Story #5008: As a user, I can enforce content specific uniqueness constaints in a repo version added

#12 Updated by rchan 5 months ago

Did not move from Sprint 54 to 55 because this issue needs some planning/grooming and not high enough priority to be done in next 3 week sprint.

#13 Updated by bmbouter about 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
  • Sprint set to Sprint 60

Taking as assigned so I can prove pulpcore's implementation as part of fixing 5008. Adding to sprint so everyone is aware.

#14 Updated by daviddavis about 2 months ago

  • Sprint/Milestone set to 0.1.0

#15 Updated by daviddavis about 2 months ago

  • Blocked by deleted (Issue #3541: Core should not add/remove content to a repository or create a repository_version without plugin input)

#16 Updated by bmbouter about 1 month ago

  • Status changed from ASSIGNED to POST

#17 Updated by bmbouter about 1 month ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Please register to edit this issue

Also available in: Atom PDF