Project

Profile

Help

Task #4028

closed

Prevent duplicate files in repositories

Added by daviddavis about 4 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 60
Quarter:

Description

Prevent the possibility of duplicate files (ie files with the same relative_path) in a single repo version. I think the places we need to check for dupes are after sync and associate.

Steps to reproduce:

http POST http://localhost:24817/pulp/api/v3/repositories/ name=foo
export REPO_HREF=$(http :24817/pulp/api/v3/repositories/ | jq -r '.results[] | select(.name == "foo") | ._href')
echo "blah1" > 1.iso
echo "blah2" > 2.iso
export ART1=$(http --form POST http://localhost:24817/pulp/api/v3/artifacts/ file@./1.iso | jq -r '._href')
export ART2=$(http --form POST http://localhost:24817/pulp/api/v3/artifacts/ file@./2.iso | jq -r '._href')
export CU1=$(http POST http://localhost:24817/pulp/api/v3/content/file/files/ relative_path=1.iso _artifact="$ART1" | jq -r '._href')
export CU2=$(http POST http://localhost:24817/pulp/api/v3/content/file/files/ relative_path=1.iso _artifact="$ART2" | jq -r '._href')
http POST ':24817'$REPO_HREF'versions/' add_content_units:="[\"$CU1\", \"$CU2\"]"
sleep 5
http ":24817${REPO_HREF}versions/1/"

You'll see the count is 2 which indicates that there are two 1.iso files in the repository.


Related issues

Related to File Support - Test #4738: File repositories shouldn't have duplicate filesCLOSED - DUPLICATEActions
Related to Pulp - Story #5008: As a user, I can enforce content specific uniqueness constaints in a repo versionCLOSED - CURRENTRELEASEbmbouter

Actions
Blocked by Pulp - Story #3934: As a plugin writer, I can have a stage that removes duplicatesCLOSED - CURRENTRELEASEamacdona@redhat.com

Actions
Actions #1

Updated by daviddavis about 4 years ago

  • Blocked by Story #3934: As a plugin writer, I can have a stage that removes duplicates added
Actions #2

Updated by daviddavis about 4 years ago

  • Tracker changed from Issue to Task
  • % Done set to 0
Actions #3

Updated by daviddavis about 4 years ago

  • Subject changed from Remove duplicate files from repositories after sync to Prevent duplicate files in repositories
  • Description updated (diff)
Actions #4

Updated by daviddavis over 3 years ago

  • Description updated (diff)
Actions #5

Updated by daviddavis over 3 years ago

When publishing we sort by created when building the manifest file[0] as #4731 points out. We should look at removing this code since it won't be needed anymore.

[0] https://github.com/pulp/pulp_file/blob/dd366601de3ae8741a7f0c2ee8f288f90f74d142/pulp_file/app/tasks/publishing.py#L73-L75

Actions #6

Updated by daviddavis over 3 years ago

  • Related to Test #4738: File repositories shouldn't have duplicate files added
Actions #7

Updated by daviddavis over 3 years ago

  • Blocked by Issue #4740: Pulpcore doesn't provide a way to guarantee uniqueness in repo versions added
Actions #8

Updated by daviddavis over 3 years ago

  • Blocked by deleted (Issue #4740: Pulpcore doesn't provide a way to guarantee uniqueness in repo versions)
Actions #9

Updated by daviddavis over 3 years ago

  • Blocked by Issue #3541: Core should not add/remove content to a repository or create a repository_version without plugin input added
Actions #10

Updated by bmbouter over 3 years ago

  • Tags deleted (Pulp 3)
Actions #11

Updated by ttereshc over 3 years ago

  • Related to Story #5008: As a user, I can enforce content specific uniqueness constaints in a repo version added
Actions #12

Updated by rchan over 3 years ago

Did not move from Sprint 54 to 55 because this issue needs some planning/grooming and not high enough priority to be done in next 3 week sprint.

Actions #13

Updated by bmbouter about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
  • Sprint set to Sprint 60

Taking as assigned so I can prove pulpcore's implementation as part of fixing 5008. Adding to sprint so everyone is aware.

Actions #14

Updated by daviddavis about 3 years ago

  • Sprint/Milestone set to 0.1.0
Actions #15

Updated by daviddavis about 3 years ago

  • Blocked by deleted (Issue #3541: Core should not add/remove content to a repository or create a repository_version without plugin input)
Actions #16

Updated by bmbouter about 3 years ago

  • Status changed from ASSIGNED to POST

Added by bmbouter about 3 years ago

Revision 2551ede9

Prevent duplicate relative_path files

When a RepositoryVersion has another file.file added with the same relative_path the original one should be removed. This is shown correctly on the RepositoryVersion added, removed fields.

This adds a test which uploads two distinct Artifacts, makes them into File content, and then associates one, and then another with a Repo. The second RepositoryVersion should have only 1 file.file content unit in it.

Required PR: https://github.com/pulp/pulpcore/pull/331

https://pulp.plan.io/issues/4028 closes #4028

Actions #17

Updated by bmbouter about 3 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #18

Updated by bmbouter almost 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF