Project

Profile

Help

Issue #7298

Sync into the imported repository fails for comps content

Added by ttereshc over 1 year ago. Updated over 1 year ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 80
Quarter:

Description

I used Centos8 BaseOS kicstart repo, hopefully any smaller repo with comps.xml will work.

To reproduce:

  1. Create a repo, sync it, and export it.
  2. Import a repo (I imported it to a different pulp instance, probably plcean and importing into the same one will work the same way).
  3. Sync from the same remote URL.
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: pulp: rq.worker:ERROR: ValueError: Cannot create repository version. More than one rpm.packagecategory content with the duplicate values for id., More than one rpm.packagegroup content with the duplicate values for id., More than one rpm.packageenvironment content with the duplicate values for id.
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: Traceback (most recent call last):
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py", line 934, in perform_job
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     rv = job.perform()
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py", line 686, in perform
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     self._result = self._execute()
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py", line 692, in _execute
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     return self.func(*self.args, **self.kwargs)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 208, in synchronize
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     dv.create()
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     loop.run_until_complete(pipeline)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py", line 790, in __exit__
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     repository.finalize_new_version(self)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/models/repository.py", line 173, in finalize_new_version
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     validate_repo_version(new_version)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/repo_version_utils.py", line 140, in validate_repo_version
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     validate_duplicate_content(version)
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/repo_version_utils.py", line 110, in validate_duplicate_content
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]:     _("Cannot create repository version. {msg}").format(msg=", ".join(error_messages))
Aug 10 16:29:41 pulp3-source-fedora31.rhgoose.example.com rq[55125]: ValueError: Cannot create repository version. More than one rpm.packagecategory content with the duplicate values for id., More than one rpm.packagegroup content with the duplicate values for id., More than one rpm.packageenvironment content with the duplicate values for id.

History

#1 Updated by ttereshc over 1 year ago

  • Description updated (diff)

#2 Updated by ttereshc over 1 year ago

  • Triaged changed from No to Yes

#3 Updated by ipanova@redhat.com over 1 year ago

check if repo version creation handles correctly duplicates for comps.xml

#4 Updated by ggainey over 1 year ago

  • Sprint set to Sprint 80

#5 Updated by ggainey over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey

I haven't been able to recreate this against pulpcore/pulp_rpm master; have tried with fixtures and with CENTOS8-BASE kickstart repo.

The process I'm going thru is essentially the following:

CENTOS8_BASE_URL="http://centos.mirror.rafal.ca/8.2.2004/BaseOS/x86_64/kickstart/"           
CENTOS8_NAME="centos8"                                                                       
CENTOS8_BASE_NAME="centos8-base"                                                             
                                                                                             
EXPORTER_NAME="centos8"                                                                      
EXPORTER_URL="/pulp/api/v3/exporters/core/pulp/"                                             
                                                                                             
CENTOS_NAME='new-centos8'                                                                    
IMPORT_NAME="test-centos8"                                                                   
IMPORTER_URL="/pulp/api/v3/importers/core/pulp/"                                             
MAPPING='{"centos8-base": "new-centos8-base"}'                                               
                                                                                             
# create repo/remote/sync                                                                    
CENTOS8_BASE_HREF=$(http POST :/pulp/api/v3/repositories/rpm/rpm/ name=${CENTOS8_BASE_NAME} | jq -r '.pulp_href')
http POST :/pulp/api/v3/remotes/rpm/rpm/ name=$CENTOS8_BASE_NAME url=$CENTOS8_BASE_URL  policy='immediate'
BASE_REMOTE_HREF=$(http :/pulp/api/v3/remotes/rpm/rpm/ | jq -r ".results[] | select(.name == \"${CENTOS8_BASE_NAME}\") | .pulp_href")
http POST :$CENTOS8_BASE_HREF'sync/' remote=$BASE_REMOTE_HREF                                
                                                                                             
# create exporter/export                                                                     
EXPORTER_HREF=$(http POST :$EXPORTER_URL name="${EXPORTER_NAME}"-exporter repositories:=[\"${CENTOS8_BASE_HREF}\"] path=/tmp/exports/) "
http POST :${EXPORTER_HREF}exports/                                                          
                                                                                             
# reset pulp                                                                                 
pclean                                                                                       
prestart                                                                                     
                                                                                             
# create destination repo/importer                                                           
CENTOS8_BASE_HREF=$(http POST :/pulp/api/v3/repositories/rpm/rpm/ name="${CENTOS_NAME}-base" | jq -r '.pulp_href')
IMPORT_HREF=$(http POST :$IMPORTER_URL name="${IMPORT_NAME}"-importer repo_mapping:="${MAPPING}")
                                                                                             
# http POST :${IMPORT_HREF}imports/ toc=/tmp/exports/<EXPORT-TOC-HERE> 

# sync once import has finished
http POST :/pulp/api/v3/remotes/rpm/rpm/ name=$CENTOS8_BASE_NAME url=$CENTOS8_BASE_URL  policy='immediate'
BASE_REMOTE_HREF=$(http :/pulp/api/v3/remotes/rpm/rpm/ | jq -r ".results[] | select(.name == \"${CENTOS8_BASE_NAME}\") | .pulp_href")
http POST :$CENTOS8_BASE_HREF'sync/' remote=$BASE_REMOTE_HREF                                

ttereshc : do you have the exact repo-URL you used when this failed?

The UpdateCollection changes happened on 12-AUG, but don't appear to have any way to have affected PackageCategoryResource. The kickstart/distr-tree work daviddavis did for import-export happened on 27-JUL, and should therefore be in place already when your test failed - unless the imported-into pulp-instance was older than that?

#6 Updated by ggainey over 1 year ago

Notes from IRC conversation:

<ttereshc> ggainey, interesting, here is what I did differently, but I don't think it should affect anything :/  I used the link form our functional tests, I didn't use mapping (I think), I used a separate instance of pulp, I probably synced with on_demand policy (after import).
<ttereshc> you can close as works for me and we'll see. I'm not sure if I'll have time to test it this week
<ggainey> ok, cool - I'll add that to the issue, and retry - maybe this is an on-demand thing. If it still works, maybe I'll close; will mull it over and see if maybe I'm missing a key step somewhere
<ggainey> I'll make sure I'm using the same repo first, maybe it's a Data Thing

#7 Updated by ggainey over 1 year ago

  • Status changed from ASSIGNED to CLOSED - WORKSFORME

I switched to using http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/kickstart/ for the remote, and also switched the resync to on_demand, and still can't make it fail. I'm going to close this as "works for me" - if you get an opportunity to retest and can make it fail again, by all means add detail here and reopen!

Also available in: Atom PDF