Project

Profile

Help

Issue #8967

"duplicate key value violates unique constraint" when syncing two repositories with identical content in parallel

Added by wilful 3 months ago. Updated 4 days ago.

Status:
NEW
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 106
Quarter:

Description

The original issue is difficult to reproduce any longer, but there are similar issues which can be. see https://pulp.plan.io/issues/8967#note-16

========================

Hi for all!

Me need added for pulp server two repositories:

http://downloads.linux.hpe.com/SDR/repo/spp/redhat/7/x86_64/current/

http://downloads.linux.hpe.com/SDR/repo/mcp/CentOS/7/x86_64/current/

But i can't do it, becouse:

    "description": "duplicate key value violates unique constraint \"rpm_package_pkgId_key\"\nDETAIL:  Key (\"pkgId\")=(ebf96fb31b880280a25d07c596bde204df50d140) already exists.\
n"

How can I find out in which repository this package is?

9029_rpm_simultaneous_sync.py (3.57 KB) 9029_rpm_simultaneous_sync.py dalley, 08/05/2021 02:54 PM

Related issues

Related to Pulp - Issue #8633: PulpImport with overlapping content can fail with unique-constraint violationCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Pulp - Issue #7828: syncing two repos with the same content at the same time results in an errorCLOSED - DUPLICATE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Issue #9029: Problems when syncing identical repositories simultaneouslyCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by wilful 3 months ago

I thought the artifact would be reused for de-duplicate. But had a conflict =(

#2 Updated by dalley 3 months ago

Hi wilful,

Could you provide a little more information? Which versions of Pulp are you running, and what steps did you take that lead you to that error?

#3 Updated by dalley 3 months ago

  • Tracker changed from Task to Issue
  • Severity set to 2. Medium
  • Triaged set to No

#4 Updated by dalley 3 months ago

  • Related to Issue #8633: PulpImport with overlapping content can fail with unique-constraint violation added

#5 Updated by dalley 3 months ago

  • Subject changed from Small question about repositories managment to "duplicate key value violates unique constraint" when syncing two repositories with identical content in parallel
  • Priority changed from Normal to High
  • Tags Katello added

#6 Updated by dalley 3 months ago

This can be reproduced if you sync the same url into two repos at the same time, or by syncing two different urls with the same repo content at the same time. It's a race condition in the sync pipeline.

@wilful, does this match your experience? Or did you experience this while syncing the repos one after another, independently and not in parallel?

#7 Updated by dalley 3 months ago

I can't seem to reproduce it on newer versions though. @wilful what version are you on?

#8 Updated by dalley 3 months ago

  • Sprint set to Sprint 100

#9 Updated by rchan 2 months ago

  • Sprint changed from Sprint 100 to Sprint 101

#10 Updated by dalley 2 months ago

  • Related to Issue #7828: syncing two repos with the same content at the same time results in an error added

#11 Updated by dalley 2 months ago

The duplicate 7828 mentions

the Oracle Linux repositories "Oracle Linux 7 (x86_64) Latest" (http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64) and "Oracle Linux 7 (x86_64) Optional Latest" (http://yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64)

So we should try that out

#12 Updated by dalley 2 months ago

  • Related to Issue #9029: Problems when syncing identical repositories simultaneously added

#13 Updated by ggainey 2 months ago

I experimented with the mentioned OLE repos on current-master and was unable to reproduce. Used this script:

pulp rpm remote create --name ol7 --url http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64 --policy on_demand
pulp rpm remote create --name ol7opt --url http://yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64 --policy on_demand
for i in {1..4}                                                                              
do                                                                                           
    echo "RUN $i"                                                                            
    pulp rpm repository create --name ol7 --remote ol7 --autopublish                         
    pulp rpm repository create --name ol7opt --remote ol7opt --autopublish                   
    pulp -b rpm repository sync --name ol7; pulp -b rpm repository sync --name ol7opt        
    while true                                                                               
    do                                                                                       
        running=`pulp task list --state running | jq length`                                 
        echo -n "."     
        sleep 5                                                                     
        if [ ${running} -eq 0 ]                                                              
        then                                                                                 
            echo "DONE"                                                                      
            break                                                                            
        fi                                                                                   
    done                                                                                     
    failed=`pulp task list --state failed | jq length`                                       
    echo "FAILURES : ${failed}"                                                              
    echo "CLEANING UP..."                                                                    
    pulp rpm repository destroy --name ol7                                                   
    pulp rpm repository destroy --name ol7opt                                                
    pulp orphans delete                                                                      
done

(Note: 4 cycles took something over an hour on my system)

#14 Updated by ipanova@redhat.com about 2 months ago

  • Sprint changed from Sprint 101 to Sprint 102

#15 Updated by dkliban@redhat.com about 2 months ago

  • Status changed from NEW to CLOSED - WORKSFORME

Based on the previous comment, I am closing.

#16 Updated by dalley about 2 months ago

I was able to reproduce this with a different traceback 3 times in a row - script attached

Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]: pulp [2d30219697a640b2a927644cbdc7f892]: pulpcore.tasking.pulpcore_worker:INFO: Task 7d27d63b-43c2-4a0e-b9f7-c1c68bc17836 failed (insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]: DETAIL:  Key (version_added_id)=(a5c43989-e695-4f07-9bdb-0f879b9cdd31) is not present in table "core_repositoryversion".
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]: )
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]: pulp [2d30219697a640b2a927644cbdc7f892]: pulpcore.tasking.pulpcore_worker:INFO:   File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 297, in _perform_task
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     result = func(*args, **kwargs)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 426, in synchronize
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     subrepo_version = dv.create()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     loop.run_until_complete(pipeline)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/lib64/python3.9/asyncio/base_events.py", line 642, in run_until_complete
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return future.result()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     await asyncio.gather(*futures)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     await self.run()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/content_stages.py", line 246, in run
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     self.new_version.add_content(Content.objects.filter(pk__in=to_add))
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py", line 763, in add_content
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     RepositoryContent.objects.bulk_create(repo_content)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 523, in bulk_create
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     obj_without_pk._state.db = self.db
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/transaction.py", line 246, in __exit__
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     connection.commit()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return func(*args, **kwargs)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     self._commit()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return self.connection.commit()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return self.connection.commit()

We have the same problem with the RPM plugin.

#17 Updated by dkliban@redhat.com about 2 months ago

  • Triaged changed from No to Yes

#18 Updated by dalley about 2 months ago

  • Description updated (diff)

#19 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 102 to Sprint 103

#20 Updated by rchan about 1 month ago

  • Sprint changed from Sprint 103 to Sprint 104

#21 Updated by rchan 18 days ago

  • Sprint changed from Sprint 104 to Sprint 105

#22 Updated by rchan 4 days ago

  • Sprint changed from Sprint 105 to Sprint 106

Please register to edit this issue

Also available in: Atom PDF