Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #8967

closed

"duplicate key value violates unique constraint" when syncing two repositories with identical content in parallel

Added by wilful over 3 years ago. Updated almost 3 years ago.

Status:
CLOSED - DUPLICATE
Priority:
High
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 111
Quarter:

Description

Ticket moved to GitHub: "pulp/pulp_rpm/2278":https://github.com/pulp/pulp_rpm/issues/2278

The original issue is difficult to reproduce any longer, but there are similar issues which can be. see https://pulp.plan.io/issues/8967#note-16

========================

Hi for all!

Me need added for pulp server two repositories:

http://downloads.linux.hpe.com/SDR/repo/spp/redhat/7/x86_64/current/

http://downloads.linux.hpe.com/SDR/repo/mcp/CentOS/7/x86_64/current/

But i can't do it, becouse:

    "description": "duplicate key value violates unique constraint \"rpm_package_pkgId_key\"\nDETAIL:  Key (\"pkgId\")=(ebf96fb31b880280a25d07c596bde204df50d140) already exists.\
n"

How can I find out in which repository this package is?

Files

9029_rpm_simultaneous_sync.py (3.57 KB) 9029_rpm_simultaneous_sync.py dalley, 08/05/2021 02:54 PM

Related issues

Related to Pulp - Issue #8633: PulpImport with overlapping content can fail with unique-constraint violationCLOSED - CURRENTRELEASEggaineyActions
Related to Pulp - Issue #7828: syncing two repos with the same content at the same time results in an errorCLOSED - DUPLICATEggaineyActions
Related to RPM Support - Issue #9029: Problems when syncing identical repositories simultaneouslyCLOSED - CURRENTRELEASEdalleyActions
Actions
Copy link
 #1

Updated by wilful over 3 years ago

I thought the artifact would be reused for de-duplicate. But had a conflict =(

Actions
Copy link
 #2

Updated by dalley over 3 years ago

Hi wilful,

Could you provide a little more information? Which versions of Pulp are you running, and what steps did you take that lead you to that error?

Actions
Copy link
 #3

Updated by dalley over 3 years ago

  • Tracker changed from Task to Issue
  • Severity set to 2. Medium
  • Triaged set to No
Actions
Copy link
 #4

Updated by dalley over 3 years ago

  • Related to Issue #8633: PulpImport with overlapping content can fail with unique-constraint violation added
Actions
Copy link
 #5

Updated by dalley over 3 years ago

  • Subject changed from Small question about repositories managment to "duplicate key value violates unique constraint" when syncing two repositories with identical content in parallel
  • Priority changed from Normal to High
  • Tags Katello added
Actions
Copy link
 #6

Updated by dalley over 3 years ago

This can be reproduced if you sync the same url into two repos at the same time, or by syncing two different urls with the same repo content at the same time. It's a race condition in the sync pipeline.

@wilful, does this match your experience? Or did you experience this while syncing the repos one after another, independently and not in parallel?

Actions
Copy link
 #7

Updated by dalley over 3 years ago

I can't seem to reproduce it on newer versions though. @wilful what version are you on?

Actions
Copy link
 #8

Updated by dalley over 3 years ago

  • Sprint set to Sprint 100
Actions
Copy link
 #9

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions
Copy link
 #10

Updated by dalley over 3 years ago

  • Related to Issue #7828: syncing two repos with the same content at the same time results in an error added
Actions
Copy link
 #11

Updated by dalley over 3 years ago

The duplicate 7828 mentions

the Oracle Linux repositories "Oracle Linux 7 (x86_64) Latest" (http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64) and "Oracle Linux 7 (x86_64) Optional Latest" (http://yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64)

So we should try that out

Actions
Copy link
 #12

Updated by dalley over 3 years ago

  • Related to Issue #9029: Problems when syncing identical repositories simultaneously added
Actions
Copy link
 #13

Updated by ggainey over 3 years ago

I experimented with the mentioned OLE repos on current-master and was unable to reproduce. Used this script:

pulp rpm remote create --name ol7 --url http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64 --policy on_demand
pulp rpm remote create --name ol7opt --url http://yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64 --policy on_demand
for i in {1..4}                                                                              
do                                                                                           
    echo "RUN $i"                                                                            
    pulp rpm repository create --name ol7 --remote ol7 --autopublish                         
    pulp rpm repository create --name ol7opt --remote ol7opt --autopublish                   
    pulp -b rpm repository sync --name ol7; pulp -b rpm repository sync --name ol7opt        
    while true                                                                               
    do                                                                                       
        running=`pulp task list --state running | jq length`                                 
        echo -n "."     
        sleep 5                                                                     
        if [ ${running} -eq 0 ]                                                              
        then                                                                                 
            echo "DONE"                                                                      
            break                                                                            
        fi                                                                                   
    done                                                                                     
    failed=`pulp task list --state failed | jq length`                                       
    echo "FAILURES : ${failed}"                                                              
    echo "CLEANING UP..."                                                                    
    pulp rpm repository destroy --name ol7                                                   
    pulp rpm repository destroy --name ol7opt                                                
    pulp orphans delete                                                                      
done

(Note: 4 cycles took something over an hour on my system)

Actions
Copy link
 #14

Updated by ipanova@redhat.com over 3 years ago

  • Sprint changed from Sprint 101 to Sprint 102
Actions
Copy link
 #15

Updated by dkliban@redhat.com over 3 years ago

  • Status changed from NEW to CLOSED - WORKSFORME

Based on the previous comment, I am closing.

Actions
Copy link
 #16

Updated by dalley over 3 years ago

I was able to reproduce this with a different traceback 3 times in a row - script attached

Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]: pulp [2d30219697a640b2a927644cbdc7f892]: pulpcore.tasking.pulpcore_worker:INFO: Task 7d27d63b-43c2-4a0e-b9f7-c1c68bc17836 failed (insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_d5113f18_fk_core_repo"
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]: DETAIL:  Key (version_added_id)=(a5c43989-e695-4f07-9bdb-0f879b9cdd31) is not present in table "core_repositoryversion".
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]: )
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]: pulp [2d30219697a640b2a927644cbdc7f892]: pulpcore.tasking.pulpcore_worker:INFO:   File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 297, in _perform_task
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     result = func(*args, **kwargs)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 426, in synchronize
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     subrepo_version = dv.create()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     loop.run_until_complete(pipeline)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/lib64/python3.9/asyncio/base_events.py", line 642, in run_until_complete
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return future.result()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     await asyncio.gather(*futures)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     await self.run()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/content_stages.py", line 246, in run
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     self.new_version.add_content(Content.objects.filter(pk__in=to_add))
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py", line 763, in add_content
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     RepositoryContent.objects.bulk_create(repo_content)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py", line 523, in bulk_create
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     obj_without_pk._state.db = self.db
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/transaction.py", line 246, in __exit__
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     connection.commit()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return func(*args, **kwargs)
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     self._commit()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return self.connection.commit()
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     raise dj_exc_value.with_traceback(traceback) from exc_value
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
Aug 05 12:46:16 pulp3-source-fedora34.localhost.example.com pulpcore-worker[75545]:     return self.connection.commit()

We have the same problem with the RPM plugin.

Actions
Copy link
 #17

Updated by dkliban@redhat.com over 3 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #18

Updated by dalley over 3 years ago

  • Description updated (diff)
Actions
Copy link
 #19

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 102 to Sprint 103
Actions
Copy link
 #20

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 103 to Sprint 104
Actions
Copy link
 #21

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions
Copy link
 #22

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 105 to Sprint 106
Actions
Copy link
 #23

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 106 to Sprint 107
Actions
Copy link
 #24

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 107 to Sprint 108
Actions
Copy link
 #25

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 108 to Sprint 109
Actions
Copy link
 #26

Updated by bmbouter about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
Actions
Copy link
 #27

Updated by pulpbot about 3 years ago

  • Status changed from ASSIGNED to POST
Actions
Copy link
 #28

Updated by bmbouter about 3 years ago

  • Project changed from Pulp to RPM Support
  • Status changed from POST to NEW
  • Assignee deleted (bmbouter)

I closed my PR because I don't see a change in pulpcore that can be made to fix this. I've summarized my findings here: https://github.com/pulp/pulpcore/pull/1717#issuecomment-965695356

Per convo in matrix I am moving to pulp_rpm to get some input there. If there is something pulpcore can do to resolve please share the idea.

Actions
Copy link
 #29

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 109 to Sprint 110
Actions
Copy link
 #30

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 110 to Sprint 111
Actions
Copy link
 #31

Updated by pulpbot almost 3 years ago

  • Description updated (diff)
  • Status changed from NEW to CLOSED - DUPLICATE
Actions
Send by e-mail Copy link

Also available in: Atom PDF