Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1925

closed

v1 sync errors

Added by jluza over 8 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
ipanova@redhat.com
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Docker:
2.0.2
Platform Release:
2.8.7
Target Release - Docker:
2.0.3
OS:
RHEL 6
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Easy Fix, Pulp 2
Sprint:
Sprint 3
Quarter:

Description

This happened during sync from crane

May 18 10:26:45 pulp-docker01 pulp: pulp.server.async.tasks:INFO: Task failed : [afb71832-8497-4a0c-8932-f9b3d82dd93a]
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152) Task pulp.server.managers.repo.sync.sync[afb71832-8497-4a0c-8932-f9b3d82dd93a] raised unexpected: NotUniqueError(u'Tried to save duplicate unique keys (E11000 duplicate key error index: pulp_28_new.units_docker_image.$image_id_1  dup key: { : "6c3a84d798dc449313787502060b6d5b4694d7527d64a7c99ba199e3b2df834e" })',)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152) Traceback (most recent call last):
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     R = retval = fun(*args, **kwargs)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 473, in __call__
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     return super(Task, self).__call__(*args, **kwargs)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 103, in __call__
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     return super(PulpTask, self).__call__(*args, **kwargs)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     return self.run(*args, **kwargs)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 760, in sync
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     sync_report = sync_repo(transfer_repo, conduit, call_config)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     return f(*args, **kwargs)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp_docker/plugins/importers/importer.py", line 84, in sync_repo
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     return self.sync_step.process_lifecycle()
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     super(PluginStep, self).process_lifecycle()
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     step.process()
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 232, in process
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     self._process_block(item=item)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 291, in _process_block
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     self.process_main(item=item)
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/pulp_docker/plugins/importers/v1_sync.py", line 150, in process_main
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     item.save()
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)   File "/usr/lib/python2.6/site-packages/mongoengine/document.py", line 390, in save
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152)     raise NotUniqueError(message % unicode(err))
May 18 10:26:45 pulp-docker01 pulp: celery.worker.job:ERROR: (16809-81152) NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error index: pulp_28_new.units_docker_image.$image_id_1  dup key: { : "6c3a84d798dc449313787502060b6d5b4694d7527d64a7c99ba199e3b2df834e" })

Related issues

Related to Packaging - Task #2083: Issues common to 2.9.1 and 2.8 streamCLOSED - NOTABUGsemyers

Actions
Actions
Copy link
 #1

Updated by rbarlow over 8 years ago

  • Platform Release changed from 2.8.2 to 2.8.4

My wild guess about this issue is that there were two concurrent syncs that both tried to create and save this same unit. One won, and this one lost. In this situation, the "loser" process should not consider the NotUniqueError to be a problem, but should catch it and simply add the unit to the repository. I think if we rework it to handle that Exception, everything will be peachy.

The uniqueness constraint can happen like that in a real-world scenario, so we need to address my wild guess even if it's not the true cause of jluza's traceback.

Actions
Copy link
 #2

Updated by ipanova@redhat.com over 8 years ago

  • Tags Easy Fix added
Actions
Copy link
 #3

Updated by mhrivnak over 8 years ago

  • Priority changed from Normal to High
  • Triaged changed from No to Yes
Actions
Copy link
 #4

Updated by semyers over 8 years ago

  • Platform Release changed from 2.8.4 to 2.8.5
Actions
Copy link
 #5

Updated by ipanova@redhat.com over 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
Actions
Copy link
 #6

Updated by ipanova@redhat.com over 8 years ago

  • Status changed from ASSIGNED to POST
Actions
Copy link
 #7

Updated by ipanova@redhat.com over 8 years ago

  • Sprint/Milestone set to 21
Actions
Copy link
 #8

Updated by semyers over 8 years ago

  • Platform Release changed from 2.8.5 to 2.8.6
Actions
Copy link
 #9

Updated by semyers over 8 years ago

  • Platform Release deleted (2.8.6)

Added by ipanova@redhat.com over 8 years ago

Revision 139659c7 | View on GitHub

Add check for duplicate unit key during v1 sync.

closes #1925 https://pulp.plan.io/issues/1925

Added by ipanova@redhat.com over 8 years ago

Revision 139659c7 | View on GitHub

Add check for duplicate unit key during v1 sync.

closes #1925 https://pulp.plan.io/issues/1925

Added by ipanova@redhat.com over 8 years ago

Revision 139659c7 | View on GitHub

Add check for duplicate unit key during v1 sync.

closes #1925 https://pulp.plan.io/issues/1925

Added by ipanova@redhat.com over 8 years ago

Revision 139659c7 | View on GitHub

Add check for duplicate unit key during v1 sync.

closes #1925 https://pulp.plan.io/issues/1925

Actions
Copy link
 #10

Updated by ipanova@redhat.com over 8 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #11

Updated by ipanova@redhat.com over 8 years ago

  • Platform Release set to 2.9.1
  • Target Release - Docker set to 2.0.3
Actions
Copy link
 #12

Updated by ipanova@redhat.com over 8 years ago

  • Platform Release changed from 2.9.1 to 2.8.6
Actions
Copy link
 #13

Updated by semyers over 8 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #14

Updated by semyers over 8 years ago

  • Platform Release changed from 2.8.6 to 2.8.7
Actions
Copy link
 #15

Updated by semyers over 8 years ago

  • Related to Task #2083: Issues common to 2.9.1 and 2.8 stream added
Actions
Copy link
 #16

Updated by pthomas@redhat.com over 8 years ago

  • Status changed from 5 to 6

verified
verified that concurrent v1 sync suceeds.
[root@ibm-x3550m3-10 ~]# rpm -qa pulp-server
pulp-server-2.8.7-0.3.beta.el7.noarch
[root@ibm-x3550m3-10 ~]#

[root@ibm-x3550m3-10 ~]# pulp-admin docker repo sync run --repo-id v1synctest
+----------------------------------------------------------------------+
                 Synchronizing Repository [v1synctest]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Retrieving v1 metadata
[|]
... completed

Copying units already in pulp
[-]
... completed

Downloading remote files
[==================================================] 100%
68 of 68 items
... completed

Saving v1 images and tags
[\]
... completed

Task Succeeded

Task Succeeded
Actions
Copy link
 #17

Updated by semyers over 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #19

Updated by bmbouter over 6 years ago

  • Sprint set to Sprint 3
Actions
Copy link
 #20

Updated by bmbouter over 6 years ago

  • Sprint/Milestone deleted (21)
Actions
Copy link
 #21

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF