Project

Profile

Help

Issue #902

race condition in condit's init_unit()

Added by mhrivnak over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.6.0
Platform Release:
2.6.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

When calling init_unit, there is a race condition here:

"/usr/lib/python2.7/site-packages/pulp/server/managers/content/query.py", line 290

When doing a sync of repos with similar content, it is possible that the conflict exhibited below will occur. The fix is likely to change to a try...except workflow that no-ops if the directory exists. Histro yfrom the downstream bug below.

I kicked off a sync of the following repos, all at once

RHEL7 Kickstart (7.0 & 7.1)
RHEL7 Common (7Server, 7.0, 7.1)
RHEL7 Server (7Server, 7.0, 7.1)

In syslog I get a few errors about File exists during the sync. But the final error, I'll upload as a follow-up post.

Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168) Exception from server requesting unit filename for relative path [OpenEXR-libs/1.7.1/7.el7/x86_64/3b8c0f8dbfc0b6ef0befc1d5f3ee82942fe825164b86c62e6b29632a9974fcff/OpenEXR-libs-1.7.1-7.el7.x86_64.rpm]
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168) Traceback (most recent call last):
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 477, in init_unit
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)     path = content_query_manager.request_content_unit_file_path(type_id, relative_path)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/query.py", line 290, in request_content_unit_file_path
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)     os.makedirs(unit_dir)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)     mkdir(name, mode)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168) OSError: [Errno 17] File exists: '/var/lib/pulp/content/rpm/OpenEXR-libs/1.7.1/7.el7/x86_64/3b8c0f8dbfc0b6ef0befc1d5f3ee82942fe825164b86c62e6b29632a9974fcff'
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168) <bound method ContainerListener.download_succeeded of <pulp_rpm.plugins.importers.yum.repomd.alternate.ContainerListener object at 0x417eb90>>
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168) Traceback (most recent call last):
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/server/content/sources/container.py", line 148, in _forward
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     method(request)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/alternate.py", line 126, in download_succeeded
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     self.content_listener.download_succeeded(report)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/listener.py", line 81, in download_succeeded
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     unit = self.sync_conduit.init_unit(model.TYPE, model.unit_key, model.metadata, model.relative_path)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 477, in init_unit
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     path = content_query_manager.request_content_unit_file_path(type_id, relative_path)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/query.py", line 290, in request_content_unit_file_path
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     os.makedirs(unit_dir)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     mkdir(name, mode)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168) ImporterConduitException: [Errno 17] File exists: '/var/lib/pulp/content/rpm/OpenEXR-libs/1.7.1/7.el7/x86_64/3b8c0f8dbfc0b6ef0befc1d5f3ee82942fe825164b86c62e6b29632a9974fcff'

Associated revisions

Revision ac370014 View on GitHub
Added by dkliban@redhat.com over 6 years ago

Added OSError 17 exception handling during race condition

Fixes #902

Revision ac370014 View on GitHub
Added by dkliban@redhat.com over 6 years ago

Added OSError 17 exception handling during race condition

Fixes #902

History

#1 Updated by dkliban@redhat.com over 6 years ago

This issue can be reproduced on an EL6 machine using the following commands:

#stop all services
sudo service httpd stop
sudo service pulp_resource_manager stop
sudo service pulp_workers stop
sudo service pulp_celerybeat stop
#make sure there is no RPM content present
sudo rm -rf /var/lib/pulp/content/rpm/*
#start all services with exception of celerybeat
sudo service httpd start
sudo service pulp_resource_manager start
sudo service pulp_workers start
#create 2 repos that point to the same feed
pulp-admin rpm repo create --repo-id repo1 --relative-url repo1 --feed http://mirror.centos.org/centos/7/os/x86_64/
pulp-admin rpm repo create --repo-id repo2 --relative-url repo2 --feed http://mirror.centos.org/centos/7/os/x86_64/
#schedule sync for both repos (ctrl + c after each command)
pulp-admin rpm repo sync run --repo-id repo1
pulp-admin rpm repo sync run --repo-id repo2
#turn on celerybeat so workers can register and start performing syncs
sudo service pulp_celerybeat start

If you look in /var/log/messages you can see exceptions similar to this one:

Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640) Exception from server requesting unit filename for relative path [efibootmgr/0.8.0/5.el7/x86_64/811bac5f4d3b412b0944405d81158398b32a0b51b8a5e818e93ca5de6229fe9d/efibootmgr-0.8.0-5.el7.x86_64.rpm]
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640) Traceback (most recent call last):
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 464, in init_unit
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)     path = content_query_manager.request_content_unit_file_path(type_id, relative_path)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/query.py", line 280, in request_content_unit_file_path
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)     os.makedirs(unit_dir)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)   File "/usr/lib64/python2.6/os.py", line 157, in makedirs
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)     mkdir(name, mode)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640) OSError: [Errno 17] File exists: '/var/lib/pulp/content/rpm/efibootmgr/0.8.0/5.el7/x86_64/811bac5f4d3b412b0944405d81158398b32a0b51b8a5e818e93ca5de6229fe9d'
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640) <bound method ContainerListener.download_succeeded of <pulp_rpm.plugins.importers.yum.repomd.alternate.ContainerListener object at 0x2b8bbd0>>
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640) Traceback (most recent call last):
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/server/content/sources/container.py", line 137, in _forward
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     method(request)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/alternate.py", line 125, in download_succeeded
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     self.content_listener.download_succeeded(report)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", line 82, in download_succeeded
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     model.relative_path)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 464, in init_unit
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     path = content_query_manager.request_content_unit_file_path(type_id, relative_path)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/query.py", line 280, in request_content_unit_file_path
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     os.makedirs(unit_dir)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib64/python2.6/os.py", line 157, in makedirs
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     mkdir(name, mode)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640) ImporterConduitException: [Errno 17] File exists: '/var/lib/pulp/content/rpm/efibootmgr/0.8.0/5.el7/x86_64/811bac5f4d3b412b0944405d81158398b32a0b51b8a5e818e93ca5de6229fe9d'

#2 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from NEW to POST

#3 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#4 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from MODIFIED to 5

#6 Updated by pthomas@redhat.com over 6 years ago

  • Status changed from 5 to 6

verified
[root@mgmt12 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@mgmt12 ~]#

followed the steps from #1 and no race condition occurred in the log

#7 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#10 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF