Issue #902: race condition in condit's init_unit() - Pulp

Actions

Send by e-mail Copy link

Issue #902

closed

race condition in condit's init_unit()

Added by mhrivnak about 9 years ago. Updated about 5 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

dkliban@redhat.com

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

3. High

Version:

2.6.0

Platform Release:

2.6.2

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Quarter:

Description

When calling init_unit, there is a race condition here:

"/usr/lib/python2.7/site-packages/pulp/server/managers/content/query.py", line 290

When doing a sync of repos with similar content, it is possible that the conflict exhibited below will occur. The fix is likely to change to a try...except workflow that no-ops if the directory exists. Histro yfrom the downstream bug below.

I kicked off a sync of the following repos, all at once

RHEL7 Kickstart (7.0 & 7.1)
RHEL7 Common (7Server, 7.0, 7.1)
RHEL7 Server (7Server, 7.0, 7.1)

In syslog I get a few errors about File exists during the sync. But the final error, I'll upload as a follow-up post.

Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168) Exception from server requesting unit filename for relative path [OpenEXR-libs/1.7.1/7.el7/x86_64/3b8c0f8dbfc0b6ef0befc1d5f3ee82942fe825164b86c62e6b29632a9974fcff/OpenEXR-libs-1.7.1-7.el7.x86_64.rpm]
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168) Traceback (most recent call last):
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 477, in init_unit
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)     path = content_query_manager.request_content_unit_file_path(type_id, relative_path)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/query.py", line 290, in request_content_unit_file_path
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)     os.makedirs(unit_dir)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168)     mkdir(name, mode)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.plugins.conduits.mixins:ERROR: (4689-91168) OSError: [Errno 17] File exists: '/var/lib/pulp/content/rpm/OpenEXR-libs/1.7.1/7.el7/x86_64/3b8c0f8dbfc0b6ef0befc1d5f3ee82942fe825164b86c62e6b29632a9974fcff'
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168) <bound method ContainerListener.download_succeeded of <pulp_rpm.plugins.importers.yum.repomd.alternate.ContainerListener object at 0x417eb90>>
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168) Traceback (most recent call last):
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/server/content/sources/container.py", line 148, in _forward
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     method(request)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/alternate.py", line 126, in download_succeeded
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     self.content_listener.download_succeeded(report)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/listener.py", line 81, in download_succeeded
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     unit = self.sync_conduit.init_unit(model.TYPE, model.unit_key, model.metadata, model.relative_path)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 477, in init_unit
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     path = content_query_manager.request_content_unit_file_path(type_id, relative_path)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/query.py", line 290, in request_content_unit_file_path
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     os.makedirs(unit_dir)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168)     mkdir(name, mode)
Apr 17 13:33:52 satellite6.rdu.salab.redhat.com pulp[4689]: pulp.server.content.sources.container:ERROR: (4689-91168) ImporterConduitException: [Errno 17] File exists: '/var/lib/pulp/content/rpm/OpenEXR-libs/1.7.1/7.el7/x86_64/3b8c0f8dbfc0b6ef0befc1d5f3ee82942fe825164b86c62e6b29632a9974fcff'

Actions

Copy link

Updated by dkliban@redhat.com about 9 years ago

This issue can be reproduced on an EL6 machine using the following commands:

#stop all services
sudo service httpd stop
sudo service pulp_resource_manager stop
sudo service pulp_workers stop
sudo service pulp_celerybeat stop
#make sure there is no RPM content present
sudo rm -rf /var/lib/pulp/content/rpm/*
#start all services with exception of celerybeat
sudo service httpd start
sudo service pulp_resource_manager start
sudo service pulp_workers start
#create 2 repos that point to the same feed
pulp-admin rpm repo create --repo-id repo1 --relative-url repo1 --feed http://mirror.centos.org/centos/7/os/x86_64/
pulp-admin rpm repo create --repo-id repo2 --relative-url repo2 --feed http://mirror.centos.org/centos/7/os/x86_64/
#schedule sync for both repos (ctrl + c after each command)
pulp-admin rpm repo sync run --repo-id repo1
pulp-admin rpm repo sync run --repo-id repo2
#turn on celerybeat so workers can register and start performing syncs
sudo service pulp_celerybeat start

If you look in /var/log/messages you can see exceptions similar to this one:

Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640) Exception from server requesting unit filename for relative path [efibootmgr/0.8.0/5.el7/x86_64/811bac5f4d3b412b0944405d81158398b32a0b51b8a5e818e93ca5de6229fe9d/efibootmgr-0.8.0-5.el7.x86_64.rpm]
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640) Traceback (most recent call last):
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 464, in init_unit
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)     path = content_query_manager.request_content_unit_file_path(type_id, relative_path)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/query.py", line 280, in request_content_unit_file_path
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)     os.makedirs(unit_dir)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)   File "/usr/lib64/python2.6/os.py", line 157, in makedirs
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640)     mkdir(name, mode)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.plugins.conduits.mixins:ERROR: (1406-64640) OSError: [Errno 17] File exists: '/var/lib/pulp/content/rpm/efibootmgr/0.8.0/5.el7/x86_64/811bac5f4d3b412b0944405d81158398b32a0b51b8a5e818e93ca5de6229fe9d'
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640) <bound method ContainerListener.download_succeeded of <pulp_rpm.plugins.importers.yum.repomd.alternate.ContainerListener object at 0x2b8bbd0>>
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640) Traceback (most recent call last):
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/server/content/sources/container.py", line 137, in _forward
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     method(request)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/alternate.py", line 125, in download_succeeded
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     self.content_listener.download_succeeded(report)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", line 82, in download_succeeded
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     model.relative_path)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 464, in init_unit
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     path = content_query_manager.request_content_unit_file_path(type_id, relative_path)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/query.py", line 280, in request_content_unit_file_path
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     os.makedirs(unit_dir)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)   File "/usr/lib64/python2.6/os.py", line 157, in makedirs
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640)     mkdir(name, mode)
Apr 21 21:47:40 pulp-rhel-6 pulp: pulp.server.content.sources.container:ERROR: (1406-64640) ImporterConduitException: [Errno 17] File exists: '/var/lib/pulp/content/rpm/efibootmgr/0.8.0/5.el7/x86_64/811bac5f4d3b412b0944405d81158398b32a0b51b8a5e818e93ca5de6229fe9d'

Actions

Copy link