Issue #995: Syncing yum repo raises error - Pulp

Actions

Send by e-mail Copy link

Issue #995

closed

Syncing yum repo raises error

Added by paji@redhat.com over 9 years ago. Updated over 5 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

ipanova@redhat.com

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

1. Low

Version:

Platform Release:

2.6.3

OS:

Triaged:

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Quarter:

Description

Unpack the attached tar gz file in /tmp
you should see a /tmp/zoo5

try the following commands

pulp-admin rpm repo create --repo-id=synergism --feed=file:///var/www/repositories/zoo5
pulp-admin rpm repo sync run --repo-id=synergism

cat /var/log/messages

You see the following the stack trace. Note in particular this error

May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184) [Errno 2] No such file or directory: u'///var/www/repositories/zoo5/PULP_DISTRIBUTION.xml'

This causes katello to think there is a 500 in pulp. Causing integration tests to fail.

May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[c0a60ed2-b5a4-41ef-a888-d8d6a3684b07]
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[cd811a9c-be90-4ff3-aa04-d1b6fd0b35a6]
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[c0a60ed2-b5a4-41ef-a888-d8d6a3684b07] succeeded in 0.036162881006s: None
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[ac1d87a3-a26a-427d-8fde-119cf670e9dd]
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from file:///var/www/repositories/zoo5/.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: file:///var/www/repositories/zoo5/
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184) [Errno 2] No such file or directory: u'///var/www/repositories/zoo5/PULP_DISTRIBUTION.xml'
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184) Traceback (most recent call last):
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/local.py", line 144, in _copy
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184)     src_handle = open(src_path, 'rb')
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184) IOError: [Errno 2] No such file or directory: u'///var/www/repositories/zoo5/PULP_DISTRIBUTION.xml'
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Sync complete.
May 20 17:32:42 katello-wisdom pulp: pulp.server.event.http:INFO: {'call_report': <TaskStatus: TaskStatus object>, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': u'yum_importer', 'exception': None, 'repo_id': u'partello', 'traceback': None, 'started': '2015-05-20T21:32:42Z', '_ns': u'repo_sync_results', 'completed': '2015-05-20T21:32:42Z', 'importer_type_id': u'yum_importer', 'error_message': None, 'summary': {'content': {'state': 'FINISHED'}, 'comps': {'state': 'FINISHED'}, 'distribution': {'state': 'FINISHED'}, 'errata': {'state': 'FINISHED'}, 'metadata': {'state': 'FINISHED'}}, 'added_count': 3, 'result': 'success', 'updated_count': 11, 'details': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'FINISHED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error_details': []}, 'comps': {'state': 'FINISHED'}, 'distribution': {'items_total': 3, 'state': 'FINISHED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'FINISHED'}, 'metadata': {'state': 'FINISHED'}}, 'id': '555cfd7ad01ec00b69cae7f4', 'removed_count': 0}}
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184) Exception from notifier of type [http]
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184) Traceback (most recent call last):
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/event/fire.py", line 76, in _do_fire
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     f(l['notifier_config'], event)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib/python2.7/site-packages/pulp/server/event/http.py", line 33, in handle_event
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     body = json.dumps(data, default=json_util.default)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     sort_keys=sort_keys, **kw).encode(obj)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     chunks = self.iterencode(o, _one_shot=True)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     return _iterencode(o, 0)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib64/python2.7/site-packages/bson/json_util.py", line 220, in default
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     raise TypeError("%%r is not JSON serializable" %% obj)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184) TypeError: <TaskStatus: TaskStatus object> is not JSON serializable
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[cd811a9c-be90-4ff3-aa04-d1b6fd0b35a6] succeeded in 0.201764149999s: <pulp.server.async.tasks.TaskResult object at 0x4dd95d0>
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[ac1d87a3-a26a-427d-8fde-119cf670e9dd] succeeded in 0.00725326400425s: None
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[ad5774d9-3534-42ef-9748-557e21b75ec5]
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[09707c71-c404-433b-86c6-0e0f28a3557b]
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[ad5774d9-3534-42ef-9748-557e21b75ec5] succeeded in 0.0815736520017s: None
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9b4fd858-bf97-4de4-ad7d-a8b9be66adc6]
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[09707c71-c404-433b-86c6-0e0f28a3557b] succeeded in 0.157991663989s: {'exception': None, 'repo_id': u'partello', 'traceback': None, 'started': '2015-05-20T21:32:42Z', '_ns':...
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9b4fd858-bf97-4de4-ad7d-a8b9be66adc6] succeeded in 0.00436354399426s: None

Files

repo.tar.gz (16.7 KB) repo.tar.gz

paji@redhat.com, 05/21/2015 12:30 AM

Actions

Copy link

Updated by paji@redhat.com over 9 years ago

Correction in the bz

pulp-admin rpm repo create --repo-id=synergism --feed=file:///tmp/zoo5

Actions

Copy link

Updated by mhrivnak over 9 years ago

Description updated (diff)

Actions

Copy link

Updated by mhrivnak over 9 years ago

Priority changed from Normal to High
Version deleted (~~Master~~)

This is seen in 2.7.0 alpha

This may be because apache gets a unique view of /tmp/ and cannot see the same files you can. Can you confirm this happens if you put the repo somewhere outside /tmp/ that we know apache can read?

Actions

Copy link

Updated by ipanova@redhat.com over 9 years ago

Status changed from NEW to ASSIGNED

Actions

Copy link

Updated by ipanova@redhat.com over 9 years ago

Assignee set to ipanova@redhat.com

Actions

Copy link

Updated by ipanova@redhat.com over 9 years ago

I do not think it is the apache issue, as if I use some other directory I get the sync finished without any errors, but still in the logs I can see the [Errno 2] No such file or directory

$ pulp-admin -u admin -p admin rpm  repo create --repo-id test1 --feed file:///home/user/pulp_development/zoo5

Successfully created repository [test1]

$ pulp-admin -u admin -p admin rpm repo sync run --repo-id test1
+----------------------------------------------------------------------+
                    Synchronizing Repository [test1]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 3/3 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
8 of 8 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
2 of 2 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

sudo less /var/log/messages|tail -20

May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from file:///home/user/pulp_development/zoo5/.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 2} for base URL: file:///home/user/pulp_development/zoo5/
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848) [Errno 2] No such file or directory: u'///home/user/pulp_development/zoo5/PULP_DISTRIBUTION.xml'
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848) Traceback (most recent call last):
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/local.py", line 144, in _copy
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848)     src_handle = open(src_path, 'rb')
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848) IOError: [Errno 2] No such file or directory: u'///home/user/pulp_development/zoo5/PULP_DISTRIBUTION.xml'
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Sync complete.
May 27 13:04:59 ina pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[b5530f9f-d1b4-4d70-a520-09e1efcd9793]
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[24a3bb05-4157-4cec-9c83-f79b3ed7ae95] succeeded in 0.1826176662s: <pulp.server.async.tasks.TaskResult object at 0x393ba10>
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[b5530f9f-d1b4-4d70-a520-09e1efcd9793] succeeded in 0.0476008197293s: None
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[b8e5ddbf-2e51-4cc0-b2e8-fd5e86b84a45] succeeded in 0.0452612806112s: None
May 27 13:04:59 ina pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[5cee603b-9739-4b45-9477-827b32c8b9d4]
May 27 13:04:59 ina pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[bd302d54-7f2d-4421-9591-4b04db288727]
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[5cee603b-9739-4b45-9477-827b32c8b9d4] succeeded in 0.231409627944s: {'exception': None, 'repo_id': u'test1', 'traceback': None, 'started': '2015-05-27T11:04:59Z', '_ns':...
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[bd302d54-7f2d-4421-9591-4b04db288727] succeeded in 0.0181207852438s: None

So it could be something related to the how the distribution is processed.

Actions

Copy link

Updated by mhrivnak over 9 years ago

Triaged changed from No to Yes

Actions

Copy link

Updated by ipanova@redhat.com over 9 years ago

Project changed from RPM Support to Nectar
Status changed from ASSIGNED to MODIFIED

this will be fixed in pulp 2.6.3 which will contain new build of nectar 1.3.2

https://github.com/pulp/nectar/commit/fc6cb788429591d1595e5947f64f15bf16b4b888

Actions

Copy link

Updated by ipanova@redhat.com over 9 years ago

Project changed from Nectar to Pulp
Platform Release set to 2.6.3

Actions

Copy link

#10

Updated by dkliban@redhat.com over 9 years ago

Status changed from MODIFIED to 5

Actions

Copy link

#12

Updated by igulina@redhat.com over 9 years ago

Blocked by Issue #1128: permission denied with selinux enforcing when sync repo with a local feed added

Actions

Copy link

#13

Updated by igulina@redhat.com over 9 years ago

Blocks Issue #1127: Incorrect error message when sync fails with Permission Denied added

Actions

Copy link

#14

Updated by igulina@redhat.com over 9 years ago

Blocks deleted (Issue #1127: Incorrect error message when sync fails with Permission Denied)

Actions

Copy link

#15

Updated by igulina@redhat.com over 9 years ago

Blocked by Issue #1127: Incorrect error message when sync fails with Permission Denied added

Actions

Copy link

#16

Updated by igulina@redhat.com over 9 years ago

QE, please verify #1127 and #1128 before verifying the current issue. Check syncing in Enforcing and Permissive SElinux modes, check syncing for feeds in different locations, e.g. /tmp and /home/user and please check it all on different architectures, i.e. Fedora and RHEL. If there are any exclusions (e.g. a feed on home directory cannot be created), please ensure it's documented.

Actions

Copy link

#17