Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #995

closed

Syncing yum repo raises error

Added by paji@redhat.com almost 9 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
ipanova@redhat.com
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version:
Platform Release:
2.6.3
OS:
Triaged:
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Unpack the attached tar gz file in /tmp
you should see a /tmp/zoo5

try the following commands

pulp-admin rpm repo create --repo-id=synergism --feed=file:///var/www/repositories/zoo5
pulp-admin rpm repo sync run --repo-id=synergism

cat /var/log/messages

You see the following the stack trace. Note in particular this error

May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184) [Errno 2] No such file or directory: u'///var/www/repositories/zoo5/PULP_DISTRIBUTION.xml'

This causes katello to think there is a 500 in pulp. Causing integration tests to fail.

May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[c0a60ed2-b5a4-41ef-a888-d8d6a3684b07]
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[cd811a9c-be90-4ff3-aa04-d1b6fd0b35a6]
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[c0a60ed2-b5a4-41ef-a888-d8d6a3684b07] succeeded in 0.036162881006s: None
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[ac1d87a3-a26a-427d-8fde-119cf670e9dd]
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from file:///var/www/repositories/zoo5/.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: file:///var/www/repositories/zoo5/
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184) [Errno 2] No such file or directory: u'///var/www/repositories/zoo5/PULP_DISTRIBUTION.xml'
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184) Traceback (most recent call last):
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/local.py", line 144, in _copy
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184)     src_handle = open(src_path, 'rb')
May 20 17:32:42 katello-wisdom pulp: nectar.downloaders.local:ERROR: (2921-33184) IOError: [Errno 2] No such file or directory: u'///var/www/repositories/zoo5/PULP_DISTRIBUTION.xml'
May 20 17:32:42 katello-wisdom pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Sync complete.
May 20 17:32:42 katello-wisdom pulp: pulp.server.event.http:INFO: {'call_report': <TaskStatus: TaskStatus object>, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': u'yum_importer', 'exception': None, 'repo_id': u'partello', 'traceback': None, 'started': '2015-05-20T21:32:42Z', '_ns': u'repo_sync_results', 'completed': '2015-05-20T21:32:42Z', 'importer_type_id': u'yum_importer', 'error_message': None, 'summary': {'content': {'state': 'FINISHED'}, 'comps': {'state': 'FINISHED'}, 'distribution': {'state': 'FINISHED'}, 'errata': {'state': 'FINISHED'}, 'metadata': {'state': 'FINISHED'}}, 'added_count': 3, 'result': 'success', 'updated_count': 11, 'details': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'FINISHED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error_details': []}, 'comps': {'state': 'FINISHED'}, 'distribution': {'items_total': 3, 'state': 'FINISHED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'FINISHED'}, 'metadata': {'state': 'FINISHED'}}, 'id': '555cfd7ad01ec00b69cae7f4', 'removed_count': 0}}
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184) Exception from notifier of type [http]
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184) Traceback (most recent call last):
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/event/fire.py", line 76, in _do_fire
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     f(l['notifier_config'], event)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib/python2.7/site-packages/pulp/server/event/http.py", line 33, in handle_event
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     body = json.dumps(data, default=json_util.default)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     sort_keys=sort_keys, **kw).encode(obj)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     chunks = self.iterencode(o, _one_shot=True)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     return _iterencode(o, 0)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)   File "/usr/lib64/python2.7/site-packages/bson/json_util.py", line 220, in default
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184)     raise TypeError("%%r is not JSON serializable" %% obj)
May 20 17:32:42 katello-wisdom pulp: pulp.server.managers.event.fire:ERROR: (2921-33184) TypeError: <TaskStatus: TaskStatus object> is not JSON serializable
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[cd811a9c-be90-4ff3-aa04-d1b6fd0b35a6] succeeded in 0.201764149999s: <pulp.server.async.tasks.TaskResult object at 0x4dd95d0>
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[ac1d87a3-a26a-427d-8fde-119cf670e9dd] succeeded in 0.00725326400425s: None
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[ad5774d9-3534-42ef-9748-557e21b75ec5]
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[09707c71-c404-433b-86c6-0e0f28a3557b]
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[ad5774d9-3534-42ef-9748-557e21b75ec5] succeeded in 0.0815736520017s: None
May 20 17:32:42 katello-wisdom pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9b4fd858-bf97-4de4-ad7d-a8b9be66adc6]
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[09707c71-c404-433b-86c6-0e0f28a3557b] succeeded in 0.157991663989s: {'exception': None, 'repo_id': u'partello', 'traceback': None, 'started': '2015-05-20T21:32:42Z', '_ns':...
May 20 17:32:42 katello-wisdom pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9b4fd858-bf97-4de4-ad7d-a8b9be66adc6] succeeded in 0.00436354399426s: None

Files

repo.tar.gz (16.7 KB) repo.tar.gz paji@redhat.com, 05/21/2015 12:30 AM
Actions
Copy link
 #1

Updated by paji@redhat.com almost 9 years ago

Correction in the bz

pulp-admin rpm repo create --repo-id=synergism --feed=file:///tmp/zoo5

Actions
Copy link
 #2

Updated by mhrivnak almost 9 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by mhrivnak almost 9 years ago

  • Priority changed from Normal to High
  • Version deleted (Master)

This is seen in 2.7.0 alpha

This may be because apache gets a unique view of /tmp/ and cannot see the same files you can. Can you confirm this happens if you put the repo somewhere outside /tmp/ that we know apache can read?

Actions
Copy link
 #4

Updated by ipanova@redhat.com almost 9 years ago

  • Status changed from NEW to ASSIGNED
Actions
Copy link
 #5

Updated by ipanova@redhat.com almost 9 years ago

  • Assignee set to ipanova@redhat.com
Actions
Copy link
 #6

Updated by ipanova@redhat.com almost 9 years ago

I do not think it is the apache issue, as if I use some other directory I get the sync finished without any errors, but still in the logs I can see the [Errno 2] No such file or directory

$ pulp-admin -u admin -p admin rpm  repo create --repo-id test1 --feed file:///home/user/pulp_development/zoo5

Successfully created repository [test1]

$ pulp-admin -u admin -p admin rpm repo sync run --repo-id test1
+----------------------------------------------------------------------+
                    Synchronizing Repository [test1]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 3/3 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
8 of 8 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
2 of 2 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

sudo less /var/log/messages|tail -20

May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from file:///home/user/pulp_development/zoo5/.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 2} for base URL: file:///home/user/pulp_development/zoo5/
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848) [Errno 2] No such file or directory: u'///home/user/pulp_development/zoo5/PULP_DISTRIBUTION.xml'
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848) Traceback (most recent call last):
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/local.py", line 144, in _copy
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848)     src_handle = open(src_path, 'rb')
May 27 13:04:59 ina pulp: nectar.downloaders.local:ERROR: (4975-54848) IOError: [Errno 2] No such file or directory: u'///home/user/pulp_development/zoo5/PULP_DISTRIBUTION.xml'
May 27 13:04:59 ina pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Sync complete.
May 27 13:04:59 ina pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[b5530f9f-d1b4-4d70-a520-09e1efcd9793]
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[24a3bb05-4157-4cec-9c83-f79b3ed7ae95] succeeded in 0.1826176662s: <pulp.server.async.tasks.TaskResult object at 0x393ba10>
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[b5530f9f-d1b4-4d70-a520-09e1efcd9793] succeeded in 0.0476008197293s: None
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[b8e5ddbf-2e51-4cc0-b2e8-fd5e86b84a45] succeeded in 0.0452612806112s: None
May 27 13:04:59 ina pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[5cee603b-9739-4b45-9477-827b32c8b9d4]
May 27 13:04:59 ina pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[bd302d54-7f2d-4421-9591-4b04db288727]
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[5cee603b-9739-4b45-9477-827b32c8b9d4] succeeded in 0.231409627944s: {'exception': None, 'repo_id': u'test1', 'traceback': None, 'started': '2015-05-27T11:04:59Z', '_ns':...
May 27 13:04:59 ina pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[bd302d54-7f2d-4421-9591-4b04db288727] succeeded in 0.0181207852438s: None

So it could be something related to the how the distribution is processed.

Actions
Copy link
 #7

Updated by mhrivnak almost 9 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #8

Updated by ipanova@redhat.com almost 9 years ago

  • Project changed from RPM Support to Nectar
  • Status changed from ASSIGNED to MODIFIED

this will be fixed in pulp 2.6.3 which will contain new build of nectar 1.3.2

https://github.com/pulp/nectar/commit/fc6cb788429591d1595e5947f64f15bf16b4b888

Actions
Copy link
 #9

Updated by ipanova@redhat.com almost 9 years ago

  • Project changed from Nectar to Pulp
  • Platform Release set to 2.6.3
Actions
Copy link
 #10

Updated by dkliban@redhat.com almost 9 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #12

Updated by igulina@redhat.com over 8 years ago

  • Blocked by Issue #1128: permission denied with selinux enforcing when sync repo with a local feed added
Actions
Copy link
 #13

Updated by igulina@redhat.com over 8 years ago

  • Blocks Issue #1127: Incorrect error message when sync fails with Permission Denied added
Actions
Copy link
 #14

Updated by igulina@redhat.com over 8 years ago

  • Blocks deleted (Issue #1127: Incorrect error message when sync fails with Permission Denied)
Actions
Copy link
 #15

Updated by igulina@redhat.com over 8 years ago

  • Blocked by Issue #1127: Incorrect error message when sync fails with Permission Denied added
Actions
Copy link
 #16

Updated by igulina@redhat.com over 8 years ago

QE, please verify #1127 and #1128 before verifying the current issue. Check syncing in Enforcing and Permissive SElinux modes, check syncing for feeds in different locations, e.g. /tmp and /home/user and please check it all on different architectures, i.e. Fedora and RHEL. If there are any exclusions (e.g. a feed on home directory cannot be created), please ensure it's documented.

Actions
Copy link
 #17

Updated by dkliban@redhat.com over 8 years ago

  • Status changed from 5 to 6
Actions
Copy link
 #18

Updated by bmbouter over 8 years ago

  • Status changed from 6 to NEW

Temporarily moving state to NEW to see if I can then move it to "CLOSED - CURRENT RELEASE".

Actions
Copy link
 #19

Updated by bmbouter over 8 years ago

  • Tracker changed from Issue to Task
  • Status changed from NEW to 6
  • Groomed set to No
  • Sprint Candidate set to No

Moving back to VERIFIED since I can't move it to "CLOSED - CURRENT RELEASE" from NEW. Trying to move it to a Task temporarily now.

Actions
Copy link
 #20

Updated by bmbouter over 8 years ago

  • Tracker changed from Task to Issue
  • Severity set to 1. Low
  • Platform Release set to 2.6.3

Moving back to an issue since that also didn't allow it to be "CLOSED - CURRENT RELEASE".

Actions
Copy link
 #21

Updated by bmbouter over 8 years ago

  • Blocked by deleted (Issue #1128: permission denied with selinux enforcing when sync repo with a local feed)
Actions
Copy link
 #22

Updated by bmbouter over 8 years ago

  • Blocked by deleted (Issue #1127: Incorrect error message when sync fails with Permission Denied)
Actions
Copy link
 #23

Updated by bmbouter over 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #25

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF