Project

Profile

Help

Issue #7449

closed

F32 sync fails with on_demand policy, DocumentTooLarge

Added by ttereshc over 3 years ago. Updated over 3 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

A user reported on the foreman forum https://community.theforeman.org/t/failure-to-sync-fedora-32-repos-pulp-yum-updater-error-command-document-too-large/20023

I’m attempting to sync Fedora 32 repositories from the Princeton mirror (I have tried another just to rule out the repo.) Unfortunately the sync fails with the exception Katello::Errors::PulpError: PLP0000: Importer indicated a failed response and the Error tab shows that the Pulp yum_importer failed with error command document too large.

I’ve found a handful of similar sounding bug reports, but they’re from a few years ago and claim to have fixed the issue. https://access.redhat.com/solutions/2957371 1

I have two reproduced this on two systems, Katello 3.15.3 and RH Satellite 6.7.2.

Expected outcome: Repository to sync correctly as it did with Fedora 32.

Reproducible?: If it’s not something with me, then you should just have to add the Fedora 32 Everything repo to a Product and attempt to sync. It’ll process metadata for a while and then error out.

Foreman and Proxy versions: Foreman 2.0.1 w/ Katello 3.15.3 also produced the error on Satellite 6.7.2

Distribution and version: Katello = CentOS 7.8 Satellite = RHEL 7.8

Other relevant data:

{“pulp_tasks”=>
[{“exception”=>nil,
“task_type”=>“pulp.server.managers.repo.sync.sync”,
“_href”=>"/pulp/api/v2/tasks/91caf2e6-6d0d-4dfb-abcd-d0e3879e40bb/",
“task_id”=>“91caf2e6-6d0d-4dfb-abcd-d0e3879e40bb”,
“tags”=>
[“pulp:repository:156f4383-2ae0-48cb-96ec-462eee4c4c22”,
“pulp:action:sync”],
“finish_time”=>“2020-08-10T20:39:33Z”,
“ns"=>“task_status”,
“start_time”=>“2020-08-10T20:23:43Z”,
“traceback”=>
“Traceback (most recent call last):\n” +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task\n" +
" R = retval = fun(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 686, in call\n" +
" return super(Task, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 108, in call\n" +
" return super(PulpTask, self).call(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call\n" +
" return self.run(*args, **kwargs)\n" +
" File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 855, in sync\n" +
" raise pulp_exceptions.PulpExecutionException((‘Importer indicated a failed response’))\n” +
“PulpExecutionException: Importer indicated a failed response\n”,
“spawned_tasks”=>,
“progress_report”=>
{“yum_importer”=>
{“content”=>
{“size_total”=>78681195936,
“items_left”=>11641,
“items_total”=>55327,
“state”=>“FAILED”,
“size_left”=>15786547076,
“details”=>
{“rpm_total”=>55327,
“rpm_done”=>43686,
“drpm_total”=>0,
“drpm_done”=>0},
“error”=>“command document too large”,
“error_details”=>},
“comps”=>{“state”=>“NOT_STARTED”},
“purge_duplicates”=>{“state”=>“NOT_STARTED”},
“distribution”=>
{“items_total”=>0,
“state”=>“NOT_STARTED”,
“error_details”=>,
“items_left”=>0},
“modules”=>{“state”=>“NOT_STARTED”},
“errata”=>{“state”=>“NOT_STARTED”},
“metadata”=>{“state”=>“FINISHED”}}},
“queue”=>“reserved_resource_worker-1@libkatello.library.unt.edu.dq2”,
“state”=>“error”,
“worker_name”=>“reserved_resource_worker-1@libkatello.library.unt.edu”,
“result”=>nil,
“error”=>
{“code”=>“PLP0000”,
“data”=>{},
“description”=>“Importer indicated a failed response”,
“sub_errors”=>},
“_id”=>{"$oid"=>“5f31accf9aae893f3610f6fd”},
“id”=>“5f31accf9aae893f3610f6fd”}],
“contents_changed”=>true,
“poll_attempts”=>{“total”=>80, “failed”=>1}}
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064) command document too large
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064) Traceback (most recent call last):
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 294, in run
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     self.update_content(metadata_files, url)
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 607, in update_content
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     self.download_rpms(metadata_files, rpms_to_download, url)
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 834, in download_rpms
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     unit = self.add_rpm_unit(metadata_files, unit)
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 759, in add_rpm_unit
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     unit.save()
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 324, in save
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     object_id = collection.save(doc, **write_concern)
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     check_keys, False, manipulate, write_concern)
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 709, in _update
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     codec_options=self.codec_options).copy()
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 216, in command
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     self._raise_connection_failure(error)
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 343, in _raise_connection_failure
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064)     raise error
Aug 11 15:20:04 host.fqdn pulp[6556]: pulp_rpm.plugins.importers.yum.sync:ERROR: [7181f260] (6556-80064) DocumentTooLarge: command document too large
Actions #1

Updated by ttereshc over 3 years ago

One package (sagemath-doc-en-9.0-2.fc32.x86_64.rpm) has a lot of files ~360 000 and it doesn’t fit in the max size of the mongodb document.

I think that the immediate policy continues sync process on failure while on_demand policy fails in such case and stops the sync process.

So to workaround the sync failure, one can sync with immediate policy. If you need that specific package, unfortunately, there is no workaround for that case.

Actions #2

Updated by ttereshc over 3 years ago

F32 repository sync works successfully with Pulp 3 and since there is no MongoDB in Pulp3, this issue can't exist. We encourage to upgrade to Pulp 3.

Actions #3

Updated by ttereshc over 3 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Also available in: Atom PDF