Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2745

closed

Error for a specific rpm is silently skipped during sync with immediate policy

Added by ipanova@redhat.com almost 7 years ago. Updated about 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

$ pulp-admin rpm repo create --repo-id=test-repo --feed=http://resources.ovirt.org/pub/ovirt-4.0/rpm/el7/ --download-policy on_demand

Successfully created repository [test-repo]

$ pulp-admin -v rpm repo sync run --repo-id test-repo

+----------------------------------------------------------------------+
                  Synchronizing Repository [test-repo]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[/]
[================================================  ] 97%
RPMs:       799/899 items
Delta RPMs: 0/0 items

Task Failed

Importer indicated a failed response

May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552) command document too large
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552) Traceback (most recent call last):
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/home/ipanova/pulp_development/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 268, in run
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     self.update_content(metadata_files, url)
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/home/ipanova/pulp_development/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 548, in update_content
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     self.download_rpms(metadata_files, rpms_to_download, url)
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/home/ipanova/pulp_development/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 775, in download_rpms
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     unit = self.add_rpm_unit(metadata_files, unit)
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/home/ipanova/pulp_development/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 700, in add_rpm_unit
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     unit.save()
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 324, in save
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     object_id = collection.save(doc, **write_concern)
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 2180, in save
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     check_keys, False, manipulate, write_concern)
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 709, in _update
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     codec_options=self.codec_options).copy()
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 216, in command
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     self._raise_connection_failure(error)
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)   File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 343, in _raise_connection_failure
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552)     raise error
May  3 14:15:05 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [11055fd6] (28670-39552) DocumentTooLarge: command document too large
May  3 14:15:05 ina pulp: pulp.server.async.tasks:INFO: [11055fd6] Task failed : [11055fd6-a277-4680-938f-0c6a742eafbb]
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552) Task pulp.server.managers.repo.sync.sync[11055fd6-a277-4680-938f-0c6a742eafbb] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552) Traceback (most recent call last):
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)     R = retval = fun(*args, **kwargs)
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)   File "/home/ipanova/pulp_development/pulp/server/pulp/server/async/tasks.py", line 506, in __call__
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)     return super(Task, self).__call__(*args, **kwargs)
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)   File "/home/ipanova/pulp_development/pulp/server/pulp/server/async/tasks.py", line 107, in __call__
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)     return super(PulpTask, self).__call__(*args, **kwargs)
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)     return self.run(*args, **kwargs)
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)   File "/home/ipanova/pulp_development/pulp/server/pulp/server/controllers/repository.py", line 827, in sync
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
May  3 14:15:05 ina pulp: celery.worker.job:ERROR: (28500-39552) PulpExecutionException: Importer indicated a failed response
May  3 14:15:05 ina pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[f35b3277-42d6-433e-bc3c-5aa41a7709b9] succeeded in 0.0117766209878s: None

With immediate policy there is not error raised but the offending rpm still does not get synced, saved, and therefore published.

$ pulp-admin -v rpm repo sync run --repo-id test-repo 

+----------------------------------------------------------------------+
                  Synchronizing Repository [test-repo]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[\]
[================================================= ] 99%
RPMs:       898/899 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
898 of 898 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[\]
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Generating HTML files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[\]
... completed

Task Succeeded

Task details:

Operations:       sync
Resources:        test-repo (repository)
State:            Successful
Start Time:       2017-05-03T12:58:04Z
Finish Time:      2017-05-03T13:13:11Z
Result:           
  Added Count:      898
  Completed:        2017-05-03T13:13:11Z
  Details:          
    Comps:            
      State: FINISHED
    Content:          
      Details:       
        Drpm Done:  0
        Drpm Total: 0
        Rpm Done:   898
        Rpm Total:  899
      Error Details: 
      Items Left:    1
      Items Total:   899
      Size Left:     40862680
      Size Total:    15620421457
      State:         FINISHED
    Distribution:     
      Error Details: 
      Items Left:    0
      Items Total:   0
      State:         FINISHED
    Errata:           
      State: FINISHED
    Metadata:         
      State: FINISHED
    Purge Duplicates: 
      State: FINISHED
  Error Message:    None
  Exception:        None
  Id:               5909d76745ef487f3924ff11
  Importer Id:      yum_importer
  Importer Type Id: yum_importer
  Removed Count:    0
  Repo Id:          test-repo
  Result:           success
  Started:          2017-05-03T12:58:04Z
  Summary:          
    Comps:            
      State: FINISHED
    Content:          
      State: FINISHED
    Distribution:     
      State: FINISHED
    Errata:           
      State: FINISHED
    Metadata:         
      State: FINISHED
    Purge Duplicates: 
      State: FINISHED
  Traceback:        None
  Updated Count:    0
Task Id:          09c39264-dd68-4263-8012-8c7862f3b32b
Progress Report:  
  Yum Importer: 
    Comps:            
      State: FINISHED
    Content:          
      Details:       
        Drpm Done:  0
        Drpm Total: 0
        Rpm Done:   898
        Rpm Total:  899
      Error Details: 
      Items Left:    1
      Items Total:   899
      Size Left:     40862680
      Size Total:    15620421457
      State:         FINISHED
    Distribution:     
      Error Details: 
      Items Left:    0
      Items Total:   0
      State:         FINISHED
    Errata:           
      State: FINISHED
    Metadata:         
      State: FINISHED
    Purge Duplicates: 
      State: FINISHED
Actions
Copy link
 #1

Updated by ipanova@redhat.com almost 7 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by bmbouter almost 7 years ago

  • Subject changed from DocumentTooLarge when syncing repo with om_demand policy to DocumentTooLarge when syncing repo with on_demand policy
Actions
Copy link
 #3

Updated by ttereshc almost 7 years ago

I think there are 2 unrelated issues here.
1. Error handling is different in case of immediate and on_demand sync.
2. As expected there can be RPM packages which metadata do not fit into mongo. And the question is do we want to improve it and how. Current limitation is a parsed filelist in db, imo.

Actions
Copy link
 #4

Updated by ttereshc almost 7 years ago

  • Subject changed from DocumentTooLarge when syncing repo with on_demand policy to Error for a specific rpm is silently skipped during sync with immediate policy
Actions
Copy link
 #6

Updated by ttereshc almost 7 years ago

As agreed at triage:
- I changed title for this issue to reflect that it is about error handling, we will triage it next time.
- I created another issue (#2747) to track DocumentTooLarge issue itself.

Actions
Copy link
 #7

Updated by ttereshc almost 7 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #8

Updated by bmbouter about 5 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

Actions
Copy link
 #9

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF