Project

Profile

Help

Issue #2899

Pulp Python plugin spams logs when downloading wrong filetypes from pypi

Added by mikea about 3 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Python:
master
Platform Release:
2.14.0
Target Release - Python:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Some python packages (setuptools, argparse, pyzqm, to name a few) contain some operating system packages (*.msi, *.rpm, etc). When pulp_python comes across these files, it pukes exceptions into the log file.

I assume that we actually want to mirror these files, but if they're not required these exceptions should be caught and logged as info or something rather than dumping uncaught exceptions into the log file

Jul 13 02:52:21 dac21654prs501 pulp: pulp_python.plugins.importers.sync:INFO: Processing package retrieved from https://pypi.python.org/packages/16/47/b2f6126fc281f9822bf41a23f5dfbd8fc95d027ba5949ac2089e26ea2414/setuptools-0.6c7-1.src.rpm.
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640) ValidationError (Package:773d6179-b1a3-45e1-9f45-c57cec6186c4) (Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640) Traceback (most recent call last):
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     event_listener_callback(*args, **kwargs)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/pulp_python_plugins-2.1a1-py2.6.egg/pulp_python/plugins/importers/sync.py", line 121, in download_succeeded
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     package.save()
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/mongoengine/document.py", line 304, in save
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     self.validate(clean=clean)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/mongoengine/base/document.py", line 413, in validate
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     raise ValidationError(message, errors=errors)

Jul 13 02:52:21 dac21654prs501 pulp: pulp_python.plugins.importers.sync:INFO: Processing package retrieved from https://pypi.python.org/packages/bf/5b/35b761a62c19b8f21032eb7f6791ec728b962264c6178260c8a8cf28cae6/argparse-1.1.win32.msi.
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056) ValidationError (Package:70bfc943-2421-4191-b72e-3982be4a129f) (Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056) Traceback (most recent call last):
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     event_listener_callback(*args, **kwargs)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/pulp_python_plugins-2.1a1-py2.6.egg/pulp_python/plugins/importers/sync.py", line 121, in download_succeeded
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     package.save()
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/mongoengine/document.py", line 304, in save
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     self.validate(clean=clean)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/mongoengine/base/document.py", line 413, in validate
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     raise ValidationError(message, errors=errors)

This us using the latest pulp_python code from github as of 2017/07/12 at 8:00PM PST. I compared the 2.0 release branch to the current master and there's only been some cosmetic changes, so it's safe to say this is an issue with at least pulp_python 2.0

Associated revisions

Revision 95801cc2 View on GitHub
Added by Mike Adams about 3 years ago

add MSI and RPM as valid package types

History

#1 Updated by mikea about 3 years ago

Looks like the problem is there are a couple of packagetypes that are missing:

(Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])

https://pypi.python.org/packages/16/47/b2f6126fc281f9822bf41a23f5dfbd8fc95d027ba5949ac2089e26ea2414/setuptools-0.6c7-1.src.rpm = packagetype: "bdist_rpm",

https://pypi.python.org/packages/bf/5b/35b761a62c19b8f21032eb7f6791ec728b962264c6178260c8a8cf28cae6/argparse-1.1.win32.msi = packagetype: "bdist_msi", 

#2 Updated by mikea about 3 years ago

I've submitted a github pull request to fix this issue: https://github.com/pulp/pulp_python/pull/110

#3 Updated by ttereshc about 3 years ago

  • Status changed from NEW to POST
  • Assignee set to mikea

#4 Updated by amacdona@redhat.com about 3 years ago

To test this issue, we should add a fixture repository for setuptools to pulp-smash

#5 Updated by amacdona@redhat.com about 3 years ago

  • Status changed from POST to MODIFIED

#6 Updated by pcreech about 3 years ago

  • Platform Release set to 2.14.0

#7 Updated by pcreech about 3 years ago

  • Status changed from MODIFIED to 5

#8 Updated by pthomas@redhat.com about 3 years ago

[root@qe-blade-07 ansible]# pulp-admin python repo update --repo-id setuptools --feed https://pypi.python.org   --package-names setuptools,argparse
Repository [setuptools] successfully updated

[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id setuptools
+----------------------------------------------------------------------+
                 Synchronizing Repository [setuptools]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# pulp-admin python repo list
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886

[root@qe-blade-07 ansible]# pulp-admin python repo create  --repo-id test --feed https://pypi.python.org   --package-names pyzqm
Repository [test] successfully created

[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id test
+----------------------------------------------------------------------+
                    Synchronizing Repository [test]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# pulp-admin python repo list
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886

Id:                  test
Display Name:        None
Description:         None
Content Unit Counts: 

[root@qe-blade-07 ansible]# pulp-admin python repo update  --repo-id test --feed https://pypi.python.org   --package-names pyzmq
Repository [test] successfully updated

[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id test
+----------------------------------------------------------------------+
                    Synchronizing Repository [test]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# 

[root@qe-blade-07 ~]# pulp-admin python repo list --details
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886
Notes:               
Scratchpad:          
Importers:           
  Config:               
    Feed:          https://pypi.python.org
    Package Names: setuptools,argparse,pyzqm
  Id:                   python_importer
  Importer Type Id:     python_importer
  Last Override Config: 
  Last Sync:            2017-07-31T19:51:22Z
  Last Updated:         2017-07-31T19:51:08Z
  Repo Id:              setuptools
  Scratchpad:           None
Distributors:        
  Auto Publish:         True
  Config:               
  Distributor Type Id:  python_distributor
  Id:                   cli_python_distributor
  Last Override Config: 
  Last Publish:         2017-07-31T19:51:24Z
  Last Updated:         2017-07-31T16:38:56Z
  Repo Id:              setuptools
  Scratchpad:           

Id:                  test
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 442
Notes:               
Scratchpad:          
Importers:           
  Config:               
    Feed:          https://pypi.python.org
    Package Names: pyzmq
  Id:                   python_importer
  Importer Type Id:     python_importer
  Last Override Config: 
  Last Sync:            2017-07-31T20:01:19Z
  Last Updated:         2017-07-31T20:00:02Z
  Repo Id:              test
  Scratchpad:           None
Distributors:        
  Auto Publish:         True
  Config:               
  Distributor Type Id:  python_distributor
  Id:                   cli_python_distributor
  Last Override Config: 
  Last Publish:         2017-07-31T20:01:20Z
  Last Updated:         2017-07-31T19:58:17Z
  Repo Id:              test
  Scratchpad:           

#9 Updated by pcreech about 3 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#10 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF