Project

Profile

Help

Issue #2899

Pulp Python plugin spams logs when downloading wrong filetypes from pypi

Added by mikea almost 3 years ago. Updated about 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Severity:
3. High
Version - Python:
master
Platform Release:
2.14.0
Blocks Release:
Target Release - Python:
OS:
Backwards Incompatible:
No
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
Yes
Sprint:

Description

Some python packages (setuptools, argparse, pyzqm, to name a few) contain some operating system packages (*.msi, *.rpm, etc). When pulp_python comes across these files, it pukes exceptions into the log file.

I assume that we actually want to mirror these files, but if they're not required these exceptions should be caught and logged as info or something rather than dumping uncaught exceptions into the log file

Jul 13 02:52:21 dac21654prs501 pulp: pulp_python.plugins.importers.sync:INFO: Processing package retrieved from https://pypi.python.org/packages/16/47/b2f6126fc281f9822bf41a23f5dfbd8fc95d027ba5949ac2089e26ea2414/setuptools-0.6c7-1.src.rpm.
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640) ValidationError (Package:773d6179-b1a3-45e1-9f45-c57cec6186c4) (Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640) Traceback (most recent call last):
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     event_listener_callback(*args, **kwargs)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/pulp_python_plugins-2.1a1-py2.6.egg/pulp_python/plugins/importers/sync.py", line 121, in download_succeeded
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     package.save()
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/mongoengine/document.py", line 304, in save
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     self.validate(clean=clean)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/mongoengine/base/document.py", line 413, in validate
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     raise ValidationError(message, errors=errors)

Jul 13 02:52:21 dac21654prs501 pulp: pulp_python.plugins.importers.sync:INFO: Processing package retrieved from https://pypi.python.org/packages/bf/5b/35b761a62c19b8f21032eb7f6791ec728b962264c6178260c8a8cf28cae6/argparse-1.1.win32.msi.
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056) ValidationError (Package:70bfc943-2421-4191-b72e-3982be4a129f) (Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056) Traceback (most recent call last):
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     event_listener_callback(*args, **kwargs)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/pulp_python_plugins-2.1a1-py2.6.egg/pulp_python/plugins/importers/sync.py", line 121, in download_succeeded
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     package.save()
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/mongoengine/document.py", line 304, in save
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     self.validate(clean=clean)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/mongoengine/base/document.py", line 413, in validate
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     raise ValidationError(message, errors=errors)

This us using the latest pulp_python code from github as of 2017/07/12 at 8:00PM PST. I compared the 2.0 release branch to the current master and there's only been some cosmetic changes, so it's safe to say this is an issue with at least pulp_python 2.0

Associated revisions

Revision 95801cc2 View on GitHub
Added by Mike Adams almost 3 years ago

add MSI and RPM as valid package types

History

#1 Updated by mikea almost 3 years ago

Looks like the problem is there are a couple of packagetypes that are missing:

(Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])

https://pypi.python.org/packages/16/47/b2f6126fc281f9822bf41a23f5dfbd8fc95d027ba5949ac2089e26ea2414/setuptools-0.6c7-1.src.rpm = packagetype: "bdist_rpm",

https://pypi.python.org/packages/bf/5b/35b761a62c19b8f21032eb7f6791ec728b962264c6178260c8a8cf28cae6/argparse-1.1.win32.msi = packagetype: "bdist_msi", 

#2 Updated by mikea almost 3 years ago

I've submitted a github pull request to fix this issue: https://github.com/pulp/pulp_python/pull/110

#3 Updated by ttereshc almost 3 years ago

  • Status changed from NEW to POST
  • Assignee set to mikea

#4 Updated by amacdona@redhat.com almost 3 years ago

  • Verification Required changed from No to Yes

To test this issue, we should add a fixture repository for setuptools to pulp-smash

#5 Updated by amacdona@redhat.com almost 3 years ago

  • Status changed from POST to MODIFIED

#6 Updated by pcreech almost 3 years ago

  • Platform Release set to 2.14.0

#7 Updated by pcreech almost 3 years ago

  • Status changed from MODIFIED to ON_QA

#8 Updated by pthomas@redhat.com almost 3 years ago

  • Smash Test set to 740
  • Verified changed from No to Yes
[root@qe-blade-07 ansible]# pulp-admin python repo update --repo-id setuptools --feed https://pypi.python.org   --package-names setuptools,argparse
Repository [setuptools] successfully updated

[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id setuptools
+----------------------------------------------------------------------+
                 Synchronizing Repository [setuptools]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# pulp-admin python repo list
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886

[root@qe-blade-07 ansible]# pulp-admin python repo create  --repo-id test --feed https://pypi.python.org   --package-names pyzqm
Repository [test] successfully created

[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id test
+----------------------------------------------------------------------+
                    Synchronizing Repository [test]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# pulp-admin python repo list
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886

Id:                  test
Display Name:        None
Description:         None
Content Unit Counts: 

[root@qe-blade-07 ansible]# pulp-admin python repo update  --repo-id test --feed https://pypi.python.org   --package-names pyzmq
Repository [test] successfully updated

[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id test
+----------------------------------------------------------------------+
                    Synchronizing Repository [test]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# 

[root@qe-blade-07 ~]# pulp-admin python repo list --details
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886
Notes:               
Scratchpad:          
Importers:           
  Config:               
    Feed:          https://pypi.python.org
    Package Names: setuptools,argparse,pyzqm
  Id:                   python_importer
  Importer Type Id:     python_importer
  Last Override Config: 
  Last Sync:            2017-07-31T19:51:22Z
  Last Updated:         2017-07-31T19:51:08Z
  Repo Id:              setuptools
  Scratchpad:           None
Distributors:        
  Auto Publish:         True
  Config:               
  Distributor Type Id:  python_distributor
  Id:                   cli_python_distributor
  Last Override Config: 
  Last Publish:         2017-07-31T19:51:24Z
  Last Updated:         2017-07-31T16:38:56Z
  Repo Id:              setuptools
  Scratchpad:           

Id:                  test
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 442
Notes:               
Scratchpad:          
Importers:           
  Config:               
    Feed:          https://pypi.python.org
    Package Names: pyzmq
  Id:                   python_importer
  Importer Type Id:     python_importer
  Last Override Config: 
  Last Sync:            2017-07-31T20:01:19Z
  Last Updated:         2017-07-31T20:00:02Z
  Repo Id:              test
  Scratchpad:           None
Distributors:        
  Auto Publish:         True
  Config:               
  Distributor Type Id:  python_distributor
  Id:                   cli_python_distributor
  Last Override Config: 
  Last Publish:         2017-07-31T20:01:20Z
  Last Updated:         2017-07-31T19:58:17Z
  Repo Id:              test
  Scratchpad:           

#9 Updated by pcreech almost 3 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#10 Updated by bmbouter about 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF