Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2899

closed

Pulp Python plugin spams logs when downloading wrong filetypes from pypi

Added by mikea over 7 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
mikea
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Python:
master
Platform Release:
2.14.0
Target Release - Python:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Some python packages (setuptools, argparse, pyzqm, to name a few) contain some operating system packages (*.msi, *.rpm, etc). When pulp_python comes across these files, it pukes exceptions into the log file.

I assume that we actually want to mirror these files, but if they're not required these exceptions should be caught and logged as info or something rather than dumping uncaught exceptions into the log file

Jul 13 02:52:21 dac21654prs501 pulp: pulp_python.plugins.importers.sync:INFO: Processing package retrieved from https://pypi.python.org/packages/16/47/b2f6126fc281f9822bf41a23f5dfbd8fc95d027ba5949ac2089e26ea2414/setuptools-0.6c7-1.src.rpm.
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640) ValidationError (Package:773d6179-b1a3-45e1-9f45-c57cec6186c4) (Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640) Traceback (most recent call last):
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     event_listener_callback(*args, **kwargs)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/pulp_python_plugins-2.1a1-py2.6.egg/pulp_python/plugins/importers/sync.py", line 121, in download_succeeded
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     package.save()
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/mongoengine/document.py", line 304, in save
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     self.validate(clean=clean)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)   File "/usr/lib/python2.6/site-packages/mongoengine/base/document.py", line 413, in validate
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-64640)     raise ValidationError(message, errors=errors)

Jul 13 02:52:21 dac21654prs501 pulp: pulp_python.plugins.importers.sync:INFO: Processing package retrieved from https://pypi.python.org/packages/bf/5b/35b761a62c19b8f21032eb7f6791ec728b962264c6178260c8a8cf28cae6/argparse-1.1.win32.msi.
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056) ValidationError (Package:70bfc943-2421-4191-b72e-3982be4a129f) (Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056) Traceback (most recent call last):
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     event_listener_callback(*args, **kwargs)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/pulp_python_plugins-2.1a1-py2.6.egg/pulp_python/plugins/importers/sync.py", line 121, in download_succeeded
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     package.save()
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/mongoengine/document.py", line 304, in save
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     self.validate(clean=clean)
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)   File "/usr/lib/python2.6/site-packages/mongoengine/base/document.py", line 413, in validate
Jul 13 02:52:21 dac21654prs501 pulp: nectar.downloaders.base:ERROR: (25009-53056)     raise ValidationError(message, errors=errors)

This us using the latest pulp_python code from github as of 2017/07/12 at 8:00PM PST. I compared the 2.0 release branch to the current master and there's only been some cosmetic changes, so it's safe to say this is an issue with at least pulp_python 2.0

Actions
Copy link
 #1

Updated by mikea over 7 years ago

Looks like the problem is there are a couple of packagetypes that are missing:

(Value must be one of ('sdist', 'bdist_egg', 'bdist_wheel', 'bdist_wininst'): ['packagetype'])

https://pypi.python.org/packages/16/47/b2f6126fc281f9822bf41a23f5dfbd8fc95d027ba5949ac2089e26ea2414/setuptools-0.6c7-1.src.rpm = packagetype: "bdist_rpm",

https://pypi.python.org/packages/bf/5b/35b761a62c19b8f21032eb7f6791ec728b962264c6178260c8a8cf28cae6/argparse-1.1.win32.msi = packagetype: "bdist_msi",

Added by Mike Adams over 7 years ago

Revision 95801cc2 | View on GitHub

add MSI and RPM as valid package types

Actions
Copy link
 #2

Updated by mikea over 7 years ago

I've submitted a github pull request to fix this issue: https://github.com/pulp/pulp_python/pull/110

Actions
Copy link
 #3

Updated by ttereshc over 7 years ago

  • Status changed from NEW to POST
  • Assignee set to mikea
Actions
Copy link
 #4

Updated by amacdona@redhat.com over 7 years ago

To test this issue, we should add a fixture repository for setuptools to pulp-smash

Actions
Copy link
 #5

Updated by amacdona@redhat.com over 7 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #6

Updated by pcreech over 7 years ago

  • Platform Release set to 2.14.0
Actions
Copy link
 #7

Updated by pcreech over 7 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #8

Updated by pthomas@redhat.com over 7 years ago 

[root@qe-blade-07 ansible]# pulp-admin python repo update --repo-id setuptools --feed https://pypi.python.org   --package-names setuptools,argparse
Repository [setuptools] successfully updated

[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# 
[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id setuptools
+----------------------------------------------------------------------+
                 Synchronizing Repository [setuptools]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# pulp-admin python repo list
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886

[root@qe-blade-07 ansible]# pulp-admin python repo create  --repo-id test --feed https://pypi.python.org   --package-names pyzqm
Repository [test] successfully created

[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id test
+----------------------------------------------------------------------+
                    Synchronizing Repository [test]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# pulp-admin python repo list
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886

Id:                  test
Display Name:        None
Description:         None
Content Unit Counts: 

[root@qe-blade-07 ansible]# pulp-admin python repo update  --repo-id test --feed https://pypi.python.org   --package-names pyzmq
Repository [test] successfully updated

[root@qe-blade-07 ansible]# pulp-admin rpm repo sync run --repo-id test
+----------------------------------------------------------------------+
                    Synchronizing Repository [test]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Task Succeeded

[root@qe-blade-07 ansible]# 

[root@qe-blade-07 ~]# pulp-admin python repo list --details
+----------------------------------------------------------------------+
                          Python Repositories
+----------------------------------------------------------------------+

Id:                  setuptools
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 886
Notes:               
Scratchpad:          
Importers:           
  Config:               
    Feed:          https://pypi.python.org
    Package Names: setuptools,argparse,pyzqm
  Id:                   python_importer
  Importer Type Id:     python_importer
  Last Override Config: 
  Last Sync:            2017-07-31T19:51:22Z
  Last Updated:         2017-07-31T19:51:08Z
  Repo Id:              setuptools
  Scratchpad:           None
Distributors:        
  Auto Publish:         True
  Config:               
  Distributor Type Id:  python_distributor
  Id:                   cli_python_distributor
  Last Override Config: 
  Last Publish:         2017-07-31T19:51:24Z
  Last Updated:         2017-07-31T16:38:56Z
  Repo Id:              setuptools
  Scratchpad:           

Id:                  test
Display Name:        None
Description:         None
Content Unit Counts: 
  Python Package: 442
Notes:               
Scratchpad:          
Importers:           
  Config:               
    Feed:          https://pypi.python.org
    Package Names: pyzmq
  Id:                   python_importer
  Importer Type Id:     python_importer
  Last Override Config: 
  Last Sync:            2017-07-31T20:01:19Z
  Last Updated:         2017-07-31T20:00:02Z
  Repo Id:              test
  Scratchpad:           None
Distributors:        
  Auto Publish:         True
  Config:               
  Distributor Type Id:  python_distributor
  Id:                   cli_python_distributor
  Last Override Config: 
  Last Publish:         2017-07-31T20:01:20Z
  Last Updated:         2017-07-31T19:58:17Z
  Repo Id:              test
  Scratchpad:
Actions
Copy link
 #9

Updated by pcreech over 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #10

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF