Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #635

closed

Scientific Linux Errata Data does not get imported

Added by dlevene@advisns.com.au almost 10 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
bcourt
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.4.0
Platform Release:
2.6.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Description of problem: Errata Data is not synced from the Scientific Linux Repos when it is available;
http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/os/repodata/repomd.xml

Version-Release number of selected component (if applicable): This importing is being done via the latest version of katello
pulp-server-2.4.0-1.el6.noarch / pulp-katello-0.3-3.el6.noarch / katello-2.0.0-0.el6.noarch

How reproducible: always

Steps to Reproduce:
1. Log into Katello and setup a new product
2. Click repo discovery and use this URL http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/
3. Select OS (or any of the other 3

Actual results: All packages are successfully synced, but no errata data is pulled in

Expected results: Packages & errata data are seen in katello

Additional info: http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/os/ is the single mirror that can be used.

Also on a side note, can you import errata data manually or manually modify a pulp sync to provide it an alternative location for errata data?

+ This bug was cloned from Bugzilla Bug #1172887 +

Actions
Copy link
 #1

Updated by bcourt almost 10 years ago

The problem was that the mirror uses bz2 compression for the metadata files instead of the standard gzip. PR is to add support for bz2.

https://github.com/pulp/pulp_rpm/pull/627

+ This comment was cloned from Bugzilla #1172887 comment 1 +

Actions
Copy link
 #2

Updated by bmbouter almost 10 years ago

  • Severity changed from High to 3. High
Actions
Copy link
 #3

Updated by bcourt almost 10 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #5

Updated by pthomas@redhat.com almost 10 years ago

  • Status changed from 5 to ASSIGNED

Fails-qa
[root@qe-blade-10 ~]# rpm -qa pulp-server
pulp-server-2.6.1-0.2.beta.el6.noarch
[root@qe-blade-10 ~]#

Importing errata seems to be failing still

[root@qe-blade-10 ~]# pulp-admin rpm repo create --repo-id scientific --feed http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

Successfully created repository [scientific]

[root@qe-blade-10 ~]# 
[root@qe-blade-10 ~]# 
[root@qe-blade-10 ~]# pulp-admin rpm repo sync run --repo-id scientific 
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

+----------------------------------------------------------------------+
                 Synchronizing Repository [scientific]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... failed

Not Found

Task Failed

Importer indicated a failed response

Mar 31 05:57:09 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
Mar 31 05:57:31 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/os/
Mar 31 05:57:31 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
Mar 31 05:57:31 qe-blade-10 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): mirror.aarnet.edu.au
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280) sync failed
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280) Traceback (most recent call last):
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 135, in run
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     self.get_errata(metadata_files)
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 483, in get_errata
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     updateinfo.process_package_element, additive_type=True)
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 563, in save_fileless_units
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     for model in package_info_generator:
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 63, in package_list_generator
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     package_info = process_func(element)
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/updateinfo.py", line 27, in process_package_element
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     'description': element.find('description').text,
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280) AttributeError: 'NoneType' object has no attribute 'text'
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280) Task pulp.server.managers.repo.sync.sync[f7d421b1-1149-444e-9d54-356f501f938d] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280) Traceback (most recent call last):
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)     R = retval = fun(*args, **kwargs)
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)     return super(Task, self).__call__(*args, **kwargs)
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)     return self.run(*args, **kwargs)
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)     raise PulpExecutionException(_('Importer indicated a failed response'))
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280) PulpExecutionException: Importer indicated a failed response
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[ec62b7e5-5602-442f-be3d-d19049efede7] succeeded in 0.0082434789947s: None

<\pre>
Actions
Copy link
 #6

Updated by bcourt almost 10 years ago

  • Status changed from ASSIGNED to POST

Added by bcourt almost 10 years ago

Revision c53af27a | View on GitHub

Fix import of scientific linux erratum that don't contain a description field.

fixes #635

Actions
Copy link
 #7

Updated by bcourt almost 10 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #8

Updated by bcourt over 9 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #9

Updated by pthomas@redhat.com over 9 years ago

  • Status changed from 5 to ASSIGNED

still failing to sync scientific linux

[root@qe-blade-11 ~]# pulp-admin rpm repo create --repo-id scientific --feed http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

Successfully created repository [scientific]

[root@qe-blade-11 ~]# pulp-admin rpm repo sync run --repo-id scientific
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

+----------------------------------------------------------------------+
                 Synchronizing Repository [scientific]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... failed

Not Found

Task Failed

Importer indicated a failed response

[root@qe-blade-11 ~]# tail -f /var/log/messages
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     R = retval = fun(*args, **kwargs)
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     return super(Task, self).__call__(*args, **kwargs)
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     return self.run(*args, **kwargs)
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     raise PulpExecutionException(_('Importer indicated a failed response'))
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632) PulpExecutionException: Importer indicated a failed response
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[06a81428-d793-458e-8ac7-15262a2404b7] succeeded in 0.00686540501192s: None
Apr 28 13:01:01 qe-blade-11 systemd: Starting Session 100 of user root.
Apr 28 13:01:01 qe-blade-11 systemd: Started Session 100 of user root.
Apr 28 13:05:44 qe-blade-11 NetworkManager[777]: <error> [1430240744.751764] [devices/nm-device.c:2303] activation_source_schedule(): (enp4s0): activation stage already scheduled
Actions
Copy link
 #10

Updated by pthomas@redhat.com over 9 years ago 


Apr 28 13:36:13 qe-blade-11 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[f17bc5a8-4e34-456c-a863-661272ac042d]
Apr 28 13:36:14 qe-blade-11 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[e42708ea-b1f3-4a2e-b910-c3a64bd49c75]
Apr 28 13:36:14 qe-blade-11 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[10cf4534-037a-4817-ae95-db1565552f8f]
Apr 28 13:36:14 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/.
Apr 28 13:36:14 qe-blade-11 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): mirror.aarnet.edu.au
Apr 28 13:36:14 qe-blade-11 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[f17bc5a8-4e34-456c-a863-661272ac042d] succeeded in 0.0333014460048s: None
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632) sync failed
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632) Traceback (most recent call last):
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 105, in run
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632)     metadata_files = self.get_metadata()
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 199, in get_metadata
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632)     raise FailedException(str(e))
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632) FailedException: Not Found
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632) Task pulp.server.managers.repo.sync.sync[e42708ea-b1f3-4a2e-b910-c3a64bd49c75] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632) Traceback (most recent call last):
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     R = retval = fun(*args, **kwargs)
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     return super(Task, self).__call__(*args, **kwargs)
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     return self.run(*args, **kwargs)
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     raise PulpExecutionException(_('Importer indicated a failed response'))
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632) PulpExecutionException: Importer indicated a failed response
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[10cf4534-037a-4817-ae95-db1565552f8f] succeeded in 0.00619069201639s: None
Actions
Copy link
 #11

Updated by bcourt over 9 years ago

  • Platform Release changed from 2.6.1 to 2.6.2
Actions
Copy link
 #12

Updated by bcourt over 9 years ago

  • Status changed from ASSIGNED to 5
  • Platform Release changed from 2.6.2 to 2.6.1

It appears that this was tested using a bad repository feed url. Closer mirrors that could be used for this are:

pulp-admin rpm repo create --repo-id sci-security --feed http://www.gtlib.gatech.edu/pub/scientific/7/x86_64/updates/security/
or
pulp-admin rpm repo create --repo-id sci-security --feed http://www.gtlib.gatech.edu/pub/scientific/7/x86_64/os/

Actions
Copy link
 #13

Updated by pthomas@redhat.com over 9 years ago

  • Status changed from 5 to 6

verified

[root@qe-blade-13 ~]# pulp-admin rpm repo sync run --repo-id sci-security
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

+----------------------------------------------------------------------+
                Synchronizing Repository [sci-security]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[/]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       8685/8685 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[=========================================         ] 83%
Distributions: 5/6 items

Errors encountered during distribution sync:

File:
http://www.gtlib.gatech.edu/pub/scientific/7/x86_64/os/images/boot.iso
Error
Code:   None
Error Message: None

Importing errata...
[-]
... completed

Importing package groups/categories...
[\]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
8685 of 8685 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
3021 of 3021 items
... completed

Publishing Comps file
[==================================================] 100%
105 of 105 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[\]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

<\pre>
Actions
Copy link
 #14

Updated by dkliban@redhat.com over 9 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #20

Updated by bmbouter almost 6 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF