Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #635

closed

Scientific Linux Errata Data does not get imported

Added by dlevene@advisns.com.au over 9 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
bcourt
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.4.0
Platform Release:
2.6.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Description of problem: Errata Data is not synced from the Scientific Linux Repos when it is available;
http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/os/repodata/repomd.xml

Version-Release number of selected component (if applicable): This importing is being done via the latest version of katello
pulp-server-2.4.0-1.el6.noarch / pulp-katello-0.3-3.el6.noarch / katello-2.0.0-0.el6.noarch

How reproducible: always

Steps to Reproduce:
1. Log into Katello and setup a new product
2. Click repo discovery and use this URL http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/
3. Select OS (or any of the other 3

Actual results: All packages are successfully synced, but no errata data is pulled in

Expected results: Packages & errata data are seen in katello

Additional info: http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/os/ is the single mirror that can be used.

Also on a side note, can you import errata data manually or manually modify a pulp sync to provide it an alternative location for errata data?

+ This bug was cloned from Bugzilla Bug #1172887 +

Actions
Copy link
 #1

Updated by bcourt over 9 years ago

The problem was that the mirror uses bz2 compression for the metadata files instead of the standard gzip. PR is to add support for bz2.

https://github.com/pulp/pulp_rpm/pull/627

+ This comment was cloned from Bugzilla #1172887 comment 1 +

Actions
Copy link
 #2

Updated by bmbouter over 9 years ago

  • Severity changed from High to 3. High
Actions
Copy link
 #3

Updated by bcourt over 9 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #5

Updated by pthomas@redhat.com over 9 years ago

  • Status changed from 5 to ASSIGNED

Fails-qa
[root@qe-blade-10 ~]# rpm -qa pulp-server
pulp-server-2.6.1-0.2.beta.el6.noarch
[root@qe-blade-10 ~]#

Importing errata seems to be failing still

[root@qe-blade-10 ~]# pulp-admin rpm repo create --repo-id scientific --feed http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

Successfully created repository [scientific]

[root@qe-blade-10 ~]# 
[root@qe-blade-10 ~]# 
[root@qe-blade-10 ~]# pulp-admin rpm repo sync run --repo-id scientific 
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

+----------------------------------------------------------------------+
                 Synchronizing Repository [scientific]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... failed

Not Found

Task Failed

Importer indicated a failed response

Mar 31 05:57:09 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
Mar 31 05:57:31 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/os/
Mar 31 05:57:31 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
Mar 31 05:57:31 qe-blade-10 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): mirror.aarnet.edu.au
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280) sync failed
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280) Traceback (most recent call last):
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 135, in run
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     self.get_errata(metadata_files)
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 483, in get_errata
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     updateinfo.process_package_element, additive_type=True)
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 563, in save_fileless_units
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     for model in package_info_generator:
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 63, in package_list_generator
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     package_info = process_func(element)
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/updateinfo.py", line 27, in process_package_element
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280)     'description': element.find('description').text,
Mar 31 05:57:41 qe-blade-10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27703-77280) AttributeError: 'NoneType' object has no attribute 'text'
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280) Task pulp.server.managers.repo.sync.sync[f7d421b1-1149-444e-9d54-356f501f938d] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280) Traceback (most recent call last):
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)     R = retval = fun(*args, **kwargs)
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)     return super(Task, self).__call__(*args, **kwargs)
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)     return self.run(*args, **kwargs)
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280)     raise PulpExecutionException(_('Importer indicated a failed response'))
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:ERROR: (27685-77280) PulpExecutionException: Importer indicated a failed response
Mar 31 05:57:41 qe-blade-10 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[ec62b7e5-5602-442f-be3d-d19049efede7] succeeded in 0.0082434789947s: None

<\pre>
Actions
Copy link
 #6

Updated by bcourt over 9 years ago

  • Status changed from ASSIGNED to POST

Added by bcourt over 9 years ago

Revision c53af27a | View on GitHub

Fix import of scientific linux erratum that don't contain a description field.

fixes #635

Actions
Copy link
 #7

Updated by bcourt over 9 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #8

Updated by bcourt over 9 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #9

Updated by pthomas@redhat.com over 9 years ago

  • Status changed from 5 to ASSIGNED

still failing to sync scientific linux

[root@qe-blade-11 ~]# pulp-admin rpm repo create --repo-id scientific --feed http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

Successfully created repository [scientific]

[root@qe-blade-11 ~]# pulp-admin rpm repo sync run --repo-id scientific
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

+----------------------------------------------------------------------+
                 Synchronizing Repository [scientific]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... failed

Not Found

Task Failed

Importer indicated a failed response

[root@qe-blade-11 ~]# tail -f /var/log/messages
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     R = retval = fun(*args, **kwargs)
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     return super(Task, self).__call__(*args, **kwargs)
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     return self.run(*args, **kwargs)
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     raise PulpExecutionException(_('Importer indicated a failed response'))
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632) PulpExecutionException: Importer indicated a failed response
Apr 28 12:42:17 qe-blade-11 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[06a81428-d793-458e-8ac7-15262a2404b7] succeeded in 0.00686540501192s: None
Apr 28 13:01:01 qe-blade-11 systemd: Starting Session 100 of user root.
Apr 28 13:01:01 qe-blade-11 systemd: Started Session 100 of user root.
Apr 28 13:05:44 qe-blade-11 NetworkManager[777]: <error> [1430240744.751764] [devices/nm-device.c:2303] activation_source_schedule(): (enp4s0): activation stage already scheduled
Actions
Copy link
 #10

Updated by pthomas@redhat.com over 9 years ago 


Apr 28 13:36:13 qe-blade-11 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[f17bc5a8-4e34-456c-a863-661272ac042d]
Apr 28 13:36:14 qe-blade-11 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[e42708ea-b1f3-4a2e-b910-c3a64bd49c75]
Apr 28 13:36:14 qe-blade-11 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[10cf4534-037a-4817-ae95-db1565552f8f]
Apr 28 13:36:14 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from http://mirror.aarnet.edu.au/pub/scientific/6.6/x86_64/.
Apr 28 13:36:14 qe-blade-11 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): mirror.aarnet.edu.au
Apr 28 13:36:14 qe-blade-11 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[f17bc5a8-4e34-456c-a863-661272ac042d] succeeded in 0.0333014460048s: None
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632) sync failed
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632) Traceback (most recent call last):
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 105, in run
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632)     metadata_files = self.get_metadata()
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 199, in get_metadata
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632)     raise FailedException(str(e))
Apr 28 13:36:15 qe-blade-11 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25280-69632) FailedException: Not Found
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632) Task pulp.server.managers.repo.sync.sync[e42708ea-b1f3-4a2e-b910-c3a64bd49c75] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632) Traceback (most recent call last):
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     R = retval = fun(*args, **kwargs)
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     return super(Task, self).__call__(*args, **kwargs)
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     return self.run(*args, **kwargs)
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632)     raise PulpExecutionException(_('Importer indicated a failed response'))
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:ERROR: (25170-69632) PulpExecutionException: Importer indicated a failed response
Apr 28 13:36:15 qe-blade-11 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[10cf4534-037a-4817-ae95-db1565552f8f] succeeded in 0.00619069201639s: None
Actions
Copy link
 #11

Updated by bcourt over 9 years ago

  • Platform Release changed from 2.6.1 to 2.6.2
Actions
Copy link
 #12

Updated by bcourt over 9 years ago

  • Status changed from ASSIGNED to 5
  • Platform Release changed from 2.6.2 to 2.6.1

It appears that this was tested using a bad repository feed url. Closer mirrors that could be used for this are:

pulp-admin rpm repo create --repo-id sci-security --feed http://www.gtlib.gatech.edu/pub/scientific/7/x86_64/updates/security/
or
pulp-admin rpm repo create --repo-id sci-security --feed http://www.gtlib.gatech.edu/pub/scientific/7/x86_64/os/

Actions
Copy link
 #13

Updated by pthomas@redhat.com over 9 years ago

  • Status changed from 5 to 6

verified

[root@qe-blade-13 ~]# pulp-admin rpm repo sync run --repo-id sci-security
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

+----------------------------------------------------------------------+
                Synchronizing Repository [sci-security]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[/]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       8685/8685 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[=========================================         ] 83%
Distributions: 5/6 items

Errors encountered during distribution sync:

File:
http://www.gtlib.gatech.edu/pub/scientific/7/x86_64/os/images/boot.iso
Error
Code:   None
Error Message: None

Importing errata...
[-]
... completed

Importing package groups/categories...
[\]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
8685 of 8685 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
3021 of 3021 items
... completed

Publishing Comps file
[==================================================] 100%
105 of 105 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[\]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

<\pre>
Actions
Copy link
 #14

Updated by dkliban@redhat.com about 9 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #20

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF