Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #609

closed

Repo sync fails with Importer indicated a failed response error when you using a proxy

Added by pthomas@redhat.com about 9 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
cduryee
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.5
Platform Release:
2.6.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Description of problem:
When you have a repo with proxy, repo sync seems to be failing

Version-Release number of selected component (if applicable):
pulp-server-2.5.0-0.16.rc.el7.noarch

How reproducible:

Steps to Reproduce:
1.create a repo with proxy
2. Repo sync
3.

Actual results:
[root@cloud-qe-19 ~]# pulp-admin rpm repo create --repo-id pulp-unittest --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/ --proxy-host http://pulp-proxy.usersys.redhat.com --proxy-port 8080 --proxy-user admin --proxy-pass admin
Successfully created repository [pulp-unittest]

[root@cloud-qe-19 ~]#
[root@cloud-qe-19 ~]#
[root@cloud-qe-19 ~]# pulp-admin rpm repo sync run --repo-id pulp-unittest+----------------------------------------------------------------------+
Synchronizing Repository [pulp-unittest]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... failed

[Errno 2] No such file or directory:
u'/var/lib/pulp/working/repos/pulp-unittest/importers/yum_importer/tmpkROymL/oth
er.xml.gz'

Task Failed

Importer indicated a failed respons

Expected results:

Additional info:

Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: sync failed
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: Traceback (most recent call last):
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 104, in run
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: metadata_files = self.get_metadata()
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 211, in get_metadata
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: metadata_files.generate_dbs()
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py", line 267, in generate_dbs
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: xml_file_handle = self.get_metadata_file_handle(filename)
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py", line 238, in get_metadata_file_handle
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: file_handle = gzip.open(file_path, 'r')
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib64/python2.7/gzip.py", line 34, in open
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: return GzipFile(filename, mode, compresslevel)
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib64/python2.7/gzip.py", line 94, in init
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: fileobj = self.myfileobj = builtin.open(filename, mode or 'rb')
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: IOError: [Errno 2] No such file or directory: u'/var/lib/pulp/working/repos/pulp-unittest/importers/yum_importer/tmpkROymL/other.xml.gz'
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: Task pulp.server.tasks.repository.sync_with_auto_publish[0a8d71ea-48a9-48e9-9c4e-dc3ba7ecc898] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: Traceback (most recent call last):
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: R = retval = fun(*args, **kwargs)
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 315, in call
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: return super(Task, self).__call__(*args, **kwargs)
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in protected_call
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: return self.run(*args, **kwargs)
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/tasks/repository.py", line 210, in sync_with_auto_publish
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: sync_result = managers.repo_sync_manager().sync(repo_id, sync_config_override=overrides)
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 113, in sync
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: raise PulpExecutionException(_('Importer indicated a failed response'))
Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: PulpExecutionException: Importer indicated a failed response

+ This bug was cloned from Bugzilla Bug #1160794 +

Actions
Copy link
 #1

Updated by cduryee about 9 years ago

adding '--max-downloads 15' will work around this issue. There appears to be some problem when the same downloader is used more than once.

+ This comment was cloned from Bugzilla #1160794 comment 1 +

Actions
Copy link
 #2

Updated by cduryee about 9 years ago

Can you retry using a different ssl-enabled repo? https://mirror.umd.edu/fedora/linux/releases/20/Fedora/x86_64/os/ worked ok for me

+ This comment was cloned from Bugzilla #1160794 comment 2 +

Actions
Copy link
 #3

Updated by cduryee about 9 years ago

I suspect something may be going on with repos.fedorapeople.org. I was unable to sync the pulp_unittest repo, the pulp repo, or the katello repo via a proxy. However I was able to sync fedora 20 from the URL above ok, and was also able to sync from the CDN via a proxy.

Additionally, I was able to repro the behavior outlined in comment #0 in both pulp 2.5 and 2.4.0 (which is known to work in the past with fedorapeople.org). I used both tinyproxy and squid and did not notice any behavior difference between the two.

I see some stuff that might or might not be unusual in the tcpdump output when the proxy connects to repos.fedorapeople.org. I'll find another place to host the pulp_unittest repo tomorrow and we'll see if it works from there. If it does I'll follow up with the fedora infra team to see if something is different about their servers.

+ This comment was cloned from Bugzilla #1160794 comment 3 +

Actions
Copy link
 #4

Updated by cduryee about 9 years ago

I was able to sync from this repo without issue via a proxy:

https://cduryee-repotest.rhcloud.com/static/pulp_unittest/

I think something is different (possibly recently changed) about repos.fedorapeople.org that is causing the issue. It is certainly worth investigation and may possibly be a Pulp bug since I can't repro with curl, but I don't think it should block 2.5.0.

Is it ok to unassign the target release from this bz?

+ This comment was cloned from Bugzilla #1160794 comment 4 +

Actions
Copy link
 #5

Updated by cduryee about 9 years ago

moving off 2.5.0 per mhrivnak

+ This comment was cloned from Bugzilla #1160794 comment 5 +

Actions
Copy link
 #6

Updated by cduryee about 9 years ago

update:

I am able to repro this behavior with a small script using python-requests 2.2.1, but not 2.4.3. I suspect this is a bug that got fixed since 2.2.1.

I did a "pip install --upgrade requests" and Pulp is able to sync the repo again. I think upgrading the dep to 2.4.3 will fix this bz.

Moving to 2.5.1.

+ This comment was cloned from Bugzilla #1160794 comment 6 +

Actions
Copy link
 #8

Updated by cduryee about 9 years ago

pulp/deps change merged to 2.5-dev and master.

I did not merge the nectar change after discussion with rbarlow.

+ This comment was cloned from Bugzilla #1160794 comment 8 +

Actions
Copy link
 #9

Updated by cduryee about 9 years ago

fixed in pulp 2.6.0-0.2.beta

+ This comment was cloned from Bugzilla #1160794 comment 9 +

Actions
Copy link
 #10

Updated by pthomas@redhat.com about 9 years ago

[root@cloud-qe-4 ~]# rpm -qa python-requests
python-requests-2.4.3-1.el7.noarch
[root@cloud-qe-4 ~]# rpm -qa pulp-server
pulp-server-2.6.0-0.5.beta.el7.noarch
[root@cloud-qe-4 ~]#

[root@cloud-qe-4 ~]# pulp-admin rpm repo create --repo-id pulp-unittest --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/ --proxy-host http://ibm-x3550m3-11.lab.eng.brq.redhat.com --proxy-port 3128
Successfully created repository [pulp-unittest]

[root@cloud-qe-4 ~]#
[root@cloud-qe-4 ~]#
[root@cloud-qe-4 ~]# pulp-admin rpm repo sync run --repo-id pulp-unittest
--------------------------------------------------------------------
Synchronizing Repository [pulp-unittest]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs: 3/3 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 3/3 items
... completed

Importing errata...
[\]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
3 of 3 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
52 of 52 items
... completed

Publishing Comps file
[==================================================] 100%
5 of 5 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

+ This comment was cloned from Bugzilla #1160794 comment 10 +

Actions
Copy link
 #11

Updated by pmoravec@redhat.com about 9 years ago

(In reply to Chris Duryee from comment #1)

adding '--max-downloads 15' will work around this issue. There appears to be
some problem when the same downloader is used more than once.

Assuming I want to create a repository via hammer or WebUI for Sat6, what (other) pulp-admin options to use for the workaround? Here is the list of options:

  1. pulp-admin rpm repo create --help | awk '{ print $1 }' | grep "^--"
    --repo-id
    --display-name
    --description
    --note
    --feed
    --validate
    --skip
    --feed-ca-cert
    --verify-feed-ssl
    --feed-cert
    --feed-key
    --proxy-host
    --proxy-port
    --proxy-user
    --proxy-pass
    --max-downloads
    --max-speed
    --remove-missing
    --retain-old-count
    --relative-url
    --serve-http
    --serve-https
    --checksum-type
    --gpg-key
    --generate-sqlite
    --host-ca
    --auth-ca
    --auth-cert
    #

+ This comment was cloned from Bugzilla #1160794 comment 11 +

Actions
Copy link
 #12

Updated by cduryee about 9 years ago

Pavel,

I would confirm first that this is indeed the issue since the stack trace it gives is a bit generic.

There is a small script in the commit message for https://github.com/pulp/pulp/pull/1290 that you can use. If you update the proxy_url and base_url and then run the script on the machine in question, you can see if this BZ is indeed the issue.

I am not sure how to set --max-downloads via hammer or the web UI. We can investigate that but it may be easier to just update the nectar RPM on the system.

+ This comment was cloned from Bugzilla #1160794 comment 12 +

Actions
Copy link
 #13

Updated by bmbouter about 9 years ago

  • Severity changed from High to 3. High
Actions
Copy link
 #14

Updated by rbarlow almost 9 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #16

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF