Project

Profile

Help

Issue #2224

Cannot sync from mirrorlists

Added by ipanova@redhat.com about 5 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.11.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 8
Quarter:

Description

$ pulp-admin -vv rpm repo create --repo-id mirror --feed "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os"  --relative-url mirror

2016-09-02 18:47:26,295 - DEBUG - sending POST request to /pulp/api/v2/repositories/
2016-09-02 18:47:27,464 - INFO - POST request to /pulp/api/v2/repositories/ with parameters {"display_name": null, "description": null, "distributors": [{"distributor_id": "yum_distributor", "auto_publish": true, "distributor_config": {"http": false, "relative_url": "mirror", "https": true}, "distributor_type_id": "yum_distributor"}, {"distributor_id": "export_distributor", "auto_publish": false, "distributor_config": {"http": false, "relative_url": "mirror", "https": true}, "distributor_type_id": "export_distributor"}], "notes": {"_repo-type": "rpm-repo"}, "importer_type_id": "yum_importer", "importer_config": {"feed": "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os"}, "id": "mirror"}
2016-09-02 18:47:27,464 - INFO - Response status : 201 

2016-09-02 18:47:27,464 - INFO - Response body :
 {
  "scratchpad": {}, 
  "display_name": null, 
  "description": null, 
  "last_unit_added": null, 
  "notes": {
    "_repo-type": "rpm-repo"
  }, 
  "last_unit_removed": null, 
  "content_unit_counts": {}, 
  "_ns": "repos", 
  "_id": {
    "$oid": "57c9ad1f45ef480c8e715874"
  }, 
  "id": "mirror", 
  "_href": "/pulp/api/v2/repositories/mirror/"
}

Successfully created repository [mirror]

$ pulp-admin rpm repo sync run --repo-id=mirror

+----------------------------------------------------------------------+
                   Synchronizing Repository [mirror]
+----------------------------------------------------------------------+

The following sync configuration options will be used:

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items

Errors encountered during distribution sync:

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

I'm afraid this change broke the mirrorlist sync support https://github.com/pulp/pulp_rpm/commit/221e97797f093191101658114b48c49a45b2eeb5#diff-5185b3f1e5a02c40a5d198fe779c75fbL328

Associated revisions

Revision 9c4e6677 View on GitHub
Added by mhrivnak about 5 years ago

Fixes a bug that broke mirrorlist sync support.

https://pulp.plan.io/issues/2224 fixes #2224

History

#1 Updated by ipanova@redhat.com about 5 years ago

  • Project changed from Pulp to RPM Support

#2 Updated by ttereshc about 5 years ago

Related to or duplicate of https://pulp.plan.io/issues/2104 ?

#3 Updated by bmbouter about 5 years ago

I don't think it is a duplicate but the error in the example is the same one as #2104

Here is a passing command that still would lack the mirror configuration but completes:

[vagrant@dev ~]$ pulp-admin -vv rpm repo create --repo-id mirror --feed "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os" --relative-url foobar
2016-09-01 20:55:31,332 - DEBUG - sending POST request to /pulp/api/v2/repositories/
2016-09-01 20:55:33,224 - INFO - POST request to /pulp/api/v2/repositories/ with parameters {"display_name": null, "description": null, "distributors": [{"distributor_id": "yum_distributor", "auto_publish": true, "distributor_config": {"http": false, "relative_url": "foobar", "https": true}, "distributor_type_id": "yum_distributor"}, {"distributor_id": "export_distributor", "auto_publish": false, "distributor_config": {"http": false, "relative_url": "foobar", "https": true}, "distributor_type_id": "export_distributor"}], "notes": {"_repo-type": "rpm-repo"}, "importer_type_id": "yum_importer", "importer_config": {"feed": "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os"}, "id": "mirror"}
2016-09-01 20:55:33,224 - INFO - Response status : 201

2016-09-01 20:55:33,225 - INFO - Response body :
 {
  "scratchpad": {},
  "display_name": null,
  "description": null,
  "last_unit_added": null,
  "notes": {
    "_repo-type": "rpm-repo"
  },
  "last_unit_removed": null,
  "content_unit_counts": {},
  "_ns": "repos",
  "_id": {
    "$oid": "57c895c5e779890c82b2baac"
  },
  "id": "mirror",
  "_href": "/pulp/api/v2/repositories/mirror/"
}

Successfully created repository [mirror]

#4 Updated by ipanova@redhat.com about 5 years ago

Brian, you are right - providing relative-url does allow to create the repo, but further sync with mirrorlist provided is not correct.

$ pulp-admin rpm repo sync run --repo-id=mirror --force-full

+----------------------------------------------------------------------+
                   Synchronizing Repository [mirror]
+----------------------------------------------------------------------+

The following sync configuration options will be used:

Force Full:  True

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items

Errors encountered during distribution sync:

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

I'm afraid this change broke the mirrorlist sync support https://github.com/pulp/pulp_rpm/commit/221e97797f093191101658114b48c49a45b2eeb5#diff-5185b3f1e5a02c40a5d198fe779c75fbL328

#5 Updated by bmbouter about 5 years ago

Perhaps mirrorlist support is broken on master. Is there a pulp smash issue on this? Also is this part of the manual testing test plan?

#6 Updated by amacdona@redhat.com about 5 years ago

  • Priority changed from Normal to Urgent
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes

#7 Updated by ipanova@redhat.com about 5 years ago

In order to make mirrorlists work, then issue mentioned in comment #4 should be fixed( it is only on master).

W/r to 2.10-dev, issue #2104 prevents from creating repo with feed in general, if using the workaround of providing the --relative-url, then repo is created and regularly synced from a mirrorlist.

#8 Updated by ipanova@redhat.com about 5 years ago

@pthomas, first of all you need to take in quotes the mirrorlist you provide in the feed because of ?& signs that are treated differently in the terminal.

#9 Updated by bmbouter about 5 years ago

@preethi: does this work in 2.9.2?

#10 Updated by ipanova@redhat.com about 5 years ago

  • Subject changed from Relative_url validation does not allow to create a repo with mirrorlists to Cannot sync from mirrorlists

#11 Updated by ipanova@redhat.com about 5 years ago

  • Subject changed from Cannot sync from mirrorlists to Relative_url validation does not allow to create a repo with mirrorlists
  • Description updated (diff)

#12 Updated by ipanova@redhat.com about 5 years ago

  • Subject changed from Relative_url validation does not allow to create a repo with mirrorlists to Cannot sync from mirrorlists

#13 Updated by pthomas@redhat.com about 5 years ago

I tested this in 2.10 beta and 2.9.2 and both tests passed


[root@cloud-qe-05 ~]# rpm -qa pulp-server
pulp-server-2.10.0-0.4.beta.el7.noarch
[root@cloud-qe-05 ~]# 

[root@cloud-qe-05 ~]# pulp-admin -vv rpm repo create --repo-id mirror --feed "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os"  --relative-url mirror
2016-09-02 13:42:13,332 - DEBUG - sending POST request to /pulp/api/v2/repositories/
2016-09-02 13:42:13,621 - INFO - POST request to /pulp/api/v2/repositories/ with parameters {"display_name": null, "description": null, "distributors": [{"distributor_id": "yum_distributor", "auto_publish": true, "distributor_config": {"http": false, "relative_url": "mirror", "https": true}, "distributor_type_id": "yum_distributor"}, {"distributor_id": "export_distributor", "auto_publish": false, "distributor_config": {"http": false, "relative_url": "mirror", "https": true}, "distributor_type_id": "export_distributor"}], "notes": {"_repo-type": "rpm-repo"}, "importer_type_id": "yum_importer", "importer_config": {"feed": "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os"}, "id": "mirror"}
2016-09-02 13:42:13,621 - INFO - Response status : 201 

2016-09-02 13:42:13,621 - INFO - Response body :
 {
  "scratchpad": {}, 
  "display_name": null, 
  "description": null, 
  "last_unit_added": null, 
  "notes": {
    "_repo-type": "rpm-repo"
  }, 
  "last_unit_removed": null, 
  "content_unit_counts": {}, 
  "_ns": "repos", 
  "_id": {
    "$oid": "57c9b9f512783b7a33c5853b"
  }, 
  "id": "mirror", 
  "_href": "/pulp/api/v2/repositories/mirror/"
}

Successfully created repository [mirror]

$ pulp-admin rpm repo sync run --repo-id=mirror
+----------------------------------------------------------------------+
                   Synchronizing Repository [mirror]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       9007/9007 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[=================================                 ] 66%
Distributions: 4/6 items

pulp-admin -vv rpm repo create --repo-id mirror --feed http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
[1] 3399
[2] 3400
[2]+  Done                    arch=x86_64

$ 2016-09-02 11:16:30,437 - DEBUG - sending POST request to /pulp/api/v2/repositories/
2016-09-02 11:16:30,721 - INFO - POST request to /pulp/api/v2/repositories/ with parameters {"display_name": null, "description": null, "distributors": [{"distributor_id": "yum_distributor", "auto_publish": true, "distributor_config": {"http": false, "relative_url": "/mirror", "https": true}, "distributor_type_id": "yum_distributor"}, {"distributor_id": "export_distributor", "auto_publish": false, "distributor_config": {"http": false, "relative_url": "/mirror", "https": true}, "distributor_type_id": "export_distributor"}], "notes": {"_repo-type": "rpm-repo"}, "importer_type_id": "yum_importer", "importer_config": {"feed": "http://mirrorlist.centos.org/?release=7"}, "id": "mirror"}
2016-09-02 11:16:30,721 - INFO - Response status : 400 

2016-09-02 11:16:30,722 - INFO - Response body :
 {
  "exception": null, 
  "args": [
    "Value for [relative_url]  must be be a relative path: /mirror"
  ], 
  "_href": "/pulp/api/v2/repositories/", 
  "traceback": null, 
  "error_message": "Value for [relative_url]  must be be a relative path: /mirror", 
  "http_request_method": "POST", 
  "http_status": 400, 
  "error": {
    "code": "PLP0000", 
    "data": {}, 
    "description": "Value for [relative_url]  must be be a relative path: /mirror", 
    "sub_errors": []
  }
}

2016-09-02 11:16:30,722 - ERROR - Exception occurred:
        href:      /pulp/api/v2/repositories/
        method:    POST
        status:    400
        error:     Value for [relative_url]  must be be a relative path: /mirror
        traceback: None
        data:      {u'args': [u'Value for [relative_url]  must be be a relative path: /mirror'], u'error': {u'code': u'PLP0000', u'data': {}, u'description': u'Value for [relative_url]  must be be a relative path: /mirror', u'sub_errors': []}}

Value for [relative_url]  must be be a relative path: /mirror

#14 Updated by Ichimonji10 about 5 years ago

The second test, listed above, doesn't correctly use quotes:

pulp-admin -vv rpm repo create --repo-id mirror --feed http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
[1] 3399
[2] 3400
[2]+  Done                    arch=x86_64

The test also ends with an HTTP 400. I think this needs to be re-tested, or have an automated test written.

#15 Updated by Ichimonji10 about 5 years ago

In my opinion, this bug is not suitable for automated testing given Pulp's current capabilities. See: https://github.com/PulpQE/pulp-smash/issues/384#issuecomment-245026519

#16 Updated by semyers about 5 years ago

  • Version set to Master

#17 Updated by ttereshc about 5 years ago

  • Sprint/Milestone set to 26

#18 Updated by mhrivnak about 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak

#19 Updated by mhrivnak about 5 years ago

I am testing that all of these feed URLs should work for syncing RPMs:

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-24&arch=x86_64
https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/

Note that the centos and fedora mirror servers behave differently. The fedora one does not return a 404 when "repodata/repomd.xml" is added to the path. Both should be tested, because pulp has to handle them differently.

This URL should work for syncing just a distribution (correcting thanks to ttereshc):

https://ci.centos.org/artifacts/sig-atomic/centos-continuous/images-alpha/installer/latest/images/

And then pick your favorite bad URL to test the "Not Found" case.

#20 Updated by mhrivnak about 5 years ago

  • Status changed from ASSIGNED to POST

#22 Updated by mhrivnak about 5 years ago

  • Status changed from POST to MODIFIED

#23 Updated by mhrivnak about 5 years ago

  • Platform Release set to 2.11.0

#25 Updated by semyers almost 5 years ago

  • Status changed from MODIFIED to 5

#26 Updated by pthomas@redhat.com almost 5 years ago

  • Status changed from 5 to 6

Tested all the mirrorlist url listed in #19

<verified>
[root@ibm-x3650m4-02-vm-02 ~]# rpm -qa |grep pulp
python-pulp-rpm-common-2.11.0-0.5.rc.el6.noarch
pulp-rpm-plugins-2.11.0-0.5.rc.el6.noarch
python-isodate-0.5.0-4.pulp.el6.noarch
pulp-python-plugins-1.1.3-1.el6.noarch
python-pulp-docker-common-2.2.0-0.3.rc.el6.noarch
pulp-rpm-admin-extensions-2.11.0-0.5.rc.el6.noarch
mod_wsgi-3.4-2.pulp.el6.x86_64
python-kombu-3.0.33-6.pulp.el6.noarch
python-pulp-common-2.11.0-0.5.rc.el6.noarch
python-pulp-repoauth-2.11.0-0.5.rc.el6.noarch
python-pulp-client-lib-2.11.0-0.5.rc.el6.noarch
pulp-server-2.11.0-0.5.rc.el6.noarch
pulp-puppet-plugins-2.11.0-0.5.rc.el6.noarch
pulp-puppet-admin-extensions-2.11.0-0.5.rc.el6.noarch
python-pulp-puppet-common-2.11.0-0.5.rc.el6.noarch
python-pulp-bindings-2.11.0-0.5.rc.el6.noarch
pulp-docker-plugins-2.2.0-0.3.rc.el6.noarch
pulp-python-admin-extensions-1.1.3-1.el6.noarch
pulp-admin-client-2.11.0-0.5.rc.el6.noarch
pulp-docker-admin-extensions-2.2.0-0.3.rc.el6.noarch
pulp-selinux-2.11.0-0.5.rc.el6.noarch
python-pulp-oid_validation-2.11.0-0.5.rc.el6.noarch
python-pulp-streamer-2.11.0-0.5.rc.el6.noarch
python-pulp-python-common-1.1.3-1.el6.noarch
[root@ibm-x3650m4-02-vm-02 ~]#
[root@mgmt13 ~]# pulp-admin rpm repo create --repo-id mirror2 --feed https://ci.centos.org/artifacts/sig-atomic/centos-continuous/images-alpha/installer/latest/images/
Successfully created repository [mirror2]

[root@mgmt13 ~]# pulp-admin rpm repo sync run --repo-id mirror2
--------------------------------------------------------------------
Synchronizing Repository [mirror2]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 5/5 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[-]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[-]
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Generating HTML files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

[root@mgmt13 ~]# pulp-admin rpm repo create --repo-id mirror3 --feed "http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-24\&arch=x86_64"
Successfully created repository [mirror3]

[root@mgmt13 ~]# pulp-admin rpm repo sync run --repo-id mirror3
--------------------------------------------------------------------
Synchronizing Repository [mirror3]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[/]
... completed

Downloading repository content...
[|]
[=== ] 6%
RPMs: 943/49722 items
Delta RPMs: 0/0 items

#27 Updated by pcreech over 4 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#29 Updated by bmbouter over 3 years ago

  • Sprint set to Sprint 8

#30 Updated by bmbouter over 3 years ago

  • Sprint/Milestone deleted (26)

#31 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF