RPM Support

Related to or duplicate of https://pulp.plan.io/issues/2104 ?

I don't think it is a duplicate but the error in the example is the same one as #2104

Here is a passing command that still would lack the mirror configuration but completes:

[vagrant@dev ~]$ pulp-admin -vv rpm repo create --repo-id mirror --feed "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os" --relative-url foobar
2016-09-01 20:55:31,332 - DEBUG - sending POST request to /pulp/api/v2/repositories/
2016-09-01 20:55:33,224 - INFO - POST request to /pulp/api/v2/repositories/ with parameters {"display_name": null, "description": null, "distributors": [{"distributor_id": "yum_distributor", "auto_publish": true, "distributor_config": {"http": false, "relative_url": "foobar", "https": true}, "distributor_type_id": "yum_distributor"}, {"distributor_id": "export_distributor", "auto_publish": false, "distributor_config": {"http": false, "relative_url": "foobar", "https": true}, "distributor_type_id": "export_distributor"}], "notes": {"_repo-type": "rpm-repo"}, "importer_type_id": "yum_importer", "importer_config": {"feed": "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os"}, "id": "mirror"}
2016-09-01 20:55:33,224 - INFO - Response status : 201

2016-09-01 20:55:33,225 - INFO - Response body :
 {
  "scratchpad": {},
  "display_name": null,
  "description": null,
  "last_unit_added": null,
  "notes": {
    "_repo-type": "rpm-repo"
  },
  "last_unit_removed": null,
  "content_unit_counts": {},
  "_ns": "repos",
  "_id": {
    "$oid": "57c895c5e779890c82b2baac"
  },
  "id": "mirror",
  "_href": "/pulp/api/v2/repositories/mirror/"
}

Successfully created repository [mirror]

Brian, you are right - providing relative-url does allow to create the repo, but further sync with mirrorlist provided is not correct.

$ pulp-admin rpm repo sync run --repo-id=mirror --force-full

+----------------------------------------------------------------------+
                   Synchronizing Repository [mirror]
+----------------------------------------------------------------------+

The following sync configuration options will be used:

Force Full:  True

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items

Errors encountered during distribution sync:

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

I'm afraid this change broke the mirrorlist sync support https://github.com/pulp/pulp_rpm/commit/221e97797f093191101658114b48c49a45b2eeb5#diff-5185b3f1e5a02c40a5d198fe779c75fbL328

Perhaps mirrorlist support is broken on master. Is there a pulp smash issue on this? Also is this part of the manual testing test plan?

In order to make mirrorlists work, then issue mentioned in comment #4 should be fixed( it is only on master).

W/r to 2.10-dev, issue #2104 prevents from creating repo with feed in general, if using the workaround of providing the --relative-url, then repo is created and regularly synced from a mirrorlist.

@pthomas, first of all you need to take in quotes the mirrorlist you provide in the feed because of ?& signs that are treated differently in the terminal.

@preethi: does this work in 2.9.2?

I tested this in 2.10 beta and 2.9.2 and both tests passed

[root@cloud-qe-05 ~]# rpm -qa pulp-server
pulp-server-2.10.0-0.4.beta.el7.noarch
[root@cloud-qe-05 ~]# 

[root@cloud-qe-05 ~]# pulp-admin -vv rpm repo create --repo-id mirror --feed "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os"  --relative-url mirror
2016-09-02 13:42:13,332 - DEBUG - sending POST request to /pulp/api/v2/repositories/
2016-09-02 13:42:13,621 - INFO - POST request to /pulp/api/v2/repositories/ with parameters {"display_name": null, "description": null, "distributors": [{"distributor_id": "yum_distributor", "auto_publish": true, "distributor_config": {"http": false, "relative_url": "mirror", "https": true}, "distributor_type_id": "yum_distributor"}, {"distributor_id": "export_distributor", "auto_publish": false, "distributor_config": {"http": false, "relative_url": "mirror", "https": true}, "distributor_type_id": "export_distributor"}], "notes": {"_repo-type": "rpm-repo"}, "importer_type_id": "yum_importer", "importer_config": {"feed": "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os"}, "id": "mirror"}
2016-09-02 13:42:13,621 - INFO - Response status : 201 

2016-09-02 13:42:13,621 - INFO - Response body :
 {
  "scratchpad": {}, 
  "display_name": null, 
  "description": null, 
  "last_unit_added": null, 
  "notes": {
    "_repo-type": "rpm-repo"
  }, 
  "last_unit_removed": null, 
  "content_unit_counts": {}, 
  "_ns": "repos", 
  "_id": {
    "$oid": "57c9b9f512783b7a33c5853b"
  }, 
  "id": "mirror", 
  "_href": "/pulp/api/v2/repositories/mirror/"
}

Successfully created repository [mirror]

$ pulp-admin rpm repo sync run --repo-id=mirror
+----------------------------------------------------------------------+
                   Synchronizing Repository [mirror]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       9007/9007 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[=================================                 ] 66%
Distributions: 4/6 items

pulp-admin -vv rpm repo create --repo-id mirror --feed http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
[1] 3399
[2] 3400
[2]+  Done                    arch=x86_64

$ 2016-09-02 11:16:30,437 - DEBUG - sending POST request to /pulp/api/v2/repositories/
2016-09-02 11:16:30,721 - INFO - POST request to /pulp/api/v2/repositories/ with parameters {"display_name": null, "description": null, "distributors": [{"distributor_id": "yum_distributor", "auto_publish": true, "distributor_config": {"http": false, "relative_url": "/mirror", "https": true}, "distributor_type_id": "yum_distributor"}, {"distributor_id": "export_distributor", "auto_publish": false, "distributor_config": {"http": false, "relative_url": "/mirror", "https": true}, "distributor_type_id": "export_distributor"}], "notes": {"_repo-type": "rpm-repo"}, "importer_type_id": "yum_importer", "importer_config": {"feed": "http://mirrorlist.centos.org/?release=7"}, "id": "mirror"}
2016-09-02 11:16:30,721 - INFO - Response status : 400 

2016-09-02 11:16:30,722 - INFO - Response body :
 {
  "exception": null, 
  "args": [
    "Value for [relative_url]  must be be a relative path: /mirror"
  ], 
  "_href": "/pulp/api/v2/repositories/", 
  "traceback": null, 
  "error_message": "Value for [relative_url]  must be be a relative path: /mirror", 
  "http_request_method": "POST", 
  "http_status": 400, 
  "error": {
    "code": "PLP0000", 
    "data": {}, 
    "description": "Value for [relative_url]  must be be a relative path: /mirror", 
    "sub_errors": []
  }
}

2016-09-02 11:16:30,722 - ERROR - Exception occurred:
        href:      /pulp/api/v2/repositories/
        method:    POST
        status:    400
        error:     Value for [relative_url]  must be be a relative path: /mirror
        traceback: None
        data:      {u'args': [u'Value for [relative_url]  must be be a relative path: /mirror'], u'error': {u'code': u'PLP0000', u'data': {}, u'description': u'Value for [relative_url]  must be be a relative path: /mirror', u'sub_errors': []}}

Value for [relative_url]  must be be a relative path: /mirror

The second test, listed above, doesn't correctly use quotes:

pulp-admin -vv rpm repo create --repo-id mirror --feed http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
[1] 3399
[2] 3400
[2]+  Done                    arch=x86_64

The test also ends with an HTTP 400. I think this needs to be re-tested, or have an automated test written.

In my opinion, this bug is not suitable for automated testing given Pulp's current capabilities. See: https://github.com/PulpQE/pulp-smash/issues/384#issuecomment-245026519

I am testing that all of these feed URLs should work for syncing RPMs:

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-24&arch=x86_64
https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/

Note that the centos and fedora mirror servers behave differently. The fedora one does not return a 404 when "repodata/repomd.xml" is added to the path. Both should be tested, because pulp has to handle them differently.

This URL should work for syncing just a distribution (correcting thanks to ttereshc):

https://ci.centos.org/artifacts/sig-atomic/centos-continuous/images-alpha/installer/latest/images/

And then pick your favorite bad URL to test the "Not Found" case.

Correct URL to sync a distribution: https://ci.centos.org/artifacts/sig-atomic/centos-continuous/images-alpha/installer/latest/images/