Project

Profile

Help

Story #3756

Create CLI for Pulp3

Added by bizhang over 2 years ago. Updated 6 months ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

This is the Pulp3 CLI Epic. When the design step is done and has reached consensus, sub-tasks will be created for CLI implementation.

MVP

  • As a user, I have a set of CLI commands that match the REST API for my Pulp server
  • I have parameters for each command that correspond to API resource parameters
  • I also have a CLI filter for every API resource filter
  • I have CLI commands for core and installed plugins
  • CLI commands for plugins that aren’t installed don’t show up
  • As a user, I can configure a file with the Pulp API URI, username, password
  • As a user, I can view all results even if they are paginated
  • As a user, I can view a help screen with help text from the API schema

Implementation

After investigation it was determined that there was no pre-existing solutions that meet all our requirements.
openapi-cli-client [0] is unmaintained and does not function withour API.
coreali-cli [1] was promising, but lacked many of the features we wanted.

We made the following considerations when we investigated how to write our own CLI

Schema based vs Pulp Bindings

A bindings based CLI would be more robust, but would require us to regenerate bindings and make code changes to the CLI everytime there is a REST API change. A bindings based CLI would also mean that plugin writiers needed to write their own CLI libraries. Based off these facts, we made the decision to go with a schema generated CLI.

OpenAPI vs CoreAPI

CoreAPI is only used by django-rest-framework. It is a lot easier to parse, mainly because it isn't as feature rich as openapi. OpenAPI is the de-facto industry standard, and is what we should use for the CLI.
The POC makes use of CoreAPI because it is a lot less complicated, but everything we do there should eventually be converted to use OpenCPI

Argparse vs Click

Click supports all the MVP use cases and is a lot easier to use than argparse. The only advantage argparse has is that argcomplete supports autocomplete hooks.

POC

There is a POC [2] that supports the following features

  • Auto-generated CLI commands with one command for each API endpoint
  • Support for plugins
  • Pagination support
  • Lookup via resource UUIDs or names
  • Help screens
  • Task polling
  • Autocompletion for commands and arguments
  • Extensibility
  • Support in the future to create our own complex commands/workflows
  • Users/plugin writers can create their own commands and ship their own cli plugins

[0] https://github.com/hjacobs/openapi-cli-client
[1] https://github.com/core-api/coreapi-cli
[2] https://github.com/werwty/pulpcli-POC

History

#1 Updated by bizhang over 2 years ago

  • Tags Pulp 3 added

#2 Updated by bizhang over 2 years ago

  • Description updated (diff)

Commands and responses from the POC

Load a new Schema

$ pulp get --url=http://pulp3.dev:8000/pulp/api/v3/?format=corejson
Schema written to /home/vagrant/.pulpcli/document.json

help screen

$ pulp repositories --help
Usage: pulp repositories [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  create
  delete          Generates a Task to delete a Repository
  list
  partial_update
  read
  update          Generates a Task to update a Repository
  versions

Autocomplete

$ pulp --autocomplete
bash completion installed in /home/vagrant/.bash_completion

Create a repository

$ pulp repositories create --name=foo
{
    "_href": "http://pulp3.dev:8000/pulp/api/v3/repositories/c3550bb7-1984-4bd4-bbc4-5bc5483008b5/",
    "_latest_version_href": null,
    "_versions_href": "http://pulp3.dev:8000/pulp/api/v3/repositories/c3550bb7-1984-4bd4-bbc4-5bc5483008b5/versions/",
    "created": "2018-06-08T19:21:48.193530Z",
    "description": "",
    "id": "c3550bb7-1984-4bd4-bbc4-5bc5483008b5",
    "name": "foo",
    "notes": {}
}

List Repositories with Pagination

$ pulp repositories list
{
    "next": "http://pulp3.dev:8000/pulp/api/v3/repositories/?cursor=cD1hc2Zk",
    "previous": null,
    "results": [
        {
            "_href": "http://pulp3.dev:8000/pulp/api/v3/repositories/0d6c48e2-9b68-4221-84db-7fa94d10eb2e/",
            "_latest_version_href": null,
            "_versions_href": "http://pulp3.dev:8000/pulp/api/v3/repositories/0d6c48e2-9b68-4221-84db-7fa94d10eb2e/versions/",
            "created": "2018-06-11T02:25:59.283773Z",
            "description": "",
            "id": "0d6c48e2-9b68-4221-84db-7fa94d10eb2e",
            "name": "asfd",
            "notes": {}
        }
    ]
}

N for next page, P for previous: 

Create a Remote

$ pulp remotes file create --name=bar --url=https://repos.fedorapeople.org/pulp/pulp/demo_repos/test_file_repo/PULP_MANIFEST
{
    "_href": "http://pulp3.dev:8000/pulp/api/v3/remotes/file/396b7eb5-97aa-4394-bc25-ca8be7dd19c4/",
    "created": "2018-06-08T19:24:02.416156Z",
    "id": "396b7eb5-97aa-4394-bc25-ca8be7dd19c4",
    "last_synced": null,
    "last_updated": "2018-06-08T19:24:02.416180Z",
    "name": "bar",
    "proxy_url": "",
    "ssl_validation": true,
    "type": "file",
    "url": "https://repos.fedorapeople.org/pulp/pulp/demo_repos/test_file_repo/PULP_MANIFEST",
    "validate": true
}

Sync Repository foo with Remote bar

$ pulp remotes file sync --repository=http://pulp3.dev:8000/pulp/api/v3/repositories/c3550bb7-1984-4bd4-bbc4-5bc5483008b5/ --id=396b7eb5-97aa-4394-bc25-ca8be7dd19c4
{
    "_href": "http://pulp3.dev:8000/pulp/api/v3/tasks/97ba7d54-4168-4895-942c-4b45c1b20a48/",
    "task_id": "97ba7d54-4168-4895-942c-4b45c1b20a48"
}

Loading -|/
{
    "_href": "http://pulp3.dev:8000/pulp/api/v3/tasks/97ba7d54-4168-4895-942c-4b45c1b20a48/",
    "created": "2018-06-08T19:32:22.081480Z",
    "created_resources": [
        "http://pulp3.dev:8000/pulp/api/v3/repositories/c3550bb7-1984-4bd4-bbc4-5bc5483008b5/versions/1/"
    ],
    "error": null,
    "finished_at": "2018-06-08T19:32:23.307490Z",
    "id": "97ba7d54-4168-4895-942c-4b45c1b20a48",
    "non_fatal_errors": [],
    "parent": null,
    "progress_reports": [
        {
            "done": 3,
            "message": "Add Content",
            "state": "completed",
            "suffix": "",
            "task": "http://pulp3.dev:8000/pulp/api/v3/tasks/97ba7d54-4168-4895-942c-4b45c1b20a48/",
            "total": 3
        },
        {
            "done": 0,
            "message": "Remove Content",
            "state": "compCreate a Publisherleted",
            "suffix": "",
            "task": "http://pulp3.dev:8000/pulp/api/v3/tasks/97ba7d54-4168-4895-942c-4b45c1b20a48/",
            "total": 0
        }
    ],
    "spawned_tasks": [],
    "started_at": "2018-06-08T19:32:22.175157Z",
    "state": "completed",
    "worker": "http://pulp3.dev:8000/pulp/api/v3/workers/41f6d8b8-1a94-4338-8f0a-7234d3ee6760/"
}

View created repository version

$ pulp repositories versions read --repository_pk=0d6c48e2-9b68-4221-84db-7fa94d10eb2e number=1
{
    "_added_href": "http://pulp3.dev:8000/pulp/api/v3/repositories/0d6c48e2-9b68-4221-84db-7fa94d10eb2e/versions/1/added_content/",
    "_content_href": "http://pulp3.dev:8000/pulp/api/v3/repositories/0d6c48e2-9b68-4221-84db-7fa94d10eb2e/versions/1/content/",
    "_href": "http://pulp3.dev:8000/pulp/api/v3/repositories/0d6c48e2-9b68-4221-84db-7fa94d10eb2e/versions/1/",
    "_removed_href": "http://pulp3.dev:8000/pulp/api/v3/repositories/0d6c48e2-9b68-4221-84db-7fa94d10eb2e/versions/1/removed_content/",
    "content_summary": {
        "file": 3
    },
    "created": "2018-06-10T23:49:15.672784Z",
    "id": "c43aed46-1fa6-4930-8dc2-de4c10bb8ef9",
    "number": 1
}

Create a Publisher

$ pulp publishers file create --name=bar
{
    "_href": "http://pulp3.dev:8000/pulp/api/v3/publishers/file/5eb270e4-88e6-4952-aba0-02983f784293/",
    "created": "2018-06-10T23:52:45.045851Z",
    "distributions": [],
    "id": "5eb270e4-88e6-4952-aba0-02983f784293",
    "last_published": null,
    "last_updated": "2018-06-10T23:52:45.045869Z",
    "name": "bar",
    "type": "file"
}

Use the bar Publisher to create a Publication

$ pulp publishers file publish --id=5eb270e4-88e6-4952-aba0-02983f784293 --repository=http://pulp3.dev:8000/pulp/api/v3/repositories/90fa0c62-4d35-4a3b-99b7-d1b1a2c9c01e/
{
    "_href": "http://pulp3.dev:8000/pulp/api/v3/tasks/5078d4c4-a152-4ac9-8b3a-e0d1c5c62e28/",
    "task_id": "5078d4c4-a152-4ac9-8b3a-e0d1c5c62e28"
}

Loading |{
    "_href": "http://pulp3.dev:8000/pulp/api/v3/tasks/5078d4c4-a152-4ac9-8b3a-e0d1c5c62e28/",
    "created": "2018-06-11T00:33:58.785480Z",
    "created_resources": [
        "http://pulp3.dev:8000/pulp/api/v3/publications/20e44b38-e3d2-455f-af1a-d8aa3e720851/"
    ],
    "error": null,
    "finished_at": "2018-06-11T00:33:58.985735Z",
    "id": "5078d4c4-a152-4ac9-8b3a-e0d1c5c62e28",
    "non_fatal_errors": [],
    "parent": null,
    "progress_reports": [],
    "spawned_tasks": [],
    "started_at": "2018-06-11T00:33:58.899002Z",
    "state": "completed",
    "worker": "http://pulp3.dev:8000/pulp/api/v3/workers/fb1e6a46-c463-4d2d-b684-04c47da827c3/"
}

Create a Distribution for the Publication

$ pulp distributions create --name=baz --base_path=foo --publication=http://pulp3.dev:8000/pulp/api/v3/publications/20e44b38-e3d2-455f-af1a-d8aa3e720851/
{
    "_href": "http://pulp3.dev:8000/pulp/api/v3/distributions/1c30053e-4921-4ad7-ac18-cb68150df4fa/",
    "base_path": "foo",
    "base_url": "pulp3.dev:8000/pulp/content/foo",
    "created": "2018-06-11T00:38:09.543133Z",
    "id": "1c30053e-4921-4ad7-ac18-cb68150df4fa",
    "name": "baz",
    "publication": "http://pulp3.dev:8000/pulp/api/v3/publications/20e44b38-e3d2-455f-af1a-d8aa3e720851/",
    "publisher": null,
    "repository": null
}

#3 Updated by daviddavis over 2 years ago

One change I think we should make (and I opened a couple Github issues for this already) is to get rid of using hrefs in the CLI. So hide fields that start with underscores ("_") like "_href". Maybe show them if users pass in an option like "--debug" or something.

Also we should accept id or name for relationship fields. So for example we currently have:

pulp remotes file sync --repository=http://pulp3.dev:8000/pulp/api/v3/repositories/c3550bb7-1984-4bd4-bbc4-5bc5483008b5/ --id=396b7eb5-97aa-4394-bc25-ca8be7dd19c4

Instead, accept id or name:

pulp remotes file sync --repository=c3550bb7-1984-4bd4-bbc4-5bc5483008b5 --id=396b7eb5-97aa-4394-bc25-ca8be7dd19c4
pulp remotes file sync --repository=foo --id=396b7eb5-97aa-4394-bc25-ca8be7dd19c4

#4 Updated by daviddavis over 2 years ago

  • Project changed from Pulp to Pulp CLI

#5 Updated by daviddavis over 2 years ago

  • Status changed from NEW to CLOSED - CURRENTRELEASE

Closing as we've now migrated to a redmine project.

#6 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to Proof of concept

#7 Updated by daviddavis 6 months ago

  • Status changed from CLOSED - CURRENTRELEASE to CLOSED - WONTFIX

Please register to edit this issue

Also available in: Atom PDF