Project

Profile

Help

Issue #4987

Pulp 3 is not validating that all required fields are being supplied to the RPM create API

Added by jcabrera over 2 years ago. Updated almost 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 56
Quarter:

Description

Used Version

pulp_source_dir: "git+https://github.com/pulp/pulpcore.git@3.0.0rc2"
pulp_plugin_source_dir: "git+https://github.com/pulp/pulpcore-plugin.git@0.1.0rc2"

pulp_install_plugins:
  pulp-rpm:
    app_label: "rpm"
    source_dir: "git+https://github.com/pulp/pulp_rpm.git@3.0.0b3"

Problem

When creating a publication I get the error

{
    "_created": "2019-06-18T12:26:28.001989Z",
    "_href": "/pulp/api/v3/tasks/76fe1068-450d-43b2-b550-b224f7acad74/",
    "created_resources": [],
    "error": {
        "code": null,
        "description": "duplicate key value violates unique constraint \"core_publishedartifact_publication_id_relative__97f785f4_uniq\"\nDETAIL:  Key (publication_id, relative_path)=(36599dd4-2698-4eb8-8362-12b391d9739a, ) already exists.\n",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 812, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 588, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 594, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 100, in publish\n    populate(publication)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 213, in populate\n    published_artifact.save()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 741, in save\n    force_update=force_update, update_fields=update_fields)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 779, in save_base\n    force_update, using, update_fields,\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 870, in _save_table\n    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 908, in _do_insert\n    using=using, raw=raw)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py\", line 1186, in _insert\n    return query.get_compiler(using=using).execute_sql(return_id)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1335, in execute_sql\n    cursor.execute(sql, params)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n"
    },
    "finished_at": "2019-06-18T12:26:28.092046Z",
    "name": "pulp_rpm.app.tasks.publishing.publish",
    "non_fatal_errors": [],
    "parent": null,
    "progress_reports": [],
    "spawned_tasks": [],
    "started_at": "2019-06-18T12:26:28.053503Z",
    "state": "failed",
    "worker": "/pulp/api/v3/workers/922cc43e-5f7b-4d08-b44a-2ce61105dab3/"
}

Steps to reproduce

Use a clean pulp-server (no repositories, packages, publication neither distributions)

Download some rpm packages in mypackage directory

sudo install yum-utils
yumdownloader --resolve --destdir=mypackages/ httpd

execute this script

#/bin/bash

# To run this script, you need to get somme rpm packages.
# Install yum-utils and execute:
# yumdownloader --resolve --destdir=mypackages/ httpd

PORT=localhost:24817

wait_task() {

    echo $2
    state=$(http $PORT$1 | jq -r '.state')
    while [ "$state" == "running" ]; do
        sleep 10
        state=$(http $PORT$1 | jq -r '.state')
        echo $state
    done

    if [ "$state" == "failed" ]; then
        echo "TASK $1 FAILED"
        http $PORT$1
        exit 1
    fi
}

REPO_HREF=$(http POST $PORT/pulp/api/v3/repositories/ name=local_repo | jq -r '._href')
echo REPO_HREF=$REPO_HREF

# add rpm packages long
for p in `ls mypackages`; do
    ls ./mypackages/$p
    ARTIFACT_REF=$(http --form POST $PORT/pulp/api/v3/artifacts/ file@./mypackages/$p | jq -r '._href')
    CONTENT_HREF=$(http POST $PORT/pulp/api/v3/content/rpm/packages/ _artifact="$ARTIFACT_REF" filename="$p" | jq -r '._href')
    echo CONTENT_HREF=$CONTENT_HREF
    TASK=$(http POST $PORT${REPO_HREF}versions/ add_content_units:="[\"$CONTENT_HREF\"]" | jq -r '.task')
    wait_task $TASK 'wait content'
done

TASK=$(http POST $PORT/pulp/api/v3/publications/rpm/rpm/ repository=$REPO_HREF | jq -r '.task')
echo TASK=$TASK
wait_task $TASK 'wait publication'
#
PUBLICATION_HREF=$(http $PORT/pulp/api/v3/publications/rpm/rpm/ | jq -r '.results[] | ._href')
echo PUBLICATION_HREF=$PUBLICATION_HREF
#
http POST $PORT/pulp/api/v3/distributions/rpm/rpm/ name='baz' base_path='foo' publication=$PUBLICATION_HREF

the script gives an error at publication creation.

Associated revisions

Revision e7d624e0 View on GitHub
Added by Lubos Mjachky over 2 years ago

Use relative_path instead of filename

The field filename seems to be redundant. Therefore, it's usages were removed from the corresponding view set. Now, relative_path is used as filename instead.

In this commit, the field relative_path is no longer nullable. Also, the field cannot be initialized to an empty string because it is used for a package creation where the path is stored as location_href.

closes #4987 https://pulp.plan.io/issues/4987

History

#1 Updated by dkliban@redhat.com over 2 years ago

I discussed this issue with the reporter on the list. The problem is that the script in the description should actually fail @

CONTENT_HREF=$(http POST $PORT/pulp/api/v3/content/rpm/packages/ _artifact="$ARTIFACT_REF" filename="$p" | jq -r '._href')

but it keeps going till the user publishes.

#2 Updated by jcabrera over 2 years ago

Based in what is done for the file plugin

https://github.com/pulp/pulp_file/blob/master/docs/_scripts/content.sh

I added relative_path to the API call

    ARTIFACT_RELATIVE_PATH=$(head /dev/urandom | tr -dc a-z | head -c5)
    echo ARTIFACT_RELATIVE_PATH=$ARTIFACT_RELATIVE_PATH
    CONTENT_HREF=$(http POST $PORT/pulp/api/v3/content/rpm/packages/ _artifact="$ARTIFACT_HREF" filename="$p" relative_path="$ARTIFACT_RELATIVE_PATH" | jq -r '._href')

This seems to work.

Based on this, there are two things that seems not clear to me

1. In the REST API doc there is no filename variable for package creation

https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/content_rpm_packages_create

2. In the rpm documentation workflow there is no relative_path if it is needed.

https://pulp-rpm.readthedocs.io/en/latest/workflows/upload.html#create-rpm-content-from-an-artifact

#3 Updated by ttereshc over 2 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 54

#4 Updated by ttereshc over 2 years ago

  • Sprint changed from Sprint 54 to Sprint 55

#5 Updated by lmjachky over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to lmjachky

#6 Updated by lmjachky over 2 years ago

  • Status changed from ASSIGNED to POST

#7 Updated by dkliban@redhat.com over 2 years ago

  • Sprint changed from Sprint 55 to Sprint 56

#8 Updated by Anonymous over 2 years ago

  • Status changed from POST to MODIFIED

#9 Updated by ttereshc almost 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF