Project

Profile

Help

Issue #3462

closed

Pulp produces incorrect crane json for removed tags

Added by jluza about 6 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Docker:
Platform Release:
2.16.1
Target Release - Docker:
master
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 35
Quarter:

Description

Reproducer:

#create repo
pulp-admin docker repo create --repo-id reproducer-repo --feed https://osbs-pulp-stage.host.prod.eng.rdu2.redhat.com:8888/ --upstream-name twaugh/rsyslog

#Sync repo with schema 2 content with more then 1 tag
pulp-admin docker repo sync run --repo-id <some_repo_id>

  1. check for synced tags
    pulp-admin docker repo search tag --repo-id reproducer-repo
  1. remove one tag by name
    pulp-admin docker repo remove tag --repo-id reproducer-repo --match 'name=<tag>'

#make sure only selected tag was removed

#publish the repo
pulp-admin docker repo publish run --repo-id reproducer-repo

#on pulp server
cat /var/lib/pulp/published/docker/v2/master/reproducer-repo/*/reproducer-repo.json | python -m json.tool

And in output you can see that removed tag is still in
"manifest_list_amd64_tags"

Tag is also pullable via "docker pull"

however tag is not present in
$ cat /var/lib/pulp/published/docker/v2/master/reproducer-repo/*/tags/list | python -m json.tool

neither is in published manifests

Pulp docker version: pulp-docker-plugins-3.1.1-2


Files

tag-orphan-remove.py (1.33 KB) tag-orphan-remove.py Used by tag-orphan-test amacdona@redhat.com, 04/10/2018 08:09 PM
tag-orphan-test.sh (1.43 KB) tag-orphan-test.sh Use with "./tag-orphan-test.sh" amacdona@redhat.com, 04/10/2018 08:09 PM
Actions #1

Updated by dalley about 6 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 34
Actions #2

Updated by ipanova@redhat.com about 6 years ago

Whenever we remove a manifest/manifest list we automatically also purge the tags associated to that manifest/manifest list in the repository.
We don't want to leave tags that reference manifest/manifest list that no longer exist.

Besides the unassociation from the repo we also complety remove record from DB
https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/importers/importer.py#L552

This^ is not called during regular unassociate_by_criteria() call when we perform single tag removal.
(Calls go into the platform and there is just unassociate call, the delete() is not performed, like in case of manifest/manifest list removal)

Further when these lines are hit https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/distributors/publish_steps.py#L259

We cannot really say by this call if the tag is actually in the repo. It might be in the repo, or it can be an orphan.

(12:37:44 PM) ipanova: i wonder why there was not added in docker plugin remove_units() for Tag
(12:38:13 PM) ipanova: we'd whether need to subclass tag removal in docker so after tag removal it would call delete()
(12:38:37 PM) ipanova: or in the place you pointed /publish_steps.py#L259 we'd need to check if they are in the repo or orphaned

Actions #3

Updated by amacdona@redhat.com almost 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com
Actions #4

Updated by ipanova@redhat.com almost 6 years ago

  • Sprint changed from Sprint 34 to Sprint 35
Actions #5

Updated by amacdona@redhat.com almost 6 years ago

I've created a pseudo-integration-test-bash-hack for TDD.

Before starting on the work, here's the output:

(pulp) [vagrant@pulp2 pulp]$ ./tag-orphan-test.sh 
+----------------------------------------------------------------------+
                   Synchronizing Repository [busybox]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading manifests
[-]
^C+----------------------------------------------------------------------+
                    Publishing Repository [busybox]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

[/]
[|]
Waiting to begin...

Task Succeeded

Present. should be present with expected behavior
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Units Removed:
  1
  1

Present. tags removed. sanity check, should still be present until republish
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
+----------------------------------------------------------------------+
                    Publishing Repository [busybox]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Present. publish. should (not?) be gone (after fix)... maybe
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
+----------------------------------------------------------------------+
                    Publishing Repository [busybox]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Force Full:  True

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Gone(After)Present(before). force full publish. should be gone (after fix)
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Gone(after)Present(before) orphans removed. sanity check, should still be present until republish
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
+----------------------------------------------------------------------+
                    Publishing Repository [busybox]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Force Full:  True

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Gone(after, before) orphans removed, should still be present until publish
{
    "    Gone: (1 tag)": null
}

Added by amacdona@redhat.com almost 6 years ago

Revision c82bfc56 | View on GitHub

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.

fixes #3462

https://pulp.plan.io/issues/3462

Added by amacdona@redhat.com almost 6 years ago

Revision c82bfc56 | View on GitHub

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.

fixes #3462

https://pulp.plan.io/issues/3462

Added by amacdona@redhat.com almost 6 years ago

Revision c82bfc56 | View on GitHub

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.

fixes #3462

https://pulp.plan.io/issues/3462

Added by amacdona@redhat.com almost 6 years ago

Revision c82bfc56 | View on GitHub

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.

fixes #3462

https://pulp.plan.io/issues/3462

Actions #6

Updated by amacdona@redhat.com almost 6 years ago

  • Status changed from ASSIGNED to POST

https://github.com/pulp/pulp_docker/pull/232

Hack tests pass. Filed a pulp-smash issue.

Actions #7

Updated by amacdona@redhat.com almost 6 years ago

This change does not include a migration, and therefore will not clean up orphaned tags from previous releases. For users who experience this problem after a upgrading, we recommend running an orphan cleanup and republish (--force-full)

Actions #8

Updated by amacdona@redhat.com almost 6 years ago

  • Status changed from POST to MODIFIED
Actions #9

Updated by amacdona@redhat.com almost 6 years ago

  • Platform Release changed from 2.15.1 to 2.16.1
  • Target Release - Docker set to master

Added by amacdona@redhat.com almost 6 years ago

Revision 847264e2 | View on GitHub

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.

fixes #3462

https://pulp.plan.io/issues/3462 (cherry picked from commit c82bfc562dcab2fbbd23f1bcc29f3c4e595f1fea)

Added by amacdona@redhat.com almost 6 years ago

Revision 847264e2 | View on GitHub

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.

fixes #3462

https://pulp.plan.io/issues/3462 (cherry picked from commit c82bfc562dcab2fbbd23f1bcc29f3c4e595f1fea)

Actions #11

Updated by bmbouter almost 6 years ago

  • Status changed from MODIFIED to 5
Actions #12

Updated by bmbouter almost 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #13

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF