Project

Profile

Help

Issue #3462

Pulp produces incorrect crane json for removed tags

Added by jluza about 1 year ago. Updated 5 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Severity:
2. Medium
Version - Docker:
Platform Release:
2.16.1
Blocks Release:
Target Release - Docker:
master
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 35

Description

Reproducer:

#create repo
pulp-admin docker repo create --repo-id reproducer-repo --feed https://osbs-pulp-stage.host.prod.eng.rdu2.redhat.com:8888/ --upstream-name twaugh/rsyslog

#Sync repo with schema 2 content with more then 1 tag
pulp-admin docker repo sync run --repo-id <some_repo_id>

  1. check for synced tags
    pulp-admin docker repo search tag --repo-id reproducer-repo
  1. remove one tag by name
    pulp-admin docker repo remove tag --repo-id reproducer-repo --match 'name=<tag>'

#make sure only selected tag was removed

#publish the repo
pulp-admin docker repo publish run --repo-id reproducer-repo

#on pulp server
cat /var/lib/pulp/published/docker/v2/master/reproducer-repo/*/reproducer-repo.json | python -m json.tool

And in output you can see that removed tag is still in
"manifest_list_amd64_tags"

Tag is also pullable via "docker pull"

however tag is not present in
$ cat /var/lib/pulp/published/docker/v2/master/reproducer-repo/*/tags/list | python -m json.tool

neither is in published manifests

Pulp docker version: pulp-docker-plugins-3.1.1-2

tag-orphan-remove.py (1.33 KB) Used by tag-orphan-test amacdona@redhat.com, 04/10/2018 08:09 PM tag-orphan-remove.py
tag-orphan-test.sh (1.43 KB) Use with "./tag-orphan-test.sh" amacdona@redhat.com, 04/10/2018 08:09 PM tag-orphan-test.sh

Associated revisions

Revision c82bfc56 View on GitHub
Added by amacdona@redhat.com about 1 year ago

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed,
they are automatically orphaned. By cleaning them up when they are
removed, we solve a problem in which orphaned tags were erroneously
published.

fixes #3462

https://pulp.plan.io/issues/3462

Revision c82bfc56 View on GitHub
Added by amacdona@redhat.com about 1 year ago

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed,
they are automatically orphaned. By cleaning them up when they are
removed, we solve a problem in which orphaned tags were erroneously
published.

fixes #3462

https://pulp.plan.io/issues/3462

Revision 847264e2 View on GitHub
Added by amacdona@redhat.com 12 months ago

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed,
they are automatically orphaned. By cleaning them up when they are
removed, we solve a problem in which orphaned tags were erroneously
published.

fixes #3462

https://pulp.plan.io/issues/3462
(cherry picked from commit c82bfc562dcab2fbbd23f1bcc29f3c4e595f1fea)

Revision 847264e2 View on GitHub
Added by amacdona@redhat.com 12 months ago

Delete tags when unassociated from repository

Each tag exists in only a single repository, so when they are removed,
they are automatically orphaned. By cleaning them up when they are
removed, we solve a problem in which orphaned tags were erroneously
published.

fixes #3462

https://pulp.plan.io/issues/3462
(cherry picked from commit c82bfc562dcab2fbbd23f1bcc29f3c4e595f1fea)

History

#1 Updated by dalley about 1 year ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 34

#2 Updated by ipanova@redhat.com about 1 year ago

Whenever we remove a manifest/manifest list we automatically also purge the tags associated to that manifest/manifest list in the repository.
We don't want to leave tags that reference manifest/manifest list that no longer exist.

Besides the unassociation from the repo we also complety remove record from DB
https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/importers/importer.py#L552

This^ is not called during regular unassociate_by_criteria() call when we perform single tag removal.
(Calls go into the platform and there is just unassociate call, the delete() is not performed, like in case of manifest/manifest list removal)

Further when these lines are hit https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/distributors/publish_steps.py#L259

We cannot really say by this call if the tag is actually in the repo. It might be in the repo, or it can be an orphan.

(12:37:44 PM) ipanova: i wonder why there was not added in docker plugin remove_units() for Tag
(12:38:13 PM) ipanova: we'd whether need to subclass tag removal in docker so after tag removal it would call delete()
(12:38:37 PM) ipanova: or in the place you pointed /publish_steps.py#L259 we'd need to check if they are in the repo or orphaned

#3 Updated by amacdona@redhat.com about 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com

#4 Updated by ipanova@redhat.com about 1 year ago

  • Sprint changed from Sprint 34 to Sprint 35

#5 Updated by amacdona@redhat.com about 1 year ago

I've created a pseudo-integration-test-bash-hack for TDD.

Before starting on the work, here's the output:

(pulp) [vagrant@pulp2 pulp]$ ./tag-orphan-test.sh 
+----------------------------------------------------------------------+
                   Synchronizing Repository [busybox]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading manifests
[-]
^C+----------------------------------------------------------------------+
                    Publishing Repository [busybox]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

[/]
[|]
Waiting to begin...

Task Succeeded

Present. should be present with expected behavior
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Units Removed:
  1
  1

Present. tags removed. sanity check, should still be present until republish
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
+----------------------------------------------------------------------+
                    Publishing Repository [busybox]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Present. publish. should (not?) be gone (after fix)... maybe
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
+----------------------------------------------------------------------+
                    Publishing Repository [busybox]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Force Full:  True

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Gone(After)Present(before). force full publish. should be gone (after fix)
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Gone(after)Present(before) orphans removed. sanity check, should still be present until republish
{
    "    Present (1 tag)": [
        "sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
        2
    ]
}
+----------------------------------------------------------------------+
                    Publishing Repository [busybox]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Force Full:  True

This command may be exited via ctrl+c without affecting the request.

Task Succeeded

Gone(after, before) orphans removed, should still be present until publish
{
    "    Gone: (1 tag)": null
}

#6 Updated by amacdona@redhat.com about 1 year ago

  • Status changed from ASSIGNED to POST
  • Smash Test set to 935

https://github.com/pulp/pulp_docker/pull/232

Hack tests pass. Filed a pulp-smash issue.

#7 Updated by amacdona@redhat.com about 1 year ago

This change does not include a migration, and therefore will not clean up orphaned tags from previous releases. For users who experience this problem after a upgrading, we recommend running an orphan cleanup and republish (--force-full)

#8 Updated by amacdona@redhat.com about 1 year ago

  • Status changed from POST to MODIFIED

#9 Updated by amacdona@redhat.com about 1 year ago

  • Platform Release changed from 2.15.1 to 2.16.1
  • Target Release - Docker set to master

#11 Updated by bmbouter 12 months ago

  • Status changed from MODIFIED to ON_QA

#12 Updated by bmbouter 12 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#13 Updated by bmbouter 5 days ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF