Issue #3462
closed
Pulp produces incorrect crane json for removed tags
Description
Reproducer:
#create repo
pulp-admin docker repo create --repo-id reproducer-repo --feed https://osbs-pulp-stage.host.prod.eng.rdu2.redhat.com:8888/ --upstream-name twaugh/rsyslog
#Sync repo with schema 2 content with more then 1 tag
pulp-admin docker repo sync run --repo-id <some_repo_id>
- check for synced tags
pulp-admin docker repo search tag --repo-id reproducer-repo
- remove one tag by name
pulp-admin docker repo remove tag --repo-id reproducer-repo --match 'name=<tag>'
#make sure only selected tag was removed
#publish the repo
pulp-admin docker repo publish run --repo-id reproducer-repo
#on pulp server
cat /var/lib/pulp/published/docker/v2/master/reproducer-repo/*/reproducer-repo.json | python -m json.tool
And in output you can see that removed tag is still in
"manifest_list_amd64_tags"
Tag is also pullable via "docker pull"
however tag is not present in
$ cat /var/lib/pulp/published/docker/v2/master/reproducer-repo/*/tags/list | python -m json.tool
neither is in published manifests
Pulp docker version: pulp-docker-plugins-3.1.1-2
Files
Updated by dalley over 6 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 34
Updated by ipanova@redhat.com over 6 years ago
Whenever we remove a manifest/manifest list we automatically also purge the tags associated to that manifest/manifest list in the repository.
We don't want to leave tags that reference manifest/manifest list that no longer exist.
Besides the unassociation from the repo we also complety remove record from DB
https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/importers/importer.py#L552
This^ is not called during regular unassociate_by_criteria() call when we perform single tag removal.
(Calls go into the platform and there is just unassociate call, the delete() is not performed, like in case of manifest/manifest list removal)
Further when these lines are hit https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/distributors/publish_steps.py#L259
We cannot really say by this call if the tag is actually in the repo. It might be in the repo, or it can be an orphan.
(12:37:44 PM) ipanova: i wonder why there was not added in docker plugin remove_units() for Tag
(12:38:13 PM) ipanova: we'd whether need to subclass tag removal in docker so after tag removal it would call delete()
(12:38:37 PM) ipanova: or in the place you pointed /publish_steps.py#L259 we'd need to check if they are in the repo or orphaned
Updated by amacdona@redhat.com over 6 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to amacdona@redhat.com
Updated by ipanova@redhat.com over 6 years ago
- Sprint changed from Sprint 34 to Sprint 35
Updated by amacdona@redhat.com over 6 years ago
- File tag-orphan-remove.py tag-orphan-remove.py added
- File tag-orphan-test.sh tag-orphan-test.sh added
I've created a pseudo-integration-test-bash-hack for TDD.
Before starting on the work, here's the output:
(pulp) [vagrant@pulp2 pulp]$ ./tag-orphan-test.sh
+----------------------------------------------------------------------+
Synchronizing Repository [busybox]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading manifests
[-]
^C+----------------------------------------------------------------------+
Publishing Repository [busybox]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
[/]
[|]
Waiting to begin...
Task Succeeded
Present. should be present with expected behavior
{
" Present (1 tag)": [
"sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
2
]
}
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Units Removed:
1
1
Present. tags removed. sanity check, should still be present until republish
{
" Present (1 tag)": [
"sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
2
]
}
+----------------------------------------------------------------------+
Publishing Repository [busybox]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Task Succeeded
Present. publish. should (not?) be gone (after fix)... maybe
{
" Present (1 tag)": [
"sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
2
]
}
+----------------------------------------------------------------------+
Publishing Repository [busybox]
+----------------------------------------------------------------------+
The following publish configuration options will be used:
Force Full: True
This command may be exited via ctrl+c without affecting the request.
Task Succeeded
Gone(After)Present(before). force full publish. should be gone (after fix)
{
" Present (1 tag)": [
"sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
2
]
}
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Succeeded
Gone(after)Present(before) orphans removed. sanity check, should still be present until republish
{
" Present (1 tag)": [
"sha256:186694df7e479d2b8bf075d9e1b1d7a884c6de60470006d572350573bfa6dcd2",
2
]
}
+----------------------------------------------------------------------+
Publishing Repository [busybox]
+----------------------------------------------------------------------+
The following publish configuration options will be used:
Force Full: True
This command may be exited via ctrl+c without affecting the request.
Task Succeeded
Gone(after, before) orphans removed, should still be present until publish
{
" Gone: (1 tag)": null
}
Added by amacdona@redhat.com over 6 years ago
Added by amacdona@redhat.com over 6 years ago
Revision c82bfc56 | View on GitHub
Delete tags when unassociated from repository
Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.
fixes #3462
Added by amacdona@redhat.com over 6 years ago
Revision c82bfc56 | View on GitHub
Delete tags when unassociated from repository
Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.
fixes #3462
Added by amacdona@redhat.com over 6 years ago
Revision c82bfc56 | View on GitHub
Delete tags when unassociated from repository
Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.
fixes #3462
Updated by amacdona@redhat.com over 6 years ago
- Status changed from ASSIGNED to POST
https://github.com/pulp/pulp_docker/pull/232
Hack tests pass. Filed a pulp-smash issue.
Updated by amacdona@redhat.com over 6 years ago
This change does not include a migration, and therefore will not clean up orphaned tags from previous releases. For users who experience this problem after a upgrading, we recommend running an orphan cleanup and republish (--force-full)
Updated by amacdona@redhat.com over 6 years ago
- Status changed from POST to MODIFIED
Applied in changeset c82bfc562dcab2fbbd23f1bcc29f3c4e595f1fea.
Updated by amacdona@redhat.com over 6 years ago
- Platform Release changed from 2.15.1 to 2.16.1
- Target Release - Docker set to master
Added by amacdona@redhat.com over 6 years ago
Revision 847264e2 | View on GitHub
Delete tags when unassociated from repository
Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.
fixes #3462
https://pulp.plan.io/issues/3462 (cherry picked from commit c82bfc562dcab2fbbd23f1bcc29f3c4e595f1fea)
Added by amacdona@redhat.com over 6 years ago
Revision 847264e2 | View on GitHub
Delete tags when unassociated from repository
Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.
fixes #3462
https://pulp.plan.io/issues/3462 (cherry picked from commit c82bfc562dcab2fbbd23f1bcc29f3c4e595f1fea)
Updated by amacdona@redhat.com over 6 years ago
Applied in changeset 847264e2014644b83bc632b6930b756919feeea1.
Updated by bmbouter over 6 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
.
Delete tags when unassociated from repository
Each tag exists in only a single repository, so when they are removed, they are automatically orphaned. By cleaning them up when they are removed, we solve a problem in which orphaned tags were erroneously published.
fixes #3462
https://pulp.plan.io/issues/3462